Token error for some scopes

[fivebanger]

Describe the bug
I have compiled a recent dev-version of Librelspot (3rd Sep., build with --release) and tried to get a token for different scopes. Some scopes returns an error instead of a valid token. I have used a single scope always, not a list of scopes. Following scopes results in an error:

user-top-read
user-read-recently-played
user-read-playback-state
user-modify-playback-state
user-read-currently-playing

To reproduce
Steps to reproduce the behavior:

1. Connect with Spotify app to librespot
2. Call function get_token() with one of the above mentioned scopes.

Log
A log sequence for scope "user-read-currently-playing":

[2024-09-08T09:37:10Z TRACE librespot_core::token] Requested token in scopes "user-read-currently-playing" unavailable or expired, requesting new token.
[2024-09-08T09:37:10Z ERROR librespot_core::mercury] error 400 for uri hm://keymaster/token/authenticated?scope=user-read-currently-playing&client_id=9a8d2f0ce77a4e248bb71fefcb557637&device_id=XXX
[2024-09-08T09:37:10Z DEBUG librespot_core::session] could not dispatch command: Service unavailable { error handling Mercury response: MercuryResponse { uri: "hm://keymaster/token/authenticated?scope=user-read-currently-playing&client_id=9a8d2f0ce77a4e248bb71fefcb557637&device_id=XXX", status_code: 400, payload: [[123, 34, 99, 111, 100, 101, 34, 58, 51, 44, 34, 101, 114, 114, 111, 114, 68, 101, 115, 99, 114, 105, 112, 116, 105, 111, 110, 34, 58, 34, 73, 110, 118, 97, 108, 105, 100, 32, 115, 99, 111, 112, 101, 34, 125]] } }

Host (what you are running librespot on):

• OS: Linux

Additional context
Although I'm running Libresopt on Linux, I can see in the log that the ANDROID_CLIENT_ID from config.rs is used.

Edit:
Steps to reproduce updated

[roderickvd]

The payload numbers are ASCII codes. Maybe they give some idea what’s going on.

[fivebanger]

The payload results in {"code":3,"errorDescription":"Invalid scope"}.

I have some update on this in general. I have two different use cases where I request tokens for some scopes by librespot's get_token() function:

• Use case 1 is based on librespot's player example, i.e. I'm opening a session, loading a track and start to play it. I would call this "session-approach".

• Use case 2 is more or less librespot's main loop where I connect to librespot by using the Spotify app (Android). I would call this "connect-approach".

For use case 1 (session-approach) I don't see any issue, I get a token for whatever scope I request it for.

For use use case 2 (connect-approach) I see the afore described error. The error is not returned for the connect-approach when using v0.4.2, whereas I have to provide my own client-id in that case when calling get_token().

I have read a comment in some issue discussion here that not all scopes are allowed for all client-ids. Unfortunately I cannot find this comment anymore. Maybe the observed behavior is expected and this is not an issue at all but just a different behavior compared to v0.4.2.

Just wondering (related to connect-approach): Even though when changing the values in config.rs

pub(crate) const KEYMASTER_CLIENT_ID: &str = "65b708073fc0480ea92a077233ca87bd";
pub(crate) const ANDROID_CLIENT_ID: &str = "9a8d2f0ce77a4e248bb71fefcb557637";
pub(crate) const IOS_CLIENT_ID: &str = "58bd3c95768941ea9eb4350aaa033eb3";

to e.g. (all KEYMASTER_CLIENT_ID)

pub(crate) const KEYMASTER_CLIENT_ID: &str = "65b708073fc0480ea92a077233ca87bd";
pub(crate) const ANDROID_CLIENT_ID: &str = "65b708073fc0480ea92a077233ca87bd";
pub(crate) const IOS_CLIENT_ID: &str = "65b708073fc0480ea92a077233ca87bd";

I can see in the log that the ANDROID_CLIENT_ID is used anyway when connected by Spotify app: hm://keymaster/token/authenticated?scope=user-read-currently-playing&client_id=9a8d2f0ce77a4e248bb71fefcb557637&device_id=XXX. I have not checked the same for the session-approach. Just wondering...

Maybe this is the expected way and the issue can get closed.

[kingosticks]

When using connect-approach, do any scopes work?

0.4 and 0.5 do work differently w.r.t session client id. When using connect in 0.5, it uses the controller's client ID for the session.client_id, which will be coming from your android app. This makes sense and I'm presuming that is the correct behaviour (and we were doing it wrong before) but I don't know.

It could be that the android client id isn't allowed to use keymaster at all, or is only allowed to use it for certain scopes. Or something else related to the type of session it is. Why not try hardcoding the client id used by TokenProvider ?

I'll also add, it's not worth spending much time on keymaster since it's the outdated way of getting a token.

[fivebanger]

When using connect-approach, do any scopes work?

AFAIR I have checked for all these scopes:

user-read-playback-state
user-modify-playback-state
user-read-currently-playing
app-remote-control
streaming
playlist-read-private
playlist-read-collaborative
playlist-modify-private
playlist-modify-public
user-follow-modify
user-follow-read
user-read-playback-position
user-top-read
user-read-recently-played
user-library-modify
user-library-read
user-read-email
user-read-private

Need to double check exactly when needed. But yes, the connect approach works for these scopes, beside the mentioned ones before which returns an error.

[fivebanger]

I have tried to prepare some debug code to demonstrate the issue but now it gets really wired.... I have now two flavors of debug code, a working one and a failing one. I have placed the get_token() debug code in two different places inside librespot's main loop in main.rs:

First place:

                spirc = Some(spirc_);
                spirc_task = Some(Box::pin(spirc_task_));

                connecting = false;

                /* debug code goes here: without asking here first, get_token(scopes_fail) results in some error later on */
                let scopes_ok: &str = "streaming";
                let scopes_fail: &str = "user-read-playback-state";
                let mut token = session.token_provider().get_token(scopes_fail).await;
                if token.is_ok() {
                    println!("Got me a token: {token:#?}");
                } else {
                    println!("Token error");
                }
                token = session.token_provider().get_token(scopes_ok).await;
                if token.is_ok() {
                    println!("Got me a token: {token:#?}");
                } else {
                    println!("Token error");
                }

            },
            _ = async {
                if let Some(task) = spirc_task.as_mut() {

Second place:

            _ = tokio::signal::ctrl_c() => {

                /* debug code goes here: get_token(scopes_fail) fails, if not asked for a token before but passes, when already asked for a token */
                let scopes_ok: &str = "streaming";
                let scopes_fail: &str = "user-read-playback-state";
                let mut token = session.token_provider().get_token(scopes_fail).await;
                if token.is_ok() {
                    println!("Got me a token: {token:#?}");
                } else {
                    println!("Token error");
                }
                token = session.token_provider().get_token(scopes_ok).await;
                if token.is_ok() {
                    println!("Got me a token: {token:#?}");
                } else {
                    println!("Token error");
                }

                break;
            },

I started with putting the debug code only in the first place because I assume that's the time when we have a valid connection. I was surprised not getting an error for scope_fail.

Then I put my debug code only in the second place since that fits more to my connect-approach use case. I'm asking from time to time for a new token. This leads to an error for scope_fail.

Then I wanted to see that the problem occurs not in the first place but in the second place and put the debug code in both. Surprisingly the scope_fail case works at second place, if I ask for a token at the first place before.

This is reproducible in my environment.

[roderickvd]

I'm not sure I completely follow. Can you post something like a Gist with two full working examples?

[fivebanger]

I have added some debug code to my fork here: https://github.com/fivebanger/librespot/tree/token_error

I'll follow up on this next week but what I can tell so far: When hard code the keymaster's ID in TokenProvider::get_token() like already proposed I don't see an error with respect to different scopes. My understanding is that this is the same like get_token(client_id) in v0.4.2 in case I would have provided the keymaster's ID. I'll also check with my personal client_id's I have requested from Spotify some years ago.

[fivebanger]

tl;dr:

• Scopes = "user-read-playback-state" results in an error if client_id = ANDROID_CLIENT_ID is used inside TokenProvider::get_token().
• TokenProvider::get_token() works for scopes = "user-read-playback-state" if hard coding either client_id = KEYMASTER_CLIENT_ID or client_id = IOS_CLIENT_ID or client_id = MY_OWN_CLIENT_ID (requested from Spotify) inside TokenProvider::get_token().
• I assume that these findings also apply to the remaining scopes mentioned afore (not checked explicitly yet).
• v0.4.2: get_token(client_id) has worked for client_id = KEYMASTER_CLIENT_ID or client_id = MY_OWN_CLIENT_ID because the client_id was provided from outside when calling get_token().
• V0.5.0 vs v0.4.2: In case librespot wants to provide the same feature set with respect to get_token() and any scope (as v0.4.2 did) I see two reasonable options:
• Option 1: client_id = KEYMASTER_CLIENT_ID is hard coded inside TokenProvider::get_token() (different behavior towards Spotify's server compared to current v0.5.0 since KEYMASTER_CLIENT_ID gets always used for requests).
• Option 2: Change TokenProvider::get_token(client_id) whereas client_id is an optional string. When calling TokenProvider::get_token(None), client_id evaluates to client_id = self.session().client_id() inside TokenProvider::get_token() (same as of current v0.5.0). When calling TokenProvider::get_token(string), the string is used inside TokenProvider::get_token(). A caller from outside can provide any appropriate client_id when calling get_token().

More detailed logs (for code, pls. refer to https://github.com/fivebanger/librespot/tree/token_error):

Case 1: Start librespot regular and ask for a token on ctrl+c:

[2024-09-21T22:22:52Z INFO  librespot_core::token] Librespot requests a token, scopes: "playlist-read"
[2024-09-21T22:22:52Z INFO  librespot_core::token] Requested token in scopes "playlist-read" unavailable or expired, requesting new token.
[2024-09-21T22:22:52Z INFO  librespot_core::token] client_id = self.session().client_id(): "9a8d2f0ce77a4e248bb71fefcb557637"
[2024-09-21T22:22:52Z INFO  librespot_core::token] Librespot requests a token, scopes: "playlist-read"
[2024-09-21T22:22:52Z INFO  librespot_core::token] Requested token in scopes "playlist-read" unavailable or expired, requesting new token.
[2024-09-21T22:22:52Z INFO  librespot_core::token] client_id = self.session().client_id(): "9a8d2f0ce77a4e248bb71fefcb557637"
[2024-09-21T22:22:53Z INFO  librespot_playback::player] Loading <Down with the Sickness> with Spotify URI <spotify:track:40rvBMQizxkIqnjPdEWY1v>
[2024-09-21T22:22:53Z INFO  librespot_core::token] Librespot requests a token, scopes: "playlist-read"
[2024-09-21T22:22:53Z INFO  librespot_core::token] use a cached token
[2024-09-21T22:22:53Z INFO  librespot_connect::spirc] Resolved 50 tracks from <"spotify:playlist:3xSO3g8XSivrk6HEYsfSnu">
[2024-09-21T22:22:53Z INFO  librespot_playback::player] <Down with the Sickness> (279213 ms) loaded
^C[2024-09-21T22:23:01Z INFO  librespot_core::token] User requests a token, client_id not hard coded, scopes: "user-read-playback-state"
[2024-09-21T22:23:01Z INFO  librespot_core::token] Requested token in scopes "user-read-playback-state" unavailable or expired, requesting new token.
[2024-09-21T22:23:01Z INFO  librespot_core::token] user, client_id = self.session().client_id(): "9a8d2f0ce77a4e248bb71fefcb557637"
[2024-09-21T22:23:01Z ERROR librespot_core::mercury] error 400 for uri hm://keymaster/token/authenticated?scope=user-read-playback-state&client_id=9a8d2f0ce77a4e248bb71fefcb557637&device_id=xxx
[2024-09-21T22:23:01Z ERROR librespot] Token error
[2024-09-21T22:23:01Z INFO  librespot] Gracefully shutting down

No additional TokenProvider::get_token() initiated by user at start-up,
First get_token() was initiated by librespot itself.
Client ID used for first token request = self.session().client_id() = ANDROID_CLIENT_ID.
Client ID for scopes = "user-read-playback-state" used on ctr+c = self.session().client_id() = ANDROID_CLIENT_ID.
Result: get_token() for scope "user-read-playback-state" fails.

Case 2: Start librespot and initiate a get_token() by user on connect. Ask for a token on ctrl+c:

[2024-09-21T22:15:57Z INFO  librespot_core::token] User requests a token, client_id not hard coded, scopes: "user-read-playback-state"
[2024-09-21T22:15:57Z INFO  librespot_core::token] Requested token in scopes "user-read-playback-state" unavailable or expired, requesting new token.
[2024-09-21T22:15:57Z INFO  librespot_core::token] user, client_id = self.session().client_id(): "65b708073fc0480ea92a077233ca87bd"
[2024-09-21T22:15:57Z INFO  librespot_core::session] Country: "DE"
[2024-09-21T22:15:57Z INFO  librespot] Token ok
[2024-09-21T22:15:57Z INFO  librespot_core::spclient] Resolved "gew4-spclient.spotify.com:443" as spclient access point
[2024-09-21T22:15:57Z INFO  librespot_core::token] Librespot requests a token, scopes: "playlist-read"
[2024-09-21T22:15:57Z INFO  librespot_core::token] Requested token in scopes "playlist-read" unavailable or expired, requesting new token.
[2024-09-21T22:15:57Z INFO  librespot_core::token] client_id = self.session().client_id(): "9a8d2f0ce77a4e248bb71fefcb557637"
[2024-09-21T22:15:57Z INFO  librespot_core::token] Librespot requests a token, scopes: "playlist-read"
[2024-09-21T22:15:57Z INFO  librespot_core::token] Requested token in scopes "playlist-read" unavailable or expired, requesting new token.
[2024-09-21T22:15:57Z INFO  librespot_core::token] client_id = self.session().client_id(): "9a8d2f0ce77a4e248bb71fefcb557637"
[2024-09-21T22:15:57Z INFO  librespot_playback::player] Loading <Down with the Sickness> with Spotify URI <spotify:track:40rvBMQizxkIqnjPdEWY1v>
[2024-09-21T22:15:57Z INFO  librespot_core::token] Librespot requests a token, scopes: "playlist-read"
[2024-09-21T22:15:57Z INFO  librespot_core::token] use a cached token
[2024-09-21T22:15:57Z INFO  librespot_connect::spirc] Resolved 50 tracks from <"spotify:playlist:3xSO3g8XSivrk6HEYsfSnu">
[2024-09-21T22:15:57Z INFO  librespot_playback::player] <Down with the Sickness> (279213 ms) loaded
^C[2024-09-21T22:16:09Z INFO  librespot_core::token] User requests a token, client_id not hard coded, scopes: "user-read-playback-state"
[2024-09-21T22:16:09Z INFO  librespot_core::token] use a cached token
[2024-09-21T22:16:09Z INFO  librespot] Token ok
[2024-09-21T22:16:09Z INFO  librespot] Gracefully shutting down

Additional TokenProvider::get_token() for scopes = "user-read-playback-state" at start-up,
First get_token() was initiated by user on connect.
Client ID used for for scopes = "user-read-playback-state" token request = self.session().client_id() = KEYMASTER_CLIENT_ID (most likely the initial value, have not followed up on that).
get_token() for scopes = "user-read-playback-state" on ctr+c was already cached before.
Result: get_token() for scope "user-read-playback-state" ok (will fail probably once the cached token has expired).

Case 3: Start librespot regular and ask for a token on ctrl+c but using hard coded client_id = KEYMASTER_CLIENT_ID:

[2024-09-21T22:19:52Z INFO  librespot_core::token] Librespot requests a token, scopes: "playlist-read"
[2024-09-21T22:19:52Z INFO  librespot_core::token] Requested token in scopes "playlist-read" unavailable or expired, requesting new token.
[2024-09-21T22:19:52Z INFO  librespot_core::token] client_id = self.session().client_id(): "9a8d2f0ce77a4e248bb71fefcb557637"
[2024-09-21T22:19:52Z INFO  librespot_core::token] Librespot requests a token, scopes: "playlist-read"
[2024-09-21T22:19:52Z INFO  librespot_core::token] Requested token in scopes "playlist-read" unavailable or expired, requesting new token.
[2024-09-21T22:19:52Z INFO  librespot_core::token] client_id = self.session().client_id(): "9a8d2f0ce77a4e248bb71fefcb557637"
[2024-09-21T22:19:52Z INFO  librespot_playback::player] Loading <Down with the Sickness> with Spotify URI <spotify:track:40rvBMQizxkIqnjPdEWY1v>
[2024-09-21T22:19:52Z INFO  librespot_core::token] Librespot requests a token, scopes: "playlist-read"
[2024-09-21T22:19:52Z INFO  librespot_core::token] use a cached token
[2024-09-21T22:19:52Z INFO  librespot_connect::spirc] Resolved 50 tracks from <"spotify:playlist:3xSO3g8XSivrk6HEYsfSnu">
[2024-09-21T22:19:52Z INFO  librespot_playback::player] <Down with the Sickness> (279213 ms) loaded
^C[2024-09-21T22:20:04Z INFO  librespot_core::token] User requests a token, client_id is hard coded, : "user-read-playback-state"
[2024-09-21T22:20:04Z INFO  librespot_core::token] Requested token in scopes "user-read-playback-state" unavailable or expired, requesting new token.
[2024-09-21T22:20:04Z INFO  librespot_core::token] patch, self.session().client_id(): "9a8d2f0ce77a4e248bb71fefcb557637"
[2024-09-21T22:20:04Z INFO  librespot_core::token] patch, client_id_patch = hard coded: "65b708073fc0480ea92a077233ca87bd"
[2024-09-21T22:20:04Z INFO  librespot] Token ok
[2024-09-21T22:20:04Z INFO  librespot] Gracefully shutting down

No additional TokenProvider::get_token() initiated by user at start-up.
First get_token() was initiated by librespot itself.
Client ID used for first token request = self.session().client_id() = ANDROID_CLIENT_ID.
Client ID for scopes = "user-read-playback-state" used on ctr+c = KEYMASTER_CLIENT_ID (hard coded).
Result: get_token() for scope "user-read-playback-state" ok.

I'm wondering why librespot itself asks always three times for a token (scopes = "playlist-read") on startup and why the second get_token() call cannot get covered by a cached token already (like seen for the third call).

[kingosticks]

We just want to do what the desktop app does really. If someone who was interested could connect to the official client from an android device and trace the http requests and find.out what ID it uses.

I'm wondering why librespot itself asks always three times for a token (scopes = "playlist-read") on startup and why the second get_token() call cannot get covered by a cached token already (like seen for the third call

This is seemingly a bug, we've had for a while. Please fix if you can.

[fivebanger]

If someone who was interested could connect to the official client from an android device and trace the http requests and find.out what ID it uses.

I doubt that's beyond my current skills... But even if, I think that's not really helpfull. My understanding is:

Librespot inits the client_id at startup according to the OS its running on, see config.rs:

impl SessionConfig {
    pub(crate) fn default_for_os(os: &str) -> Self {
        let device_id = uuid::Uuid::new_v4().as_hyphenated().to_string();
        let client_id = match os {
            "android" => ANDROID_CLIENT_ID,
            "ios" => IOS_CLIENT_ID,
            _ => KEYMASTER_CLIENT_ID,
        }
        .to_owned();

That's why I see client_id = KEYMASTER_CLIENT_ID when calling get_token() direktly on connect.

The client_id gets updated when librespot receives an update frame, which comes from Spotify most likely? See SpircTask, handle_remote_update()

        for entry in update.device_state.metadata.iter() {
            match entry.type_() {
                "client_id" => self.session.set_client_id(entry.metadata()),
                "brand_display_name" => self.session.set_client_brand_name(entry.metadata()),
                "model_display_name" => self.session.set_client_model_name(entry.metadata()),
                _ => (),
            }
        }

That's the only place I have found set_client_id(). So the client_id used further on comes from Spotify, nothing to tweak or change at this point. Even a http trace as you have proposed would not change that fact, I guess.

BTW, within the same function handle_remote_update() we check for a changed client_id which then will emit an event.

        if self.device.is_active() && new_client_id != old_client_id {
            info!("spirc, new client_id: {:#?}", new_client_id);

            self.player.emit_session_client_changed_event(
                new_client_id,
                self.session.client_name(),
                self.session.client_brand_name(),
                self.session.client_model_name(),
            );
        }

My understanding of the intention here is to signal that another device has connected to the outside. But if Spotify returns ANDROID_CLIENT_ID for (all?) Andriod devices, how will the if() statement ever becomes "true" in case just another Android device connects? Maybe client_id defaults to OS specific during a re-connect? I cannot check since I don't have another Spotify account yet.

BTW 2, due to impl SessionConfig {...} I doubt the get_token example would never run on an Android device once compiled for Android? The client_id would default to ANDROID_CLIENT_ID which does not allow all used scopes, e.g. "user-read-playback-state".

We just want to do what the desktop app does really.

I absolutely appreciate to implement librespot according to the desktop app. However, is it worth to follow the discussion about an alternative implementation of TokenProvider::get_token() here? Something like

    pub async fn get_token(&self, scopes: &str, client_id_opt: Option<&str>) -> Result<Token, Error> {
        //let client_id = self.session().client_id();
        let client_id = match client_id_opt {
            Some(client_id_opt) => client_id_opt.to_string(),
            None => self.session().client_id(),
        };
        if client_id.is_empty() {
            return Err(Error::invalid_argument("Client ID cannot be empty"));
        }

or simply an additional function like

    pub async fn get_token_with_client_id(&self, scopes: &str, client_id: &str) -> Result<Token, Error> {...}

This would not change any flow inside standard librespot but would enable an user sitting on top of the source code to request a token according to its needed scopes and maybe also using its own client_id. If this discussion is not desired I would like to stop here regarding the get_token() error issue since the error is then just an observation with respect to the current implementation.

This is seemingly a bug, we've had for a while. Please fix if you can.

Maybe better to open an additional issue to not mix topics?

Edit:
I just realized that the get_token() example has changed. So maybe my doubts are not correct at all, forget about this.

[kingosticks]

That's the only place I have found set_client_id(). So the client_id used further on comes from Spotify, nothing to tweak or change at this point. Even a http trace as you have proposed would not change that fact, I guess.

No, we are free to ignore that client ID update if we decide that's more useful to us. This is exactly what happens in 0.4.

However, is it worth to follow the discussion about an alternative implementation of TokenProvider::get_token() here?

I think putting it back to how it worked in 0.4 is reasonable (your second example?). But keep in mind that the desktop app doesn't use keymaster any more and we will integrate the new method soon as it's ready (#1344). After that, maybe we keep keymaster, or maybe we don't. Personally, I think it would be more useful to have a look at that new Login5-based approach and ensure it handles client ID in a way that works for you.

Maybe better to open an additional issue to not mix topics?

Yes please.

[fivebanger]

That's the only place I have found set_client_id(). So the client_id used further on comes from Spotify, nothing to tweak or change at this point. Even a http trace as you have proposed would not change that fact, I guess.

No, we are free to ignore that client ID update if we decide that's more useful to us. This is exactly what happens in 0.4.

With respect to tracing the desktop app's traffic: Maybe the original app ignores the client_id update and still uses the KEYMASTER_ID which could be figured out by tracing the traffic?

I think putting it back to how it worked in 0.4 is reasonable (your second example?). But keep in mind that the desktop app doesn't use keymaster any more and we will integrate the new method soon as it's ready (#1344). After that, maybe we keep keymaster, or maybe we don't. Personally, I think it would be more useful to have a look at that new Login5-based approach and ensure it handles client ID in a way that works for you.

I just realized that there is a lot of change going on with respect to authentication under the hood. I'm confused currently by Oauth and login5 and whether the one is an alternative for the other or if these are completely different topics and what keymaster means in this context.

As someone using librespot as kind of a lib with a small wrapper around (to be able to use Python's subprocess to communicate with librespot) I have more or less two requirements:

• Get audio via pipe backend and rodio backend
• Get a token which can be used to feed spotipy to make use of Spotify's web-api (to get rid of spotipy's OAuth flow)

My current understanding is that TokenProvider::get_token() gets maybe removed in future and is considered to be deprecated (from Spotify's side?). So it would be more future-proof to check the PR for login5 you have mentioned? And there is no need to dig into that TokenProvider::get_token() topic since it's deprecated? I'll start to get familiar with the changes of the login5 PR....

Maybe better to open an additional issue to not mix topics?

Yes please.

Before opening a new issue, what exactly you do consider to be a bug? The fact that librespot asks three times at startup for a token? Or the fact that asking the second time does not use a cached token? Or both?