Problem with the connection to Netatmo API since 2023-12-04 #77

jostrasser · 2023-12-04T18:47:47Z

Describe the bug
Homebridge unable to start, plugin is preventing start because there are API issues with the Netatmo API.

[04/12/2023, 19:26:12] [eveatmo-platform] WARN - Netatmo: Error: getHealthyHomeCoachData error: Invalid access token
[04/12/2023, 19:26:29] [homebridge-eveatmo] This plugin is taking long time to load and preventing Homebridge from starting. See https://homebridge.io/w/JtMGR for more info.
[04/12/2023, 19:26:49] [homebridge-eveatmo] This plugin is taking long time to load and preventing Homebridge from starting. See https://homebridge.io/w/JtMGR for more info.
[04/12/2023, 19:27:09] [homebridge-eveatmo] This plugin is taking long time to load and preventing Homebridge from starting. See https://homebridge.io/w/JtMGR for more info.
[04/12/2023, 19:27:29] [homebridge-eveatmo] This plugin is taking long time to load and preventing Homebridge from starting. See https://homebridge.io/w/JtMGR for more info.
[04/12/2023, 19:27:49] [homebridge-eveatmo] This plugin is taking long time to load and preventing Homebridge from starting. See https://homebridge.io/w/JtMGR for more info.

Version you use

homebridge version: 1.7.0
node version: 20.10.0
plugin version: latest

To Reproduce
Restart homebridge service

The text was updated successfully, but these errors were encountered:

jostrasser · 2023-12-04T18:48:24Z

I think we are facing another API issue. Contacted Netatmo support and still waiting for feedback.
I´ll keep you posted. BR/JO

novski · 2023-12-04T21:23:37Z

I have the same. The workaround here does not work on my side.

mrmaximas · 2023-12-04T21:47:21Z

anyone using api in two places at once like me? homebridge-eveatmo + node-red-contrib-netatmo-dashboard?

mrmaximas · 2023-12-04T21:59:53Z

I reset keys, deleted netatmo-token.json, generated a new access token with full r/w/a rights for all device types, put new client id, secret and refresh token into plugin settings and everything works again.

jostrasser · 2023-12-05T08:37:49Z

anyone using api in two places at once like me? homebridge-eveatmo + node-red-contrib-netatmo-dashboard?

Yes, I am also using both: node-red and homebridge-eveatmo. :)
homebridge-eveatmo is working again after generating a new refresh token, node-red is broken and a fresh token does not solve the issue.

jostrasser · 2023-12-05T08:39:40Z

I reset keys, deleted netatmo-token.json, generated a new access token with full r/w/a rights for all device types, put new client id, secret and refresh token into plugin settings and everything works again.

Have you tested a restart of the bridge? Is it solved permanently after creating the new tokens?
I created only a fresh access and refresh token...
In my case a restart of the homebridge is breaking it again!

@skrollme I think there are adjustments required on the plugin if Netatmo changed their Auth method again... :(

mrmaximas · 2023-12-05T14:13:46Z

Have you tested a restart of the bridge? Is it solved permanently after creating the new tokens?

after full HB restart i have a problem again (((

jostrasser · 2023-12-05T14:19:16Z

Have you tested a restart of the bridge? Is it solved permanently after creating the new tokens?

after full HB restart i have a problem again (((

Yep, Is expected... :(

jostrasser · 2023-12-05T14:19:52Z

@skrollme

Info from Netatmo DEV Support:

We just did a modification on the token retrieval process :
 
When you refreshed an access_token using the associated endpoint [https://api.netatmo.com/oauth2/token, Netatmo servers](https://api.netatmo.com/oauth2/token,%C2%A0Netatmo%C2%A0servers) responded with a couple of tokens : an access_token and a refresh_token. If the previous access_token was still valid, the refresh_token value was never renewed
 
Starting from the 04/12/2023, this behavior changed to be compliant with the recommendations of the [RFC of the OAuth2 Authorization Framework](https://www.rfc-editor.org/rfc/rfc6749) (section 10.4) and improving the security of the data of our users
When refreshing tokens, access_token and refresh_token values will be automatically renewed and former tokens invalidated
 
So, if you do not update and use the new refresh_token value when refreshing your access_token, your users will be disconnected after 3 hours and you will retrieve an “invalid_grant” error
To fix it, you need to update the tokens value as soon as you get the newly generated ones.

mrmaximas · 2023-12-05T14:22:27Z

Yep, Is expected... :(

By the way i make additional "application" in dev.netatmo for NR.. Waiting for fix.
In the new reality one token will not work in two systems as it will change after each request.

fmarzocca · 2023-12-05T15:18:59Z

In the new reality one token will not work in two systems as it will change after each request.

Unless the 2 systems don't share the same json file with the tokens

mrmaximas · 2023-12-05T15:21:24Z

Unless the 2 systems don't share the same json file with the tokens

they're two different systems.

fmarzocca · 2023-12-05T15:23:29Z

they're two different systems.

yes, but there are many ways to share a file between 2 different systems (provided they are using the same Netatmo account)

mrmaximas · 2023-12-05T15:41:58Z

many ways to share a file between 2 different systems

It won't work.
Each time the api is accessed, the token will be renew, and the second system will get the expired token.

fmarzocca · 2023-12-05T15:43:40Z

You need to save the file at each request, and to keep the json file synchronized between the 2 systems

mrmaximas · 2023-12-05T15:45:47Z

You need to save the file at each request, and to keep the json file synchronized between the 2 systems

What's the point? I added a second app and have a couple of tokens that i can use at the frequency i need.

fmarzocca · 2023-12-05T15:49:03Z

What's the point? I added a second app and have a couple of tokens that i can use at the frequency i need.

Good, no point. But you said before you said that "one token will not work in two systems " and that's not true. Ok.

mrmaximas · 2023-12-05T15:52:23Z

But you said before you said that "one token will not work in two systems " and that's not true. Ok.

you are right, maybe it will work only in one case - when systems are unity and request data with the same frequency, but I have homebridge for homekit and node-red for export to influx then grafana.

jostrasser · 2023-12-05T16:42:57Z

Additional info from Netatmo:

The access_token must be refreshed every 3 hours (10800 seconds). This has always been the case
The difference was that if you refreshed your access_token before its expiration, refresh_token value always remained the same. If you wanted to refresh it but after expiration, the refresh_token value changed
 
Now, the 2 values change at each new /token request. So, you must catch the new generated refresh_token value each time a new request is done to the endpoint, and perform the next /token call with this new value

Mindphazer73 · 2023-12-05T18:00:37Z

On my Netatmo app on my iPhone, I need to re-connect each time i use the app (I guess after the famous 3 hours)
What a great idea from Netatmo, if they want their users to run away :-(

mrmaximas · 2023-12-05T18:06:25Z

On my Netatmo app on my iPhone, I need to re-connect each time i use the app (I guess after the famous 3 hours) What a great idea from Netatmo, if they want their users to run away :-(

I'm guessing it's not Netatmo's idea, but the very important Jedi Legrand engineers. :-)

PvdGulik · 2023-12-05T18:34:48Z

On my Netatmo app on my iPhone, I need to re-connect each time i use the app (I guess after the famous 3 hours) What a great idea from Netatmo, if they want their users to run away :-(

I don't recognize this till now...... Maybe later.....

Mindphazer73 · 2023-12-05T19:41:10Z

I'm guessing it's not Netatmo's idea, but the very important Jedi Legrand engineers. :-)

Yeah you're probably right :-D

jostrasser · 2023-12-07T14:44:49Z

Another update from Netatmo.
This is the reason why all the apps, plugins, integrations are back working again:

Indeed, teams had to do a rollback to fix a problem for multiple connections (if a 2nd device connects, it asks for a token and invalidates the access for the 1st connected device). I reported that it was a problem for third-party apps The plan is still to put in place this change. They are working on a solution, but I don't have for now more information about a release date

I requested detailed information when the next change will take place.

Mindphazer73 · 2023-12-07T16:07:45Z

Well, it doesn't work for me
I still have invalid token access

Pajonk100 · 2023-12-07T17:05:27Z

Well, it doesn't work for me I still have invalid token access

same problem...

jostrasser · 2023-12-07T17:21:40Z

@Mindphazer73 & @Pajonk100

Please try the following:

Create a new Access and Refresh token with Postman
Manually enter both tokens into the netatmo-token.json file
Enter the same Refresh token into the config.json of Homebridge
Restart Homebridge service

This should actually work.

PvdGulik · 2023-12-07T17:32:32Z

I just created a new refresh token on dev.netatmo.com, and copied/pasted in Homebridge Eveatmo plugin in the settings, and voila, working again 😉 [07/12/2023, 18:29:53] [eveatmo platform] Initializing eveatmo platform... [07/12/2023, 18:29:53] [eveatmo platform] Authenticating using 'refresh_token' grant [07/12/2023, 18:29:57] [eveatmo platform] Loading new data from API for: weatherstation [07/12/2023, 18:29:57] Loading 3 accessories...

…

________________________________ Van: Johannes Strasser ***@***.***> Verzonden: donderdag 7 december 2023 18:21 Aan: skrollme/homebridge-eveatmo ***@***.***> CC: Tebogo ***@***.***>; Comment ***@***.***> Onderwerp: Re: [skrollme/homebridge-eveatmo] Problem with the connection to Netatmo API since 2023-12-04 (Issue #77) @Mindphazer73<https://github.com/Mindphazer73> & @Pajonk100<https://github.com/Pajonk100> Please try the following: 1. Create a new Access and Refresh token with Postman 2. Manually enter both tokens into the netatmo-token.json file 3. Enter the same Refresh token into the config.json of Homebridge 4. Restart Homebridge service This should actually work. — Reply to this email directly, view it on GitHub<#77 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/ACH4FIROVQOQZEQHRT562LDYIH3K7AVCNFSM6AAAAABAGOL7BOVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTQNBVG43TENBTGI>. You are receiving this because you commented.Message ID: ***@***.***>

Pajonk100 · 2023-12-07T17:34:47Z

@Mindphazer73 & @Pajonk100

Please try the following:

Create a new Access and Refresh token with Postman

Manually enter both tokens into the netatmo-token.json file

Enter the same Refresh token into the config.json of Homebridge

Restart Homebridge service

This should actually work.

problem was in netatmo-token.json - old tokens before my testing what was wrong... Now its working :) thx :D

mrmaximas · 2023-12-07T17:35:40Z

@Mindphazer73 & @Pajonk100

Please try the following:

Create a new Access and Refresh token with Postman

Manually enter both tokens into the netatmo-token.json file

Enter the same Refresh token into the config.json of Homebridge

Restart Homebridge service

This should actually work.

all Hombridge or plugin Bridge only?

mrmaximas · 2023-12-07T17:36:57Z

@Mindphazer73 & @Pajonk100
Please try the following:

Create a new Access and Refresh token with Postman

Manually enter both tokens into the netatmo-token.json file

Enter the same Refresh token into the config.json of Homebridge

Restart Homebridge service

This should actually work.

problem was in netatmo-token.json - old tokens before my testing what was wrong... Now its working :) thx :D

In my case it was always enough to delete netatmo-token.json and put a new refresh token in plugin config

Mindphazer73 · 2023-12-07T18:06:39Z

Thanks
I just deleted netatmo-token.json, and restarted Homebridge, and it works again

jostrasser · 2023-12-07T18:08:25Z

@mrmaximas

all Hombridge or plugin Bridge only?

Only for the eveatmo plugin and a restart of the Homebridge service.

In my case it was always enough to delete netatmo-token.json and put a new refresh token in plugin config

Should also work if the plugin is able to create it again. I ran into an issue in the past, that's the reason why I suggested to replace the keys. ;)

I hope @skrollme can create a fix until Netatmo redo the change on their platform.

mrmaximas · 2023-12-07T18:10:35Z

Only for the eveatmo plugin and a restart of the Homebridge service.

A couple of days ago it stopped working after a full restart of homebridge. Since I'm already tired of generating tokens, I'm not touching or restarting anything yet.

jostrasser · 2023-12-07T18:18:11Z

A couple of days ago it stopped working after a full restart of homebridge. Since I'm already tired of generating tokens, I'm not touching or restarting anything yet.

Yes, I had the same behavior after a Homebridge restart. If the bridge is not restarted, everything should continue to run, also if Netatmo will redo the change.

The sticking point was the creation of new keys when the bridge restarts I think. So the goal should be to make it persistent again.

skrollme · 2024-04-08T18:44:29Z

Is this still an open case, @jostrasser ?

jostrasser · 2024-04-08T18:56:20Z

Is this still an open case, @jostrasser ?

Hey @skrollme ,
Right now there is no issue because they rolled back the Auth method again. I think this was related to it. Netatmo confirmed me "issues with 3rd party apps" and confirmed this as reason for the rollback.

They have planned to enable the new auth method again (May 29th 2024) but I don't know if the current implantation is working afterwards. I hope so ;)

General: If the API is inaccessible the plugin won't start / is hanging at "Starting".

It would be cool if you can add a timeout with a following retry (every 5mins) until the API comes back online. This would keep Homebridge up and running even if the API is not working correctly.

Thanks!

ADD: INFOMAIL FROM NETATMO

Today, when you refresh an Access Token using the associated endpoint https://api.netatmo.com/oauth2/token, the Netatmo servers respond with a pair of tokens: an Access Token and a Refresh Token.

If the previous Access Token is still valid, the newly returned access token is identical but has an expiration time extended by 3 hours.

The Refresh Token is not renewed.

Starting May 29, 2024, this behavior will change to comply with the OAuth2 Authorization Framework RFC recommendations (section 10.4) and improve the security of our users' data.

When refreshing the tokens, the Access Token and Refresh Token will be different from the previous ones and the old tokens will be invalidated.

What does this mean for you?

If you already store the tokens returned by the API when refreshing your tokens, this change will not affect you.

If you do not update the tokens when refreshing them, your users will be logged out because the old tokens will be invalidated.

If for a given user, you have stored the same token in different places (for example: an application and its widget), you must have a way to synchronize them because the new token will be applied for the entire application.

skrollme · 2024-04-08T19:31:25Z

General: If the API is inaccessible the plugin won't start / is hanging at "Starting".

Correct, I think this is the main problem here.

It would be cool if you can add a timeout with a following retry (every 5mins) until the API comes back online. This would keep Homebridge up and running even if the API is not working correctly.

I'm not sure if the plugin's overall architecture supports this kind of approach. This would also be tricky to stage this environment/situation for testing purposes 🤔

jostrasser · 2024-04-08T19:45:41Z

I'm not sure if the plugin's overall architecture supports this kind of approach. This would also be tricky to stage this environment/situation for testing purposes 🤔

Understood. If there is no way to workaround a hanging startup of Homebridge if the API is down we only can disable the plugin until the API is back to get Homebridge back online (or we have to wait for the API).

For me it's okay because I am backing up my HomeKit telemetry and can easily restore it even I loose the room assignments of the sensors. (can be done with the "Controller for HomeKit" app)

If you are planning to overwork the plugin: please keep this in mind as a feature request please 😉

Thank you!

jostrasser added the bug label Dec 4, 2023

jostrasser assigned skrollme Dec 4, 2023

mrmaximas mentioned this issue Dec 5, 2023

Since 2023/12/04: Unable to refresh the access token guidone/node-red-contrib-netatmo-dashboard#28

Open

jostrasser mentioned this issue Apr 8, 2024

Access Token Expired #71

Closed

jostrasser closed this as not planned Won't fix, can't repro, duplicate, stale Apr 9, 2024

Problem with the connection to Netatmo API since 2023-12-04 #77

Problem with the connection to Netatmo API since 2023-12-04 #77

Comments

jostrasser commented Dec 4, 2023

jostrasser commented Dec 4, 2023

novski commented Dec 4, 2023

mrmaximas commented Dec 4, 2023 • edited Loading

mrmaximas commented Dec 4, 2023

jostrasser commented Dec 5, 2023 • edited Loading

jostrasser commented Dec 5, 2023 • edited Loading

mrmaximas commented Dec 5, 2023

jostrasser commented Dec 5, 2023

jostrasser commented Dec 5, 2023

mrmaximas commented Dec 5, 2023

fmarzocca commented Dec 5, 2023

mrmaximas commented Dec 5, 2023

fmarzocca commented Dec 5, 2023

mrmaximas commented Dec 5, 2023 • edited Loading

fmarzocca commented Dec 5, 2023

mrmaximas commented Dec 5, 2023 • edited Loading

fmarzocca commented Dec 5, 2023

mrmaximas commented Dec 5, 2023

jostrasser commented Dec 5, 2023

Mindphazer73 commented Dec 5, 2023

mrmaximas commented Dec 5, 2023

PvdGulik commented Dec 5, 2023

Mindphazer73 commented Dec 5, 2023

jostrasser commented Dec 7, 2023

Mindphazer73 commented Dec 7, 2023

Pajonk100 commented Dec 7, 2023

jostrasser commented Dec 7, 2023

PvdGulik commented Dec 7, 2023 via email

Pajonk100 commented Dec 7, 2023

mrmaximas commented Dec 7, 2023

mrmaximas commented Dec 7, 2023 • edited Loading

Mindphazer73 commented Dec 7, 2023

jostrasser commented Dec 7, 2023 • edited Loading

mrmaximas commented Dec 7, 2023

jostrasser commented Dec 7, 2023

skrollme commented Apr 8, 2024

jostrasser commented Apr 8, 2024 • edited Loading

skrollme commented Apr 8, 2024

jostrasser commented Apr 8, 2024 • edited Loading

mrmaximas commented Dec 4, 2023 •

edited

Loading

jostrasser commented Dec 5, 2023 •

edited

Loading

jostrasser commented Dec 5, 2023 •

edited

Loading

mrmaximas commented Dec 5, 2023 •

edited

Loading

mrmaximas commented Dec 5, 2023 •

edited

Loading

mrmaximas commented Dec 7, 2023 •

edited

Loading

jostrasser commented Dec 7, 2023 •

edited

Loading

jostrasser commented Apr 8, 2024 •

edited

Loading

jostrasser commented Apr 8, 2024 •

edited

Loading