v 13.2 keeps crashing and restarting #1538
Comments
|
Unfortunately, there is nothing useful in the log as there is no reason for the crash. Without a stack trace there's not much to go on. However, can you please try the 13.2.1-beta.0 released yesterday. It might also be worth upgrading to NodeJS 22. |
|
I did try 13.2.1-beta.0 and the same thing kept happening. I also just upgraded to nodejs 20 and tried both versions again with the same result. Reverting back to 13.1 fixes it. |
|
Yes, I was hoping for Node 22 though. I don't really understand what could have changed that would cause a crash like this, there really aren't any major changes between the two versions, just some very minor things and dependency updates. Most of the significant changes are with cameras (like ffmpeg bump/werift) and you don't even appear to have cameras. Regardless, without a trace of the crash, there's not anything actionable here, so I'm not sure how we can proceed. |
|
Is there anything I can do on my end or any other logs I can try to pull that would help? |
|
It would be really useful to see full logs, including base homebridge logs, not just the subset you shared. Also, maybe there would be more information if you enabled debug in the Homebridge config and plugin config. |
|
I also just noticed this the logs and appears to be coming from the ring plugin. No clue if it's related though.
|
|
I enabled debug and this also shows
|
|
Reading between the lines of those two errors, my suspicion is that this is caused by changes in push-receiver that were implemented to make it try to connect forever. Full logs from both 13.1 and 13.2 would be useful to try to follow the flow as I can't do this with log snippets. |
|
I did a bit of troubleshooting and can reproduce a crash if I reject connections on TCP port 5228 at my firewall. My guess is you are leveraging some type of DNS based adblock that is blocking resolution of mtalk.google.com, maybe something like Adguard DNS or perhaps PiHole. See #1306 (comment) for a similar issue and how the user resolved it. As to why it worked in versions prior to 13.1.0, those versions would have produced the connection error, but did not have the onReady promise that was introduced in upstream and, realistically, so push notifications were likely not working for your prior version either, but wouldn't have caused a crash as the older push-receiver library did not include a promise rejection for socket connection failures. This wouldn't have caused any problem with functionality in your case because push notifications are not used for hub devices, only devices like cameras/intercoms, which you don't appear to have (this may change in the future as there is some useful information for alarm available in push notifications). I think I can code up a simple fix to keep the crash from happening, but you should determine why your environment can't connect to TCP port 5228. |
|
Interesting. I just searched my firewall logs for anything related to mtalk.google.com and all of that traffic flows through on TCP 5228 and not a single one is blocked. So TCP 5228 appears to be open and working as normal. I also turned off everything related to any type of ad blocking and I still get the crashes. It's seems to be something else happening. |
|
Again, full logs would help, not just snippets that you believe are important. We won't make much progress without them. The error above is clearly from push-receiver having the socket connection closed. And, regardless, it still seems to be the cause of the crash. |
|
Sure thing. Here are the logs with 13.1 running and then the switch to 13.2. Debug is turned on for the logs. Thanks for taking a look. |
|
Thanks, I'll dig through them and see if I can get any other clues. The issue here is that I can't actually figure out a way to trap this in ring-client-api, and we really need to fixes in the newer push-receiver, so now I'll probably have to open an issue in upstream and try to get yet another fix put in there. |
|
If you need anything else on my end to troubleshoot, just let me know. |
|
Logs show that 13.1.0 also crashes, just not as often, and there are still issues with connection from push receiver even there. Maybe when you disabled adblock the DNS block results were still cached. Not that we don't need to try to fix this crash, but something in your network is causing this behavior. Maybe a router reboot or something would help. It is weird that your not really seeing the full debug information I would expect. Do you definitely have debug enabled in Homebrdige settings as well (and have restarted). It might be interesting if you set the NODE_DEBUG=net environemnt on your docker container so we can see the network level debug data. If you are uncomfortable sending full logs in an open forum, you can send them to me via email (same username at gmail). |
|
A thought, perhaps this issue is not actually network level, but rather authentication with push receiver failing. Have you fully re-initialized your authentication since upgrading to 13.x versions of the plugin, by stopping the plugin, deleting the device from Ring Control Center, and then re-authenticating? |
|
So if 13.1 is also crashes it must not have been very often as you said since I never noticed. On 13.2 it would loop through crashed very quickly which is the only reason why I noticed things weren't staying connected. I just deleted all devices from ring control center, reauthenticated the connection in homebridge, upgraded to 13.2 again, but it still crashes. For the NODE_DEBUG=net, do you just want me to add that to my docker compose for homebridge? And then I assume you want to see the container logs and not the ring plugin logs like I sent before? I can go ahead and email those to you. Thanks! |
|
If you set NODE_DEBUG in the docker compose I would still expect that extra debugging to appear in the Homebridge UI logs. As I already explained, 13.2 includes a new push-receiver which creates an unhandled promise on socker disconnect, so that crash is happening every single time your push connection is disconnected, which should not happen often under normal operation. That is different from the prior version, which would not create this crashing behavior, but there is still at least one crash in 13.1 in the logs you sent. The problem is, the logs you sent don't include the crash traces in many cases, so it's very difficult to know the cause. I'm not sure why your logs are missing these details. It feels like maybe plugin logs vs full homebridge logs. |
|
Ok I just emailed you the full homebridge logs after adding the environment variable to my docker compose and redeploying the stack. |
|
Thanks, I received them, although I don't see the node debugging I would expect. Regardless, one workaround you can use for now is to set "--unhandled-rejections none" in NODE_OPTIONS in the Homebridge settings. That will keep the thrown error from causes the process to crash. I've opened an upstream issue to try to get that resolved, but it will take time to get a fix upstream, get it incorporated into ring-client-api, etc. In the end, there's still an underlying issue why you can't connect to push-receiver and maintain the connection, but I can't really help you with that. It's either network or authentication. |
|
Got it. I just set --unhandled-rejections none as you suggested, upgraded to 13.2 and the crashes stopped. I'll keep that setting for now. Thank you for your help with this. |
|
@cantchooseaname8 would you mind giving |
|
No problem. I just updated to that beta and removed --unhandled-rejections. I'm now getting a different error. This keeps repeating over and over: My firewall logs show that TCP 5228 has traffic passing through without anything being blocked, so I don't think that's the cause. Not sure if there's something else though. I also removed any adblocks just to be safe, but I still get that error repeating. |
|
Yes, this is expected since the root cause of the problem is not solved, i.e. your system is unable to connect to the Firebase push notification servers, for whatever reason. Previously however, this would cause the system to crash, now these errors are logged as they should be. You must still troubleshoot the cause of the error in question. This would have been happening silently even for previous versions of the plugin. The crash itself was introduced in 13.2.0, due to upstream changes, but the problem would have existed in previous versions and simply failed to be logged. With 13.2.1-beta.1 the crash is prevented, but the malfunction is logged. In the end, something is causing the websocket connection to FCM servers to fail. I've seen this in probably a dozen other cases and it has, to this point, never failed to be network related. I've seen some people get this because they are running servers in clouds or are using various VPNs, which FCM servers may block. Assuming you haven't already followed the standard Notification Troubleshooting Steps, you could try those as that will reset the push credentials in case that is the issue, but normally, if credentials where the issue, it would not even get to the point of attempting to establish the websocket connection. |
|
Understood. I'll take a close look at my network and firewall to make sure everything is routing correctly. Thanks for your help! |
|
@cantchooseaname8 As another way to troubleshoot and see if it's at least making the initial connection, you can open the terminal in the Homebridge UI and run the following: Obviously you might get a different IP address. One thing I've seen in some other cases, mtalk.google.com will resolve to IPv6 addresses, but I've seen tons of firewalls not configured properly for IPv6. For example, some will silently drop the packets, which will make the socket connection take too long to start up and it will timeout, causing the process to repeat. So that's one thing to check. Or maybe the IPv6 rules don't allow 5228, for example. If a ping looks good and is resolving to valid IP addresses, then you can try this:
This will attempt to connect to port 5228 and should report if it least the base level TCP connection can be estabished. I've also had one report of a customers firewall, which had an IPS/IDS (Intrusion Detection/Prevention) setup running which was actually blocking the connection later in the process so perhaps that's something to check. In that case it was some Snort rule. A few users have reported just a router reboot/power cycle solved the problem. I'm happy to try to help more, but there's not much going on here at a basic level, the code just gets the auth data and calls the API to open a websocket, the websocket either stays connected or it doesn't, and, every time the connection fails, it tries again. |
|
Thank you! I put my firewall into “emergency access” mode which basically just pauses all rules, routes, etc. The error disappeared after doing that so it’s definitely something with my firewall. Even though it’s not blocking that particular port, there must still be something going on at some point in the process that is making it finicky. I’ll go through and start bringing everything back one by one to hopefully narrow it down. |
|
Thanks for reporting back. Would love to hear what the final solution turns out to be as, while issues with push service connectivity are fairly rare, they do happen, and it seems like the cause is always something unexpected. |
|
I'm honestly not sure where the issue was. It might have been buried in a list or custom route somewhere in my firewall. I ended up just adding a new custom route for that specific traffic so I know it's being handled exactly as it should be. Everything is still working great now. Thanks again for all of your help with this! |
|
Regardless, thanks for reporting and testing. Way better to have an error message vs it just crashing! I'll go ahead and close this for now. I think the plan is to cut a new minor version with this fix in the next few days. |
|
@tsightler I am using 13.2.1 and I am getting the same error, however my firewall is open and custom routing is created. When i run the command i can see the port is open I stoped the Network Intrusion detection and DNS filtering but behavior continue, and i keep getting the error. @cantchooseaname8 what custom rule do you add, i use Unifi and added custom rule by domain from my internal network to all connection to mtalk.google.com . |
|
I found the reason for the issue - its the DNS that return the IP for the mtalk.google.com - some of them are not responsive. Clearing cache of DNS fix the issue |
I was going to ask why ping was so high, like it's picking some server far away from you. As mtalk.google.com is a highly distributed service, I would expect low latency pings for pretty much anyone. Also, the DNS lifetime for mtalk.google.com is only 180 seconds so that would imply some local DNS caching is not honoring the lifetime. You can always use dig and other tools to further into DNS issues. |
Yeah , it appears some of the DoH servers was returning address that is far away from me. I notice when pinging from my UniFi router I always have 30ms and IP located on google regional network, and for some reason the raspberry where is HB had cached this IP and always ping it and try to connect it - and most probably it was IP not for my region and it rejects me. I have such behavior in the past when using DoH servers and not my provider DNS sometimes they don’t honor where I am located and give me IP which if for Asia Pacific for example . Thank you and have a great evening ! |
|
Thanks for sharing! Out of curiosity, where were you configuring DoH? Something on Unifi or some internal DNS proxy like PiHole or something else? Just kind of curious as it might help future users as well. |
Both options are possible in Unifi router, in Unifi Settngs - Security you can use build in list of SoH servers or you can add your own server which can be Adguard or Pihole. I use build in servers as they are kinda the same as in AdGuard.
|
Is there an existing issue for this?
Describe The Bug
Upgraded to v13.2 and it keeps crashing and restarting over and over. Downgrading back to v13.1 solves the issue. Below are the logs. Most irrelevant logs about my many accessories were removed to show the process ending and starting over and over.
To Reproduce
No response
Expected behavior
v13.1 maintained a stable connection, but v13.2 keeps crashing.
Relevant log output
Screenshots
No response
Homebridge Ring Config
{ "refreshToken": "alarmOnEntryDelay": true, "hideLightGroups": true, "hideInHomeDoorbellSwitch": true, "hideAlarmSirenSwitch": true, "hideDeviceIds": [ "b46a9713-19aa-4838-a2dc-fd19be8488e3", "aae08fdf-aa4f-4bd7-a74a-80cef405baa7", "8533af49-379c-4e96-b922-5c5391079255", "703d3582-101e-4e48-8f3b-ceeecfb17e4f", "ae1ab059-8693-4a44-b499-40593a8b4d27", "aaf20bb9-746f-4fad-bcc8-f595aa4711db", "def3dda1-16bb-4ea8-b175-bd27da4119a7", "07a04a55-6d97-4e25-a75c-3cdc40e67c4a", "2a6dd443-50e9-4a8f-b201-27e2a4d8ad83", "0f9ae647-c04b-49a2-9aea-4261143637a5", "b567fe23-f8b7-4d41-bb66-f326b4941c59", "ec9be2df-5b3f-49a0-954b-449cf2a4b6b1" ], "locationIds": [" ], "platform": "Ring", "_bridge": { "username": "port": 46745 } }Additional context
No response
OS
Ubuntu (using docker)
Node.js Version
v20.18.0
NPM Version
Homebridge/HOOBs Version
v1.8.5
Homebridge Ring Plugin Version
v13.2
Operating System
Ubuntu (using docker)
The text was updated successfully, but these errors were encountered: