[Question]: p2p streaming aborts after 120 data packages only on Hyper-V clients

[thieren]

Client version

2.1.1 (tested 2.0.0 also)

Node version

16.16.0

Operating System type

Linux

Operating system version

tested ubuntu and debian in latest LTS versions

Describe the bug

Here is a real treat if you love debugging. ;-)

@bropat I am not really sure if there is something, that can be done about this, but at least you should know that this issue exists I think.

I have been using Hyper-V with a Debian installation as homebridge instance for a while now. This weekend our eufy-plugin suddenly stopped working (e.g. the streaming wouldn't start).
After debugging this for a while I found that the stream did indeed start, but after exactly 120 DATA VIDEO packages the sequence number suddenly reverts to 0 and all following packages are received with either a 0 or a 1 value as sequence number.
The stream itself therefore times out and is killed after 5 seconds.

Since I couldn't get my head around this, I tried a few things and found, that this is working fine, if run on windows, or on a linux machine under VMWare. But as soon as you fire up a linux with Hyper-V this happens.

Since I didn't change anything and since this happens with version 2.0.0 as well as the latest 2.1.1 I believe this is due to an update of Hyper-V or something which may introduce a weird network error.

What I have tested so far:

• Debian with Hyper-V (not working)
• Ubuntu with Hyper-V (not working)
• Ubuntu with Hyper-V on totally different hardware (not working)
• Ubuntu with VMWare (first Hardware) -> works
• Windows (first Hardware) -> works
• macOS -> works

An example log is attached.

To reproduce

Fire up Ubuntu or Debian with Hyper-V and try p2p-streaming

Screenshots & Logfiles

sequence-number-hyper-v.log

Additional context

No response

[bropat]

Looked at the log and from the behavior I'm guessing a UDP communication problem, because the P2P protocol relies on the client acknowledging receipt of a data packet, otherwise the remote peer resends the same packet. When streaming video, you receive a lot of data packets in a short time, so there is a latency in the communication between the two. In concrete terms, it could be that after 121 data packets the remote station only notices that it has not received an acknowledgement of the first data packet (sequence 0) from the client. Therefore it tries to send it again and this several times, hoping for an answer from the client (the same for sequence 1). After some time the remote peer terminates the connection, because it assumes that the client is no longer reachable.

To check if it is as I suspect, I would need a trace of the network traffic (tcpdump / Wireshark).

[thieren]

Looked at the log and from the behavior I'm guessing a UDP communication problem, because the P2P protocol relies on the client acknowledging receipt of a data packet, otherwise the remote peer resends the same packet. When streaming video, you receive a lot of data packets in a short time, so there is a latency in the communication between the two. In concrete terms, it could be that after 121 data packets the remote station only notices that it has not received an acknowledgement of the first data packet (sequence 0) from the client. Therefore it tries to send it again and this several times, hoping for an answer from the client (the same for sequence 1). After some time the remote peer terminates the connection, because it assumes that the client is no longer reachable.

To check if it is as I suspect, I would need a trace of the network traffic (tcpdump / Wireshark).

I can record the traffic and send you the dump. Is there a safe way to send this to you?

But I am a little pessimistic as to whether you'll see what you're describing. I mean I think you're right that this is possibly due to failed ACK packets that are not send by the Hyper-V network stack but if I'll record the traffic inside the VM I'd most likely see these ACKs be send - just whether these ever reach the Homebase will not be recognisable - or am I missing something.

Anyway, will report back when I have the dump.

Any chance you could try to reproduce this? I have heard of at least one other user on discord who is having also problems with Hyper-V but it's not yet clear whether it's the same bug.

[bropat]

Of course, I need the dump from the physical machine running Hyper-V (physical network card), not from inside the VM.

You can restrict the trace to the communication with the local Ip address of the station. You can share the dump with me via Google Drive, OneDrive or similar.

At the moment I can't replicate it because I'm on holiday and don't have a Windows PC with me. ;)

[thieren]

Unfortunately I am not much of an expert when it comes to network traffic sniffing.

I have two physical network interfaces and four virtual network adapters from hyper-v on this hyper-v host. I have tried recording the traffic with Wireshark on every one on them while starting the stream command. In the vm the issue persists (so I get the connection, start and stop events from the station are recognised and so on, and after 120 packets the stream aborts)

But none of the dumps had any traffic to the station ip, so I don't really know what I could send you here.

[srrsparky]

I think I have replicated this issue - I have been struggling for a few days to get the integration to work on Windows 11 & Hyper-V. It would make the rtsp stream, but then drop out almost immediately.

I made a new instance of HASS using VirtualBox and the integration is working.

I have a T8210 2k battery doorbell.

On a side note - you can enable RTSP directly on this doorbell now too with a little hack. https://www.reddit.com/r/EufyCam/comments/nal27f/enabling_rtsp_on_battery_doorbell/

[thieren]

I think I have replicated this issue - I have been struggling for a few days to get the integration to work on Windows 11 & Hyper-V. It would make the rtsp stream, but then drop out almost immediately.

I made a new instance of HASS using VirtualBox and the integration is working.

I have a T8210 2k battery doorbell.

On a side note - you can enable RTSP directly on this doorbell now too with a little hack. https://www.reddit.com/r/EufyCam/comments/nal27f/enabling_rtsp_on_battery_doorbell/

Thx for replicating the issue and confirming that this is indeed a hyper-v bug.
I think it would be good to test also if this is also an issue with windows guests on hyper-v or if only Unix systems are affected.

[jasonwfrancis1]

Hey I installed Homebridge on Hyper-V running on Windows 10 today (i run hoobs on a raspberry pi, and I really want to use this plugin!) and I think I am getting the same issue. The video stream just will not start.

VIdeo stream was stopped after transmission of 14 chunks.
FFmpeg exited with code: null and signal: SIGKILL (Forced)

I tried enabling experimental and RTSP and that works for a short time, but after a while the plugin returns a 404 error saying the stream cannot be found.

[thieren]

Hey I installed Homebridge on Hyper-V running on Windows 10 today (i run hoobs on a raspberry pi, and I really want to use this plugin!) and I think I am getting the same issue. The video stream just will not start.

VIdeo stream was stopped after transmission of 14 chunks. FFmpeg exited with code: null and signal: SIGKILL (Forced)

I tried enabling experimental and RTSP and that works for a short time, but after a while the plugin returns a 404 error saying the stream cannot be found.

Hey @jasonwfrancis1

For now I can now advise to not use hyper-v. If you have to run in a vm, I'd suggest trying VMWare (for me that works fine) or Virtualbox.

Also please don't confuse this library with the homebridge-eufy-security plugin. Most of the things you referenced are only related to the plugin and not this library directly.

Regarding the issue with the experimental rtsp stream I'd suggest you open a new issue in the plugin repo.

[bropat]

@thieren @jasonwfrancis1

Could you try the following and see if that also solves the streaming problem?

bropat/eufy-security-ws#130 (comment)

Let me know, please.

[thieren]

@thieren @jasonwfrancis1

Could you try the following and see if that also solves the streaming problem?

bropat/eufy-security-ws#130 (comment)

Let me know, please.

Will do as soon as I have the time. Thx for having a deeper look into this!

[bropat]

@thieren

With Gen 2 VMs this fixes also the issue:

ethtool --offload eth0 rx off tx off

[jasonwfrancis1]

Thanks, I’m on vacation currently so will try it when I get back.

…

Sent from my iPhone On 29 Aug 2022, at 08:59, bropat ***@***.***> wrote:  @thieren<https://nam12.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fthieren&data=05%7C01%7C%7C18dfebdcef234898589708da8983a698%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C637973495776727912%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=EFoyltcH6l5DvrG3lqFHNZ396wQqrk6owys1Cy4z4xs%3D&reserved=0> With Gen 2 VMs this fixes also the issue: ethtool --offload eth0 rx off tx off — Reply to this email directly, view it on GitHub<https://nam12.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fbropat%2Feufy-security-client%2Fissues%2F185%23issuecomment-1229812627&data=05%7C01%7C%7C18dfebdcef234898589708da8983a698%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C637973495776727912%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=Yk2UilJ34KXrX2RS4JiktA8e8%2BCAVDrkG3wYPoggzq0%3D&reserved=0>, or unsubscribe<https://nam12.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FABQHZM5US7PDN5QXD5MWYH3V3RGUPANCNFSM537SH6WQ&data=05%7C01%7C%7C18dfebdcef234898589708da8983a698%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C637973495776727912%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=%2BNDJwyhRmAblXnUykQoNrPGIJwzHPYnxT%2FJPOqxmrMk%3D&reserved=0>. You are receiving this because you were mentioned.Message ID: ***@***.***>

[thieren]

@thieren

With Gen 2 VMs this fixes also the issue:

ethtool --offload eth0 rx off tx off

I can confirm this fixed it for me.

[ericflickner]

This fixed the issue for me. I put this and some other commands to install ethtool, then run the above ethtool cmd in the startup script. Without this cmd run on startup, my live streams do not work. With the cmd run on startup, my live streams work each time. Even between container and homebridge server restarts.