Audio lost and only fix is to restart

[sm0tsc]

Hi since some time xlxd stops to pass audio. Callsigns etc is passed
Any clue how to debug? Or fix. I don´t find anything specific in logs. XLX010 is using last version
//SM0TSC

[LX1IQ]

Hi,
Normally it is not the XLX, which is crashing, but the usb port of the AMBed server.
You should see very high packet loses in the log.
Most of the time it is sufficient to kill the AMBed task and start it again.
In the download section, you can find a small script, which allows you to run the AMBed as a service.
You can schedule a restart of the service at night.
We need to add some sort of watchdog to the AMBed. So if the dongle get unresponsive the service can restarts by its own.

73, Luc

[sm0tsc]

This also occurs for d-star to d-star connections //Johan Den fre 8 maj 2020 kl 09:46 skrev LX1IQ <[email protected]>:

…

Hi, Normally it is not the XLX, which is crashing, but the usb port of the AMBed server. You should see very high packet loses in the log. Most of the time it is sufficient to kill the AMBed task and start it again. In the download section, you can find a small script, which allows you to run the AMBed as a service. You can schedule a restart of the service at night. We need to add some sort of watchdog to the AMBed. So if the dongle get unresponsive the service can restarts by its own. 73, Luc — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#168 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABSGDX26B4IUANMKWYXVZYLRQO2ELANCNFSM4M3UOHEA> .

[LX1IQ]

Do to the fact that all traffic is always sent to the AMBed server. All modes are affected.

73, Luc

[sm0tsc]

Ok can you give some example on how the logentry will look like? //Johan Den fre 8 maj 2020 kl 09:53 skrev LX1IQ <[email protected]>:

…

Do to the fact that all traffic is always sent to the AMBed server. All modes are affected. 73, Luc — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#168 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABSGDX2RY2RCDWXAQEFH6VDRQO27LANCNFSM4M3UOHEA> .

[sm0tsc]

I ususally restart xlxd wich makes it work again.. ambeD is on another site //Johan Den fre 8 maj 2020 kl 09:46 skrev LX1IQ <[email protected]>:

…

Hi, Normally it is not the XLX, which is crashing, but the usb port of the AMBed server. You should see very high packet loses in the log. Most of the time it is sufficient to kill the AMBed task and start it again. In the download section, you can find a small script, which allows you to run the AMBed as a service. You can schedule a restart of the service at night. We need to add some sort of watchdog to the AMBed. So if the dongle get unresponsive the service can restarts by its own. 73, Luc — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#168 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABSGDX26B4IUANMKWYXVZYLRQO2ELANCNFSM4M3UOHEA> .

[LX1IQ]

You are going to see in the XLX log very high packet lost for the AMBed.
If you kill the AMbed, it takes a few seconds until the XLX disconnects the AMBed server with a keep alive timeout.
At this point, D-Star to D-Star should already come back.

73, Luc

[sm0tsc]

Why does it get "fixed" by restarting xlxd then? I usually restart only xlxd when this happens. As ambed is on another host in my lab //Johan Den fre 8 maj 2020 kl 10:00 skrev LX1IQ <[email protected]>:

…

You are going to see in the XLX log very high packet lost for the AMBed. If you kill the AMbed, it takes a few seconds until the XLX disconnects the AMBed server with a keep alive timeout. At this point, D-Star to D-Star should already come back. 73, Luc — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#168 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABSGDX4PQT4SEWKDXBNQ7IDRQO3ZVANCNFSM4M3UOHEA> .

[LX1IQ]

If you stop the XLX, of course the connection on the AMBed side is disconnecting too.
When you start the XLX back on, a new connection is established to the AMBed server which is resetting the dongle connection, connected to the AMbed as well.

[sm0tsc]

Tnx info I will do a script restarting ambed.. Hopefully we can monitor it somehow in the future //Johan Den fre 8 maj 2020 kl 10:10 skrev LX1IQ <[email protected]>:

…

If you stop the XLX, of course the connection on the AMBed side is disconnecting too. When you start the XLX back on, a new connection is established to the AMBed server which is resetting the the dongle connection, connected to the AMbed as well. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#168 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABSGDXZ4U7M5MFO45FWQYZ3RQO47LANCNFSM4M3UOHEA> .

[sm0tsc]

Seems that the easiest way will be adding this to the systemd script Restart=always RuntimeMaxSec=604800 (and a suitable number of seconds) //SM0TSC Den fre 8 maj 2020 kl 10:10 skrev LX1IQ <[email protected]>:

…

If you stop the XLX, of course the connection on the AMBed side is disconnecting too. When you start the XLX back on, a new connection is established to the AMBed server which is resetting the the dongle connection, connected to the AMbed as well. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#168 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABSGDXZ4U7M5MFO45FWQYZ3RQO47LANCNFSM4M3UOHEA> .

[sm0tsc]

My ambed.service as an example [Unit] Description=Ambed Service Requires=network.target After=syslog.target network.target [Service] Type=simple Restart=always RuntimeMaxSec=14400 RestartSec=3 StandardOutput=null WorkingDirectory=/usr/local/bin ExecStartPre=-/sbin/rmmod ftdi_sio ExecStartPre=-/sbin/rmmod usbserial ExecStart=/usr/bin/screen -S ambed -D -m /usr/local/bin/ambed 192.168.2.203 ExecStop=/usr/bin/screen -S ambed -X quit [Install] WantedBy=multi-user.target Den fre 8 maj 2020 kl 10:10 skrev LX1IQ <[email protected]>:

…

If you stop the XLX, of course the connection on the AMBed side is disconnecting too. When you start the XLX back on, a new connection is established to the AMBed server which is resetting the the dongle connection, connected to the AMbed as well. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#168 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABSGDXZ4U7M5MFO45FWQYZ3RQO47LANCNFSM4M3UOHEA> .

[sm0tsc]

Seems not to help. Only helps with restarting xlxd.. Any more ideas?

//Johan

[LX1IQ]

Any errors in the log ?

[sm0tsc]

Nothing that I can see out of the ordinary I see some ambed timeouts but no other errors. Restarted ambed with no change, then after some mnutes I restarted xlxd and trafic started flowing again
No ambedstats messages while error persists

Attached is syslog

//Johan
syslog.zip

[bv3un]

same problem here .when the audio not pass through .the "timed out" is appear on /var/log/message
after restart ambed .it becomes normal . xlxd is 2.4.1 /ambed 1.3.4 or 1.3.5 same problem

no packet loss on ambed log .only timed out message on xlxd log as below

Mar 3 05:24:09 debian-reflector xlxd: ambed 170 of 216 packets timed out
Mar 3 06:58:07 debian-reflector xlxd: ambed 152 of 190 packets timed out
Mar 3 08:45:26 debian-reflector xlxd: ambed 123 of 173 packets timed out
Mar 3 12:54:10 debian-reflector xlxd: ambed 207 of 576 packets timed out
Mar 3 12:54:33 debian-reflector xlxd: ambed 252 of 252 packets timed out
Mar 3 12:55:27 debian-reflector xlxd: ambed 2475 of 2538 packets timed out
Mar 3 12:56:20 debian-reflector xlxd: ambed 2273 of 2292 packets timed out
Mar 3 12:57:04 debian-reflector xlxd: ambed 174 of 177 packets timed out
Mar 3 12:58:37 debian-reflector xlxd: ambed 2246 of 2268 packets timed out
Mar 3 12:59:28 debian-reflector xlxd: ambed 235 of 237 packets timed out
Mar 3 13:22:26 debian-reflector xlxd: ambed 74 of 75 packets timed out

[LX1IQ]

Coming back to this issue.
The problem is definitively the FTDI chip (USB port ) on the ambed server side which is crashing.
If you install the latest ambed server software you should see something like this in the log of the ambed server.
We need some more investigations to see if we get always the same error which would make it much easier to fix.

Oct 27 06:25:12 ambed01 ambed[546]: Stream Open from XLX270
Oct 27 06:25:12 ambed01 ambed[546]: Vocodec channel USB-3012_A:0 -> USB-3012_A:1 open
Oct 27 06:25:12 ambed01 ambed[546]: Opened stream 88
Oct 27 06:25:19 ambed01 ambed[546]: FTDI_read_bytes(12) failed : 7
Oct 27 06:26:26 ambed01 ambed[546]: Vocodec channel USB-3012_A:0 -> USB-3012_A:1 closed
Oct 27 06:26:26 ambed01 ambed[546]: 0 of 3718 packets lost
Oct 27 06:26:26 ambed01 ambed[546]: Stream 88 closed

73, Luc

[ShaYmez]

The trouble is, we can put a watchdog on the AMBEd service but the XLXd stream still does not recover. You have to physically restart the XLXD service to recover! Is there a fix for this yet? Maybe add a watchdog for XLX to restart in the event of a failure?