UART serial stall

[holmbuar]

@pablogs9 with reference to #937, could you please test my UART serial publisher example at

https://github.com/torlarse/micro_ros_raspberrypi_pico_sdk/blob/humble-uart-vscode/examples/uart/publisher/publisher.c

With the current timeout setting, it spins around 2-500 times, then stall.

My setup is Pico with Debug Probe, free running a DEBUG build, but not in a debug session. For UART testing, I suppose connecting two Picos would render the same result.

[pablogs9]

Does it works if instead of having a timer, you publish in a loop with delay inside main?

[holmbuar]

same result really, the longer delay, the more spins before stalling. For example, a second delay might reach 1200 spins, while 100 ms delay could reach 80.

[pablogs9]

If instead of having one call to rclc_executor_spin_period you use something like:

while(1) {
  your_delay_func(100);
  rclc_executor_spin_some(&executor, RCL_MS_TO_NS(100));
}

does it change anything?

[holmbuar]

I changed to rclc_executor_spin_period, after testing with the spin_some() function. It seemed like the spin_period was more stable than spin_some, but eventually stalled.

[pablogs9]

Is there any possibility of your transport allocating memory somehow? Could you implement some custom allocators and check if something is continuously allocating memory?

I'm wondering if this is a stack override or some kind of heap smash

[holmbuar]

I have not gotten to read up on custom allocators yet. One thing I notice though, is that when I run the agent in debug mode

docker run -it --rm -v /dev:/dev --privileged --net=host microros/micro-ros-agent:humble serial --dev /dev/ttyACM0 -b 115200 -v6

and just test a custom service call

ros2 service call /motor_data custom_interfaces/srv/MotorData

the Pico transmits debug messages to the agent with a frequency of ~10Hz, and does not stop. It is only when I publish something, for example

RCSOFTCHECK(rcl_publish(&publisher, &msg, NULL));
msg.data = delta_time;

that the Pico stalls. I don't know if this is relevant, just an observation.

[pablogs9]

Please paste the output of the agent in the situation you mention

[holmbuar]

sure, here is a snippet:

[1691506748.793534] debug    | SerialAgentLinux.cpp | recv_message             | [==>> SER <<==]        | client_key: 0x79F1AFA6, len: 13, data: 
0000: 81 00 00 00 0B 01 05 00 04 00 05 00 80
[1691506748.793689] debug    | SerialAgentLinux.cpp | send_message             | [** <<SER>> **]        | client_key: 0x79F1AFA6, len: 13, data: 
0000: 81 00 00 00 0A 01 05 00 06 00 00 00 80
[1691506749.794075] debug    | SerialAgentLinux.cpp | recv_message             | [==>> SER <<==]        | client_key: 0x79F1AFA6, len: 13, data: 
0000: 81 00 00 00 0B 01 05 00 04 00 05 00 80
[1691506749.794277] debug    | SerialAgentLinux.cpp | send_message             | [** <<SER>> **]        | client_key: 0x79F1AFA6, len: 13, data: 
0000: 81 00 00 00 0A 01 05 00 06 00 00 00 80
[1691506750.794267] debug    | SerialAgentLinux.cpp | recv_message             | [==>> SER <<==]        | client_key: 0x79F1AFA6, len: 13, data: 
0000: 81 00 00 00 0B 01 05 00 04 00 05 00 80
[1691506750.794370] debug    | SerialAgentLinux.cpp | send_message             | [** <<SER>> **]        | client_key: 0x79F1AFA6, len: 13, data: 
0000: 81 00 00 00 0A 01 05 00 06 00 00 00 80
[1691506751.794720] debug    | SerialAgentLinux.cpp | recv_message             | [==>> SER <<==]        | client_key: 0x79F1AFA6, len: 13, data: 
0000: 81 00 00 00 0B 01 05 00 04 00 05 00 80
[1691506751.794973] debug    | SerialAgentLinux.cpp | send_message             | [** <<SER>> **]        | client_key: 0x79F1AFA6, len: 13, data: 
0000: 81 00 00 00 0A 01 05 00 06 00 00 00 80
[1691506752.795185] debug    | SerialAgentLinux.cpp | recv_message             | [==>> SER <<==]        | client_key: 0x79F1AFA6, len: 13, data: 
0000: 81 00 00 00 0B 01 05 00 04 00 05 00 80
[1691506752.795383] debug    | SerialAgentLinux.cpp | send_message             | [** <<SER>> **]        | client_key: 0x79F1AFA6, len: 13, data: 
0000: 81 00 00 00 0A 01 05 00 06 00 00 00 80
[1691506753.795454] debug    | SerialAgentLinux.cpp | recv_message             | [==>> SER <<==]        | client_key: 0x79F1AFA6, len: 13, data: 
0000: 81 00 00 00 0B 01 05 00 04 00 05 00 80
[1691506753.795639] debug    | SerialAgentLinux.cpp | send_message             | [** <<SER>> **]        | client_key: 0x79F1AFA6, len: 13, data: 
0000: 81 00 00 00 0A 01 05 00 06 00 00 00 80
[1691506754.795872] debug    | SerialAgentLinux.cpp | recv_message             | [==>> SER <<==]        | client_key: 0x79F1AFA6, len: 13, data: 
0000: 81 00 00 00 0B 01 05 00 04 00 05 00 80
[1691506754.796063] debug    | SerialAgentLinux.cpp | send_message             | [** <<SER>> **]        | client_key: 0x79F1AFA6, len: 13, data: 
0000: 81 00 00 00 0A 01 05 00 06 00 00 00 80
[1691506755.796289] debug    | SerialAgentLinux.cpp | recv_message             | [==>> SER <<==]        | client_key: 0x79F1AFA6, len: 13, data: 
0000: 81 00 00 00 0B 01 05 00 04 00 05 00 80
[1691506755.796455] debug    | SerialAgentLinux.cpp | send_message             | [** <<SER>> **]        | client_key: 0x79F1AFA6, len: 13, data: 
0000: 81 00 00 00 0A 01 05 00 06 00 00 00 80
[1691506756.171862] debug    | Replier.cpp        | read_fn                  | [==>> DDS <<==]        | client_key: 0x00000000, len: 28, data: 
0000: 01 0F 02 E2 19 18 CF FD 01 00 00 00 00 00 12 03 00 00 00 00 01 00 00 00 00 00 00 00
[1691506756.171926] debug    | SerialAgentLinux.cpp | send_message             | [** <<SER>> **]        | client_key: 0x79F1AFA6, len: 40, data: 
0000: 81 80 05 00 09 01 20 00 00 0F 00 08 01 0F 02 E2 19 18 CF FD 01 00 00 00 00 00 12 03 00 00 00 00
0020: 01 00 00 00 00 00 00 00
[1691506756.177808] debug    | SerialAgentLinux.cpp | send_message             | [** <<SER>> **]        | client_key: 0x79F1AFA6, len: 13, data: 

[holmbuar]

This turned out to be a non-issue, it's just a matter of enabling UART in

micro_ros_raspberrypi_pico_sdk/CMakeLists.txt

Line 29 in b2f2734

pico_enable_stdio_usb(pico_micro_ros_example 1)

Possible clarification, for future users, in #955