Rosbag duration in index_recorded_data.py

[Nanoseb]

In the generated html some durations are quite off, for example the bag from the 4th of September 2017 "14:49:47 [horten_fleetrace]" lasted only 1 minute but seems to be a bit longer ;)

[takluyver]

Here's where it comes from:

sailing-robot/utilities/index_recorded_data.py

Line 80 in cbc725b

duration = b.get_end_time() - b.get_start_time()

[Nanoseb]

maybe related to ros/ros_comm#952

[Nanoseb]

Ok the issue is more scary than that :/
The time is ok because the bagfile is not complete, 14:49:47 [horten_fleetrace] for example should be quite long but it lasts 1min and weight 13M... so we've lost some data.

[takluyver]

Sometimes we get .bag.active files - I think when ROS isn't shut down cleanly. We copied some code we found to fix them up into usable .bag files, but I haven't looked into what it's doing. I assumed you lost some data in that case, but I would have expected it to be the last couple of minutes rather than most of your run.

If the Raspi lost power while ROS was running, that may cause further data loss - data written to file will initially be cached in volatile memory, so it can still be lost on a power failure. Maybe the system caches writes quite heavily to minimise write cycles on the flash storage.

If I found the time, I'd love to dig into this issue more. If Rosbag is doing something that fails badly when it loses power, a better recording system could be really valuable to a lot of projects.

[smaria]

Interesting. i was under the assumption the map plots are generated from the bags. However, looking at /dbg_distance_to_waypoint for 14:49:47 [horten_fleetrace], no waypoint is reached. Where does the complete plotted triangle race come from, if not from said bag?

(as a side note, it is rather inconvenient that the message types recorded in a rosbag are not contained in the bag, I currently don't have a workspace for sailrobot at hand, and it does put a question mark on data longevity, so I can't easily look at the actual gps data)

[Nanoseb]

Map plots are generated thanks to the dump of the gps trace (.csv file formatted according to the rules). See https://www.southampton.ac.uk/~yc6n13/sailing-robot/gps-trace_horten_fleetrace_2017-09-04T14.49.52.csv

[smaria]

Thanks for explaining.
It seems to be accepted that rosbag drops messages if the storage device is not fast enough.
https://answers.ros.org/question/240948/rosbag-record-dropping-and-corrupting-data/
That seems an unlikely reason for our problem, both considering the low amount of data we generate and that the rosbag just stops rather than missing a couple of messages.

It may be worth setting the rosbag recording node to respawn, at least in the model launch file this is not the case: https://github.com/Maritime-Robotics-Student-Society/sailing-robot/blob/master/src/sailing_robot/launch/model_launch_file.launch

This won't get us to the bottom of this, but should this be due to rosbag crashing, the next time it happens we should notice two bags being recorded and we will only miss a smaller amount of data. Is it possible to dig out the ROS log files from that date? Is there any chance this rosbag is not the correct one for the map data?

There is also one more unsolved report of rosbag dropping data: https://answers.ros.org/question/62833/rosbag-not-recording-every-message/

[takluyver]

At the moment the indexing uses the rosbag to define when a run starts and ends. If the recording restarts, you'll likely see an extra run in the index without the GPS trace or parameter dump. (Or the indexing might crash if it assumes the GPS trace is there). I'm sure there are ways to work round that.