tartan_dataset_management

Collection of tools to manage ROSbag recordings data from AV vehicle

Metadata Generator Usage

This script generates metadata for ROSbag MCAP files. The metadata is compiled into a resources.json file that complies with the EIDF requirements

Features

• Reads MCAP files and extracts metadata such as:
  • Duration of the log
  • Topics and message counts
  • File size
  • File hash (MD5)
• Generates a JSON file (resources.json) with metadata for all MCAP files in a given directory.
• Metadata includes:
  • File name
  • Identifier
  • Description
  • Format
  • License
  • Size
  • Hash
  • Issued date
  • Modified date
  • Duration
  • Topics and message counts

Usage

1. Setup

Ensure all dependencies are installed. You can use the following command to install required packages:

pip install mcap

2. Running the Script

To generate the metadata JSON file, follow these steps:

• Place all your MCAP files in a directory.

• The default directory is /recorded_datasets/edinburgh

• Run the script:

  python metadata_generator.py

If you want to generate metadata for files in a specified path, run the script:

python metadata_generator.py -p path/to/file

3. Output

The script will generate a resources.json file in the specified directory. This JSON file will contain metadata for each MCAP file in the directory.

Upload Vehicle Data

This script automates the process of uploading rosbags from the IPAB-RAD autonomous vehicle server to a cloud instance within the EIDF (Edinburgh International Data Facility) infrastructure. It streamlines data collection and transfer by first compressing the rosbags using the MCAP CLI, and then uploading the compressed files. This ensures efficient handling and storage of large datasets generated by vehicle sensors.

1. Dependencies

Host Machine (Cloud instance)

• Install iperf3:

  sudo apt-get install iperf3

• Set up SSH key authentication:
  • Generate an SSH key:

    ssh-keygen -t rsa -b 2048

  • Copy the SSH key to the remote machine:

    ssh-copy-id username@hostname_ip

Remote Machine (Vehicle Server)

• Install iperf3:

  sudo apt-get install iperf3

• Install MCAP CLI 0.0.47 for compressing rosbags:

  wget -O $HOME/mcap https://github.com/foxglove/mcap/releases/download/releases%2Fmcap-cli%2Fv0.0.47/mcap-linux-amd64 && chmod +x $HOME/mcap

2. Usage

To execute the script, use the following command from the host machine:

python3 upload_vehicle_data.py -config <path_to_yaml_config>

or

python3 upload_vehicle_data.py -c <path_to_yaml_config>

Replace <path_to_yaml_config> with the path to your YAML configuration file. Use the -d or --debug option to enable detailed information:

python3 upload_vehicle_data.py -c <path_to_yaml_config> -d

3. YAML Parameters

• remote_user (str): Username for the remote machine.
• remote_ip (str): IP address of the remote machine (default: 129.215.117.104).
• remote_directory (str): Directory on the remote machine containing rosbags (default: /mnt/mydrive/rosbags).
• cloud_upload_directory (str): Remote directory for uploading compressed files (default: /mnt/vdb/data).
• clean_up (bool): Whether to delete all rosbags from the vehicle machine after uploading.
• upload_attempts (int): Number of attempts to upload each rosbag file to the cloud host (default: 3).
• mcap_path (str): Path to the mcap CLI binary. Found using which mcap if installed.
• parallel_processes (int): Number of parallel processes for compression and upload.

Example YAML configuration:

remote_user: "username"
remote_ip: "129.215.117.104"
remote_directory: "/mnt/mydrive/rosbags"
cloud_upload_directory: "/mnt/vdb/data"
clean_up: false
upload_attempts: 3
mcap_path: "$HOME/mcap"
parallel_processes: 1

4. Script Workflow

• Measures available bandwidth using iperf3.
• Creates a remote temporary directory for storing compressed .mcap files.
• Lists all .mcap files in the specified directory.
• Displays the total number of files, their combined size, and the estimated upload time based on the measured bandwidth.
• Prompts for user confirmation before proceeding with the upload.
• Compresses each .mcap file using mcap CLI with zstd level 2.
• Uploads each compressed file to the remote server using rsync.
• Verifies the integrity of each uploaded file.
• Deletes the original and compressed files from the vehicle after successful upload and verification, if clean_up is set to true.

5. Logging

The script logs its activities to upload_vehicle_data_<timestamp>.log. This log file includes:

• Information about the processed files.
• Bandwidth measurements.
• Any errors encountered during the process.