data format
The sampled data can be stored either in CSV or in binary RLD format, which is explained below.
A RLD file consists of a file header and a data structure. The header contains a constant size lead-in, a comment and a list of channel structures, describing all sampled channels. The data part contains a list of data blocks. All blocks are aligned to 32-bit and the data is saved in little endian.
The following diagram shows an overview of the file structure. Red blocks define data, whereas all other colors are only used to show the logical structure of the file.
A file version number (see lead-in) is used to track changes and extensions to the data format and guarantee backward compatibility for data import scripts. A list at the end this section keeps track of changes introduced in different RocketLogger releases: File Format Revisions.
The file header lead-in contains general measurement information and has a fixed size.
| Offset | Size (byte) | Field | Content |
|---|---|---|---|
| 0x00 | 4 | Magic Number |
%RLD (= 0x444C5225) magic number (unsigned integer) |
| 0x04 | 2 | File Version | Current file version number (unsigned integer) |
| 0x06 | 2 | Header Length | Total header length in bytes (unsigned integer) |
| 0x08 | 4 | Data Block Size | Data block size in samples (unsigned integer) |
| 0x0C | 4 | Data Block Count | Number of data blocks (unsigned integer), last block may not be full |
| 0x10 | 8 | Sample Count | Total number of samples taken (unsigned integer) |
| 0x18 | 2 | Sampling Rate | Sampling rate in SPS (unsigned integer) |
| 0x1A | 6 x 1 | MAC Address | MAC address as device identifier, stored in big endian (unsigned integer) |
| 0x20 | 8 | Start Time Seconds | Start time seconds in UNIX UTC time (integer) |
| 0x28 | 8 | Start Time Nanoseconds | Start time nanoseconds (integer) |
| 0x30 | 4 | Comment Length | Length of header comment in bytes (unsigned integer), needs to be a multiple of 4 for alignment purposes |
| 0x34 | 2 | Binary Channel Count | Number of binary channels (unsigned integer), nb |
| 0x36 | 2 | Analog Channel Count | Number of analog channels (unsigned integer), na |
Total size: 56 bytes
An additional comment can be put here in ASCII format. Its size is defined in the header lead-in and it is aligned to 32-bit using bit stuffing.
This part contains detailed information about all sampled channels. There can be an custom number of channels (defined in lead-in field "Binary Channel Count" and "Analog Channel Count"). The channel order determines the channel position in the data block. Each channel can have a different sample size ("Data Size" field).
| Offset | Size (byte) | Field | Content |
|---|---|---|---|
| 0x00 | 4 | Channel Unit | Encoded channel unit 1 (integer) |
| 0x04 | 4 | Channel Scale | Channel scaling in power of tens, ignored for binary channels (integer) |
| 0x08 | 2 | Data Size | Channel sample size in bytes, ignored for binary channels (unsigned integer) |
| 0x0A | 2 | Valid Data Link | Link to additional range valid binary channel (unsigned integer, using zero based indexing) |
| 0x0C | 16 x 1 | Channel Name | Name of channel (char array) |
Total size per channel: 28 bytes
| Code | Unit |
|---|---|
| 0 | Unit-less |
| 1 | Voltage (in V) |
| 2 | Current (in A) |
| 3 | Binary |
| 4 | Data Valid (binary) |
| 5 | Illuminance (in lux) |
| 6 | Temperature (in centigrades) |
| 7 | Integer |
| 8 | Percent |
| 9 | Pressure (in bar) |
| -1 | Undefined |
The channel data is stored in a sequence of blocks. The number of blocks is defined in lead-in field Data Block Count. Every data block starts with two different timestamps, which both specify the time at which the first samples of each block was read. The first timestamp is stored in UNIX format (UTC) and is read from the Linux OS real-time clock (clock_gettime() using CLOCK_REALTIME). This timestamp is adjusted by the operating system when time updates are received via network using any time synchronization protocol. The second timestamp is read from the Linux OS raw monotonic timer and is unaffected by these measurements (clock_gettime() using CLOCK_MONOTONIC_RAW). It is also stored in UNIX format with a fixed, but unspecified time reference in the past and always refers to the (drifting) hardware clock . More details about the differences of these timers can be found in the Linux manpage of clock_gettime(2).
The timestamps are followed by a specific number of samples (defined in lead-in field Data Block Size). The last block may not be full. The binary channels are merged into bit-stuffed 32-bit values, right aligned with the first channel at LSB. The number of binary values nbv is equal to nbv = ceil(nb/32), where nb is lead-in field Binary Channel Count. nbv may be zero.
An overview over a data block can be seen in the following figure.
| Offset | Size (byte) | Field | Content |
|---|---|---|---|
| 0x00 | 8 | Seconds | Timestamp (real-time clock) of first sample in UNIX UTC seconds (integer) |
| 0x08 | 8 | Nanoseconds | Nanoseconds of timestamp (real-time clock) of first sample (integer) |
| 0x10 | 8 | Seconds | Timestamp (raw monotonic clock) of first sample in seconds (integer) |
| 0x18 | 8 | Nanoseconds | Nanoseconds of timestamp (monotonic clock) of first sample (integer) |
| 0x20 | nbv x 4 (nbv = ceil(nb/32)) | Binary Values | Merged binary channels, right aligned, stuffed to 32-bit |
| 0x#0 | CH1 size (defined in channel field Data Size) | Analog Channel 1 | First sample of analog channel one (integer of header defined size) |
| ... |
-
Unsupported development format without specification. Can be used for own experimental modification to the format.
-
Additions:
- Initial definition of the data format Changes:
- none Remarks:
- First stable and supported version
-
Additions:
- unit definitions for environmental sensors
- introduce separate "undefined" unit Changes:
- channel links are now zero indexed (before one based, but undocumented)
- rename unit with index 0x00 to "unit-less" Remarks:
- Updated indexing of the channel link requires update to data import scripts
-
Additions:
- unit definitions for time difference
The CSV file written by the RocketLogger has a header section of 9 rows containing measurement and device information.
Separated by a blank line, the channel data header including the units of data follows.
Starting at row 12, the actual measurement data is stored. Similar to the binary format, data is stored in blocks of header defined size. The first sample of every block is timestamped in the UNIX format (UTC) and nanosecond resolution. The remaining samples of the block have an empty timestamp.
The columns of the CSV files are separated by the comma (,) character. A sample to illustrate the structure of the CSV file is shown below.
RocketLogger CSV File
File Version,3
Block Size,1000
Block Count,4
Sample Count,4000
Sample Rate,1000
MAC Address,12:34:56:78:90:ab
Start Time,Fri Dec 1 18:46:59 2017
Comment,Your file comment
,DI1,DI2,DI3,DI4,DI5,DI6,I1L_valid,I2L_valid,I1H [nA],I1L [10pA],V1 [10nV],V2 [10nV],I2H [nA],I2L [10pA],V3 [10nV],V4 [10nV]
1512154019.573057418,0,0,0,0,0,0,1,1,48004,-69945,19608482,99129414,31489,-262,-242,-599037712
,0,0,0,0,0,0,1,1,46238,-70173,19623435,99129535,31016,-507,-849,-599040138
,0,0,0,0,0,0,1,1,44156,-70085,19638144,99129171,31899,-735,-1334,-599039531
...
Limitations:
For longer term and/or high-data rate measurements the binary format should be used, because text based storing of data is storage inefficient and demands for much higher file system throughput. Corruption of CSV files might occur if the file system bandwidth cannot handle the higher measurement data rates.
Writing to CSV files is only supported for voltages, currents and binary inputs.