data format
The sampled data can be stored either in CSV or in binary RLD format, which is explained below.
A RLD file consists of a file header and a data section. The header contains a constant size lead-in, a comment and a list of channel structures describing all sampled channels. The data part contains a list of data blocks. All blocks are aligned to 32-bit boundaries and the data is saved in little endian format.
The following diagram shows an overview of the file structure. Red blocks define actually stored data, whereas other colors are used to highlight the logical structure of the file.
A file version number (see lead-in) is used to track changes and extensions to the data format and guarantee backward compatibility for data import scripts. A list at the end this section keeps track of changes introduced in different RocketLogger releases: File Format Revisions.
The file header lead-in contains general measurement information and has a fixed size.
| Offset | Size (byte) | Field | Content |
|---|---|---|---|
| 0x00 | 4 | Magic Number |
%RLD (= 0x444C5225) magic number (unsigned integer) |
| 0x04 | 2 | File Version | Current file version number (unsigned integer) |
| 0x06 | 2 | Header Length | Total header length in bytes (unsigned integer) |
| 0x08 | 4 | Data Block Size | Data block size in samples (unsigned integer) |
| 0x0C | 4 | Data Block Count | Number of data blocks (unsigned integer), last block may not be full |
| 0x10 | 8 | Sample Count | Total number of samples taken (unsigned integer) |
| 0x18 | 2 | Sampling Rate | Sampling rate in SPS (unsigned integer) |
| 0x1A | 6 x 1 | MAC Address | MAC address as device identifier, stored in big endian (unsigned integer) |
| 0x20 | 8 | Start Time Seconds | Start time seconds in UNIX UTC time (integer) |
| 0x28 | 8 | Start Time Nanoseconds | Start time in nanoseconds (integer) |
| 0x30 | 4 | Comment Length | Length of header comment in bytes (unsigned integer), needs to be a multiple of 4 for alignment purposes |
| 0x34 | 2 | Binary Channel Count | Number of binary channels (unsigned integer), nb |
| 0x36 | 2 | Analog Channel Count | Number of analog channels (unsigned integer), na |
Total size: 56 bytes
An additional comment can be put here in ASCII format. Its size is defined in the header lead-in and it is aligned to 32-bit using bit stuffing.
This part contains detailed information about all sampled channels. There can be an custom number of channels (defined in lead-in field Binary Channel Count and Analog Channel Count). The channel order determines the channel position in the data block. Each channel can have a different sample size (Data Size field).
| Offset | Size (byte) | Field | Content |
|---|---|---|---|
| 0x00 | 4 | Channel Unit | Encoded channel unit 1 (integer) |
| 0x04 | 4 | Channel Scale | Channel scaling in power of tens, ignored for binary channels (integer) |
| 0x08 | 2 | Data Size | Channel sample size in bytes, ignored for binary channels (unsigned integer) |
| 0x0A | 2 | Valid Data Link | Link to additional range valid binary channel (unsigned integer, using zero based indexing) |
| 0x0C | 16 x 1 | Channel Name | Name of channel (char array) |
Total size per channel: 28 bytes
| Code | Unit |
|---|---|
| 0 | Unit-less |
| 1 | Voltage (in V) |
| 2 | Current (in A) |
| 3 | Binary |
| 4 | Data Valid (binary) |
| 5 | Illuminance (in lux) |
| 6 | Temperature (in centigrade) |
| 7 | Integer |
| 8 | Percent |
| 9 | Pressure (in bar) |
| -1 | Undefined |
The channel data is stored in a sequence of blocks. The number of blocks is defined in the
lead-in field Data Block Count. Every data block starts with two different
timestamps, which both specify the time at which the first samples of each block was read.
The first timestamp is stored in UNIX format (UTC) and is read from the Linux OS real-time clock
(clock_gettime() using CLOCK_REALTIME). This timestamp is adjusted by the operating system
when time updates are received via network using any time synchronization protocol. The second
timestamp is read from the Linux OS raw monotonic timer and is unaffected by these measurements
(clock_gettime() using CLOCK_MONOTONIC_RAW). It is also stored in UNIX format with a fixed,
but unspecified time reference in the past and always refers to the (drifting) hardware clock.
More details about the differences of these timers can be found in the
Linux manpage of clock_gettime(2).
The timestamps are followed by a specific number of samples (defined in lead-in field Data Block Size). The last block may not be full. The binary channels are merged into bit-stuffed 32-bit values, right aligned with the first channel at LSB. The number of binary values nbv is equal to nbv = ceil(nb/32), where nb is lead-in field Binary Channel Count. nbv may be zero.
An overview of the data block structure is given on the diagram below.
| Offset | Size (byte) | Field | Content |
|---|---|---|---|
| 0x00 | 8 | Seconds | Timestamp (real-time clock) of first sample in UNIX UTC seconds (integer) |
| 0x08 | 8 | Nanoseconds | Nanoseconds of timestamp (real-time clock) of first sample (integer) |
| 0x10 | 8 | Seconds | Timestamp (raw monotonic clock) of first sample in seconds (integer) |
| 0x18 | 8 | Nanoseconds | Nanoseconds of timestamp (monotonic clock) of first sample (integer) |
| 0x20 | nbv x 4 (nbv = ceil(nb/32)) | Binary Values | Merged binary channels, right aligned, stuffed to 32-bit |
| 0x#0 | CH1 size (defined in channel field Data Size) | Analog Channel 1 | First sample of analog channel one (integer of header defined size) |
| ... |
-
Unsupported development format without specification. Can be used for own experimental modification to the format.
-
Additions:
- Initial definition of the data format
Changes:
- none
Remarks:
- First stable and supported version
-
Additions:
- unit definitions for environmental sensors
- introduce separate "undefined" unit
Changes:
- channel links are now zero indexed (before one based, but undocumented)
- rename unit with index 0x00 to "unit-less"
Remarks:
- Updated indexing of the channel link requires update to data import scripts
-
Additions:
- unit definitions for time difference
Changes:
- none
The CSV file written by the RocketLogger has a header section of 9 rows containing measurement and
device information. Separated by a blank line, the channel data header including the units of data
follows. Starting at row 12, the actual measurement data is stored. Similar to the binary format,
data is stored in blocks of header defined size. The first sample of every block is timestamped in
the UNIX format (UTC) and nanosecond resolution. The remaining samples of the block have an empty
timestamp. The columns of the CSV files are separated by the comma (,) character. A sample to
illustrate the structure of the CSV file is given below.
RocketLogger CSV File
File Version,3
Block Size,1000
Block Count,4
Sample Count,4000
Sample Rate,1000
MAC Address,12:34:56:78:90:ab
Start Time,Fri Dec 1 18:46:59 2017
Comment,Your file comment
,DI1,DI2,DI3,DI4,DI5,DI6,I1L_valid,I2L_valid,I1H [nA],I1L [10pA],V1 [10nV],V2 [10nV],I2H [nA],I2L [10pA],V3 [10nV],V4 [10nV]
1512154019.573057418,0,0,0,0,0,0,1,1,48004,-69945,19608482,99129414,31489,-262,-242,-599037712
,0,0,0,0,0,0,1,1,46238,-70173,19623435,99129535,31016,-507,-849,-599040138
,0,0,0,0,0,0,1,1,44156,-70085,19638144,99129171,31899,-735,-1334,-599039531
...
Limitations
For longer term and/or high-data rate measurements the binary format should be used, because text based storing of data is storage inefficient and demands for much higher file system throughput. Corruption of CSV files might occur if the file system bandwidth cannot handle the higher measurement data rates.
Writing to CSV files is only supported for voltages, currents and binary inputs.