Suggestion: serialize/deserialize with the serde crate

[rjzak]

The serde crate is great for reading/writing data to disk with different formats. It works with the bincode crate to write data as raw bytes, which this crate does.

However, the code in this crate is difficult to read, especially with the extensive use of the libc crate. I'd like to suggest re-writing to use serde, bincode, with a more idiomatic Rust style, with a few C helper functions as needed. Instead, this feels like a C project with uncomfortable Rust wrappers.

I'm happy to help and do the bulk of this if it would be accepted.

[cccs-douglass]

Just off the top, I'll pretty happily concede this crate is pretty rough around the edges and was actually written as a shared object library for c/python that incidentally could be used from rust. I think you're right that the c cruft is getting in the way. I would probably move all the c related stuff to a separate folder, probably a non-uploaded crate that only builds the shared library. Something like:

Cargo.toml  <- workspace definition
/cart-container/Cargo.toml <- The usable rust crate
/cart-container/src/... 
/exported/Cargo.toml <- Configured to depend on above and export c interface
/exported/src/lib.rs

At the moment I don't think the part of the library that is actually meant to be used from rust uses the libc crate at all. That should all be in the wrapper functions meant to export. The principle rust interface is basically only the functions pack_stream,unpack_stream, and the Digester trait. The encoder should definitely come with a builder to simplify setting the arguments.

The actual file format this library writes/reads is already fixed so we can't use bincode so much that it would effect the encoded output, but it could be configured to write fixed length numbers LE so that we could use it to read the header information and clean up that part of the code. The format is meant to be a byte stream container so I'm not sure how much more we could use serde itself (beyond the use of serde_json since we require the metadata to be json and possible use of bincode as above), could you give me an example?

[rjzak]

This example https://github.com/second-state/rust-by-example-ext/blob/master/examples/serde/bincode/src/main.rs has this output:

i32 number 5 serializes into byte array [5, 0, 0, 0]
f32 number 3.14 serializes into byte array [195, 245, 72, 64]
Vec<u8> [1, 2, 3] serializes into byte array [3, 0, 0, 0, 0, 0, 0, 0, 1, 2, 3]
Vec<f32> [3.141, 2.718, 1.618] serializes into byte array [3, 0, 0, 0, 0, 0, 0, 0, 37, 6, 73, 64, 182, 243, 45, 64, 160, 26, 207, 63]
tuple (1, "hello", 4.5, true) serializes into byte array [1, 0, 0, 0, 5, 0, 0, 0, 0, 0, 0, 0, 104, 101, 108, 108, 111, 0, 0, 144, 64, 1]
nested tuple ((1, 2), (3.141, 'a'), true) serializes into byte array [1, 2, 0, 37, 6, 73, 64, 97, 1]
struct Point serializes into byte array [0, 0, 128, 63, 0, 0, 0, 64]
struct Point serializes into byte array [0, 0, 64, 64, 0, 0, 128, 64]
struct Line serializes into byte array [2, 0, 0, 0, 0, 0, 0, 0, 0, 0, 128, 63, 0, 0, 0, 64, 0, 0, 64, 64, 0, 0, 128, 64, 1, 243, 4, 53, 64, 11, 0, 0, 0, 0, 0, 0, 0, 97, 32, 116, 104, 105, 110, 32, 108, 105, 110, 101]

Which appears to be Big Endian, which is odd and maybe problematic (but easier to read). But I do realise the CaRT format is more complex, and has internal byte sequences to mark some of the data. But serde & bincode with custom options is an option https://docs.rs/bincode/latest/bincode/config/index.html but I haven't done this so I don't know if it'll work.