Feature proposal: Thermo raw file reader

[david-bouyssie]

I would propose to implement a reader for Thermo RAW files based on their C# libraries.

I was able to bind the ThermoRawFileParser in this repo:
https://github.com/david-bouyssie/ThermoRawFileParserBindings

It works by streaming the raw file as mzML chunks and then decoding the mzML chunks on the code using the C bindings (relying on Mono).
It's a bit hacky but it allowed to implement an efficient .raw -> .mzDB converter:
https://github.com/mzdb/mzdb4s/blob/master/io-thermo/native/src/main/scala/com/github/mzdb4s/io/thermo/RawFileStreamer.scala

I would propose to port this code to Rust.

[david-bouyssie]

I recently tried to hack around that, and I think it is a very important feature. Not only for Rusteomics but also for tools like Sage. Having native raw file support in Sage would be super great for end-users.

Short story first. I developed some years a Scala Native tool to convert thermo raw files to mzdb in a cross-platform way (at least supporting and Linux). This was a requirement for another project involving some people from EuBIC (https://github.com/wombat-p/) and more specifically the Proline workflow).
The C# interop in this conversion tool relies on the obsolete, ugly, but still working, Embeddinator-4000. The later generates glue code for different target languages, and is leveraging a Mono API that enables C# FFI.

Now I'm happy to announce that I was able to port (partially) this solution to Rust 🎉
It is a bit hacky as it relies on a custom fork of the Compomics ThermoRawFileParser that streams the raw file content as mzML chunks, that can then be decoded from the caller side. Vectors of spectra values (mz, intensity) are however treated in a specific manner. They are memory-allocated from the C# side then the memory address and arrays length are sent back to the caller, that can consume the corresponding data, avoiding back and forth Base64 serialization round-trips.
Using this trick I was able to speed up the conversion process significantly.

In the future I think it would be better to rely on dotnet rather Mono for performance and stability reasons. This could be achieved thanks to the DNNE Microsoft tool. Unfortunately, despite a lot of trials on my side (and also @Luxxii one), we were never able to get a working solution 😕
However we could imagine a more decoupled streaming system relying on IPC. We could spawn a background dotnet process that could serve the raw filen content as a stream (using an efficient binary serialization protocol). It would be similar to the current solution but would work inter process rather than intra process. This eliminates possibility for low-level hacky stuffs (as the one currently used) but could give us something more stable and sustainable.

[douweschulte]

Having support for .raw files would be very beneficial for any proteomics tool. For a slight personal note, this could be THE reason for me to fully commit to merging my rustyms into rusteomics as I would not be able to support .raw files on my own. On the implementation, I do not have a very defined standing I did not ever try C# with Rust interop. But I would vote for a more stable and dependable solution over a faster one in general.

[david-bouyssie]

Good news, I have a first working prototype:)

I'll patiently wait for final agreement before pushing it to Rusteomics.
I think a separate crate that would be a dependency of mzio could be considered.
It will avoid polluting mzio with some hacky mono related stuffs.

[Luxxii]

@david-bouyssie actually this sounds amazing! Would love to help here out a bit! For me, i started to access .raw files directly via python (using fisher-py, which uses dotnet mono) having very fast access as well as a stable access to older and newer RAWs (as far as i can test it).

I could also retry all the steps which we tried in a hackathon in a more strategic manner. Maybe there is a good way today to access through rust C# libraries.

Besides this, an idea to spawn an dotnet instance and accesing it through it sounds also reasonable! Linking the page of netcore-hosting here. We may also might try to retry it with this crate using fisher-py as an example. Just giving suggestions!

[david-bouyssie]

Thank you for your feedback.
I think DNNE relies on netcorehost and for now I'm convinced the compatibility issues comes from the Thermo DLLs. With home made libraries I have no issues to use DNNE.

Regarding fisher_py, I came across it by looking at existing solutions but never tested.
I just see they have an MGF writer example:
https://github.com/ethz-institute-of-microbiology/fisher_py/blob/main/examples/generate_mgf_file_example.py

Would be interesting to try my Rust prototype combined with the mzio MGF writer to see how it competes on large raw files. A second test could consist in writing the MGF file at the Python level while using the Rust raw file access.
I'm pretty sure that PythonNET is adding some overhead (because of Mono) but this has to be tested.

To summarize we have at least three possible technical ways of accessing Thermo raw data:

• tight coupling using Mono -> works
• tight coupling using dotntet -> doesn't work yet?
• loose coupling using IPC -> should work but be less efficient

So we could maybe consider maybe multiple backends? This would allow to compare them and to use them in an interchangeable way.

[Luxxii]

Oh my mistake, PythonNet relies on mono!

Before I would go the multiple backend route, i would vote for any implementation first (the one easiest to implement/realize) and then improve it. Maybe there is no need for multiple backends for users. Maybe there will be other C# tools we need/want to include in rusteomics. But this is too early to discuss or implement it, just to get started :)

[david-bouyssie]

There are several stuffs that could be helpful actually:

• my fork of the Compomics ThermoRawFileParser is actually a bit old. The main complain I have about ThermoRawFileParser, is the lack of modularity within the project. There are a lot of new features that were added over the past years but they were not implemented as optional modules. So it means the project comes now with (too) many dependencies.
  A first task could consist in creating a PR for the ThermoRawFileParser to increase modularity.
• Some meta-data such as the acquisition method summary (text format) are not exported (although I would have to check if still true in the latest version).
• ThermoRawFileParser is serializing data in the XML format to match the mzML specs. We could also revise this and append support for binary serialization (such as MessagePack or Protocol Buffers).

IMO, having a lightweight fork of ThermoRawFileParser doing binary serialization we would allow to:

• update the thermo-streamer crate to adopt binary deserialization
• experiment with a loosely coupled backend using the same binary deserialization. This would require to update the ThermoRawFileParser fork in order to support IPC. I was considering nng for this purpose but there are some IPC alternatives. We may want to do some quick&dirty benchmarks before adopting a technical solution.

Now you know my Christmas wish-list 😆

[david-bouyssie]

@Luxxii actually you were right, PythonNET can work both on Mono and .NET core (since 2019) on Linux, and can also work with .NET Framework on Windows:
https://pythonnet.github.io/pythonnet/python.html#loading-a-runtime

However it is not clear to me if fisher_py works out of the box without Mono on Linux (https://github.com/ethz-institute-of-microbiology/fisher_py#system-requirements):
If you have trouble problems installing fisher_py it is probably because of pythonnet not being able to compile. To resolve this the usualy path is to install mono

[david-bouyssie]

I would like to create a new repo under the rusteomcis organization.
Could you please thumbs up/down this message?
And also in replies give name suggestions.
I was consider thermo-streamer, in the sense that the serialization/deserialization process is done in-memory, but I'm open to other name suggestions (like thermo-embeddinator, thermo-rs).
Ideally the name of this repo would be the name of the corresponding crate.
Once we have a list I'll create another dedicated poll.