XNAT - what format for IGI data?

[thompson318]

In GitLab by @MattClarkson on Mar 5, 2019, 13:37

Can we start a discussion of what formats/protocols/requirements we need in order to upload sessions of IGI data to an XNAT server. This would be relevant to a variety of projects, e.g. SmartLiver, Skull Based Navigation, dvLogger etc.
Some ideas and implementations have been captured in various projects, probably inconsistently.

1. PLUS uses .mha for ultrasound and tracking, synchronised.
2. GIFT-Surg uses a video uploaded in a DICOM wrapper - @DzhoshkunShakir - can you sumarise?
3. DaVinci uses dvLogger, I spoke with @EddieEdwards
4. Lots of formats required for Skull Based Navigation - @JonathanShapey

Requirements include:

1. Maintain original data, rather than synchronised / or just synchronised tracking+images? / or both?
2. Few files, so not storing each tracking frame as a separate file, too many to search.

Please can others add ideas and thoughts? Before next Thursday?

@StephenThompson @ThomasDowrick @mianahmad @EddieEdwards @MirekJanatka @DavidPerez-Suarez @RemiDelaunay @RolandGuichard @AnastasisGeorgoulas @DzhoshkunShakir @tomvercauteren

Thanks

[thompson318]

In GitLab by @MattClarkson on Mar 7, 2019, 16:19

changed the description

[thompson318]

In GitLab by @JonathanShapey on Mar 7, 2019, 16:24

Please see this gitlab page for details regarding data formats used in the SkullBaseNavigation project

https://weisslab.cs.ucl.ac.uk/WEISS/SoftwareRepositories/skullbasenavigation/issues/18

[thompson318]

In GitLab by @tomvercauteren on Mar 12, 2019, 19:46

That's a tough question. While the pre-operative data part is rather straightforward, I am not sure how to best handle intra-operative data in general.

One of the key elements is to make sure we have access to synchronised data, stemming from multiple sources. This should be available both for real-time purposes but also for post-processing analysis.

Intra-operative data of interest could include quite a variety of sources:

• video-like streams (ultrasound, surgical microscope, camera embedded in the surgical light)
• snapshot images (fluoroscopy - used for validation as the proposal focuses on radiation free)
• snapshot volumes (reconstructed 3D US, intra-op CT for validation)
• small vector-like data streams (tracking information, robot kinematics, sensing data, laser feedback)
  snapshot small data (event flags)
• etc.

On my side, I am not very familiar with how capturing, structuring and sharing of such varied data is currently done. Given this scope, it's probably wise not to start from scratch though.

I assume ROS can do most of this (http://wiki.ros.org/Bags), at least to a certain extent? However it looks like videos are stored as a series of individual images which would lead to quite large amounts of data. Is this an issue in practice? Similarly, is synchronisation and frame drop a big issue with ROS (http://wiki.ros.org/ROS/Tutorials/Recording%20and%20playing%20back%20data#rosbag.2BAC8-Tutorials.2BAC8-Recording_and_playing_back_data.The_limitations_of_rosbag_record.2BAC8-play)?

How good is the native PLUS / Slicer / OpenIGTLink (https://plustoolkit.github.io/features.html) way for this? I guess that it's less versatile than ROS but might be wrong as a lot can be done already with their sequence files (http://perk-software.cs.queensu.ca/plus/doc/nightly/user/FileSequenceFile.html).

[thompson318]

In GitLab by @MattClarkson on Mar 13, 2019, 09:28

Thanks @tomvercauteren, very good points. There is a DICOM standard for surgery, and Heinz Lemke runs a workgroup each year at CARS, but i don't believe anything official ha s been produced. Its been talked about for years. And even if it was, it might be too industrial and heavy weight.

Im cc'ing @GeorgeDwyer for his opinion, and I'll revisit this in the coming days when I get some headspace.

[thompson318]

In GitLab by @MattClarkson on Mar 13, 2019, 09:31

@EddieEdwards in the meantime can you summarise all that comes out of DaVinci logger. cc'ing @DanailStoyanov for leadership and inspiration ;-)

@EddieEdwards - I plan to get hold of Tom Doel and ask what a vanilla XNAT can upload. It might be contingent on file extensions. So, one option is to just dump a session from the dvLogger into a big zip file and upload it. But I suspect you could end up with files that are too large. So, we may be forced to break it down into pieces.

So, what format and file extensions are each file that comes out of dvLogger? i.e. before any of us post-process it.

Post processing tools could be an independent python package for researchers to muck around with.

[thompson318]

In GitLab by @GeorgeDwyer on Mar 13, 2019, 10:19

I've found ROS bags pretty useful for quickly collecting data, in terms of synchronisation it just takes a time stamp when it (rosbag record) receives the message (a lot of messages also contain a timestamp of when they were created). So it won't do any actual synchronisation for you but if you're recording a sequence with a couple imaging devices and robots it'll grab all the messages you set it too get, I haven't experienced it becoming overloaded and missing messages yet. Generally I've used the rosbag api and wrote my own logger that triggers off my slowest data source or an external trigger and captures everything else at the same time. Here's a small example of a bag (https://drive.google.com/file/d/1uzRmyITQxy0-aQUPpeSW5hBKxVt4pldR/view?usp=sharing). @tomvercauteren got the main disadvantage though 1680 stereo frames (250x250) with kinematics takes almost 800mb.

[thompson318]

In GitLab by @DzhoshkunShakir on Mar 13, 2019, 10:30

We haven't done much with wrapping video files into DICOM. It was an idea we had for storing endoscopic videos. For the record, by endoscopic video, I mean HEVC-encoded video in MP4 containers. From the GS perspective, it'd be interesting to synchronise endoscopic video with EM tracking data (both coming primarily from Leuven), with the latter saved in the rosbag format, as far as I know. This combination is of interest for validating mosaicing algorithms. I believe right now Sophia Bano is leading that effort, so maybe she's the best person to comment on the synchronisation requirements.

One thing worth investigating is whether timestamping of individual frames within a video is a supported feature in video encoding. This would be useful for synchronising video data with data from other sources without a tight coupling.

[thompson318]

In GitLab by @ThomasDowrick on Mar 13, 2019, 10:46

How useful/practical are rosbags if the main application doesn't use ROS as standard?

For example, in the Skull Navigation project, where we are sending OpenIGTLink messages to Slicer, we would have run a ROS master node and an instance of ROS-OpenIGTLink bridge, send the data back out over OpenIGTLink and then rosbag it. My intitution is that the overheads would outweight the ease of use of rosbags.

For situations, such as the above, where it may not be practical to use ROS, is there any mileage in trying to implement something that has a similar data structure (https://wiki.ros.org/Bags/Format/2.0)?

[thompson318]

In GitLab by @StephenThompson on Mar 13, 2019, 14:35

It's going to be extremely hard to find specific formats that meet the needs of all users, and no one else has managed it.

Could we just come up with a standard "SNAPPY" header that contains information on the software (+version) used to record the data and the software (+version) that can be used to play it back, together with usual header stuff, (data, patient ID, checksum) this could go in scikit-surgerycore (or scikit-surgeryxnat?).

The format for the actual data can then be decided based on the use case. We can supply some basic ones, but in general try and be permissive. The main requirement then is that you write a data playback library that behaves in the same way as the library used to grab the data, implementing a get_data(timestamp) method.

[thompson318]

In GitLab by @MattClarkson on Mar 14, 2019, 08:48

cc'ing @tomdoel - cos Im sure he'll want to be included ;-)

[thompson318]

In GitLab by @MattClarkson on Mar 14, 2019, 08:53

Questions for Tom Doel:

1. Is XNAT suitable for Image Guided Surgery data? I believe it was primarily intended for DICOM/NifTI etc, but can be coerced to accept other formats, and used as a generic data store. Correct me if Im wrong.

If Q1 is 'yes', then:

2. What file formats are supported?
3. What determines which file formats? Is it easy to extend, and add more. Im hoping its just file extensions.

[thompson318]

In GitLab by @MattClarkson on Mar 14, 2019, 08:54

Questions for @MirekJanatka @EddieEdwards:

Please can you document here the file/directory structure and all file format extensions that come out of dvLogger.

Can you do this this week? Thanks.

[thompson318]

In GitLab by @EddieEdwards on Mar 14, 2019, 13:37

Hi Matt and everyone else.

Thanks for getting the conversation going on this issue.

The dVLogger is a pretty good model for collection of intraoperative data and works as follows.

All data is streamed to an encrypted (VeraCrypt) volume on an external USB drive. The box itself has two inputs each for SDI, HDMI, VGA and audio 1/4" jack L/R as well as ethernet to connect to the da Vinci kinematics. We collect stereoendoscopic video through the two SDI inputs and robot kinematics as standard. We have also shown that we can collect other video sources at the same time.

For any video device that's connected the video itself is recorded as DIVX in an AVI container (.avi) and there is a separate file containing the timestamps for each frame. This is in a proprietary format (.issi) that we currently have to read using MatLab. For convenience in the projects I'm working on we convert this to CSV. The timestamps are different for each video source.

The kinematics generates a more complex .issi file containing information about the robot joints as well as timestamps.

The data recordings get split up into different directories but the video files can sometimes run to over 1GB. Is there a file size limit for XNAT? This could be an issue. If not then I think a zip file containing all the data would be a sensible way to store these recordings.

The dVLogger is a great device for recording and retrospective work, but for live systems we will need to sort out our own methods and formats of course.

We can transfer data just by downloading off the USB drive and that's what we do at the moment. So we bypass any need for XNAT currently. It would be great to store the data on XNAT and in particular to be able to relate the intraoperative data to the same patient's preop/postop images.

I am very much of the opinion that the data stored on XNAT should be the original data as unadulterated as possible. If we write a better converter or improve our synchronisation methods then we always have the original data to go back to and reprocess.

I hope this is of some use to the discussion,

Eddie.

[thompson318]

In GitLab by @MattClarkson on Mar 14, 2019, 16:06

For tracking data in general:

• csv file per tracked tool
• ASCI not binary
• 1 line per frame containing

timestamp, w, x, y, z, t_x, t_y, t_z, vendor1, vendor2, .... vendorN.

• timestamp = default python time() returns seconds since epoch, fractions of second after decimal place
• w,x,y,z = quaternion
• t_x, t_y, t_z = translation
• then any other vendor specific fields

Note: I haven't compared this with ROS fields yet, or daVinci kinematics. We were just thinking about requirements, and trying to keep it simple. It would be easily readable by most things like numpy pandas etc.

Note sure if we need header: tool name, patient ID?

[thompson318]

In GitLab by @tomvercauteren on Mar 14, 2019, 16:41

@EddieEdwards Many thanks for the feedback which is super useful.

@GeorgeDwyer Here is the feedback from Leuven on using ROS for recording stereo:

Our experience with ros and stereo camera is not good - and we are saving data on same pc that is connect with the frame grabbers. We did not find a good solution at the moment (but we did not look for it very hard), but ros streaming has serious issues in this.

@MattClarkson I would advise against simple csv as it will not be easy to upgrade / modify. Protobuf, JSON or similar would probably be better if you want to implement your own solution.

[thompson318]

In GitLab by @MattClarkson on Mar 15, 2019, 08:52

All good points indeed!

Re: csv, i was essentially hoping that just tracking data wouldn't need upgrading. If I was going more complex, say JSON, like you suggest, then maybe one could put multiple tools in one file. But basically, you need something you can continually add too, and read serially. i.e. you dont want to have to read the whole block into memory, it can get very large. Would any format jump to mind with that requirement?

Re: stereo. I agree. While I like message architectures for loose coupling, flexibility, etc, they are harder for non software engineers to understand, and sometimes, you just need direct access to a video card, and to synchronise it more directly. .... Also, the bandwidthe of stereo HD (or 4K in future) may always be too high.

Depends on requirements.

Keep the suggestions coming.

[thompson318]

In GitLab by @tomvercauteren on Mar 15, 2019, 14:00

HDF5 would be in the usual contender list of flexible file formats but I am not sure it makes much sense for videos.

Anyway, reading on how ROS2 plans to address this issue is really interesting: ros2/design#160

TL;DR Given the feedback we decided that there is no single best storage format, which suits all needs and use-cases. We therefore focus on a flexible plugin API, which supports multiple storage formats. We start off with SQLite as our default storage format because of its simplicity and community support. We simultaneously will work on a plugin which reads legacy ros1 bags.
The idea is that the API is powerful enough to easily provide further storage formats, which might be faster than SQLite and can be provided by the ROS community.

Things are still being discussed but I like how things are shaped in their memo: https://github.com/ros2/design/blob/c4ae1c276f4c8efd7bca1365362827c9075d46c5/articles/rosbags.md

[thompson318]

In GitLab by @tomvercauteren on Mar 18, 2019, 11:04

For the record, DICOM RTV / DICOM-RTV (Real-Time Video) is apparently progressing well:

• http://dicom.nema.org/Dicom/News/Current/docs/sups/sup202.pdf
• https://b-com.com/en/news/world-first-achievement-bcom-and-rennes-university-hospital

[thompson318]

In GitLab by @DzhoshkunShakir on Mar 19, 2019, 09:20

Re: csv + speed

In one project we ended up using a combination of protobuf and NumPy, because using merely protobuf was limiting in terms of performance.

Background: protobuf has a type called bytes for supporting arbitrary-length binary data. So when your data is just a sequence of bytes represented by this type, protobuf can read data off disk very fast. But if you have e.g. floats, then the only way I could find to represent those was via repeated protobuf floating-point types. And this seriously degraded data loading performance.

So we decided to use protobuf for the metadata (e.g. resolution, pixel dimensions), and NumPy for reading the actual data. The former allows for versioning the format, while the latter provides a fast way to load data.

[thompson318]

In GitLab by @MattClarkson on Mar 19, 2019, 10:47

Thanks everyone. Im just finishing a lecture course, then I'll get time to digest.

[tomdoel]

I realised we hadn't recorded the answers to these questions:

1. Is XNAT suitable for Image Guided Surgery data? I believe it was primarily intended for DICOM/NifTI etc, but can be coerced to accept other formats, and used as a generic data store. Correct me if Im wrong.

Yes, if you're just looking for a secure structured file storage where you group your data by subject/session (perhaps alongside DICOM/NIFTI data) and where you want data upload/download and perhaps some offline processing, then XNAT is fine for this purpose.

If you want realtime processing or want to play or edit videos on the website, XNAT may not be the best bet.

If Q1 is 'yes', then:

1. What file formats are supported?

2. What determines which file formats? Is it easy to extend, and add more. Im hoping its just file extensions.

You can store any file format, it's just that XNAT has additional built-in support for DICOM/nifti; for example:

• You need to upload non-DICOM data using the REST API (which is fairly straightforward for example using pyxnat or curl). The XNAT Desktop tool only supports DICOM.
• XNAT will automatically generate metadata from known datatypes. If you want to add metadata for your own data (which is not essential but can be useful for search or automated processing), you can for example create a custom data type by adding a schema in a custom plugin, and then providing the metadata as a csv file when uploading.
• The built-in image viewers probably don't support video, although you could always write your own...