Failed with int16 data

[pieper]

I downloaded data from the TCIA / IDC C4KC-KiTS dataset and loaded a CT series. I got the error below. The int16 data type is very common for real world data so it would be great if this module worked with it.

Creating SAM predictor ... Done

Slicing along x direction
Traceback (most recent call last):
  File "/home/exouser/Downloads/Slicer-5.3.0-2023-07-08-linux-amd64/slicer.org/Extensions-31860/TomoSAM/lib/Slicer-5.3/qt-scripted-modules/tomosam.py", line 504, in onPushEmbeddingsCreate
    self.ui.PathLineEdit_emb.currentPath = self.logic.create_embeddings(output_filepath)
  File "/home/exouser/Downloads/Slicer-5.3.0-2023-07-08-linux-amd64/slicer.org/Extensions-31860/TomoSAM/lib/Slicer-5.3/qt-scripted-modules/tomosamLib/tomosamLogic.py", line 119, in create_embeddings
    self.predictor.set_image(np.repeat(img_slice[:, :, np.newaxis], 3, axis=2))
  File "/home/exouser/Downloads/Slicer-5.3.0-2023-07-08-linux-amd64/lib/Python/lib/python3.9/site-packages/segment_anything/predictor.py", line 56, in set_image
    input_image = self.transform.apply_image(image)
  File "/home/exouser/Downloads/Slicer-5.3.0-2023-07-08-linux-amd64/lib/Python/lib/python3.9/site-packages/segment_anything/utils/transforms.py", line 31, in apply_image
    return np.array(resize(to_pil_image(image), target_size))
  File "/home/exouser/Downloads/Slicer-5.3.0-2023-07-08-linux-amd64/lib/Python/lib/python3.9/site-packages/torchvision/transforms/functional.py", line 335, in to_pil_image
    raise TypeError(f"Input type {npimg.dtype} is not supported")
TypeError: Input type int16 is not supported

[pieper]

As an update: I used numpy to adjust the CT data range to be 0-255 stored in a uint8 array and then created a new node and copied over the ijkToRAS matrix and the DICOM.instanceUIDs attribute from the source node. This got me past the error and the embeddings are being calculated now.

[pieper]

Here are the steps to make a volume that passes the embedding:

>>> n = getNode("4:*")
>>> a = arrayFromVolume(n)
>>> aa = a + 1024
>>> aa = aa / aa.max()
>>> aa.max()
1.0
>>> aa *= 255
>>> nn = addVolumeFromArray(aa.astype('uint8'))
>>> m = vtk.vtkMatrix4x4()
>>> n.GetIJKToRASMatrix(m)
>>> nn.SetIJKToRASMatrix(m)
>>> nn.SetAttribute("DICOM.instanceUIDs", n.GetAttribute("DICOM.instanceUIDs"))

[pieper]

Once I used this data, the embeddings were calculated and I was able to create segments and enter the point selection mode. However my points never seemed to be accepted - I would stay in markups place mode and never got a segmentation no matter where I clicked. There were no messages on the console. I'm not sure if this is due to the data still not being what's expected or if there's something else wrong.

These tests were done on a linux ubuntu 22.04 on a machine with an 8 GB fraction of an A100. I could see that the embedding calculation was using the GPU using nvtop.

[fedesemeraro]

Can you try saving the data to a tif file, place the pkl embeddings into the same folder as your tif, and import it into TomoSAM? This should trigger the correct volume and embeddings import

[pieper]

I'll try with one of your tif files, but in general tif is not a good medical image file format since it doesn't preserve the image orientations see these notes and we prefer not to truncate to 8 bits if we can.

[nikgiannotti]

Following this conversation with interest, any update on whether we can use TOMOSAM with uint8 array DICOM/Nifti files and avoid .tif? Thank you!