Cannot open OME-Zarr created with bioformats2raw

[dpshepherd]

Hi all,

We have a OME-Zarr (190 GB) created with a freshly installed bioformats2raw, installed into a new conda environment (python=3.8) using the ome channel on conda-forge on a Window 10 machine. The top-level file structure looks as follows:

lung.zarr
├── 0
├── OME
├── .zattrs
└── .zgroup

The call to bioformats2raw was:
bioformats2raw image.pattern d:/lung_2x/lung.zarr where image.pattern contains fused_ch<00-16>.tif. Each .tif file is ~14 GB, is a 3D stack created by the BigStitcher plugin in FIJI, and contains one channel.

Using a brand new conda environment (python=3.9) with the latest Napari and napari-ome-zarr, installed using pip install napari[all] napari-ome-zarr, we try to open the file using 'napari d:/lung_2x/lung.zarr and get the following error: ValueError: No plugin found capable of reading 'D:\\lung_2x\\lung.zarr'.

If we instead call Napari using to napari d:/lung.zarr/0 it does open the data, with the channels appearing on one of the sliders.

Any suggestions on how to get Napari to properly open this as a OME-Zarr?

Thanks.

[sbesson]

@dpshepherd thanks for raising this important issue. Overall, I think the behavior you are seeing is expecting.

For non-HCS filesets, the output of bioformat2raw is a Zarr fileset which maps the representation of Bio-Formats series concepts with a top-level group containing N Zarr groups called 0...n (just 0 for a single image) mapping to the individual series (images) of the dataset plus an OME group containing the metadata. The reason for this layout is primarily compatibility with downstream tools such as raw2ometiff which can then reconstruct multi-image filesets together with their original metadata as OME-TIFF.

From the OME-NGFF perspective, each of the 0...n groups is effectively an NGFF image as defined by https://ngff.openmicroscopy.org/0.2/#image-layout and including notably the multiscales metadata. So napari d:/lung.zarr/0 is definitely the call that should open the first image of the output of bioformats2raw. For comparison, the omero zarr --bf command which exports a single OMERO image using bioformats2raw under the hood includes a step which simply renames this 0 folder as <image_id>.zarr to clarify the expectation - https://github.com/ome/omero-cli-zarr/blob/fb318edb2ae4bfe62f97683501e41418ed82dfd4/src/omero_zarr/cli.py#L326-L327

In the mid-term, we are definitely hoping to formalize the layout of bioformat2raw. This might build on top of the ongoing work on the collections specification which is currently being discussed in ome/ngff#31.

I'll leave @chris-allan and @melissalinkert to add anything I missed.

[dpshepherd]

@sbesson thanks for the clarification. Given that we have 17 (typo, not 16) independent channels for the same 3D voxels, is there a "better" way to use bioformats2raw? Or is the process we have used the preferred method?

We can also do this conversion in Python, but wanted to try out bioformats2raw before going that route.

[chris-allan]

What that napari user interface behaviour signals to me is that Bio-Formats is not detecting your channels correctly to begin with and consequently you have bad input going into bioformats2raw. A .pattern file containing fused_ch<00-16>.tif indicates 17 channels, not 16, and the FilePatternReader will not allow missing files. The fact that bioformats2raw is not blowing up spectacularly before conversion starts is telling.

You first need to ensure that the .pattern file you are using to signal a multi-file aggregation is representing your desired input dimensions which you can do with the Bio-Formats tools (showinf in particular) like so:

showinf -nopix image.pattern

I have mocked your use case up using .fake files:

$ cat image.pattern
fused_ch<01-16>.fake.ini
$ cat fused_ch01.fake.ini
sizeX=8192
sizeY=8192
pixelType=uint16
$ sha256sum fused_*fake.ini
fd1490d32868d4dce5336b71bee69b26cfdb83499d4b2059fad4969522f1a744  fused_ch01.fake.ini
fd1490d32868d4dce5336b71bee69b26cfdb83499d4b2059fad4969522f1a744  fused_ch02.fake.ini
fd1490d32868d4dce5336b71bee69b26cfdb83499d4b2059fad4969522f1a744  fused_ch03.fake.ini
fd1490d32868d4dce5336b71bee69b26cfdb83499d4b2059fad4969522f1a744  fused_ch04.fake.ini
fd1490d32868d4dce5336b71bee69b26cfdb83499d4b2059fad4969522f1a744  fused_ch05.fake.ini
fd1490d32868d4dce5336b71bee69b26cfdb83499d4b2059fad4969522f1a744  fused_ch06.fake.ini
fd1490d32868d4dce5336b71bee69b26cfdb83499d4b2059fad4969522f1a744  fused_ch07.fake.ini
fd1490d32868d4dce5336b71bee69b26cfdb83499d4b2059fad4969522f1a744  fused_ch08.fake.ini
fd1490d32868d4dce5336b71bee69b26cfdb83499d4b2059fad4969522f1a744  fused_ch09.fake.ini
fd1490d32868d4dce5336b71bee69b26cfdb83499d4b2059fad4969522f1a744  fused_ch10.fake.ini
fd1490d32868d4dce5336b71bee69b26cfdb83499d4b2059fad4969522f1a744  fused_ch11.fake.ini
fd1490d32868d4dce5336b71bee69b26cfdb83499d4b2059fad4969522f1a744  fused_ch12.fake.ini
fd1490d32868d4dce5336b71bee69b26cfdb83499d4b2059fad4969522f1a744  fused_ch13.fake.ini
fd1490d32868d4dce5336b71bee69b26cfdb83499d4b2059fad4969522f1a744  fused_ch14.fake.ini
fd1490d32868d4dce5336b71bee69b26cfdb83499d4b2059fad4969522f1a744  fused_ch15.fake.ini
fd1490d32868d4dce5336b71bee69b26cfdb83499d4b2059fad4969522f1a744  fused_ch16.fake.ini

This is what my showinf output looks like (it should be similar to what you would want to see, in particular sizeC=16):

$ showinf -nopix image.pattern
... >8 snip ...
Series #0 :
 Image count = 16
 RGB = false (1)
 Interleaved = false
 Indexed = false (true color)
 Width = 8192
 Height = 8192
 SizeZ = 1
 SizeT = 1
 SizeC = 16
 Tile size = 8192 x 64
 Thumbnail size = 128 x 128
 Endianness = intel (little)
 Dimension order = XYZCT (certain)
 Pixel type = uint16
 Valid bits per pixel = 16
 Metadata complete = true
 Thumbnail series = false
 -----
 Plane #0 <=> Z 0, C 0, T 0
 Plane #6 <=> Z 0, C 6, T 0
 Plane #7 <=> Z 0, C 7, T 0
 Plane #8 <=> Z 0, C 8, T 0
 Plane #9 <=> Z 0, C 9, T 0
 Plane #10 <=> Z 0, C 10, T 0
 Plane #15 <=> Z 0, C 15, T 0


Reading global metadata

Reading metadata

Converting this with bioformats2raw image.pattern lung2.zarr and opening it up with napari lung2.zarr/0 gives me a napari that looks like this (after some manual tweaking of the channel scaling and turning several channels off):

As @sbesson has already mentioned, yes, napari lung2.zarr/0 is the correct way to open the converted image.

[dpshepherd]

I will test your suggestions later, but 16 channels was a typo, sorry about that. We do have 17, <00-16> is correct.

Thanks.

[dpshepherd]

Here is the output of showinf -nopix image.pattern. This shows you are correct and the different files are being interpreted as the time axis, not channel axis. We don't have control over the .tif file creation, since they are being built by the BigStitcher tool. BigStitcher aligns and fuses a large number of independent tiles acquired with a custom light sheet fluorescence microscope.

Is there a way to force the assignment to the channel axis? If we have to fix individual TIFF files, we will probably just start writing this with Python, since we are quite comfortable in Python and can use one of the existing ome-ngff packages.

D:\lung_2x_fused>C:\Users\qi2lab\miniconda3\envs\bioformats\bin\..\share\bftools-6.7.0-0\showinf.bat -nopix image.pattern
Checking file format [File pattern]
Initializing reader
FilePatternReader initializing image.pattern
 *** One or more readers is misbehaving. See the debug output for more information. e.g.:
     loci.formats.in.APLReader@2fba3fc4 -> java.lang.NullPointerException('null') ***
TiffDelegateReader initializing D:\lung_2x_fused\fused_ch00.tif
Reading IFDs
Populating metadata
Checking comment style
Populating OME metadata
TiffDelegateReader initializing D:\lung_2x_fused\fused_ch00.tif
Reading IFDs
Populating metadata
Checking comment style
Populating OME metadata
TiffDelegateReader initializing D:\lung_2x_fused\fused_ch00.tif
TiffDelegateReader initializing D:\lung_2x_fused\fused_ch00.tif
TiffDelegateReader initializing D:\lung_2x_fused\fused_ch00.tif
TiffDelegateReader initializing D:\lung_2x_fused\fused_ch00.tif
Initialization took 0.248s

Reading core metadata
filename = D:\lung_2x_fused\image.pattern
TiffDelegateReader initializing D:\lung_2x_fused\fused_ch00.tif
TiffDelegateReader initializing D:\lung_2x_fused\fused_ch01.tif
Reading IFDs
Populating metadata
Checking comment style
Populating OME metadata
TiffDelegateReader initializing D:\lung_2x_fused\fused_ch02.tif
Reading IFDs
Populating metadata
Checking comment style
Populating OME metadata
TiffDelegateReader initializing D:\lung_2x_fused\fused_ch03.tif
Reading IFDs
Populating metadata
Checking comment style
Populating OME metadata
TiffDelegateReader initializing D:\lung_2x_fused\fused_ch04.tif
Reading IFDs
Populating metadata
Checking comment style
Populating OME metadata
TiffDelegateReader initializing D:\lung_2x_fused\fused_ch05.tif
Reading IFDs
Populating metadata
Checking comment style
Populating OME metadata
TiffDelegateReader initializing D:\lung_2x_fused\fused_ch06.tif
Reading IFDs
Populating metadata
Checking comment style
Populating OME metadata
TiffDelegateReader initializing D:\lung_2x_fused\fused_ch07.tif
Reading IFDs
Populating metadata
Checking comment style
Populating OME metadata
TiffDelegateReader initializing D:\lung_2x_fused\fused_ch08.tif
Reading IFDs
Populating metadata
Checking comment style
Populating OME metadata
TiffDelegateReader initializing D:\lung_2x_fused\fused_ch09.tif
Reading IFDs
Populating metadata
Checking comment style
Populating OME metadata
TiffDelegateReader initializing D:\lung_2x_fused\fused_ch10.tif
Reading IFDs
Populating metadata
Checking comment style
Populating OME metadata
TiffDelegateReader initializing D:\lung_2x_fused\fused_ch11.tif
Reading IFDs
Populating metadata
Checking comment style
Populating OME metadata
TiffDelegateReader initializing D:\lung_2x_fused\fused_ch12.tif
Reading IFDs
Populating metadata
Checking comment style
Populating OME metadata
TiffDelegateReader initializing D:\lung_2x_fused\fused_ch13.tif
Reading IFDs
Populating metadata
Checking comment style
Populating OME metadata
TiffDelegateReader initializing D:\lung_2x_fused\fused_ch14.tif
Reading IFDs
Populating metadata
Checking comment style
Populating OME metadata
TiffDelegateReader initializing D:\lung_2x_fused\fused_ch15.tif
Reading IFDs
Populating metadata
Checking comment style
Populating OME metadata
TiffDelegateReader initializing D:\lung_2x_fused\fused_ch16.tif
Reading IFDs
Populating metadata
Checking comment style
Populating OME metadata
Used files:
        D:\lung_2x_fused\image.pattern
        D:\lung_2x_fused\fused_ch00.tif
        D:\lung_2x_fused\fused_ch01.tif
        D:\lung_2x_fused\fused_ch02.tif
        D:\lung_2x_fused\fused_ch03.tif
        D:\lung_2x_fused\fused_ch04.tif
        D:\lung_2x_fused\fused_ch05.tif
        D:\lung_2x_fused\fused_ch06.tif
        D:\lung_2x_fused\fused_ch07.tif
        D:\lung_2x_fused\fused_ch08.tif
        D:\lung_2x_fused\fused_ch09.tif
        D:\lung_2x_fused\fused_ch10.tif
        D:\lung_2x_fused\fused_ch11.tif
        D:\lung_2x_fused\fused_ch12.tif
        D:\lung_2x_fused\fused_ch13.tif
        D:\lung_2x_fused\fused_ch14.tif
        D:\lung_2x_fused\fused_ch15.tif
        D:\lung_2x_fused\fused_ch16.tif
Series count = 1
TiffDelegateReader initializing D:\lung_2x_fused\fused_ch00.tif
Reading IFDs
Populating metadata
Checking comment style
Populating OME metadata
TiffDelegateReader initializing D:\lung_2x_fused\fused_ch00.tif
TiffDelegateReader initializing D:\lung_2x_fused\fused_ch00.tif
TiffDelegateReader initializing D:\lung_2x_fused\fused_ch00.tif
TiffDelegateReader initializing D:\lung_2x_fused\fused_ch00.tif
TiffDelegateReader initializing D:\lung_2x_fused\fused_ch00.tif
Series #0 :
        Image count = 4947
        RGB = false (1)
        Interleaved = false
        Indexed = false (false color)
        Width = 5046
        Height = 5199
        SizeZ = 291
        SizeT = 17
        SizeC = 1
        Tile size = 5046 x 103
        Thumbnail size = 124 x 128
        Endianness = motorola (big)
        Dimension order = XYCZT (certain)
        Pixel type = uint16
        Valid bits per pixel = 16
        Metadata complete = true
        Thumbnail series = false
        -----
        Plane #0 <=> Z 0, C 0, T 0
        Plane #2471 <=> Z 143, C 0, T 8
        Plane #2472 <=> Z 144, C 0, T 8
        Plane #2473 <=> Z 145, C 0, T 8
        Plane #2474 <=> Z 146, C 0, T 8
        Plane #2475 <=> Z 147, C 0, T 8
        Plane #4946 <=> Z 290, C 0, T 16


Reading global metadata
BitsPerSample: 16
Compression: Uncompressed
ImageJ: 1.53f
ImageLength: 5199
ImageWidth: 5046
MetaDataPhotometricInterpretation: Monochrome
MetaMorph: no
NewSubfileType: 0
NumberOfChannels: 1
PhotometricInterpretation: BlackIsZero
ResolutionUnit: None
SamplesPerPixel: 1
Spacing: 0.23000000000000026
Unit: \u00B5m
X Origin: 0
XResolution: 4.347826
Y Origin: 0
YResolution: 4.347826
loop: false
max: 695.0
min: 0.0
zorigin: 7154.495000000008

Reading metadata

[dpshepherd]

If we change the filename from fused_ch<00-16>.tif to fused_c<00-16>.tif, the showinf command now recognizes each file as a unique channel. We'll try converting to OME-Zarr that way and opening in Napari.

The pattern file page states that "ch" should be recognized as a channel axis, which is why we did not alter it on the first try.

[sbesson]

Thanks @dpshepherd, I was also under the assumption that ch should have been a sufficient channel prefix. We would probably need sample files to investigate the source of this issue at the Bio-Formats level. In the meantime, glad to hear switching to _c is sufficient for your needs.

[dpshepherd]

The napari-ome-zarr plugin also reads this OME-Zarr with the channels on a slider, instead of in the layers as @chris-allan showed above.

We will create smaller examples files that still have this issue and host them for download. Might take a few days.

[chris-allan]

I certainly used ch in my mock test case with .fake files. There's definitely some other interplay here with the multi-dimensional TIFFs. If I have some time over the next couple of days I'll try to expand the .fake case with dimensions and see where I get to.

[sbesson]

@dpshepherd while you are generating example files, could you give us the output of showinf.bat -nopix D:\lung_2x_fused\fused_ch00.tif? This might provide some clues on why the stitching and the dimension concatenation fails to group the TIFFs as channels

[dpshepherd]

Note that we renamed fused_ch00.tif to fused_c00.tif.

D:\lung_2x_fused>C:\Users\qi2lab\miniconda3\envs\bioformats\bin\..\share\bftools-6.7.0-0\showinf.bat -nopix fused_c00.tif
 *** One or more readers is misbehaving. See the debug output for more information. e.g.:
     loci.formats.in.APLReader@bb9e6dc -> java.lang.NullPointerException('null') ***
Checking file format [Tagged Image File Format]
Initializing reader
TiffDelegateReader initializing fused_c00.tif
Reading IFDs
Populating metadata
Checking comment style
Populating OME metadata
Initialization took 0.101s

Reading core metadata
filename = fused_c00.tif
Series count = 1
Series #0 :
        Image count = 291
        RGB = false (1)
        Interleaved = false
        Indexed = false (false color)
        Width = 5046
        Height = 5199
        SizeZ = 291
        SizeT = 1
        SizeC = 1
        Tile size = 5046 x 103
        Thumbnail size = 124 x 128
        Endianness = motorola (big)
        Dimension order = XYCZT (certain)
        Pixel type = uint16
        Valid bits per pixel = 16
        Metadata complete = true
        Thumbnail series = false
        -----
        Plane #0 <=> Z 0, C 0, T 0
        Plane #143 <=> Z 143, C 0, T 0
        Plane #144 <=> Z 144, C 0, T 0
        Plane #145 <=> Z 145, C 0, T 0
        Plane #146 <=> Z 146, C 0, T 0
        Plane #147 <=> Z 147, C 0, T 0
        Plane #290 <=> Z 290, C 0, T 0


Reading global metadata
BitsPerSample: 16
Compression: Uncompressed
ImageJ: 1.53f
ImageLength: 5199
ImageWidth: 5046
MetaDataPhotometricInterpretation: Monochrome
MetaMorph: no
NewSubfileType: 0
NumberOfChannels: 1
PhotometricInterpretation: BlackIsZero
ResolutionUnit: None
SamplesPerPixel: 1
Spacing: 0.23000000000000026
Unit: \u00B5m
X Origin: 0
XResolution: 4.347826
Y Origin: 0
YResolution: 4.347826
loop: false
max: 695.0
min: 0.0
zorigin: 7154.495000000008

Reading metadata

[sbesson]

Thanks @dpshepherd, I tried to reproduce the issue using the _ch prefix, fake files of the same dimensions as the ones reported above and the same minimal example as the one using above:

(base) sbesson@ls30630:pattern $ cat image.pattern 
fused_ch<00-16>.fake
(base) sbesson@ls30630:pattern $ cat fused_ch00.fake.ini 
sizeX=5046
sizeY=5199
sizeZ=291
pixelType=uint16(base) sbesson@ls30630:pattern $ shasum *.fake.ini
295cb47ca81ed6b4978f1d205a4b992311c50fc3  fused_ch00.fake.ini
295cb47ca81ed6b4978f1d205a4b992311c50fc3  fused_ch01.fake.ini
295cb47ca81ed6b4978f1d205a4b992311c50fc3  fused_ch02.fake.ini
295cb47ca81ed6b4978f1d205a4b992311c50fc3  fused_ch03.fake.ini
295cb47ca81ed6b4978f1d205a4b992311c50fc3  fused_ch04.fake.ini
295cb47ca81ed6b4978f1d205a4b992311c50fc3  fused_ch05.fake.ini
295cb47ca81ed6b4978f1d205a4b992311c50fc3  fused_ch06.fake.ini
295cb47ca81ed6b4978f1d205a4b992311c50fc3  fused_ch07.fake.ini
295cb47ca81ed6b4978f1d205a4b992311c50fc3  fused_ch08.fake.ini
295cb47ca81ed6b4978f1d205a4b992311c50fc3  fused_ch09.fake.ini
295cb47ca81ed6b4978f1d205a4b992311c50fc3  fused_ch10.fake.ini
295cb47ca81ed6b4978f1d205a4b992311c50fc3  fused_ch11.fake.ini
295cb47ca81ed6b4978f1d205a4b992311c50fc3  fused_ch12.fake.ini
295cb47ca81ed6b4978f1d205a4b992311c50fc3  fused_ch13.fake.ini
295cb47ca81ed6b4978f1d205a4b992311c50fc3  fused_ch14.fake.ini
295cb47ca81ed6b4978f1d205a4b992311c50fc3  fused_ch15.fake.ini
295cb47ca81ed6b4978f1d205a4b992311c50fc3  fused_ch16.fake.ini
(base) sbesson@ls30630:pattern $ showinf -nopix image.pattern 
Checking file format [File pattern]
Initializing reader
FilePatternReader initializing image.pattern
FakeReader initializing /private/tmp/pattern/fused_ch00.fake
FakeReader initializing /private/tmp/pattern/fused_ch00.fake
FakeReader initializing /private/tmp/pattern/fused_ch00.fake
FakeReader initializing /private/tmp/pattern/fused_ch00.fake
FakeReader initializing /private/tmp/pattern/fused_ch00.fake
FakeReader initializing /private/tmp/pattern/fused_ch00.fake
Initialization took 0.116s

Reading core metadata
filename = /private/tmp/pattern/image.pattern
FakeReader initializing /private/tmp/pattern/fused_ch00.fake
FakeReader initializing /private/tmp/pattern/fused_ch01.fake
FakeReader initializing /private/tmp/pattern/fused_ch02.fake
FakeReader initializing /private/tmp/pattern/fused_ch03.fake
FakeReader initializing /private/tmp/pattern/fused_ch04.fake
FakeReader initializing /private/tmp/pattern/fused_ch05.fake
FakeReader initializing /private/tmp/pattern/fused_ch06.fake
FakeReader initializing /private/tmp/pattern/fused_ch07.fake
FakeReader initializing /private/tmp/pattern/fused_ch08.fake
FakeReader initializing /private/tmp/pattern/fused_ch09.fake
FakeReader initializing /private/tmp/pattern/fused_ch10.fake
FakeReader initializing /private/tmp/pattern/fused_ch11.fake
FakeReader initializing /private/tmp/pattern/fused_ch12.fake
FakeReader initializing /private/tmp/pattern/fused_ch13.fake
FakeReader initializing /private/tmp/pattern/fused_ch14.fake
FakeReader initializing /private/tmp/pattern/fused_ch15.fake
FakeReader initializing /private/tmp/pattern/fused_ch16.fake
Used files:
	/private/tmp/pattern/image.pattern
	/private/tmp/pattern/fused_ch00.fake
	/private/tmp/pattern/fused_ch01.fake
	/private/tmp/pattern/fused_ch02.fake
	/private/tmp/pattern/fused_ch03.fake
	/private/tmp/pattern/fused_ch04.fake
	/private/tmp/pattern/fused_ch05.fake
	/private/tmp/pattern/fused_ch06.fake
	/private/tmp/pattern/fused_ch07.fake
	/private/tmp/pattern/fused_ch08.fake
	/private/tmp/pattern/fused_ch09.fake
	/private/tmp/pattern/fused_ch10.fake
	/private/tmp/pattern/fused_ch11.fake
	/private/tmp/pattern/fused_ch12.fake
	/private/tmp/pattern/fused_ch13.fake
	/private/tmp/pattern/fused_ch14.fake
	/private/tmp/pattern/fused_ch15.fake
	/private/tmp/pattern/fused_ch16.fake
	/private/tmp/pattern/fused_ch00.fake.ini
	/private/tmp/pattern/fused_ch01.fake.ini
	/private/tmp/pattern/fused_ch02.fake.ini
	/private/tmp/pattern/fused_ch03.fake.ini
	/private/tmp/pattern/fused_ch04.fake.ini
	/private/tmp/pattern/fused_ch05.fake.ini
	/private/tmp/pattern/fused_ch06.fake.ini
	/private/tmp/pattern/fused_ch07.fake.ini
	/private/tmp/pattern/fused_ch08.fake.ini
	/private/tmp/pattern/fused_ch09.fake.ini
	/private/tmp/pattern/fused_ch10.fake.ini
	/private/tmp/pattern/fused_ch11.fake.ini
	/private/tmp/pattern/fused_ch12.fake.ini
	/private/tmp/pattern/fused_ch13.fake.ini
	/private/tmp/pattern/fused_ch14.fake.ini
	/private/tmp/pattern/fused_ch15.fake.ini
	/private/tmp/pattern/fused_ch16.fake.ini
Series count = 1
FakeReader initializing /private/tmp/pattern/fused_ch00.fake
FakeReader initializing /private/tmp/pattern/fused_ch00.fake
FakeReader initializing /private/tmp/pattern/fused_ch00.fake
FakeReader initializing /private/tmp/pattern/fused_ch00.fake
FakeReader initializing /private/tmp/pattern/fused_ch00.fake
FakeReader initializing /private/tmp/pattern/fused_ch00.fake
Series #0 :
	Image count = 4947
	RGB = false (1) 
	Interleaved = false
	Indexed = false (true color)
	Width = 5046
	Height = 5199
	SizeZ = 291
	SizeT = 1
	SizeC = 17
	Tile size = 5046 x 103
	Thumbnail size = 124 x 128
	Endianness = intel (little)
	Dimension order = XYZCT (certain)
	Pixel type = uint16
	Valid bits per pixel = 16
	Metadata complete = true
	Thumbnail series = false
	-----
	Plane #0 <=> Z 0, C 0, T 0
	Plane #2471 <=> Z 143, C 8, T 0
	Plane #2472 <=> Z 144, C 8, T 0
	Plane #2473 <=> Z 145, C 8, T 0
	Plane #2474 <=> Z 146, C 8, T 0
	Plane #2475 <=> Z 147, C 8, T 0
	Plane #4946 <=> Z 290, C 16, T 0


Reading global metadata

Reading metadata

Here the detection and the combination of all the constituent files as a multi-channel image work as expected. As Chris said, one variation between our environments is the underlying file format and there might be some unexpected interaction with these TIFF files. Another difference is the operating system under which the command-line utilities are executed.

[dpshepherd]

@sbesson - yes, I am also wondering about environment, OS, and these specific TIFF files. Our server is still tied up, but making smaller files for this test is the next job up. I also am curious if we should build the latest bioformats2raw from GitHub instead of using the ome conda channel? We can also try the conversion on our server (Linux Mint 19) instead.

Did you make a OME-Zarr out of these newest test files? Even when showinf recognized the image pattern as channels instead of time, the napari-ome-zarr plugin still assigned the other dimension to a slider instead of layers.

[sbesson]

@dpshepherd: sorry for the delay. I have been working on this in the background. I generated a Zarr dataset from the fake-based pattern fileset described above using bioformat2raw installed in a Conda environment (see below) and a simple bioformats2raw image.pattern fused.zarr conversion command.

Conda packages

``` (zarr) [sbesson@pilot-zarr2-dev ~]$ conda --version conda 4.10.3 (zarr) [sbesson@pilot-zarr2-dev ~]$ conda list # packages in environment at /home/sbesson/miniconda3/envs/zarr: # # Name Version Build Channel _libgcc_mutex 0.1 conda_forge conda-forge _openmp_mutex 4.5 1_gnu conda-forge aiobotocore 1.4.1 pypi_0 pypi aiohttp 3.7.4.post0 pypi_0 pypi aioitertools 0.8.0 pypi_0 pypi alsa-lib 1.2.3 h516909a_0 conda-forge appdirs 1.4.4 pyh9f0ad1d_0 conda-forge asciitree 0.3.3 py_2 conda-forge async-timeout 3.0.1 pypi_0 pypi attrs 21.2.0 pypi_0 pypi awscli 1.22.14 py39hf3d152e_0 conda-forge bioformats2raw 0.3.0 0 ome bioformats2raw-libs 0.3.0 0 ome blosc 1.21.0 h9c3ff4c_0 conda-forge botocore 1.20.106 pypi_0 pypi brotlipy 0.7.0 py39h3811e60_1001 conda-forge bzip2 1.0.8 h7f98852_4 conda-forge ca-certificates 2021.10.8 ha878542_0 conda-forge cairo 1.16.0 h6cf1ce9_1008 conda-forge certifi 2021.10.8 py39hf3d152e_1 conda-forge cffi 1.14.6 py39he32792d_0 conda-forge chardet 4.0.0 py39hf3d152e_1 conda-forge cloudpickle 2.0.0 pypi_0 pypi colorama 0.4.3 py_0 conda-forge cryptography 3.4.7 py39hbca0aa6_0 conda-forge cycler 0.10.0 pypi_0 pypi dask 2021.9.0 pypi_0 pypi docutils 0.15.2 py39hf3d152e_3 conda-forge expat 2.4.1 h9c3ff4c_0 conda-forge fasteners 0.16 pyhd8ed1ab_0 conda-forge fontconfig 2.13.1 hba837de_1005 conda-forge freetype 2.10.4 h0708190_1 conda-forge fsspec 2021.8.1 pypi_0 pypi future 0.18.2 py39hf3d152e_3 conda-forge gettext 0.19.8.1 h0b5b191_1005 conda-forge giflib 5.2.1 h36c2ea0_2 conda-forge graphite2 1.3.13 h58526e2_1001 conda-forge harfbuzz 2.9.1 h83ec7ef_0 conda-forge icu 68.1 h58526e2_0 conda-forge idna 2.10 pyh9f0ad1d_0 conda-forge imageio 2.9.0 pypi_0 pypi jbig 2.1 h7f98852_2003 conda-forge jmespath 0.10.0 pyh9f0ad1d_0 conda-forge jpeg 9d h36c2ea0_0 conda-forge kiwisolver 1.3.2 pypi_0 pypi lcms2 2.12 hddcbb42_0 conda-forge ld_impl_linux-64 2.36.1 hea4e1c9_2 conda-forge lerc 2.2.1 h9c3ff4c_0 conda-forge libblas 3.9.0 11_linux64_openblas conda-forge libcblas 3.9.0 11_linux64_openblas conda-forge libdeflate 1.7 h7f98852_5 conda-forge libffi 3.3 h58526e2_2 conda-forge libgcc-ng 11.1.0 hc902ee8_8 conda-forge libgfortran-ng 11.1.0 h69a702a_8 conda-forge libgfortran5 11.1.0 h6c583b3_8 conda-forge libglib 2.68.4 h3e27bee_0 conda-forge libgomp 11.1.0 hc902ee8_8 conda-forge libiconv 1.16 h516909a_0 conda-forge liblapack 3.9.0 11_linux64_openblas conda-forge libopenblas 0.3.17 pthreads_h8fe5266_1 conda-forge libpng 1.6.37 h21135ba_2 conda-forge libstdcxx-ng 11.1.0 h56837e0_8 conda-forge libtiff 4.3.0 hf544144_1 conda-forge libuuid 2.32.1 h7f98852_1000 conda-forge libwebp-base 1.2.1 h7f98852_0 conda-forge libxcb 1.13 h7f98852_1003 conda-forge libxml2 2.9.12 h72842e0_0 conda-forge locket 0.2.1 pypi_0 pypi lz4-c 1.9.3 h9c3ff4c_1 conda-forge matplotlib 3.4.3 pypi_0 pypi monotonic 1.5 py_0 conda-forge msgpack-python 1.0.2 py39h1a9c180_1 conda-forge multidict 5.1.0 pypi_0 pypi ncurses 6.2 h58526e2_4 conda-forge networkx 2.6.3 pypi_0 pypi numcodecs 0.9.1 py39he80948d_0 conda-forge numpy 1.21.2 py39hdbf815f_0 conda-forge olefile 0.46 pyh9f0ad1d_1 conda-forge ome-zarr 0.2.1 pypi_0 pypi omero-cli-zarr 0.2.1 pypi_0 pypi omero-py 5.10.0 py39_1 ome openjdk 11.0.9.1 h5cc2fde_1 conda-forge openjpeg 2.4.0 hb52868f_1 conda-forge openssl 1.1.1l h7f98852_0 conda-forge packaging 21.0 pypi_0 pypi partd 1.2.0 pypi_0 pypi pcre 8.45 h9c3ff4c_0 conda-forge pillow 8.3.2 py39ha612740_0 conda-forge pip 21.2.4 pyhd8ed1ab_0 conda-forge pixman 0.40.0 h36c2ea0_0 conda-forge pthread-stubs 0.4 h36c2ea0_1001 conda-forge pyasn1 0.4.8 py_0 conda-forge pycparser 2.20 pyh9f0ad1d_2 conda-forge pyopenssl 20.0.1 pyhd8ed1ab_0 conda-forge pyparsing 2.4.7 pypi_0 pypi pysocks 1.7.1 py39hf3d152e_3 conda-forge python 3.9.7 h49503c6_0_cpython conda-forge python-dateutil 2.8.2 pyhd8ed1ab_0 conda-forge python_abi 3.9 2_cp39 conda-forge pywavelets 1.1.1 pypi_0 pypi pyyaml 5.4.1 py39h3811e60_1 conda-forge raw2ometiff 0.3.0 0 ome raw2ometiff-libs 0.3.0 0 ome readline 8.1 h46c0cb4_0 conda-forge requests 2.25.1 pyhd3deb0d_0 conda-forge rsa 4.7.2 pyh44b312d_0 conda-forge s3fs 2021.8.1 pypi_0 pypi s3transfer 0.5.0 pyhd8ed1ab_0 conda-forge scikit-image 0.18.3 pypi_0 pypi scipy 1.7.1 pypi_0 pypi setuptools 58.0.4 py39hf3d152e_0 conda-forge six 1.16.0 pyh6c4a22f_0 conda-forge sqlite 3.36.0 h9cd32fc_1 conda-forge tifffile 2021.8.30 pypi_0 pypi tk 8.6.11 h27826a3_1 conda-forge toolz 0.11.1 pypi_0 pypi typing-extensions 3.10.0.2 pypi_0 pypi tzdata 2021a he74cb21_1 conda-forge urllib3 1.26.6 pyhd8ed1ab_0 conda-forge wheel 0.37.0 pyhd8ed1ab_1 conda-forge wrapt 1.12.1 pypi_0 pypi xorg-fixesproto 5.0 h7f98852_1002 conda-forge xorg-inputproto 2.3.2 h7f98852_1002 conda-forge xorg-kbproto 1.0.7 h7f98852_1002 conda-forge xorg-libice 1.0.10 h7f98852_0 conda-forge xorg-libsm 1.2.3 hd9c2040_1000 conda-forge xorg-libx11 1.7.2 h7f98852_0 conda-forge xorg-libxau 1.0.9 h7f98852_0 conda-forge xorg-libxdmcp 1.1.3 h7f98852_0 conda-forge xorg-libxext 1.3.4 h7f98852_1 conda-forge xorg-libxfixes 5.0.3 h7f98852_1004 conda-forge xorg-libxi 1.7.10 h7f98852_0 conda-forge xorg-libxrender 0.9.10 h7f98852_1003 conda-forge xorg-libxtst 1.2.3 h7f98852_1002 conda-forge xorg-recordproto 1.14.2 h7f98852_1002 conda-forge xorg-renderproto 0.11.1 h7f98852_1002 conda-forge xorg-xextproto 7.3.0 h7f98852_1002 conda-forge xorg-xproto 7.0.31 h7f98852_1007 conda-forge xz 5.2.5 h516909a_1 conda-forge yaml 0.2.5 h516909a_0 conda-forge yarl 1.6.3 pypi_0 pypi zarr 2.9.5 pyhd8ed1ab_0 conda-forge zeroc-ice36-python 3.6.5 py39h2bc3f7f_7 ome zlib 1.2.11 h516909a_1010 conda-forge zstd 1.5.0 ha95c52a_0 conda-forge ```

The output of the data generation has been uploaded to a temporary public bucket. Locally running

$ napari https://uk1s3.embassy.ebi.ac.uk/napari-ome-zarr_21/fused.zarr/0/

gives me the expected output with channels appearing as layers rather than sliders

I have napari 0.4.12 with napari-ome-zarr==0.3.1 and ome-zarr==0.2.1 on a local Python 3.9.7 Conda environment.

Trying to identify the divergence between our environments, what is the behavior when you are pointing napari at this dataset?

[dpshepherd]

I have napari==0.4.12, napari-ome-zarr==0.3.1, ome-zarr==0.2.1 on local Python 3.9.7 conda environment on a Windows 10 64-bit machine.

When I load the data from the bucket using,
napari https://uk1s3.embassy.ebi.ac.uk/napari-ome-zarr_21/fused.zarr/0/
I get the following two messages on the console,

11:36:37 WARNING version mismatch: detected:FormatV02, requested:FormatV03
11:36:43 WARNING version mismatch: detected:FormatV03, requested:FormatV02

and then the data loads as expected and I get a viewer that looks exactly like yours.

This suggests to me that it is either the TIFF files that BigStitcher created or bioformats2raw. Our server is finally freed up, so we can generate smaller files in the next day or two.

[will-moore]

Hi @dpshepherd did you make any progress here? Is this issue still open?
If so, I wonder if I could rename it, since the discussion is really about channel support within napari d:/lung.zarr/0 and I've created a new issue for adding support for bioformats2raw to allow napari d:/lung.zarr/ at #71.
Cheers

[dpshepherd]

Hi @will-moore -

We've moved away from OME-NGFF, so this can be closed as far as I'm concerned. We wont' be working on it anymore.