idr0001-graml-sysgro to NGFF #683

will-moore · 2024-01-08T12:56:28Z

idr0001 has 192 x 96-Well Plates, 6 acquisitions each.
1 Plate converted below is 47 GB.
Approx 4.5 TB in total.
bioformats2raw took ~30mins to convert 1 Plate => approx 4 days in total.

NB: The need to convert multi-acquisition Flex data (idr0001) is because the support for that hasn't been ported from IDR to mainline BioFormats: ome/bioformats#3537

will-moore · 2024-01-08T13:30:33Z

convert first plate for testing...
on pilot-zarr1-dev

~/bioformats2raw-0.6.0-24/bin/bioformats2raw "/uod/idr/filesets/idr0001-graml-sysgro/20151116-verified/JL_120731_S6A/Meas_01(2012-07-31_10-41-12)/001001001.flex" JL_120731_S6A
...
2024-01-08 13:36:06,658 [main] WARN  loci.formats.FormatHandler - mismatch between image count, names and factors (count=36864, names=32, factors=32)
2024-01-08 13:36:06,661 [main] WARN  loci.formats.FormatHandler - mismatch between image count, names and factors (count=36864, names=32, factors=32)
2024-01-08 13:36:06,664 [main] WARN  loci.formats.FormatHandler - mismatch between image count, names and factors (count=36864, names=32, factors=32)
2024-01-08 13:36:06,668 [main] WARN  loci.formats.FormatHandler - mismatch between image count, names and factors (count=36864, names=32, factors=32)
2024-01-08 13:36:06,671 [main] WARN  loci.formats.FormatHandler - mismatch between image count, names and factors (count=36864, names=32, factors=32)
...

(at 2pm)...

Done about 14:15 (approx 45 mins).

will-moore · 2024-01-08T14:23:05Z

make bucket etc...
then:

(base) [wmoore@pilot-zarr1-dev ~]$ ./mc cp -r /data/idr0001/JL_120731_S6A.ome.zarr uk1s3/idr0001/zarr
...0731_S6A.ome.zarr/OME/METADATA.ome.xml: 96.57 GiB / 96.57 GiB ━━━━━━━━━━━━━━ 48.81 MiB/s 33m45s

will-moore · 2024-01-08T14:45:52Z

Looks like we are getting duplicates of each of the 6 acquisitions for each Plate.
We would expect each Well to have 6 Fields but we get 12 (6 pairs of duplicates):
https://hms-dbmi.github.io/vizarr/?source=https://uk1s3.embassy.ebi.ac.uk/idr0001/zarr/JL_120731_S6A.ome.zarr/A/1/

sbesson · 2024-01-09T09:10:04Z

There are a series of issues we might need to review here. For reference, the source data corresponding to the example above is https://ftp.ebi.ac.uk/pub/databases/IDR/idr0001-graml-sysgro/20151116-verified/JL_120731_S6A/.

As can be seen above, there are 6 measurement folders, which are interpreted as plate acquisitions using the IDR Flex reader. Looking at the file listing for each measurement, there are 2 fields of views per well and plate acquisition.

001001001.flex
001001002.flex
001002001.flex
001002002.flex
...

In IDR, each well of each plate acquisition only contains 1 field per view and the fileset only includes the Flex files ending with 001.flex. I cannot comment know whether this is to be treated as an issue or an active decision made during the loading of this historical study /cc @joshmoore

If the intent is to create a Zarr dataset matching the current IDR representation, I think of two options:

either point bioformats2raw at a secondary structure where only the ...001.flex files are present
oruse omero-cli-zarr against live IDR

will-moore · 2024-01-09T10:24:09Z

Let's try omero-cli-zarr export for comparison...
On idr-ftp...
Update to latest ome/omero-cli-zarr#147

conda activate omero_zarr_export
pip uninstall omero-cli-zarr

# cloned https://github.com/ome/omero-cli-zarr/pull/147 and merged https://github.com/ome/omero-cli-zarr/pull/156
$ cd /data/idr0001/omero_cli_zarr/omero-cli-zarr
$ pip install -e .

$ omero zarr export Plate:2552 --name_by name

...export status... 5 out of 96 Wells done in 35 mins. 7 mins per Well is 11 hours per Plate!

... completed at 22:08 - ~11.5 hours.

Upload...

./mc cp -r /data/idr0001/omero_cli_zarr/JL_120731_S6B.ome.zarr uk1s3/idr0001/zarr

Looks good at https://ome.github.io/ome-ngff-validator/?source=https://uk1s3.embassy.ebi.ac.uk/idr0001/zarr/JL_120731_S6B.ome.zarr/

will-moore · 2024-01-10T06:32:07Z

Tried exporting Polygons as labels... but this fails as the Polygons are overlapping:

(omero_zarr_export) [wmoore@pilot-zarr1-dev omero_cli_zarr]$ omero zarr polygons Plate:2552 --source-image=JL_120731_S6B.ome.zarr

Found 51 mask shapes in 51 ROIs
Unique dimensions: {'T': {None}, 'C': {None}, 'Z': {None}}
source_image JL_120731_S6B.ome.zarr/D/7/0
Ignoring dimensions {'C', 'T', 'Z'}
Traceback (most recent call last):
  File "/home/wmoore/miniconda3/envs/omero_zarr_export/bin/omero", line 11, in <module>
    sys.exit(main())
  File "/home/wmoore/miniconda3/envs/omero_zarr_export/lib/python3.9/site-packages/omero/main.py", line 125, in main
    rv = omero.cli.argv()
  File "/home/wmoore/miniconda3/envs/omero_zarr_export/lib/python3.9/site-packages/omero/cli.py", line 1784, in argv
    cli.invoke(args[1:])
  File "/home/wmoore/miniconda3/envs/omero_zarr_export/lib/python3.9/site-packages/omero/cli.py", line 1222, in invoke
    stop = self.onecmd(line, previous_args)
  File "/home/wmoore/miniconda3/envs/omero_zarr_export/lib/python3.9/site-packages/omero/cli.py", line 1299, in onecmd
    self.execute(line, previous_args)
  File "/home/wmoore/miniconda3/envs/omero_zarr_export/lib/python3.9/site-packages/omero/cli.py", line 1381, in execute
    args.func(args)
  File "/data/idr0001/omero_cli_zarr/omero-cli-zarr/src/omero_zarr/cli.py", line 125, in _wrapper
    return func(self, *args, **kwargs)
  File "/data/idr0001/omero_cli_zarr/omero-cli-zarr/src/omero_zarr/cli.py", line 333, in polygons
    plate_shapes_to_zarr(plate, ["Polygon"], args)
  File "/data/idr0001/omero_cli_zarr/omero-cli-zarr/src/omero_zarr/masks.py", line 113, in plate_shapes_to_zarr
    saver.save(list(masks.values()), args.label_name)
  File "/data/idr0001/omero_cli_zarr/omero-cli-zarr/src/omero_zarr/masks.py", line 334, in save
    labels, fill_colors, properties = self.masks_to_labels(
  File "/data/idr0001/omero_cli_zarr/omero-cli-zarr/src/omero_zarr/masks.py", line 565, in masks_to_labels
    raise Exception(
Exception: Shape 624314 overlaps with existing labels

will-moore · 2024-01-10T08:50:31Z

Run mkngff on idr0125-pilot with

$ sudo mkdir /idr0001 && sudo /opt/goofys --endpoint https://uk1s3.embassy.ebi.ac.uk/ -o allow_other idr0001 /idr0001
$ ls /idr0001/zarr/
JL_120731_S6A.ome.zarr  JL_120731_S6B.ome.zarr

as omero-server

omero mkngff sql 16452 --clientpath="https://uk1s3.embassy.ebi.ac.uk/idr0001/zarr/JL_120731_S6B.ome.zarr" "/idr0001/zarr/JL_120731_S6B.ome.zarr" > "idr0001/16452.sql"

$ psql -U omero -d idr -h $DBHOST -f idr0001/16452.sql 
UPDATE 576
BEGIN
 mkngff_fileset 
----------------
        5289241
(1 row)

COMMIT


$ omero mkngff symlink /data/OMERO/ManagedRepository 16452 "/idr0001/zarr/JL_120731_S6B.ome.zarr" --bfoptions
...
Creating symlink /data/OMERO/ManagedRepository/demo_2/2015-11/23/16-25-19.109_mkngff/JL_120731_S6B.ome.zarr -> /idr0001/zarr/JL_120731_S6B.ome.zarr
Checking for prefix_dir /data/OMERO/ManagedRepository/demo_2/2015-11/23/16-25-19.109
write bfoptions to: /data/OMERO/ManagedRepository/demo_2/2015-11/23/16-25-19.109_mkngff/JL_120731_S6B.ome.zarr.bfoptions

http://localhost:1040/webclient/?show=image-1230224 ...

Fails with:

2024-01-10 09:00:57,168 INFO  [                      omero.cmd.SessionI] (.Server-11) Unregistered servant:078c4a67-66a5-46f8-822f-8e196ea78f82/a88d46c1-8860-4398-b072-50d9939ab731omero.api.IQuery(omero.api._IQueryTie@cbaf0b18)
2024-01-10 09:00:57,168 INFO  [                      omero.cmd.SessionI] (.Server-11) Removed servant from adapter: a88d46c1-8860-4398-b072-50d9939ab731omero.api.IQuery
2024-01-10 09:01:05,954 DEBUG [                   loci.formats.Memoizer] (l.Server-0) start[1704877049382] time[216572] tag[loci.formats.Memoizer.setId]
2024-01-10 09:01:05,954 DEBUG [                   loci.formats.Memoizer] (.Server-10) start[1704876989274] time[276679] tag[loci.formats.Memoizer.setId]
2024-01-10 09:01:05,956 ERROR [         ome.io.bioformats.BfPixelBuffer] (l.Server-0) Failed to instantiate BfPixelsWrapper with /data/OMERO/ManagedRepository/demo_2/2015-11/23/16-25-19.109_mkngff/JL_120731_S6B.ome.zarr/.zattrs
2024-01-10 09:01:05,956 ERROR [         ome.io.bioformats.BfPixelBuffer] (.Server-10) Failed to instantiate BfPixelsWrapper with /data/OMERO/ManagedRepository/demo_2/2015-11/23/16-25-19.109_mkngff/JL_120731_S6B.ome.zarr/.zattrs
2024-01-10 09:01:05,957 ERROR [                ome.io.nio.PixelsService] (.Server-10) Error instantiating pixel buffer: /data/OMERO/ManagedRepository/demo_2/2015-11/23/16-25-19.109_mkngff/JL_120731_S6B.ome.zarr/.zattrs
java.lang.RuntimeException: java.lang.ClassCastException: class java.lang.Long cannot be cast to class java.lang.String (java.lang.Long and java.lang.String are in module java.base of loader 'bootstrap')
        at ome.io.bioformats.BfPixelBuffer.reader(BfPixelBuffer.java:79)
        at ome.io.bioformats.BfPixelBuffer.setSeries(BfPixelBuffer.java:124)
        at ome.io.nio.PixelsService.createBfPixelBuffer(PixelsService.java:898)
        at ome.io.nio.PixelsService._getPixelBuffer(PixelsService.java:653)
        at ome.io.nio.PixelsService.getPixelBuffer(PixelsService.java:571)
        at ome.services.RenderingBean$12.doWork(RenderingBean.java:2205)
        at jdk.internal.reflect.GeneratedMethodAccessor295.invoke(Unknown Source)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.base/java.lang.reflect.Method.invoke(Method.java:566)
        at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:333)

will-moore · 2024-01-10T09:28:43Z

Error is raised at https://github.com/ome/omero-romio/blob/1d30fafc4e06c5511cfbb24c25a753925ffb2eb4/src/main/java/ome/io/bioformats/BfPixelBuffer.java#L79
but that's not where the ClassCastException comes from.

sbesson · 2024-01-10T11:09:26Z

Error is raised at https://github.com/ome/omero-romio/blob/1d30fafc4e06c5511cfbb24c25a753925ffb2eb4/src/main/java/ome/io/bioformats/BfPixelBuffer.java#L79
but that's not where the ClassCastException comes from.

I suspect the error comes from the underlying reader and is rethrown from the Bio-Formats pixel buffer. Is there more information in the following lines of the stack trace?

will-moore · 2024-01-10T11:25:21Z

Ah, yes...

Caused by: java.lang.ClassCastException: class java.lang.Long cannot be cast to class java.lang.String (java.lang.Long and java.lang.String are in module java.base of loader 'bootstrap')
        at loci.formats.in.ZarrReader.parsePlate(ZarrReader.java:755)
        at loci.formats.in.ZarrReader.initFile(ZarrReader.java:353)
        at loci.formats.FormatReader.setId(FormatReader.java:1443)
        at loci.formats.ImageReader.setId(ImageReader.java:849)
        at ome.io.nio.PixelsService$3.setId(PixelsService.java:869)
        at loci.formats.ReaderWrapper.setId(ReaderWrapper.java:650)
        at loci.formats.ChannelFiller.setId(ChannelFiller.java:234)
        at loci.formats.ReaderWrapper.setId(ReaderWrapper.java:650)
        at loci.formats.ChannelSeparator.setId(ChannelSeparator.java:293)
        at loci.formats.ReaderWrapper.setId(ReaderWrapper.java:650)
        at loci.formats.Memoizer.setId(Memoizer.java:690)
        at ome.io.bioformats.BfPixelsWrapper.<init>(BfPixelsWrapper.java:52)
        at ome.io.bioformats.BfPixelBuffer.reader(BfPixelBuffer.java:73)
        ... 82 common frames omitted

will-moore · 2024-01-10T11:30:25Z

That line is https://github.com/ome/ZarrReader/blob/main/src/loci/formats/in/ZarrReader.java#L755

String acqStartTime = (String) acquistion.get("starttime");

The schema states that this is an integer:
https://github.com/ome/ngff/blob/main/0.4/schemas/plate.schema#L35

cc @dgault

will-moore · 2024-01-11T11:09:55Z

We want to create a copy of the original data without ...2.flex files but without duplicating all the data!

On pilot-zarr1-dev

export PLATE=JL_120801_S7A
cd /data/idr0001/raw

mkdir $PLATE
cd $PLATE
for i in $(ls /uod/idr/filesets/idr0001-graml-sysgro/20151116-verified/$PLATE/); do
  mkdir "$i"
  for f in $(ls "/uod/idr/filesets/idr0001-graml-sysgro/20151116-verified/$PLATE/$i"); do
    ln -s "/uod/idr/filesets/idr0001-graml-sysgro/20151116-verified/$PLATE/$i/$f" "$i/$f"
  done
  rm "$i"/*2.flex
done

$ ~/bioformats2raw-0.6.0-24/bin/bioformats2raw "JL_120801_S7A/Meas_01(2012-08-01_10-57-32)/001001001.flex"  ngff/JL_120801_S7A.ome.zarr
...

Conversion took approx 30 mins.

(bioformats2raw) [wmoore@pilot-zarr1-dev ngff]$ ~/mc cp -r /data/idr0001/raw/ngff/JL_120801_S7A.ome.zarr uk1s3/idr0001/zarr
...7A.ome.zarr/OME/METADATA.ome.xml: 47.15 GiB / 47.15 GiB ━━━━━━━━━━━━━━━━ 49.82 MiB/s 16m9s

will-moore · 2024-01-11T13:07:34Z

omero mkngff sql 16454 --clientpath="https://uk1s3.embassy.ebi.ac.uk/idr0001/zarr/JL_120801_S7A.ome.zarr" "/idr0001/zarr/JL_120801_S7A.ome.zarr" > "idr0001/16454.sql"

$ psql -U omero -d idr -h $DBHOST -f idr0001/16454.sql 
UPDATE 576
BEGIN
 mkngff_fileset 
----------------
        5289242
(1 row)

COMMIT

$ omero mkngff symlink /data/OMERO/ManagedRepository 16454 "/idr0001/zarr/JL_120801_S7A.ome.zarr" --bfoptions
...
Creating dir at /data/OMERO/ManagedRepository/demo_2/2015-11/23/17-31-08.324_mkngff
Creating symlink /data/OMERO/ManagedRepository/demo_2/2015-11/23/17-31-08.324_mkngff/JL_120801_S7A.ome.zarr -> /idr0001/zarr/JL_120801_S7A.ome.zarr
Checking for prefix_dir /data/OMERO/ManagedRepository/demo_2/2015-11/23/17-31-08.324
write bfoptions to: /data/OMERO/ManagedRepository/demo_2/2015-11/23/17-31-08.324_mkngff/JL_120801_S7A.ome.zarr.bfoptions

This Plate renders OK in webclient 👍
NB: it avoids the ZarrReader bug above since it doesn't have acquisition starttime or endtime metadata.

will-moore · 2024-01-11T14:15:38Z

Create symlinks...

cd /data/idr0001/raw

for PLATE in $(ls /uod/idr/filesets/idr0001-graml-sysgro/20151116-verified/); do
echo $PLATE
mkdir $PLATE
for i in $(ls /uod/idr/filesets/idr0001-graml-sysgro/20151116-verified/$PLATE/); do
  mkdir "$PLATE/$i"
  for f in $(ls "/uod/idr/filesets/idr0001-graml-sysgro/20151116-verified/$PLATE/$i"); do
    ln -s "/uod/idr/filesets/idr0001-graml-sysgro/20151116-verified/$PLATE/$i/$f" "$PLATE/$i/$f"
  done
  rm -f "$PLATE/$i"/*2.flex
done
done

Count flex files for each Plate/Acquisition in plates.csv (in IDR):

for PLATE in $(cat plates.csv); do
echo $PLATE >> counts3.log
for i in $(ls /uod/idr/filesets/idr0001-graml-sysgro/20151116-verified/$PLATE/); do
 ls "$PLATE/$i" | wc >> counts3.log
done
done

cat counts3.log grep -v 96
# ignored Plates where all acquisitions have 96 files. That leaves...
JL_120801_S7B
     95      95    1425
JL_120814_S17A
     95      95    1425
JL_120816_S19B
     95      95    1425
JL_120907_S2A
     95      95    1425
JL_121214_J2_1
     95      95    1425
JL_121217_J7_1
     95      95    1425
JL_121219_J4_3
     95      95    1425
JL_121220_J5_3
     95      95    1425
JL_130126_J10_3
     93      93    1395
JL_130126_J10_4
     91      91    1365
JL_130126_J9_3
     94      94    1410
JL_130126_J9_4
     93      93    1395
JL_130127_J10_5
     94      94    1410
JL_130127_J9_5
     93      93    1395
JL_130127_J9_6
     94      94    1410
JL_130128_J9_7
     95      95    1425
JL_130128_J9_8
     95      95    1425
     95      95    1425
     95      95    1425
     95      95    1425
     95      95    1425
JL_130304_R1_3
     95      95    1425
X_110213_S2_Blue1000
     95      95    1425
X_110331_S13
     95      95    1425
X_110425_S6
     95      95    1425
X_110429_S24
     95      95    1425

will-moore · 2024-01-11T15:56:25Z

Checking for counts of ...2.flex files which I've "deleted" (removed symlinks) above.
Most plates/acquisitions have 0 or 96 but some have other counts. Getting complex!

for PLATE in $(ls /uod/idr/filesets/idr0001-graml-sysgro/20151116-verified/); do
echo $PLATE >> counts2.log
for i in $(ls /uod/idr/filesets/idr0001-graml-sysgro/20151116-verified/$PLATE/); do
 ls "/uod/idr/filesets/idr0001-graml-sysgro/20151116-verified/$PLATE/$i" | grep "2.flex" | wc >> counts2.log
done
done

will-moore · 2024-01-11T16:27:24Z

On pilot-zarr1-dev

$ cd /data/idr0001/raw
$ cat plates.csv
JL_120731_S6A
JL_120731_S6B
JL_120801_S7A
...
$ cat plates.csv | wc
    192     192    2828

for i in $(cat plates.csv); do
  first_flex=$(find $i -name *.flex -print -quit)
  echo $first_flex
  ~/bioformats2raw-0.6.0-24/bin/bioformats2raw "$first_flex"  ngff/$i.ome.zarr
done

will-moore · 2024-01-12T09:27:14Z

Try viewing some of the 22 Plates above which have fewer than 96 .flex files...
Are these OK on IDR??

Plate JL_120801_S7B only has 3 Acquisitions in IDR and these images are viewable. It is the 4th Acquisition that has only 95 flex files
JL_120814_S17A appears to be all OK.
Plate JL_120816_S19B - The 2nd batch of 50 thumbnails doesn't load - No images viewable message = Error instantiating pixel buffer: /data/OMERO/ManagedRepository/demo_2/2015-11/24/03-19-34.711/Meas_01(2012-08-16_10-38-18)/001001001.flex
JL_120907_S2A - Error instantiating pixel buffer: /data/OMERO/ManagedRepository/demo_2/2015-11/24/18-09-54.256/Meas_01(2012-09-07_03-04-56)/001001001.flex
E.g. plate JL_130126_J10_3 https://idr.openmicroscopy.org/webclient/?show=well-522426
Try to view Preview tab:

    serverExceptionClass = ome.conditions.ResourceError
    message = Error instantiating pixel buffer: /data/OMERO/ManagedRepository/demo_2/2015-10/01/19-56-57.181/Meas_01(2013-01-26_16-42-53)/005004001.flex
}

will-moore · 2024-01-12T13:51:55Z

Since none of the Plates with missing .flex files above have successfully been converted by the script running above,
let's test one on it's own (NB: this is the symlinked data with the *2.flex files removed):

screen -S idr0001_test:

$ ~/bioformats2raw-0.6.0-24/bin/bioformats2raw JL_120801_S7B/Meas_01\(2012-08-01_18-40-47\)/001001001.flex ngff/JL_120801_S7B.ome.zarr
...
2024-01-12 13:51:27,012 [main] WARN  loci.formats.FormatHandler - parsing /data/idr0001/raw/JL_120801_S7B/Meas_06(2012-08-02_01-08-20)/008010001.flex
2024-01-12 13:51:27,669 [main] WARN  loci.formats.FormatHandler - parsing /data/idr0001/raw/JL_120801_S7B/Meas_06(2012-08-02_01-08-20)/008011001.flex
2024-01-12 13:51:28,343 [main] WARN  loci.formats.FormatHandler - parsing /data/idr0001/raw/JL_120801_S7B/Meas_06(2012-08-02_01-08-20)/008012001.flex
Exception in thread "main" picocli.CommandLine$ExecutionException: Error while calling command (com.glencoesoftware.bioformats2raw.Converter@16150369): java.lang.ArithmeticException: / by zero
        at picocli.CommandLine.executeUserObject(CommandLine.java:1962)
        at picocli.CommandLine.access$1300(CommandLine.java:145)
        at picocli.CommandLine$RunLast.executeUserObjectOfLastSubcommandWithSameParent(CommandLine.java:2352)
        at picocli.CommandLine$RunLast.handle(CommandLine.java:2346)
        at picocli.CommandLine$RunLast.handle(CommandLine.java:2311)
        at picocli.CommandLine$AbstractParseResultHandler.handleParseResult(CommandLine.java:2172)
        at picocli.CommandLine.parseWithHandlers(CommandLine.java:2550)
        at picocli.CommandLine.parseWithHandler(CommandLine.java:2485)
        at picocli.CommandLine.call(CommandLine.java:2761)
        at com.glencoesoftware.bioformats2raw.Converter.main(Converter.java:2192)
Caused by: java.lang.ArithmeticException: / by zero
        at loci.formats.in.FlexReader.groupFiles(FlexReader.java:1402)
        at loci.formats.in.FlexReader.initFlexFile(FlexReader.java:581)
        at loci.formats.in.FlexReader.initFile(FlexReader.java:390)
        at loci.formats.FormatReader.setId(FormatReader.java:1389)
        at loci.formats.ImageReader.setId(ImageReader.java:849)
        at com.glencoesoftware.bioformats2raw.Converter.getBaseReaderClass(Converter.java:2032)
        at com.glencoesoftware.bioformats2raw.Converter.convert(Converter.java:540)
        at com.glencoesoftware.bioformats2raw.Converter.call(Converter.java:516)
        at com.glencoesoftware.bioformats2raw.Converter.call(Converter.java:107)
        at picocli.CommandLine.executeUserObject(CommandLine.java:1953)
        ... 9 more

Also try it on the original data where the number of 2.flex files for each acquisition matches the *1.flex file counts.

(bioformats2raw) [wmoore@pilot-zarr1-dev raw]$ ~/bioformats2raw-0.6.0-24/bin/bioformats2raw /uod/idr/filesets/idr0001-graml-sysgro/20151116-verified/JL_120801_S7B/Meas_01\(2012-08-01_18-40-47\)/001001001.flex ngff/JL_120801_S7B.ome.zarr

This actually worked and has the correct number of Fields!

(base) [wmoore@pilot-zarr1-dev ngff]$ ls -alh !$
ls -alh JL_120801_S7B.ome.zarr/A/1
total 0
drwxrwxr-x.  8 wmoore wmoore  88 Jan 12 14:28 .
drwxrwxr-x. 14 wmoore wmoore 169 Jan 12 14:13 ..
drwxrwxr-x.  6 wmoore wmoore 100 Jan 12 14:12 0
drwxrwxr-x.  6 wmoore wmoore 100 Jan 12 14:15 1
drwxrwxr-x.  6 wmoore wmoore 100 Jan 12 14:18 2
drwxrwxr-x.  6 wmoore wmoore 100 Jan 12 14:21 3
drwxrwxr-x.  6 wmoore wmoore 100 Jan 12 14:24 4
drwxrwxr-x.  6 wmoore wmoore 100 Jan 12 14:28 5

Plate has the same number of images (.zattrs) as previous S7A plate!

$ find JL_120801_S7A.ome.zarr/ -name ".zattrs" | wc
    674     674   24906
$ find JL_120801_S7B.ome.zarr/ -name ".zattrs" | wc
    674     674   24906

will-moore · 2024-01-12T16:06:27Z

Another Plate that failed with symlinked data (without *2.flex files) - run against original data...

(bioformats2raw) [wmoore@pilot-zarr1-dev raw]$ ~/bioformats2raw-0.6.0-24/bin/bioformats2raw /uod/idr/filesets/idr0001-graml-sysgro/20151116-verified/JL_120814_S17A/Meas_01\(2012-08-14_10-49-32\)/001001001.flex ngff/JL_120814_S17A.ome.zarr
...

will-moore · 2024-06-24T11:11:46Z

5 months later

Investigation of possible Bio-Formats fix at ome/bioformats#3537 finds that this won't be easy/viable solution. So we need to revive the NGFF conversion work...

Summary from first read of history above:

using bioformats2raw-0.6.0-24 to convert requires creating plate structure that lacks ...2.flex files to avoid duplicating fields. But this has variable success depending on plate structure
using omero-cli-zarr takes 11 hours per plate (192 plates will take 3 months!!)

will-moore · 2024-07-03T15:58:21Z

If we need to use omero-cli-zarr, old idr is still available at ssh 45.88.81.175 etc...

will-moore · 2024-07-07T12:45:31Z

Would be good to convert all plates as above with bioformats2raw, and check which ones work with mkngff. Then possibly use omero-cli-zarr to export the ones that fail (hopefully smaller number) which is kinda slow.

Plates converted are ~47 GB so we only have space to convert 1 plate on each of pilot-zarr1-dev and pilot-zarr2-dev`.

(base) [wmoore@pilot-zarr1-dev ~]$ df -h /
Filesystem      Size  Used Avail Use% Mounted on
/dev/vda1       100G   25G   76G  25% /

(base) [wmoore@pilot-zarr2-dev ~]$ df -h /
Filesystem      Size  Used Avail Use% Mounted on
/dev/vda1       100G   20G   81G  20% /

sbesson · 2024-07-07T13:32:52Z

Plates converted are ~47 GB so we only have space to convert 1 plate on each of pilot-zarr1-dev and pilot-zarr2-dev`.
(base) [wmoore@pilot-zarr1-dev ~]$ df -h /
Filesystem      Size  Used Avail Use% Mounted on
/dev/vda1       100G   25G   76G  25% /
(base) [wmoore@pilot-zarr2-dev ~]$ df -h /
Filesystem      Size  Used Avail Use% Mounted on
/dev/vda1       100G   20G   81G  20% /

Converted data should never end up in the root partition. There is a dedidated /data partition on each of this node.

(base) [sbesson@pilot-zarr1-dev ~]$ df -h | grep /data
/dev/vdb                    4.9T  4.8T  112G  98% /data

(base) [sbesson@pilot-zarr2-dev ~]$ df -h | grep /data
/dev/vdb                    750G   89G  661G  12% /data

If 5TB is sufficient, let's review and discuss the cleanup the existing /data volume on these resources. Otherwise, let's define what the ideal requirements in terms of compute and storage and create a new pilot.

joshmoore · 2024-07-08T14:47:44Z

Thanks, @sbesson. @will-moore, were you looking to convert e.g. all of idr0001 at once rather than, e.g., moving each plate off to S3?

will-moore · 2024-08-19T13:04:25Z

Need env for omero cli zarr...
Want to install conda etc...
First need to install wget!

$ ssh -A 45.88.81.175 -L 1080:localhost:80
Last login: Mon Aug 19 10:45:01 2024 from 82.132.231.200

NB: webclient is broken e.g. http://localhost:1080/webclient/

  File "/opt/omero/web/venv3/lib64/python3.9/site-packages/omeroweb/webclient/views.py", line 490, in _load_template
    active_group = request.session.get("active_group") or conn.getEventContext().groupId

AttributeError: 'NoneType' object has no attribute 'getEventContext'

Try to use omero-cli-zarr...

[wmoore@prod121-proxy ~]$ sudo yum install wget

[wmoore@prod121-proxy ~]$ wget https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh -O ~/miniconda3/miniconda.sh
$ bash ~/miniconda3/miniconda.sh -b -u -p ~/miniconda3

$ ~/miniconda3/bin/conda init bash
$ source .bashrc

$ conda create -n omeropy python=3.9 conda-forge::zeroc-ice==3.6.5 omero-py

We want to use ome/omero-cli-zarr#147 (not merged yet) so we need to checkout that branch etc.
Need to install git!

(omeropy) [wmoore@prod121-proxy idr0001]$ sudo yum install git

Merged ome/omero-cli-zarr#147 with origin to fix scm version issue...

14:01...

omero zarr export Plate:2551
...

Completed ~ 10:30 pm - 8.5 hours for a Plate...

Rename plate, since I didn't use --name_by name during export...
and upload to s3...

$ mv 2551.ome.zarr JL_120731_S6A.ome.zarr

$ /home/wmoore/mc cp -r JL_120731_S6A.ome.zarr uk1s3/idr/zarr/v0.4/idr0101A/

(base) [wmoore@prod121-proxy idr0001]$ /home/wmoore/mc cp -r JL_120731_S6A.ome.zarr uk1s3/idr/zarr/v0.4/idr0101A/
.../JL_120731_S6A.ome.zarr/H/9/5/4/1/9/0/0: 32.53 GiB / 32.53 GiB ┃▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓┃ 37.98 MiB/s 14m36s

EDIT... 29th August...
Ooops! typo in the upload "idr0101A" -> "idr0001A".

(base) [wmoore@prod121-proxy idr0001]$ /home/wmoore/mc cp -r JL_120731_S6A.ome.zarr uk1s3/idr/zarr/v0.4/idr0001A/
...0001/JL_120731_S6A.ome.zarr.zip: 24.53 GiB / 24.53 GiB ┃▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓┃ 134.05 MiB/s 3m7s

Ahhh! - typo again!!! JL_120731_S6A.ome.zarr.zip because I have deleted local unzipped data!
Unzip... Upload AGAIN!

$ unzip JL_120731_S6A.ome.zarr.zip

$  /home/wmoore/mc cp -r JL_120731_S6A.ome.zarr uk1s3/idr/zarr/v0.4/idr0001A/
...31_S6A.ome.zarr/H/9/5/4/1/9/0/0: 57.06 GiB / 57.06 GiB ┃▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓┃ 89.84 MiB/s 10m50s
$ df -h ./
Filesystem      Size  Used Avail Use% Mounted on
/dev/vda5        79G   64G   16G  81% /

Cleanup:

time /home/wmoore/mc rm --recursive --force uk1s3/idr/zarr/v0.4/idr0101A/JL_120731_S6A.ome.zarr
/home/wmoore/mc rm --force uk1s3/idr/zarr/v0.4/idr0001A/JL_120731_S6A.ome.zarr.zip

will-moore · 2024-08-29T11:42:28Z

Install goofys and mount bia-integrator-data and idr buckets...

$ wget https://github.com/kahing/goofys/releases/latest/download/goofys
$ chmod +x goofys

# didn't need this yet!
$ sudo mkdir /bia-integrator-data && sudo ./goofys --endpoint https://uk1s3.embassy.ebi.ac.uk/ -o allow_other bia-integrator-data /bia-integrator-data

$ sudo mkdir /uk1s3_idr && sudo ./goofys --endpoint https://uk1s3.embassy.ebi.ac.uk/ -o allow_other idr /uk1s3_idr

(base) [wmoore@prod121-proxy ~]$ ls /uk1s3_idr/zarr/v0.4/idr0001A
2551.zarr  JL_120731_S6A.ome.zarr

All good!

will-moore · 2024-08-29T13:33:07Z

mkngff...

conda activate omero-py
pip install 'omero-mkngff @ git+https://github.com/joshmoore/omero-mkngff@main'

First plate: Fileset 16451

$ omero login
$ time omero mkngff sql 16451 --clientpath="https://uk1s3.embassy.ebi.ac.uk/idr/zarr/v0.4/idr0001A/JL_120731_S6A.ome.zarr" "/uk1s3_idr/zarr/v0.4/idr0001A/JL_120731_S6A.ome.zarr" > "idr0001/16451.sql"

On pilot-idrngff... Following https://github.com/IDR/mkngff_upgrade_scripts

(venv3) (base) [wmoore@pilot-idrngff-omeroreadwrite ~]$ sudo mkdir /uk1s3_idr && sudo /opt/goofys --endpoint https://uk1s3.embassy.ebi.ac.uk/ -o allow_other idr /uk1s3_idr

$ psql -U omero -d idr -h $DBHOST -f 16451.sql 
UPDATE 576
BEGIN
 mkngff_fileset 
----------------
        6320797
(1 row)

COMMIT


$ omero login   (as demo user)
$ omero mkngff symlink /data/OMERO/ManagedRepository 16451 "/uk1s3_idr/zarr/v0.4/idr0001A/JL_120731_S6A.ome.zarr" --bfoptions --clientpath="https://uk1s3.embassy.ebi.ac.uk/idr/zarr/v0.4/idr0001A/JL_120731_S6A.ome.zarr"

Using session for demo@localhost:4064. Idle timeout: 10 min. Current group: Public
Checking for prefix_dir /data/OMERO/ManagedRepository/demo_2/2015-11/23/16-25-15.579
Creating dir at /data/OMERO/ManagedRepository/demo_2/2015-11/23/16-25-15.579_mkngff
Creating symlink /data/OMERO/ManagedRepository/demo_2/2015-11/23/16-25-15.579_mkngff/JL_120731_S6A.ome.zarr -> /uk1s3_idr/zarr/v0.4/idr0001A/JL_120731_S6A.ome.zarr
Checking for prefix_dir /data/OMERO/ManagedRepository/demo_2/2015-11/23/16-25-15.579
write bfoptions to: /data/OMERO/ManagedRepository/demo_2/2015-11/23/16-25-15.579_mkngff/JL_120731_S6A.ome.zarr.bfoptions

View image...
10-15 mins...

Traceback (most recent call last):

  File "/opt/omero/web/venv3/lib64/python3.9/site-packages/omero_api_RenderingEngine_ice.py", line 1192, in load
    return _M_omero.api.RenderingEngine._op_load.invoke(self, ((), _ctx))

omero.ResourceError: exception ::omero::ResourceError
{
    serverStackTrace = ome.conditions.ResourceError: Error instantiating pixel buffer: /data/OMERO/ManagedRepository/demo_2/2015-11/23/16-25-15.579_mkngff/JL_120731_S6A.ome.zarr/.zattrs
	at ome.io.nio.PixelsService.createBfPixelBuffer(PixelsService.java:907)
	at ome.io.nio.PixelsService._getPixelBuffer(PixelsService.java:653)

    serverExceptionClass = ome.conditions.ResourceError
    message = Error instantiating pixel buffer: /data/OMERO/ManagedRepository/demo_2/2015-11/23/16-25-15.579_mkngff/JL_120731_S6A.ome.zarr/.zattrs
}

<WSGIRequest: GET '/webclient/metadata_preview/well/590809/?_=1724937458855'>

Blitz.log

2024-08-29 13:28:52,846 DEBUG [                   loci.formats.Memoizer] (.Server-10) start[1724937495656] time[637189] tag[loci.formats.Memoizer.setId]
2024-08-29 13:28:52,847 ERROR [         ome.io.bioformats.BfPixelBuffer] (.Server-10) Failed to instantiate BfPixelsWrapper with /data/OMERO/ManagedRepository/demo_2/2015-11/23/16-25-15.579_mkngff/JL_120731_S6A.ome.zarr/.zattrs
2024-08-29 13:28:52,848 ERROR [                ome.io.nio.PixelsService] (.Server-10) Error instantiating pixel buffer: /data/OMERO/ManagedRepository/demo_2/2015-11/23/16-25-15.579_mkngff/JL_120731_S6A.ome.zarr/.zattrs
java.lang.RuntimeException: java.lang.ClassCastException: class java.lang.Long cannot be cast to class java.lang.Integer (java.lang.Long and java.lang.Integer are in module java.base of loader 'bootstrap')
        at ome.io.bioformats.BfPixelBuffer.reader(BfPixelBuffer.java:79)
        at ome.io.bioformats.BfPixelBuffer.setSeries(BfPixelBuffer.java:124)
        at ome.io.nio.PixelsService.createBfPixelBuffer(PixelsService.java:898)
        at ome.io.nio.PixelsService._getPixelBuffer(PixelsService.java:653)
        at ome.io.nio.PixelsService.getPixelBuffer(PixelsService.java:571)
        at ome.services.RenderingBean$12.doWork(RenderingBean.java:2205)
        at jdk.internal.reflect.GeneratedMethodAccessor317.invoke(Unknown Source)
...
Caused by: java.lang.ClassCastException: class java.lang.Long cannot be cast to class java.lang.Integer (java.lang.Long and java.lang.Integer are in module java.base of loader 'bootstrap')
        at loci.formats.in.ZarrReader.parsePlate(ZarrReader.java:764)
        at loci.formats.in.ZarrReader.initFile(ZarrReader.java:361)
        at loci.formats.FormatReader.setId(FormatReader.java:1480)
        at loci.formats.ImageReader.setId(ImageReader.java:864)
        at ome.io.nio.PixelsService$3.setId(PixelsService.java:869)
        at loci.formats.ReaderWrapper.setId(ReaderWrapper.java:692)
        at loci.formats.ChannelFiller.setId(ChannelFiller.java:258)
        at loci.formats.ReaderWrapper.setId(ReaderWrapper.java:692)
        at loci.formats.ChannelSeparator.setId(ChannelSeparator.java:317)
        at loci.formats.ReaderWrapper.setId(ReaderWrapper.java:692)
        at loci.formats.Memoizer.setId(Memoizer.java:726)
        at ome.io.bioformats.BfPixelsWrapper.<init>(BfPixelsWrapper.java:52)
        at ome.io.bioformats.BfPixelBuffer.reader(BfPixelBuffer.java:73)

This line of ZarrReader:

          Integer acqStartTime = (Integer) acquistion.get("starttime");

will-moore · 2024-08-30T11:10:00Z

Edited the plate /.zattrs file above to remove starttime attrs and replaced on s3:

$ /home/wmoore/mc cp JL_120731_S6A.ome.zarr/.zattrs uk1s3/idr/zarr/v0.4/idr0001A/JL_120731_S6A.ome.zarr/
...e/idr0001/JL_120731_S6A.ome.zarr/.zattrs: 13.99 KiB / 13.99 KiB ┃▓▓▓▓▓▓▓▓▓▓▓▓┃ 32.17 KiB/s 0s

Try to delete memo file on pilot-idrngff and re-create...
Don't see any memo file at:

bash-5.1$ ls -alh !$
ls -alh /data/OMERO/BioFormatsCache/data/OMERO/ManagedRepository/demo_2/2015-11/23/16-25-15.579_mkngff/JL_120731_S6A.ome.zarr
total 0

2024-08-30 10:28:11,565 DEBUG [                   loci.formats.Memoizer] (.Server-20) saved to temp file: /data/OMERO/BioFormatsCache/data/OMERO/ManagedRepository/demo_2/2015-11/23/16-25-15.579_mkngff/JL_120731_S6A.ome.zarr/..zattrs.bfmemo14210388543534098629
2024-08-30 10:28:11,566 DEBUG [                   loci.formats.Memoizer] (.Server-20) start[1725013691240] time[326] tag[loci.formats.Memoizer.saveMemo]
2024-08-30 10:28:11,566 DEBUG [                   loci.formats.Memoizer] (.Server-20) saved memo file: /data/OMERO/BioFormatsCache/data/OMERO/ManagedRepository/demo_2/2015-11/23/16-25-15.579_mkngff/JL_120731_S6A.ome.zarr/..zattrs.bfmemo (1318254 bytes)
2024-08-30 10:28:11,566 DEBUG [                   loci.formats.Memoizer] (.Server-20) start[1725012959020] time[732546] tag[loci.formats.Memoizer.setId]

732546 ms is 12 minutes.

We can now view images for all acquisitions...
BUT Wells are in the wrong place:

With new thumbnails generated by saving rendering settings, we see:
The first 6 images (A1 - A6) are actually the 6 Fields from A1. The next 6 images (A7 - A12) are the 6 Fields from A2.

The next 6 images (B1 - B6) correspond to the 6 Fields from A3 (as seen in vizarr):

https://hms-dbmi.github.io/vizarr/?source=https://uk1s3.embassy.ebi.ac.uk/idr/zarr/v0.4/idr0001A/JL_120731_S6A.ome.zarr/A/3

will-moore · 2024-09-02T10:12:44Z

NB: few plates uploaded above:

$ aws --profile embassy s3 ls idr0001/zarr/
                           PRE JL_120731_S6A.ome.zarr/
                           PRE JL_120731_S6B.ome.zarr/
                           PRE JL_120801_S7A.ome.zarr/

See #683 (comment)
"Plate renders OK in webclient" - but did it have the Well/acquisition layout issues in previous comment above??
https://ome.github.io/ome-ngff-validator/?source=https://uk1s3.embassy.ebi.ac.uk/idr0001/zarr/JL_120801_S7A.ome.zarr/

TODO: try mkngff with that plate

will-moore · 2024-09-02T10:14:58Z

@sbesson pointed out that pilot-idrngff doesn't have latest ZarrReader. Try to update and check again... (delete memo etc)

will-moore · 2024-09-06T13:08:15Z

Update OMEZarrReader...
Can't get latest release from https://artifacts.openmicroscopy.org/ui/repos/tree/General/ome.releases/ome/OMEZarrReader (asks for login to JFrog!) so used latest build instead...

on pilot-idrngff, as omero-server user

wget https://merge-ci.openmicroscopy.org/jenkins/job/BIOFORMATS-build/182/label=testintegration/artifact/bio-formats-build/ZarrReader/target/OMEZarrReader-0.5.3-SNAPSHOT-jar-with-dependencies.jar
bash-5.1$ mv OMEZarrReader-0.5.3-SNAPSHOT-jar-with-dependencies.jar OMEZarrReader-0.5.3-build_182.jar
bash-5.1$ rm /opt/omero/server/OMERO.server/lib/client/OMEZarrReader_b45.jar
bash-5.1$ cp OMEZarrReader-0.5.3-build_182.jar /opt/omero/server/OMERO.server/lib/client/
bash-5.1$ rm /opt/omero/server/OMERO.server/lib/server/OMEZarrReader_b45.jar
bash-5.1$ cp OMEZarrReader-0.5.3-build_182.jar /opt/omero/server/OMERO.server/lib/server/

restarted...
Delete previous memo file for plate above:

$ ls -alh /data/OMERO/BioFormatsCache/data/OMERO/ManagedRepository/demo_2/2015-11/23/16-25-15.579_mkngff/JL_120731_S6A.ome.zarr
total 1.3M
drwxrwxr-x. 2 omero-server omero-server   37 Aug 30 10:28 .
drwxrwxr-x. 3 omero-server omero-server   44 Aug 29 13:18 ..
-rw-rw-r--. 1 omero-server omero-server 1.3M Aug 30 10:28 ..zattrs.bfmemo

$ rm /data/OMERO/BioFormatsCache/data/OMERO/ManagedRepository/demo_2/2015-11/23/16-25-15.579_mkngff/JL_120731_S6A.ome.zarr/..zattrs.bfmemo

View plate in webclient. 14:08...

will-moore · 2024-09-06T16:07:21Z

Unfortunately, upgrading ZarrReader didn't fix the Plate/Wells layout issue described above. No change.

Issue created at ome/ZarrReader#96

will-moore · 2024-09-10T09:25:11Z

As suggested by Seb, let's try to import the zarr plate from scratch from remote s3 at https://uk1s3.embassy.ebi.ac.uk/idr/zarr/v0.4/idr0001A/JL_120731_S6A.ome.zarr.
Follow the idr0125 workflow as described at https://github.com/IDR/SubmissionWorkflow/issues/85

Download metadata-only plate to temp location for import...

ssh pilot-idrngff
conda create -n awscli python=3.12
conda activate awscli
pip install awscli

# as sudo -u omero-server -s

time aws s3 sync --no-sign-request --exclude '*' --include "*.z*" --include "*.xml" --endpoint-url https://uk1s3.embassy.ebi.ac.uk s3://idr/zarr/v0.4/idr0001A/JL_120731_S6A.ome.zarr JL_120731_S6A.ome.zarr
...
real	1m21.225s

Import the data...

screen -r idr0125_import
omero login
time omero import --depth=100 --transfer=ln_s --skip=all JL_120731_S6A.ome.zarr --file /tmp/idr0001_20241010.log  --errs /tmp/idr0001_20241010.err

No imports found

Ah, I guess Bio-Formats doesn't recognise a plate, without bioformats2raw.layout and OME/METADATA.ome.xml ?

will-moore · 2024-09-10T14:40:34Z

@sbesson @joshmoore What's the easiest way to get Bio-Formats to recognise a plate for import?

Does it need plate.zarr/OME/METADATA.ome.xml and if so, how complete does the xml have to be? If it's too complex to create manually, I could try to export it from the original plate with bioformats2raw ?
If it doesn't need the xml, is it sufficient to add the bioformats2raw.layout to the .zattrs??

sbesson · 2024-09-10T16:14:38Z

I am unaware /OME/METADATA.ome.xml is a requirement for ZarrReader (although it might speed up initialization time?). Looking at an example of existing production IDR plate that have been converted to NGFF using omero-cli-zarr, I was able to run

[sbesson@pilot-idrngff-omeroreadwrite ~]$ /opt/omero/server/OMERO.server/bin/omero import -f  --debug=DEBUG /bia-integrator-data/S-BIAD867/103d9428-b86b-4f4e-84d8-966b5d89aae1/103d9428-b86b-4f4e-84d8-966b5d89aae1.zarr/ --depth=10
2024-09-10 16:08:39,375 272        [      main] INFO          ome.formats.importer.ImportConfig - OMERO.blitz Version: 5.7.1
...
2024-09-10 16:12:29,277 205352     [      main] INFO      ome.formats.importer.ImportCandidates - 6082 file(s) parsed into 1 group(s) with 1 call(s) to setId in 198510ms. (204557ms total) [0 unknowns]
#======================================
# Group: /bia-integrator-data/S-BIAD867/103d9428-b86b-4f4e-84d8-966b5d89aae1/103d9428-b86b-4f4e-84d8-966b5d89aae1.zarr/A/10/0/0/0/0/0/0 SPW: false Reader: loci.formats.in.ZarrReader
...
/bia-integrator-data/S-BIAD867/103d9428-b86b-4f4e-84d8-966b5d89aae1/103d9428-b86b-4f4e-84d8-966b5d89aae1.zarr/H/9/0/3/1/8/0/0
/bia-integrator-data/S-BIAD867/103d9428-b86b-4f4e-84d8-966b5d89aae1/103d9428-b86b-4f4e-84d8-966b5d89aae1.zarr/H/9/0/3/1/9/0/0

I don't know what's incorrect with your plate.

will-moore · 2024-10-04T15:18:58Z

on pilot-idrngff as omero-server

$ cd
$ /opt/omero/server/OMERO.server/bin/omero import -f  --debug=DEBUG --depth=100 JL_120731_S6A.ome.zarr --file /tmp/idr0001_20241004.log  --errs /tmp/idr0001_20241004.err

ends with:

2024-10-04 15:12:24,129 854        [      main] INFO      ome.formats.importer.ImportCandidates - 4234 file(s) parsed into 0 group(s) with 0 call(s) to setId in 0ms. (90ms total) [0 unknowns]

/tmp/idr0001_20241004.err

2024-10-04 15:12:23,571 296        [      main] INFO          ome.formats.importer.ImportConfig - OMERO.blitz Version: 5.7.1
2024-10-04 15:12:23,591 316        [      main] INFO          ome.formats.importer.ImportConfig - Bioformats version: 8.0.0-SNAPSHOT revision: 2acec68a47dcea9e6ff24aac1f24acfb4ea0d5f2 date: 25 April 2024
2024-10-04 15:12:23,663 388        [      main] INFO   formats.importer.cli.CommandLineImporter - Log levels -- Bio-Formats: DEBUG OMERO.importer: DEBUG
2024-10-04 15:12:23,688 413        [      main] DEBUG         ome.formats.importer.ImportConfig - Loaded savedDirectory from ome.formats.importer.util.IniFileLoader@bcec361
2024-10-04 15:12:23,689 414        [      main] DEBUG         ome.formats.importer.ImportConfig - Loaded savedDirectory from default
2024-10-04 15:12:23,689 414        [      main] DEBUG         ome.formats.importer.ImportConfig - Replacing port load value with default
2024-10-04 15:12:23,701 426        [      main] DEBUG                    loci.formats.ClassList - Could not find loci.formats.in.URLReader
java.lang.ClassNotFoundException: loci.formats.in.URLReader
        at java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:581)
        at java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:178)
        at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:527)
        at java.base/java.lang.Class.forName0(Native Method)
        at java.base/java.lang.Class.forName(Class.java:315)
        at loci.formats.ClassList.parseLine(ClassList.java:196)
        at loci.formats.ClassList.parseFile(ClassList.java:258)
        at loci.formats.ClassList.<init>(ClassList.java:138)
        at loci.formats.ClassList.<init>(ClassList.java:122)
        at loci.formats.ImageReader.getDefaultReaderClasses(ImageReader.java:82)
        at ome.formats.importer.OMEROWrapper.createReader(OMEROWrapper.java:141)
        at ome.formats.importer.OMEROWrapper.<init>(OMEROWrapper.java:103)
        at ome.formats.importer.OMEROWrapper.<init>(OMEROWrapper.java:95)
        at ome.formats.importer.OMEROWrapper.<init>(OMEROWrapper.java:90)
        at ome.formats.importer.cli.CommandLineImporter.<init>(CommandLineImporter.java:142)
        at ome.formats.importer.cli.CommandLineImporter.main(CommandLineImporter.java:1021)
2024-10-04 15:12:23,706 431        [      main] DEBUG                    loci.formats.ClassList - Could not find loci.formats.in.SlideBook6Reader
java.lang.ClassNotFoundException: loci.formats.in.SlideBook6Reader
        at java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:581)
        at java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:178)
        at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:527)
        at java.base/java.lang.Class.forName0(Native Method)
        at java.base/java.lang.Class.forName(Class.java:315)
        at loci.formats.ClassList.parseLine(ClassList.java:196)
        at loci.formats.ClassList.parseFile(ClassList.java:258)
        at loci.formats.ClassList.<init>(ClassList.java:138)
        at loci.formats.ClassList.<init>(ClassList.java:122)
        at loci.formats.ImageReader.getDefaultReaderClasses(ImageReader.java:82)
        at ome.formats.importer.OMEROWrapper.createReader(OMEROWrapper.java:141)
        at ome.formats.importer.OMEROWrapper.<init>(OMEROWrapper.java:103)
        at ome.formats.importer.OMEROWrapper.<init>(OMEROWrapper.java:95)
        at ome.formats.importer.OMEROWrapper.<init>(OMEROWrapper.java:90)
        at ome.formats.importer.cli.CommandLineImporter.<init>(CommandLineImporter.java:142)
        at ome.formats.importer.cli.CommandLineImporter.main(CommandLineImporter.java:1021)
	2024-10-04 15:12:23,708 433        [      main] DEBUG                    loci.formats.ClassList - Could not find loci.formats.in.ScreenReader
java.lang.ClassNotFoundException: loci.formats.in.ScreenReader
        at java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:581)
        at java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:178)
        at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:527)
        at java.base/java.lang.Class.forName0(Native Method)
        at java.base/java.lang.Class.forName(Class.java:315)
        at loci.formats.ClassList.parseLine(ClassList.java:196)
        at loci.formats.ClassList.parseFile(ClassList.java:258)
        at loci.formats.ClassList.<init>(ClassList.java:138)
        at loci.formats.ClassList.<init>(ClassList.java:122)
        at loci.formats.ImageReader.getDefaultReaderClasses(ImageReader.java:82)
        at ome.formats.importer.OMEROWrapper.createReader(OMEROWrapper.java:141)
        at ome.formats.importer.OMEROWrapper.<init>(OMEROWrapper.java:103)
        at ome.formats.importer.OMEROWrapper.<init>(OMEROWrapper.java:95)
        at ome.formats.importer.OMEROWrapper.<init>(OMEROWrapper.java:90)
        at ome.formats.importer.cli.CommandLineImporter.<init>(CommandLineImporter.java:142)
        at ome.formats.importer.cli.CommandLineImporter.main(CommandLineImporter.java:1021)
2024-10-04 15:12:23,712 437        [      main] DEBUG                    loci.formats.ClassList - Could not find ch.epfl.biop.formats.in.ZeissQuickStartCZIReader
java.lang.ClassNotFoundException: ch.epfl.biop.formats.in.ZeissQuickStartCZIReader
        at java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:581)
        at java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:178)
        at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:527)
        at java.base/java.lang.Class.forName0(Native Method)
        at java.base/java.lang.Class.forName(Class.java:315)
        at loci.formats.ClassList.parseLine(ClassList.java:196)
        at loci.formats.ClassList.parseFile(ClassList.java:258)
        at loci.formats.ClassList.<init>(ClassList.java:138)
        at loci.formats.ClassList.<init>(ClassList.java:122)
        at loci.formats.ImageReader.getDefaultReaderClasses(ImageReader.java:82)
        at ome.formats.importer.OMEROWrapper.createReader(OMEROWrapper.java:141)
        at ome.formats.importer.OMEROWrapper.<init>(OMEROWrapper.java:103)
        at ome.formats.importer.OMEROWrapper.<init>(OMEROWrapper.java:95)
        at ome.formats.importer.OMEROWrapper.<init>(OMEROWrapper.java:90)
        at ome.formats.importer.cli.CommandLineImporter.<init>(CommandLineImporter.java:142)
        at ome.formats.importer.cli.CommandLineImporter.main(CommandLineImporter.java:1021)
2024-10-04 15:12:24,015 740        [      main] DEBUG   org.scijava.nativelib.NativeLibraryUtil - processor is INTEL_64 os.arch is amd64
2024-10-04 15:12:24,015 740        [      main] DEBUG   org.scijava.nativelib.NativeLibraryUtil - architecture is LINUX_64 os.name is linux
2024-10-04 15:12:24,016 741        [      main] DEBUG   org.scijava.nativelib.NativeLibraryUtil - architecture is LINUX_64 os.name is linux
2024-10-04 15:12:24,016 741        [      main] DEBUG   org.scijava.nativelib.NativeLibraryUtil - platform specific path is natives/linux_64/
2024-10-04 15:12:24,016 741        [      main] DEBUG    org.scijava.nativelib.BaseJniExtractor - mappedLib is libturbojpeg.so
2024-10-04 15:12:24,016 741        [      main] DEBUG    org.scijava.nativelib.BaseJniExtractor - Couldn't find resource natives/linux_64/libturbojpeg.so
2024-10-04 15:12:24,016 741        [      main] DEBUG   org.scijava.nativelib.NativeLibraryUtil - platform specific path is linux_64/
2024-10-04 15:12:24,016 741        [      main] DEBUG    org.scijava.nativelib.BaseJniExtractor - mappedLib is libturbojpeg.so
2024-10-04 15:12:24,017 742        [      main] DEBUG    org.scijava.nativelib.BaseJniExtractor - Couldn't find resource linux_64/libturbojpeg.so
2024-10-04 15:12:24,017 742        [      main] DEBUG   org.scijava.nativelib.NativeLibraryUtil - platform specific path is META-INF/lib/linux_64/
2024-10-04 15:12:24,017 742        [      main] DEBUG    org.scijava.nativelib.BaseJniExtractor - mappedLib is libturbojpeg.so
2024-10-04 15:12:24,017 742        [      main] DEBUG    org.scijava.nativelib.BaseJniExtractor - URL is jar:file:/opt/omero/server/OMERO.server-5.6.9-ice36/lib/client/turbojpeg-8.0.0-SNAPSHOT.jar!/META-INF/lib/linux_64/libturbojpeg.so
2024-10-04 15:12:24,017 742        [      main] DEBUG    org.scijava.nativelib.BaseJniExtractor - URL path is file:/opt/omero/server/OMERO.server-5.6.9-ice36/lib/client/turbojpeg-8.0.0-SNAPSHOT.jar!/META-INF/lib/linux_64/libturbojpeg.so
2024-10-04 15:12:24,017 742        [      main] DEBUG    org.scijava.nativelib.BaseJniExtractor - Extracting 'jar:file:/opt/omero/server/OMERO.server-5.6.9-ice36/lib/client/turbojpeg-8.0.0-SNAPSHOT.jar!/META-INF/lib/linux_64/libturbojpeg.so' to '/tmp/nativelib-loader_599140003118702579/libturbojpeg.so'
2024-10-04 15:12:24,038 763        [      main] INFO      ome.formats.importer.ImportCandidates - Depth: 100 Metadata Level: MINIMUM
2024-10-04 15:12:24,087 812        [      main] DEBUG     ome.formats.importer.cli.ErrorHandler - SCANNING: Depth:0 Num: 4234 Tot:  n/a File: 731_S6A.ome.zarr
2024-10-04 15:12:24,128 853        [      main] DEBUG     ome.formats.importer.cli.ErrorHandler - SCANNING: Depth:0 Num: 4234 Tot: 4234 File: 731_S6A.ome.zarr
2024-10-04 15:12:24,129 854        [      main] INFO      ome.formats.importer.ImportCandidates - 4234 file(s) parsed into 0 group(s) with 0 call(s) to setId in 0ms. (90ms total) [0 unknowns]

sbesson · 2024-10-08T07:15:00Z

I am unable to reproduce #683 (comment). Using the same server and the converted data mounted as described in #683 (comment), I have

[sbesson@pilot-idrngff-omeroreadwrite ~]$ /opt/omero/server/OMERO.server/bin/omero import -f /idr0001/zarr/JL_120731_S6A.ome.zarr/ --depth=10 --file idr0001.out --errs idr0001.err
[sbesson@pilot-idrngff-omeroreadwrite ~]$ cat idr0001.err 
2024-10-07 18:32:16,100 298        [      main] INFO          ome.formats.importer.ImportConfig - OMERO.blitz Version: 5.7.1
2024-10-07 18:32:16,121 319        [      main] INFO          ome.formats.importer.ImportConfig - Bioformats version: 8.0.0-SNAPSHOT revision: 2acec68a47dcea9e6ff24aac1f24acfb4ea0d5f2 date: 25 April 2024
2024-10-07 18:32:16,195 393        [      main] INFO   formats.importer.cli.CommandLineImporter - Log levels -- Bio-Formats: ERROR OMERO.importer: INFO
2024-10-07 18:32:16,589 787        [      main] INFO      ome.formats.importer.ImportCandidates - Depth: 10 Metadata Level: MINIMUM
2024-10-07 23:47:14,271 18898469   [      main] INFO      ome.formats.importer.ImportCandidates - 265165 file(s) parsed into 1 group(s) with 1 call(s) to setId in 1025839ms. (18897678ms total) [0 unknowns]
[sbesson@pilot-idrngff-omeroreadwrite ~]$ head idr0001.out 
#======================================
# Group: /idr0001/zarr/JL_120731_S6A.ome.zarr/A/1/0/0/0/0/0/0/0 SPW: false Reader: loci.formats.in.ZarrReader
/idr0001/zarr/JL_120731_S6A.ome.zarr/A/1/0/0/0/0/0/0/0
/idr0001/zarr/JL_120731_S6A.ome.zarr/.zattrs
/idr0001/zarr/JL_120731_S6A.ome.zarr/.zgroup
/idr0001/zarr/JL_120731_S6A.ome.zarr/A/.zgroup
/idr0001/zarr/JL_120731_S6A.ome.zarr/A/1/.zattrs
/idr0001/zarr/JL_120731_S6A.ome.zarr/A/1/.zgroup
/idr0001/zarr/JL_120731_S6A.ome.zarr/A/1/0/.zattrs
/idr0001/zarr/JL_120731_S6A.ome.zarr/A/1/0/.zgroup

will-moore added this to NGFF conversion Jan 8, 2024

will-moore moved this to convert all data to NGFF in NGFF conversion Jan 8, 2024

dgault mentioned this issue Jan 11, 2024

Update handling and parsing of acquisition metadata ome/ZarrReader#75

Merged

will-moore mentioned this issue Sep 6, 2024

Multiple PlateAcquisitions: Images in wrong Wells ome/ZarrReader#96

Open

idr0001-graml-sysgro to NGFF #683

idr0001-graml-sysgro to NGFF #683

Comments

will-moore commented Jan 8, 2024 • edited Loading

will-moore commented Jan 8, 2024 • edited Loading

will-moore commented Jan 8, 2024 • edited Loading

will-moore commented Jan 8, 2024 • edited Loading

sbesson commented Jan 9, 2024

will-moore commented Jan 9, 2024 • edited Loading

will-moore commented Jan 10, 2024

will-moore commented Jan 10, 2024 • edited Loading

will-moore commented Jan 10, 2024

sbesson commented Jan 10, 2024

will-moore commented Jan 10, 2024

will-moore commented Jan 10, 2024

will-moore commented Jan 11, 2024 • edited Loading

will-moore commented Jan 11, 2024 • edited Loading

will-moore commented Jan 11, 2024 • edited Loading

will-moore commented Jan 11, 2024

will-moore commented Jan 11, 2024

will-moore commented Jan 12, 2024 • edited Loading

will-moore commented Jan 12, 2024 • edited Loading

will-moore commented Jan 12, 2024

will-moore commented Jun 24, 2024

5 months later

will-moore commented Jul 3, 2024

will-moore commented Jul 7, 2024

sbesson commented Jul 7, 2024

joshmoore commented Jul 8, 2024

will-moore commented Aug 19, 2024 • edited Loading

will-moore commented Aug 29, 2024 • edited Loading

will-moore commented Aug 29, 2024 • edited Loading

will-moore commented Aug 30, 2024

will-moore commented Sep 2, 2024

will-moore commented Sep 2, 2024

will-moore commented Sep 6, 2024 • edited Loading

will-moore commented Sep 6, 2024 • edited Loading

will-moore commented Sep 10, 2024 • edited Loading

will-moore commented Sep 10, 2024

sbesson commented Sep 10, 2024

will-moore commented Oct 4, 2024

sbesson commented Oct 8, 2024

will-moore commented Jan 8, 2024 •

edited

Loading

will-moore commented Jan 8, 2024 •

edited

Loading

will-moore commented Jan 8, 2024 •

edited

Loading

will-moore commented Jan 8, 2024 •

edited

Loading

will-moore commented Jan 9, 2024 •

edited

Loading

will-moore commented Jan 10, 2024 •

edited

Loading

will-moore commented Jan 11, 2024 •

edited

Loading

will-moore commented Jan 11, 2024 •

edited

Loading

will-moore commented Jan 11, 2024 •

edited

Loading

will-moore commented Jan 12, 2024 •

edited

Loading

will-moore commented Jan 12, 2024 •

edited

Loading

will-moore commented Aug 19, 2024 •

edited

Loading

will-moore commented Aug 29, 2024 •

edited

Loading

will-moore commented Aug 29, 2024 •

edited

Loading

will-moore commented Sep 6, 2024 •

edited

Loading

will-moore commented Sep 6, 2024 •

edited

Loading

will-moore commented Sep 10, 2024 •

edited

Loading