Testing mkngff on idr-testing #672

will-moore · 2023-09-06T15:49:53Z

will-moore · 2023-09-06T16:05:58Z

ssh idr-testing.openmicroscopy.org -At ssh omeroreadwrite -A

# also run this in another terminal:
ssh idr-testing.openmicroscopy.org -A -L 1080:omeroreadwrite:80

sudo mkdir /bia-integrator-data && sudo /opt/goofys --endpoint https://uk1s3.embassy.ebi.ac.uk/ -o allow_other bia-integrator-data /bia-integrator-data

Install mkngff latest PR IDR/omero-mkngff#8 into /opt/omero/server/venv3..

sudo -u root -s
source /opt/omero/server/venv3/bin/activate

pip install 'omero-mkngff @ git+https://github.com/will-moore/omero-mkngff@always_update_pixels'
... Resolved https://github.com/will-moore/omero-mkngff to commit 231431164e3692298864fcc52fb3f0c663c8f595
Successfully installed omero-mkngff-0.1.0.dev0

will-moore · 2023-09-06T16:38:33Z

Going to test idr0051 (3 images), following IDR/omero-mkngff#2 workflow...

idr0051.csv

idr0054/Tonsil_1.ome.zarr,S-BIAD800/a0e1b9c0-5c07-4600-b114-7b4697900e39,1591301
idr0054/Tonsil_2.ome.zarr,S-BIAD800/e24d29ea-0e39-4368-a2e4-5e6320c91573,1591302
idr0054/Tonsil_3.ome.zarr,S-BIAD800/f49ada41-43bf-47ff-99b9-bdf8cc311ce3,5025553

$ psql -U omero -d idr -h $DBHOST -c "select uuid from (select * from session where node = 0 and owner = 0 and defaulteventtype = 'Sessions' order by id desc limit 1) x order by x.id asc limit 1;"

                 uuid
--------------------------------------
 01b61552-49fc-4543-bf40-...
(1 row)

$ psql -U omero -d idr -h $DBHOST -c "select uuid from (select * from session where node = 0 and owner = 0 and defaulteventtype = 'Sessions' order by id desc limit 3) x order by x.id asc limit 3;"
                 uuid                 
--------------------------------------
 9dee647a-d1ed-483b-8e67-...
 7b025064-588b-4d17-8eb4-...
 01b61552-49fc-4543-bf40-...
(3 rows)

for r in $(cat $IDRID.csv); do
>   biapath=$(echo $r | cut -d',' -f2)
>   uuid=$(echo $biapath | cut -d'/' -f2)
>   fsid=$(echo $r | cut -d',' -f3)
>   omero mkngff sql --symlink_repo /data/OMERO/ManagedRepository --secret=$SECRET $fsid "/bia-integrator-data/$biapath/$uuid.zarr" > "$fsid.sql"
> done
Using session for demo@localhost:4064. Idle timeout: 10 min. Current group: Public
Found prefix demo_2/Blitz-0-Ice.ThreadPool.Server-5/2019-03/15 // 15-27-40.023 for fileset 1591301
Checking for prefix_dir /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-5/2019-03/15/15-27-40.023
Creating dir at /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-5/2019-03/15/15-27-40.023_mkngff
Creating symlink /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-5/2019-03/15/15-27-40.023_mkngff/a0e1b9c0-5c07-4600-b114-7b4697900e39.zarr -> /bia-integrator-data/S-BIAD800/a0e1b9c0-5c07-4600-b114-7b4697900e39/a0e1b9c0-5c07-4600-b114-7b4697900e39.zarr
Using session for demo@localhost:4064. Idle timeout: 10 min. Current group: Public
Found prefix demo_2/Blitz-0-Ice.ThreadPool.Server-10/2019-03/15 // 15-28-44.081 for fileset 1591302
Checking for prefix_dir /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-10/2019-03/15/15-28-44.081
Creating dir at /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-10/2019-03/15/15-28-44.081_mkngff
Creating symlink /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-10/2019-03/15/15-28-44.081_mkngff/e24d29ea-0e39-4368-a2e4-5e6320c91573.zarr -> /bia-integrator-data/S-BIAD800/e24d29ea-0e39-4368-a2e4-5e6320c91573/e24d29ea-0e39-4368-a2e4-5e6320c91573.zarr
Using session for demo@localhost:4064. Idle timeout: 10 min. Current group: Public
Found prefix demo_2/Blitz-0-Ice.ThreadPool.Server-13/2021-06/07 // 23-05-45.568 for fileset 5025553
Checking for prefix_dir /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-13/2021-06/07/23-05-45.568
Creating dir at /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-13/2021-06/07/23-05-45.568_mkngff
Creating symlink /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-13/2021-06/07/23-05-45.568_mkngff/f49ada41-43bf-47ff-99b9-bdf8cc311ce3.zarr -> /bia-integrator-data/S-BIAD800/f49ada41-43bf-47ff-99b9-bdf8cc311ce3/f49ada41-43bf-47ff-99b9-bdf8cc311ce3.zarr

UPDATE 1
BEGIN
 mkngff_fileset 
----------------
        6311582
(1 row)
COMMIT
UPDATE 1
BEGIN
 mkngff_fileset 
----------------
        6311583
(1 row)
COMMIT
UPDATE 1
BEGIN
 mkngff_fileset 
----------------
        6311584
(1 row)
COMMIT

will-moore · 2023-09-06T16:42:51Z

Image above is not viewable...
https://idr-testing.openmicroscopy.org/webclient/imgData/5025551/
returns

"Exception": "Error instantiating pixel buffer: /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-5/2019-03/15/15-27-40.023_mkngff/a0e1b9c0-5c07-4600-b114-7b4697900e39.zarr/OME/METADATA.ome.xml"

sbesson · 2023-09-06T16:49:47Z

Was the bucket mounted both on the read-write VM and the read-only VMs (which are the ones accessed by the idr-testing endpoint)?

will-moore · 2023-09-06T16:52:15Z

Update ZarrReader to include all current PRs, as described... IDR/omero-mkngff#2 (comment)

wget https://merge-ci.openmicroscopy.org/jenkins/job/BIOFORMATS-build/lastSuccessfulBuild/default/artifact/bio-formats-build/ZarrReader/target/OMEZarrReader-0.3.2-SNAPSHOT-jar-with-dependencies.jar
...

will-moore · 2023-09-06T16:53:21Z

@sbesson no, I'll try... - I also wondered if the ZarrReader updates need to be applied across all VMs?

sbesson · 2023-09-06T16:56:37Z

@sbesson no, I'll try... - I also wondered if the ZarrReader updates need to be applied across all VMs?

If you are departing from the release 0.3.1 artifact was deployed as part of IDR/deployment#380, the same JAR will definitely need to be copied under lib/server on all VMs and the omero-server service restarted accordingly

will-moore · 2023-09-06T17:05:36Z

Mount bia-integrator-data on all the readonly servers...

ssh -A idr-testing.openmicroscopy.org

for server in omeroreadonly-1 omeroreadonly-2 omeroreadonly-3 omeroreadonly-4; do ssh $server "sudo mkdir /bia-integrator-data && sudo /opt/goofys --endpoint https://uk1s3.embassy.ebi.ac.uk/ -o allow_other bia-integrator-data /bia-integrator-data"; done

@sbesson What about the creation of symlinks by mkngff? Does that need to happen on multiple servers?

sbesson · 2023-09-07T09:04:11Z

That should work out of the box as the OMERO binary repository should be shared between the read-write VM and all read-only VMs.

will-moore · 2023-09-07T09:39:05Z

I updated ZarrReader on omeroreadonly-1 omeroreadonly-2 omeroreadonly-3 omeroreadonly-4 as at IDR/omero-mkngff#2 (comment):

wget https://merge-ci.openmicroscopy.org/jenkins/job/BIOFORMATS-build/lastSuccessfulBuild/default/artifact/bio-formats-build/ZarrReader/target/OMEZarrReader-0.3.2-SNAPSHOT-jar-with-dependencies.jar

But I'm still seeingError instantiating pixel buffer at https://idr-testing.openmicroscopy.org/webclient/imgData/5025551/

I'was trying to check the Blitz.log on omeroreadwrite at the same time as refreshing that URL, but I don't see anything because I guess this is being handled by a different server. I guess I have to try them in turn...

will-moore · 2023-09-07T09:48:26Z

On omeroreadonly-2 I saw this $ less /opt/omero/server/OMERO.server/var/log/Blitz-0.log...

2023-09-07 09:44:29,669 INFO  [         ome.security.basic.EventHandler] (l.Server-4)  Auth:    user=52,group=3,event=null(User),sess=bb36fb91-d187-4988-8d0e-ef595314e803
2023-09-07 09:44:29,670 INFO  [      ome.services.OmeroFilePathResolver] (l.Server-4) Metadata only file, resulting path: /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-5/2019-03/15/15-27-40.023_mkngff/a0e1b9c0-5c07-4600-b114-7b4697900e39.zarr/OME/METADATA.
ome.xml
2023-09-07 09:44:30,259 INFO  [                loci.formats.ImageReader] (l.Server-4) ZarrReader initializing /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-5/2019-03/15/15-27-40.023_mkngff/a0e1b9c0-5c07-4600-b114-7b4697900e39.zarr/OME/METADATA.ome.xml
2023-09-07 09:44:33,844 WARN  [              loci.formats.FormatHandler] (l.Server-4) Mismatch with group key paths and original OME-XML metadata, original ordering wont be maintained
2023-09-07 09:44:34,734 DEBUG [                   loci.formats.Memoizer] (l.Server-4) start[1694079869675] time[5059] tag[loci.formats.Memoizer.setId]
2023-09-07 09:44:34,735 ERROR [         ome.io.bioformats.BfPixelBuffer] (l.Server-4) Failed to instantiate BfPixelsWrapper with /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-5/2019-03/15/15-27-40.023_mkngff/a0e1b9c0-5c07-4600-b114-7b4697900e39.zarr/OME/ME
TADATA.ome.xml
2023-09-07 09:44:34,735 ERROR [                ome.io.nio.PixelsService] (l.Server-4) Error instantiating pixel buffer: /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-5/2019-03/15/15-27-40.023_mkngff/a0e1b9c0-5c07-4600-b114-7b4697900e39.zarr/OME/METADATA.om
e.xml
java.lang.RuntimeException: java.lang.ClassCastException: class com.esotericsoftware.kryo.Kryo$DefaultInstantiatorStrategy cannot be cast to class com.esotericsoftware.kryo.util.DefaultInstantiatorStrategy (com.esotericsoftware.kryo.Kryo$DefaultInstantiatorStrategy and com.eso
tericsoftware.kryo.util.DefaultInstantiatorStrategy are in unnamed module of loader 'app')
        at ome.io.bioformats.BfPixelBuffer.reader(BfPixelBuffer.java:79)
        at ome.io.bioformats.BfPixelBuffer.setSeries(BfPixelBuffer.java:124)
        at ome.io.nio.PixelsService.createBfPixelBuffer(PixelsService.java:898)
        at ome.io.nio.PixelsService._getPixelBuffer(PixelsService.java:653)
        at ome.io.nio.PixelsService.getPixelBuffer(PixelsService.java:571)
        at ome.services.RenderingBean$12.doWork(RenderingBean.java:2205)
        at jdk.internal.reflect.GeneratedMethodAccessor224.invoke(Unknown Source)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.base/java.lang.reflect.Method.invoke(Method.java:566)
...
	at IceInternal.ThreadPool.run(ThreadPool.java:395)
	at IceInternal.ThreadPool.access$300(ThreadPool.java:12)
	at IceInternal.ThreadPool$EventHandlerThread.run(ThreadPool.java:832)
	at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: java.lang.ClassCastException: class com.esotericsoftware.kryo.Kryo$DefaultInstantiatorStrategy cannot be cast to class com.esotericsoftware.kryo.util.DefaultInstantiatorStrategy (com.esotericsoftware.kryo.Kryo$DefaultInstantiatorStrategy and com.esotericsoftware.kryo.util.DefaultInstantiatorStrategy are in unnamed module of loader 'app')
	at loci.formats.Memoizer$KryoDeser.<init>(Memoizer.java:125)
	at loci.formats.Memoizer.getDeser(Memoizer.java:789)
	at loci.formats.Memoizer.saveMemo(Memoizer.java:990)
	at loci.formats.Memoizer.setId(Memoizer.java:733)
	at ome.io.bioformats.BfPixelsWrapper.<init>(BfPixelsWrapper.java:52)
	at ome.io.bioformats.BfPixelBuffer.reader(BfPixelBuffer.java:73)
	... 82 common frames omitted
2023-09-07 09:43:18,832 INFO  [                 org.perf4j.TimingLogger] (l.Server-0) start[1694079795148] time[3684] tag[omero.call.exception]

will-moore · 2023-09-07T10:17:29Z

David (slack) "Kryo will be related to the memoizer, I haven’t seen that particular issue though, perhaps a version mismatch with the dependency?
if it is also using the latest merge of Bio-Formats then that may be using different versions"

sbesson · 2023-09-07T13:07:17Z

One thing worth investigating is that the latest OMERO.server release, 5.6.8, was deployed on test120 as opposed to OMERO 5.6.7 on pilot-idr0125. In terms of underlying library changes, Bio-Formats was upgraded from 6.12.0 to 6.14.0 - see https://www.openmicroscopy.org/2023/07/25/omero-5-6-8.html which in turns included a large update of low-level dependencies and notably kryo - see https://www.openmicroscopy.org/2023/05/02/bio-formats-6-13-0.html.

Would it be worth to revert test120 to OMERO.server 5.6.7 to confirm whether the issue is related to this dependency upgrade ?

will-moore · 2023-09-07T13:58:11Z

@sbesson yes, if that's the easiest option to get this working, and we don't need the 5.6.8, that would be great

will-moore · 2023-09-07T15:14:43Z

Thanks @sbesson that's working now. After updating ZarrReader again on the 4 servers I can see NGFF images.

NB: I noticed a couple of warnings when restarting servers on omeroreadonly-3 and omeroreadonly-4. Not sure what that means?

[wmoore@test120-omeroreadonly-3 ~]$ sudo service omero-server restart
Redirecting to /bin/systemctl restart omero-server.service
Message from syslogd@localhost at Sep  7 15:02:01 ...
 haproxy[18172]: backend omero4064-1 has no server available!

sbesson · 2023-09-07T16:08:36Z

That's the HAproxy application which is balancing the load between read-only servers and responding to the service getting restarted. If you look at /var/log/messages under the proxy server, you will see that it the connection is recovered eventually

Sep  7 15:02:01 localhost haproxy[18172]: Server omero4064-1/omero4064-192_168_4_90 is DOWN, reason: Layer4 connection problem, info: "Connection refused", check duration: 0ms. 0 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
Sep  7 15:02:01 localhost haproxy[18172]: backend omero4064-1 has no server available!
Sep  7 15:02:02 localhost haproxy[18172]: Server omero4064/omero4064-192_168_4_90 is DOWN, reason: Layer4 connection problem, info: "Connection refused", check duration: 0ms. 1 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
Sep  7 15:02:17 localhost haproxy[18172]: Server omero4064-1/omero4064-192_168_4_90 is UP, reason: Layer4 check passed, check duration: 0ms. 1 active and 0 backup servers online. 0 sessions requeued, 0 total in queue.
Sep  7 15:02:18 localhost haproxy[18172]: Server omero4064/omero4064-192_168_4_90 is UP, reason: Layer4 check passed, check duration: 0ms. 2 active and 0 backup servers online. 0 sessions requeued, 0 total in queue.
Sep  7 15:03:09 localhost haproxy[18172]: Server omero4064/omero4064-192_168_4_68 is DOWN, reason: Layer4 connection problem, info: "Connection refused", check duration: 0ms. 1 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
Sep  7 15:03:10 localhost haproxy[18172]: Server omero4064-0/omero4064-192_168_4_68 is DOWN, reason: Layer4 connection problem, info: "Connection refused", check duration: 0ms. 0 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
Sep  7 15:03:10 localhost haproxy[18172]: backend omero4064-0 has no server available!
Sep  7 15:03:26 localhost haproxy[18172]: Server omero4064-0/omero4064-192_168_4_68 is UP, reason: Layer4 check passed, check duration: 0ms. 1 active and 0 backup servers online. 0 sessions requeued, 0 total in queue.
Sep  7 15:03:27 localhost haproxy[18172]: Server omero4064/omero4064-192_168_4_68 is UP, reason: Layer4 check passed, check duration: 0ms. 2 active and 0 backup servers online. 0 sessions requeued, 0 total in que

will-moore · 2023-09-08T17:53:14Z

Going to test a plate on idr-testing...
idr0125...

  idr0025/10x images plate 3.ome.zarr,S-BIAD846/3c534b4f-12be-4881-a84a-af6b65e142ea,23152
  idr0025/10x images plate 1.ome.zarr,S-BIAD846/52304cdf-4eba-4f0a-84b1-690e0d66add9,23151
  idr0025/10x images plate 2.ome.zarr,S-BIAD846/72cc291b-a4e0-4807-bd23-22e9ad75c0dd,23201

oops - forgot to remove whitespace from the csv above, but it still seems to find the right strings eventually...

$ for r in $(cat $IDRID.csv); do
>   biapath=$(echo $r | cut -d',' -f2)
>   uuid=$(echo $biapath | cut -d'/' -f2)
>   fsid=$(echo $r | cut -d',' -f3)
>   omero mkngff sql --symlink_repo /data/OMERO/ManagedRepository --secret=$SECRET $fsid "/bia-integrator-data/$biapath/$uuid.zarr" > "$fsid.sql"
> done
bash: idr0025/10x.sql: No such file or directory
usage: /opt/omero/server/venv3/bin/omero mkngff sql [-h] [--secret SECRET]
                                                    [--zarr_name ZARR_NAME]
                                                    [--symlink_repo SYMLINK_REPO]
                                                    fileset_id symlink_target
/opt/omero/server/venv3/bin/omero mkngff sql: error: argument fileset_id: invalid int value: 'images'
usage: /opt/omero/server/venv3/bin/omero mkngff sql [-h] [--secret SECRET]
                                                    [--zarr_name ZARR_NAME]
                                                    [--symlink_repo SYMLINK_REPO]
                                                    fileset_id symlink_target
/opt/omero/server/venv3/bin/omero mkngff sql: error: argument fileset_id: invalid int value: 'plate'
Using session for demo@localhost:4064. Idle timeout: 10 min. Current group: Public
Found prefix demo_2/2017-03/13 // 15-26-46.818 for fileset 23152
Checking for prefix_dir /data/OMERO/ManagedRepository/demo_2/2017-03/13/15-26-46.818
Creating dir at /data/OMERO/ManagedRepository/demo_2/2017-03/13/15-26-46.818_mkngff
Creating symlink /data/OMERO/ManagedRepository/demo_2/2017-03/13/15-26-46.818_mkngff/3c534b4f-12be-4881-a84a-af6b65e142ea.zarr -> /bia-integrator-data/S-BIAD846/3c534b4f-12be-4881-a84a-af6b65e142ea/3c534b4f-12be-4881-a84a-af6b65e142ea.zarr
...

will-moore · 2023-09-11T10:43:47Z

$ for r in $(cat $IDRID.csv); do
>   fsid=$(echo $r | cut -d',' -f3)
>   psql -U omero -d idr -h $DBHOST -f "$fsid.sql"
> done
psql: error: idr0025/10x.sql: No such file or directory
UPDATE 384
BEGIN
 mkngff_fileset 
----------------
        6311585
(1 row)

COMMIT
psql: error: idr0025/10x.sql: No such file or directory
UPDATE 384
BEGIN
 mkngff_fileset 
----------------
        6311586
(1 row)

COMMIT
psql: error: idr0025/10x.sql: No such file or directory
UPDATE 384
BEGIN
 mkngff_fileset 
----------------
        6311587
(1 row)

COMMIT

Viewed first plate image (to start memo file generation - 11:42)...

will-moore · 2023-09-11T12:51:26Z

ON http://localhost:1080/webclient/ got this while waiting on memo file....

DatabaseBusyException
 message = cannot create transaction

https://idr-testing.openmicroscopy.org/webclient/?show=image-5025551
now loads right panel but image doesn't display.

https://idr-testing.openmicroscopy.org/webclient/imgData/3260451/ takes time and fails with 504 Gateway Time-out.

# on readonly-1

$ grep -A 2 "saved memo" /opt/omero/server/OMERO.server/var/log/Blitz-0.log | grep -A 2 "mkngff"

2023-09-07 15:08:44,105 DEBUG [                   loci.formats.Memoizer] (l.Server-1) saved memo file: /data/OMERO/BioFormatsCache/data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-10/2019-03/15/15-28-44.081_mkngff/e24d29ea-0e39-4368-a2e4-5e6320c91573.zarr/OME/.METADATA.ome.xml.bfmemo (46996 bytes)
2023-09-07 15:08:44,105 DEBUG [                   loci.formats.Memoizer] (l.Server-1) start[1694099317085] time[7019] tag[loci.formats.Memoizer.setId]
2023-09-07 15:08:44,111 INFO  [                ome.io.nio.PixelsService] (l.Server-1) Creating BfPixelBuffer: /data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-10/2019-03/15/15-28-44.081_mkngff/e24d29ea-0e39-4368-a2e4-5e6320c91573.zarr/OME/METADATA.ome.xml Series: 0
--
2023-09-07 19:41:25,964 DEBUG [                   loci.formats.Memoizer] (l.Server-4) saved memo file: /data/OMERO/BioFormatsCache/data/OMERO/ManagedRepository/demo_2/Blitz-0-Ice.ThreadPool.Server-5/2023-01/30/17-28-12.832/.front.companion.ome.bfmemo (1616957 bytes)

restart all servers... and web servers...

ssh -A idr-testing.openmicroscopy.org

for server in omeroreadwrite omeroreadonly-1 omeroreadonly-2 omeroreadonly-3 omeroreadonly-4; do ssh $server "sudo service omero-server restart"; done

$ for server in omeroreadwrite omeroreadonly-1 omeroreadonly-2 omeroreadonly-3 omeroreadonly-4; do ssh $server "sudo service omero-web restart"; done

But still see Gateway Timeout at https://idr-testing.openmicroscopy.org/webclient/imgData/3260451/

will-moore · 2023-09-11T13:14:45Z

Going to update ZarrReader to hopefully improve performance...

for each of omeroreadwrite omeroreadonly-1 omeroreadonly-2 omeroreadonly-3 omeroreadonly-4...

ssh omeroreadwrite
sudo -u omero-server -s
cd
wget https://merge-ci.openmicroscopy.org/jenkins/job/BIOFORMATS-build/lastSuccessfulBuild/default/artifact/bio-formats-build/ZarrReader/target/OMEZarrReader-0.3.2-SNAPSHOT-jar-with-dependencies.jar
mv OMEZarrReader-0.3.2-SNAPSHOT-jar-with-dependencies.jar OMEZarrReader.jar
cp OMEZarrReader.jar /opt/omero/server/OMERO.server/lib/client/
cp OMEZarrReader.jar /opt/omero/server/OMERO.server/lib/server/

Then restarted all 5 servers in loop as above.

for server in omeroreadwrite omeroreadonly-1 omeroreadonly-2 omeroreadonly-3 omeroreadonly-4; do ssh $server "sudo service omero-server restart"; done

will-moore · 2023-09-11T13:40:43Z

Now try set-id on the 2nd Plate of idr0025 above (only tried Plate 1 so far)...
We want to specify which server to test with (instead of letting nginx choose with ip-hash)

ssh idr-testing.openmicroscopy.org -L 1080:omeroreadwrite:80

First checked Plate 1 image above http://localhost:1080/webclient/?show=image-3260451 and this was viewable!
Checked memo time, using prefix for this Fileset "590_mkngff"...

[wmoore@test120-omeroreadwrite ~]$ grep -A 2 "saved memo" /opt/omero/server/OMERO.server/var/log/Blitz-0.log | grep -A 2 "590_mkngff"
2023-09-11 13:21:00,608 DEBUG [                   loci.formats.Memoizer] (l.Server-9) saved memo file: /data/OMERO/BioFormatsCache/data/OMERO/ManagedRepository/demo_2/2017-03/13/15-19-51.590_mkngff/52304cdf-4eba-4f0a-84b1-690e0d66add9.zarr/OME/.METADATA.ome.xml.bfmemo (873699 bytes)
2023-09-11 13:21:00,608 DEBUG [                   loci.formats.Memoizer] (l.Server-9) start[1694438408707] time[51900] tag[loci.formats.Memoizer.setId]
2023-09-11 13:21:00,608 INFO  [                ome.io.nio.PixelsService] (l.Server-9) Creating BfPixelBuffer: /data/OMERO/ManagedRepository/demo_2/2017-03/13/15-19-51.590_mkngff/52304cdf-4eba-4f0a-84b1-690e0d66add9.zarr/OME/METADATA.ome.xml Series: 0

However - this isn't the original memo file regeneration, which I attempted to start this morning.
Couldn't see logs for that on any of the 5 servers.

Now tried viewing image on Plate 2...

will-moore · 2023-09-11T15:25:08Z

Image from Plate 2 of idr0025 is now "viewable" (Preview panel loads in webclient)...

[wmoore@test120-omeroreadwrite ~]$ grep -A 2 "saved memo" /opt/omero/server/OMERO.server/var/log/Blitz-0.log | grep -A 2 "747_mkngff"
2023-09-11 15:06:01,615 DEBUG [                   loci.formats.Memoizer] (l.Server-8) saved memo file: /data/OMERO/BioFormatsCache/data/OMERO/ManagedRepository/demo_2/2017-03/17/15-15-54.747_mkngff/72cc291b-a4e0-4807-bd23-22e9ad75c0dd.zarr/OME/.METADATA.ome.xml.bfmemo (873699 bytes)
2023-09-11 15:06:01,615 DEBUG [                   loci.formats.Memoizer] (l.Server-8) start[1694439010614] time[5751000] tag[loci.formats.Memoizer.setId]
2023-09-11 15:06:01,615 INFO  [                ome.io.nio.PixelsService] (l.Server-8) Creating BfPixelBuffer: /data/OMERO/ManagedRepository/demo_2/2017-03/17/15-15-54.747_mkngff/72cc291b-a4e0-4807-bd23-22e9ad75c0dd.zarr/OME/METADATA.ome.xml Series: 0

5751000 ms is 95 minutes.

Unfortunately the Images are rendering black:

https://idr-testing.openmicroscopy.org/webclient/?show=image-3261251

will-moore · 2023-09-12T11:32:44Z

Testing with today's merge OMEZarrReader.jar on the 3rd idr0025 Plate

$ grep -A 2 "saved memo" /opt/omero/server/OMERO.server/var/log/Blitz-0.log | grep -A 2 "46.818_mkngff"
2023-09-12 11:08:39,631 DEBUG [                   loci.formats.Memoizer] (l.Server-2) saved memo file: /data/OMERO/BioFormatsCache/data/OMERO/ManagedRepository/demo_2/2017-03/13/15-26-46.818_mkngff/3c534b4f-12be-4881-a84a-af6b65e142ea.zarr/OME/.METADATA.ome.xml.bfmemo (873699 bytes)
2023-09-12 11:08:39,631 DEBUG [                   loci.formats.Memoizer] (l.Server-2) start[1694516674666] time[244964] tag[loci.formats.Memoizer.setId]
2023-09-12 11:08:39,631 INFO  [                ome.io.nio.PixelsService] (l.Server-2) Creating BfPixelBuffer: /data/OMERO/ManagedRepository/demo_2/2017-03/13/15-26-46.818_mkngff/3c534b4f-12be-4881-a84a-af6b65e142ea.zarr/OME/METADATA.ome.xml Series: 0

244964 ms is 4 minutes

sbesson · 2023-10-02T16:16:36Z

As discussed during the weekly IDR calls, the following commands aim to measure the initialization time for several of the IDR datasets converted to NGFF in order to benchmark/assess the current performance of ZarrReader against the original reader (ScreenReader for many of these). These measurements use the Bio-Formats command-line utility downloaded from the OME website with the bioformats_package JAR

idr0016

[sbesson@prod118-omeroreadwrite ~]$ ./bftools/showinf -nopix /uod/idr/metadata/idr0016-wawer-bioactivecompoundprofiling/screens/20607.screen | grep Init
Initializing reader
Initialization took 1136.254s
[sbesson@prod118-omeroreadwrite ~]$ ./bftools/showinf -nopix /uod/idr/metadata/idr0016-wawer-bioactivecompoundprofiling/screens/20607.screen | grep Init
Initializing reader
Initialization took 130.914s

idr0033

[sbesson@prod118-omeroreadwrite ~]$ ./bftools/showinf -nopix /uod/idr/metadata/idr0033-rohban-pathways/screens/41744.screen  | grep Init
 Initializing reader
 Initialization took 1145.868s                                                                                                                                                                                                                    
[sbesson@prod118-omeroreadwrite ~]$ ./bftools/showinf -nopix /uod/idr/metadata/idr0033-rohban-pathways/screens/41744.screen  | grep Init
Initializing reader
Initialization took 318.314s

idr0025

[sbesson@prod118-omeroreadwrite ~]$ ./bftools/showinf -nopix  /uod/idr/metadata/idr0025-stadler-proteinatlas/screens/10x\ images\ plate\ 2.screen | grep Init
Initializing reader
Initialization took 70.219s
[sbesson@prod118-omeroreadwrite ~]$ ./bftools/showinf -nopix  /uod/idr/metadata/idr0025-stadler-proteinatlas/screens/10x\ images\ plate\ 2.screen | grep Init
Initializing reader
Initialization took 5.566s

Overall, these timings should be used for an informal comparison with the initialization times described in the issue description. Note a few comments:

by design, the commands above only measure the reader initialization time which I expect to be the bulk of the memo file regeneration. If necessary, we could look into a more involved testing workflow by deleting the memo files in a pilot with the current software and regenerating them using the same approach as the one above
by design, the commands above only measure the reader initialization time which I expect to be the bulk of the memo file regeneration. If necessary, we could look into a more involved testing workflow by deleting the memo files in a pilot with the current software and regenerating them using the same approach as the one above
Bio-Formats showinf utility is invoked a few times successfully. The first measurement effectively reflects the initialization tme without any caching while the subsequent ones benefit from some of the filesystem level caching associated with the first command. Freeing the cache using https://linux-mm.org/Drop_Caches allows to partly restore the initialization time associated with the first invocation.
For a faithful comparison, it would be useful to know underI don't know under which regime the ZarrReader memo file regeneration might be working

will-moore · 2023-10-04T13:47:49Z

Comparing the first initialisation time for pattern files above with the times to create bfmemo for equivalent NGFF filesets (from description above):

idr0016 - pattern: 19 mins, NGFF: 7.5 mins
idr0033 - pattern: 19 mins, NGFF: 81 mins
idr0025 - pattern: 1.2 mins, NGFF: 4 - 12 mins

So, NGFF is ~ 2x quicker for idr0016 but ~ 4x slower for the others.

joshmoore · 2023-10-09T09:44:29Z

👍 for the data but still concerning.

will-moore added this to NGFF conversion Sep 6, 2023

will-moore moved this to create new Filesets in idr-next in NGFF conversion Sep 6, 2023

will-moore mentioned this issue Sep 11, 2023

ZarrReader performance improvements ome/ZarrReader#64

Merged

will-moore moved this from check_pixels to Other issues (not studies) in NGFF conversion May 21, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Testing mkngff on idr-testing #672

Testing mkngff on idr-testing #672

will-moore commented Sep 6, 2023 •

edited

Loading

will-moore commented Sep 6, 2023 •

edited

Loading

will-moore commented Sep 6, 2023

will-moore commented Sep 6, 2023

sbesson commented Sep 6, 2023

will-moore commented Sep 6, 2023

will-moore commented Sep 6, 2023

sbesson commented Sep 6, 2023

will-moore commented Sep 6, 2023

sbesson commented Sep 7, 2023

will-moore commented Sep 7, 2023

will-moore commented Sep 7, 2023

will-moore commented Sep 7, 2023

sbesson commented Sep 7, 2023

will-moore commented Sep 7, 2023

will-moore commented Sep 7, 2023

sbesson commented Sep 7, 2023

will-moore commented Sep 8, 2023 •

edited

Loading

will-moore commented Sep 11, 2023

will-moore commented Sep 11, 2023 •

edited

Loading

will-moore commented Sep 11, 2023

will-moore commented Sep 11, 2023 •

edited

Loading

will-moore commented Sep 11, 2023 •

edited

Loading

will-moore commented Sep 12, 2023

sbesson commented Oct 2, 2023

will-moore commented Oct 4, 2023

joshmoore commented Oct 9, 2023

Testing mkngff on idr-testing #672

Testing mkngff on idr-testing #672

Comments

will-moore commented Sep 6, 2023 • edited Loading

will-moore commented Sep 6, 2023 • edited Loading

will-moore commented Sep 6, 2023

will-moore commented Sep 6, 2023

sbesson commented Sep 6, 2023

will-moore commented Sep 6, 2023

will-moore commented Sep 6, 2023

sbesson commented Sep 6, 2023

will-moore commented Sep 6, 2023

sbesson commented Sep 7, 2023

will-moore commented Sep 7, 2023

will-moore commented Sep 7, 2023

will-moore commented Sep 7, 2023

sbesson commented Sep 7, 2023

will-moore commented Sep 7, 2023

will-moore commented Sep 7, 2023

sbesson commented Sep 7, 2023

will-moore commented Sep 8, 2023 • edited Loading

will-moore commented Sep 11, 2023

will-moore commented Sep 11, 2023 • edited Loading

will-moore commented Sep 11, 2023

will-moore commented Sep 11, 2023 • edited Loading

will-moore commented Sep 11, 2023 • edited Loading

will-moore commented Sep 12, 2023

sbesson commented Oct 2, 2023

will-moore commented Oct 4, 2023

joshmoore commented Oct 9, 2023

will-moore commented Sep 6, 2023 •

edited

Loading

will-moore commented Sep 6, 2023 •

edited

Loading

will-moore commented Sep 8, 2023 •

edited

Loading

will-moore commented Sep 11, 2023 •

edited

Loading

will-moore commented Sep 11, 2023 •

edited

Loading

will-moore commented Sep 11, 2023 •

edited

Loading