idr0016-wawer-bioactivecompoundprofiling S-BIAD851 #638

dominikl · 2023-02-22T11:47:47Z

https://github.com/IDR/idr0016-wawer-bioactivecompoundprofiling

Sample plate conversion failed with:

(base) [dlindner@pilot-zarr2-dev idr0016]$ time /home/dlindner/bioformats2raw/bin/bioformats2raw --memo-directory ../memo 24320.screen 24320.ome.zarr
OpenJDK 64-Bit Server VM warning: You have loaded library /tmp/opencv_openpnp6586654250319720590/nu/pattern/opencv/linux/x86_64/libopencv_java342.so which might have disabled stack guard. The VM will try to fix the stack guard now.
It's highly recommended that you fix the library with 'execstack -c <libfile>', or link it with '-z noexecstack'.
[Fatal Error] :1:84: Character reference "&#0" is an invalid XML character.
Exception in thread "main" picocli.CommandLine$ExecutionException: Error while calling command (com.glencoesoftware.bioformats2raw.Converter@63a65a25): java.lang.RuntimeException: org.xml.sax.SAXParseException; lineNumber: 1; columnNumber: 84; Character reference "&#0" is an invalid XML character.
        at picocli.CommandLine.executeUserObject(CommandLine.java:1962)
        at picocli.CommandLine.access$1300(CommandLine.java:145)
        at picocli.CommandLine$RunLast.executeUserObjectOfLastSubcommandWithSameParent(CommandLine.java:2352)
        at picocli.CommandLine$RunLast.handle(CommandLine.java:2346)
        at picocli.CommandLine$RunLast.handle(CommandLine.java:2311)
        at picocli.CommandLine$AbstractParseResultHandler.handleParseResult(CommandLine.java:2172)
        at picocli.CommandLine.parseWithHandlers(CommandLine.java:2550)
        at picocli.CommandLine.parseWithHandler(CommandLine.java:2485)
        at picocli.CommandLine.call(CommandLine.java:2761)

This error Character reference "&#0" is an invalid XML character is already referenced by IDR/bioformats#29 .

The text was updated successfully, but these errors were encountered:

sbesson · 2023-05-29T10:50:49Z

Note that this study will have the same caveats as of #640 (comment) in terms of channel order. So similar decisions will need to be done in terms of the conversion we want to perform.

will-moore · 2023-07-10T13:08:53Z

Since we've decided to use omero-cli-zarr for idr0036 #640 (comment) we should do the same here...

will-moore · 2023-07-10T13:57:33Z

Going to try on a different machine since pilot-zarr1-dev and pilot-zarr2-dev are at capacity...

Update to use ome/omero-cli-zarr#146

$ ssh -A ome-zarr-dev1.openmicroscopy.org
$ conda activate omero_zarr_export
$ pip uninstall omero-cli-zarr
$ pip install git+https://github.com/will-moore/omero-cli-zarr.git@fix_downsample_image_path
...
Successfully installed omero-cli-zarr-0.1.dev451+g983576f

Just use my home dir...

$ df -h ./
Filesystem                   Size  Used Avail Use% Mounted on
/dev/mapper/VolGroup00-root  994G   34G  960G   4% /

Listing all 413 Plate IDs:

6208 4907 4906 6151 6210 6153 6154 6155 4908 6156 6157 6158 6159 4909 6160 6161 4911 6162 6163 6164 6165 6166 6167 4910 6168 6169 4912 6170 4913 6171 6172 4914 6173 6174 6175 6176 6177 6178 6179 4915 4917 4916 4951 4953 4952 4954 4956 4955 4958 4959 4957 6180 4962 4961 4960 4963 4964 4965 4966 4967 4968 4969 4970 4971 4973 4972 4974 4975 4976 4977 4978 6181 6182 6183 6184 6185 6186 4979 4980 6187 6188 6189 6190 6191 6192 6193 6194 4981 6195 6196 6197 6198 6199 6200 4982 6201 6202 6203 6204 6205 6206 6207 4983 4984 4986 4985 4987 4989 4988 4990 4991 4992 4993 4994 4995 4996 4997 4998 4999 5001 5000 5002 5004 5003 5005 5006 5007 5008 5010 5009 5011 5012 5014 5013 5015 5017 5016 5019 5018 5020 5021 5023 5022 5024 5025 5026 5029 5027 5028 5032 5031 5030 5033 5035 5034 5036 5037 5038 5039 5040 5041 5042 5044 5043 5047 5046 5045 5050 5048 5049 5052 5051 5053 5054 5056 5055 5059 5058 5057 5062 5060 5061 5063 5065 5064 5066 5068 5067 5069 5071 5070 5072 5074 5073 5075 5076 5077 5080 5079 5078 5081 5082 5083 5084 5085 5086 5087 5088 5089 5091 5090 5092 5094 5093 5095 5096 5097 5098 5101 5100 5099 5102 5103 5104 5105 5106 5107 5151 5152 5153 5154 5155 5156 5157 5158 5159 5160 5161 5162 5163 5164 5165 5166 5167 5168 5169 5170 5171 5172 5173 5174 5175 5176 5177 5178 5179 5180 5181 5182 5183 5184 5185 5186 5187 5188 5189 5190 5191 5192 5193 5194 5195 5196 5197 5198 5199 5200 5201 5202 5203 5204 5205 5206 5207 5208 5209 5210 5211 5212 5213 5214 5215 5216 5217 5218 5219 5220 5221 5222 5223 5224 5225 5226 5227 5228 5229 5230 5231 5232 5233 5234 5235 5236 5237 5238 5239 5240 5241 5242 5243 5244 5245 5246 5247 5248 5249 5251 5250 5252 5253 5254 5255 5256 5257 5259 5258 5260 5261 5262 5263 5264 5265 5266 5267 5268 5269 5270 5271 5272 5273 5274 5275 5276 5277 5278 5279 5280 5281 5282 5283 5284 5285 5286 5287 5288 5289 5290 5291 5292 5293 5294 5295 5296 5297 5298 5299 5300 5301 5302 5351 5303 5304 5305 5306 5307 5308 5380 5353 5354 5355 5356 5357 5358 5359 5360 5361 5362 5363 5364 5365 5366 5367 5368 5369 5370 5371 5372 5373 5374 5375 5376 5377 5378

start with export of 10 Plates.

screen -S idr0016_ngff
mkdir idr0016 && cd idr0016
omero login.   # idr-testing public/public
for id in 6208 4907 4906 6151 6210 6153 6154 6155 4908 6156; do
  echo $id;
  omero zarr export Plate:$id;
done

will-moore · 2023-07-10T22:46:22Z

3 plates completed so far...
Zipping...

for i in */; do zip -mr "${i%/}.zip" "$i"; done

will-moore · 2023-07-11T00:55:44Z

Ooops - forgot to rename from plateID.zarr to plateName.ome.zarr as I did for idr0036...
Unzipped each of 3 zips and renamed...

$ ls -lh
total 0
drwxr-xr-x 18 wmoore lsd 180 Jul 10 17:20 24277.ome.zarr
drwxr-xr-x 18 wmoore lsd 180 Jul 10 19:51 24278.ome.zarr
drwxr-xr-x 18 wmoore lsd 180 Jul 10 22:14 24279.ome.zarr

Then zipped again...

will-moore · 2023-07-11T03:28:35Z

Upload failed...

(base) [wmoore@ome-zarr-dev1 bin]$ ./ascp -P33001 -i ../etc/asperaweb_id_dsa.openssh -d ~/idr0016 [email protected]:5f/136e8d-xxxxxx
24277.ome.zarr.zip                                                           95% 5277MB 97.8Mb/s    00:25 ETAPartial Completion: 5414415K bytes transferred in 517 seconds
 (85719K bits/sec), in 3 files, 1 directory; 3 files failed.

Session Stop  (Error: Session data transfer timeout (server), Session data transfer timeout)

Deleted zips on BioStudies and ran again...

will-moore · 2023-07-11T04:30:48Z

Uploaded 1 zip, then timed-out on next one. Repeated again - for each zip (only uploaded 1 at a time before time-out).
Last one uploaded with:

(base) [wmoore@ome-zarr-dev1 bin]$ ./ascp -P33001 -i ../etc/asperaweb_id_dsa.openssh -d ~/idr0016 [email protected]:5f/136e8d-xxxxxxxxxx
24279.ome.zarr.zip                        100% 6188MB 83.4Mb/s    08:51    
Completed: 6336955K bytes transferred in 532 seconds
 (97506K bits/sec), in 1 file, 1 directory.

will-moore · 2023-07-11T05:02:13Z

Testing on s3...

$ aws --endpoint-url https://uk1s3.embassy.ebi.ac.uk s3 mb s3://idr0016
make_bucket: idr0016
$ aws --endpoint-url https://uk1s3.embassy.ebi.ac.uk s3api put-bucket-policy --bucket idr0016 --policy file://policy.json
$ aws --endpoint-url https://uk1s3.embassy.ebi.ac.uk s3api put-bucket-cors --bucket idr0016  --cors-configuration file://cors.json

$ cd /idr0016
$ unzip 24279.ome.zarr.zip && rm 24279.ome.zarr.zip
$ cd
$ ./mc cp -r idr0016/ uk1s3/idr0016/zarr
...79.ome.zarr/P/9/5/3/4/0/0: 6.61 GiB / 6.61 GiB ━━━━━━━━━━━━━━━━━ 24.22 MiB/s 4m39s

Looks good and valid...
https://ome.github.io/ome-ngff-validator/?source=https://uk1s3.embassy.ebi.ac.uk/idr0016/zarr/24279.ome.zarr

https://hms-dbmi.github.io/vizarr/?source=https://uk1s3.embassy.ebi.ac.uk/idr0016/zarr/24279.ome.zarr

will-moore · 2023-07-11T06:04:38Z

Above export still running...
Start another, also using idr-testing (same session).
Use ome/omero-cli-zarr#147 so we don't have to manually rename every plate after export...

batch2 of 100 plates...

screen -S idr0016_export2
conda activate omero_zarr_export
pip install git+https://github.com/will-moore/omero-cli-zarr.git@name_option
cd idr0016
mkdir batch2
cd batch2
for id in 6157 6158 6159 4909 6160 6161 4911 6162 6163 6164 6165 6166 6167 4910 6168 6169 4912 6170 4913 6171 6172 4914 6173 6174 6175 6176 6177 6178 6179 4915 4917 4916 4951 4953 4952 4954 4956 4955 4958 4959 4957 6180 4962 4961 4960 4963 4964 4965 4966 4967 4968 4969 4970 4971 4973 4972 4974 4975 4976 4977 4978 6181 6182 6183 6184 6185 6186 4979 4980 6187 6188 6189 6190 6191 6192 6193 6194 4981 6195 6196 6197 6198 6199 6200 4982 6201 6202 6203 6204 6205 6206 6207 4983 4984 4986 4985 4987 4989 4988 4990; do
  echo $id;
  omero zarr export Plate:$id --name_by name;
done

will-moore · 2023-07-11T12:54:59Z

Lost connection with IDR part-way through initial 10 plates (and batch2 of 100 plates)...
Restarted, repeating the part-exported plate and the remaining 3 of 10...

ssh -A ome-zarr-dev1.openmicroscopy.org
screen -S idr0016_export
for id in 6154 6155 4908 6156; do
  echo $id;
  omero zarr export Plate:$id;
done

start from scratch for all 100 plates (only part of 1 plate done so far)

cd batch2
for id in 6157 6158 6159 4909 6160 6161 4911 6162 6163 6164 6165 6166 6167 4910 6168 6169 4912 6170 4913 6171 6172 4914 6173 6174 6175 6176 6177 6178 6179 4915 4917 4916 4951 4953 4952 4954 4956 4955 4958 4959 4957 6180 4962 4961 4960 4963 4964 4965 4966 4967 4968 4969 4970 4971 4973 4972 4974 4975 4976 4977 4978 6181 6182 6183 6184 6185 6186 4979 4980 6187 6188 6189 6190 6191 6192 6193 6194 4981 6195 6196 6197 6198 6199 6200 4982 6201 6202 6203 6204 6205 6206 6207 4983 4984 4986 4985 4987 4989 4988 4990; do
  echo $id;
  omero zarr export Plate:$id --name_by name;
done

will-moore · 2023-07-12T01:55:18Z

Remaining of the first batch of 10 plates exported OK.
Renamed to plateName.ome.zarr using e.g. https://idr.openmicroscopy.org/webclient/?show=plate-6210 to lookup...

(base) [wmoore@ome-zarr-dev1 idr0016]$ ls -alh
total 4.0K
drwxr-xr-x 11 wmoore lsd  161 Jul 11 21:13 .
drwx------ 30 wmoore lsd 4.0K Jul 11 13:00 ..
drwxr-xr-x 18 wmoore lsd  180 Jul 10 22:14 24279.ome.zarr
drwxr-xr-x 18 wmoore lsd  180 Jul 11 21:00 4908.zarr
drwxr-xr-x 18 wmoore lsd  180 Jul 11 00:59 6151.zarr
drwxr-xr-x 18 wmoore lsd  180 Jul 11 06:23 6153.zarr
drwxr-xr-x 18 wmoore lsd  180 Jul 11 15:28 6154.zarr
drwxr-xr-x 18 wmoore lsd  180 Jul 11 18:14 6155.zarr
drwxr-xr-x 18 wmoore lsd  180 Jul 11 23:52 6156.zarr
drwxr-xr-x 18 wmoore lsd  180 Jul 11 03:47 6210.zarr
drwxr-xr-x  7 wmoore lsd  116 Jul 12 00:47 batch2
(base) [wmoore@ome-zarr-dev1 idr0016]$ mv 4908.zarr 24297.ome.zarr
(base) [wmoore@ome-zarr-dev1 idr0016]$ mv 6151.zarr 24280.ome.zarr
(base) [wmoore@ome-zarr-dev1 idr0016]$ mv 6153.zarr 24294.ome.zarr
(base) [wmoore@ome-zarr-dev1 idr0016]$ mv 6154.zarr 24295.ome.zarr
(base) [wmoore@ome-zarr-dev1 idr0016]$ mv 6155.zarr 24296.ome.zarr
(base) [wmoore@ome-zarr-dev1 idr0016]$ mv 6156.zarr 24300.ome.zarr
(base) [wmoore@ome-zarr-dev1 idr0016]$ mv 6210.zarr 24293.ome.zarr

moved into batch1 dir to zip...

$ cd batch1/
(base) [wmoore@ome-zarr-dev1 batch1]$ ls -lh
total 0
drwxr-xr-x 18 wmoore lsd 180 Jul 10 22:14 24279.ome.zarr
drwxr-xr-x 18 wmoore lsd 180 Jul 11 00:59 24280.ome.zarr
drwxr-xr-x 18 wmoore lsd 180 Jul 11 03:47 24293.ome.zarr
drwxr-xr-x 18 wmoore lsd 180 Jul 11 06:23 24294.ome.zarr
drwxr-xr-x 18 wmoore lsd 180 Jul 11 15:28 24295.ome.zarr
drwxr-xr-x 18 wmoore lsd 180 Jul 11 18:14 24296.ome.zarr
drwxr-xr-x 18 wmoore lsd 180 Jul 11 21:00 24297.ome.zarr
drwxr-xr-x 18 wmoore lsd 180 Jul 11 23:52 24300.ome.zarr
(base) [wmoore@ome-zarr-dev1 batch1]$ for i in */; do zip -mr "${i%/}.zip" "$i"; done

will-moore · 2023-07-12T04:05:42Z

Current status of batch2 export of 100 plates...
Just under 3 hours per plate...

(base) [wmoore@ome-zarr-dev1 ~]$ ls -lh ~/idr0016/batch2
total 0
drwxr-xr-x 18 wmoore lsd 180 Jul 11 16:21 24301.ome.zarr
drwxr-xr-x 18 wmoore lsd 180 Jul 11 19:01 24302.ome.zarr
drwxr-xr-x 18 wmoore lsd 180 Jul 11 21:49 24303.ome.zarr
drwxr-xr-x 18 wmoore lsd 180 Jul 12 00:37 24304.ome.zarr
drwxr-xr-x 18 wmoore lsd 180 Jul 12 03:13 24305.ome.zarr
drwxr-xr-x 12 wmoore lsd 126 Jul 12 04:49 24306.ome.zarr

Space is enough for 100 plates (approx 6.6 GB per plate)...

$ df -h ./
Filesystem                   Size  Used Avail Use% Mounted on
/dev/mapper/VolGroup00-root  994G  139G  855G  14% /

Zipping progress of batch1 - about 25 minutes per plate...

(base) [wmoore@ome-zarr-dev1 ~]$ ls -lh ~/idr0016/batch1
total 33G
-rw-r--r--  1 wmoore lsd 6.1G Jul 12 03:28 24280.ome.zarr.zip
-rw-r--r--  1 wmoore lsd 6.1G Jul 12 03:53 24293.ome.zarr.zip
-rw-r--r--  1 wmoore lsd 6.1G Jul 12 04:17 24294.ome.zarr.zip
-rw-r--r--  1 wmoore lsd 6.0G Jul 12 04:42 24295.ome.zarr.zip
drwxr-xr-x 18 wmoore lsd  180 Jul 11 18:14 24296.ome.zarr
drwxr-xr-x 18 wmoore lsd  180 Jul 11 21:00 24297.ome.zarr
drwxr-xr-x 18 wmoore lsd  180 Jul 11 23:52 24300.ome.zarr
-rw-------  1 wmoore lsd 4.2G Jul 12 05:03 ziLKSu9D

will-moore · 2023-07-12T06:26:25Z

Upload remaining 7 zips of batch1.
Timeout failure again...

(base) [wmoore@ome-zarr-dev1 bin]$ ./ascp -P33001 -i ../etc/asperaweb_id_dsa.openssh -d ~/idr0016/batch1/idr0016 [email protected]:5f/136e8d-xxxxxx

24280.ome.zarr.zip                                                                                                                   0%  562MB 95.1Mb/s   - error - 
                                                                                                                                Error 35 [Data transfer timeout]     
Partial Completion: 588018K bytes transferred in 120 seconds
 (40105K bits/sec), in 7 files, 1 directory; 7 files failed.

Session Stop  (Error: Session data transfer timeout)

will-moore · 2023-07-12T10:25:22Z

Move 7 zips to minio objectstore...

(base) [wmoore@ome-zarr-dev1 idr]$ mv ~/idr0016/batch1/idr0016/* /uod/idr/objectstore/minio/idr/idr0016/

These are then available to download from e.g. https://minio-dev.openmicroscopy.org/idr/idr0016/24280.ome.zarr.zip

will-moore · 2023-07-12T12:17:54Z

Want to use idr-ftp machine to aspera the data to BioStudies (as we did for idr0012)...

Try to rsync to ssh ome-zarr-dev1.openmicroscopy.org from there but can't ssh...

(base) [wmoore@idrftp-ftp ~]$ ssh ome-zarr-dev1.openmicroscopy.org
ssh: Could not resolve hostname ome-zarr-dev1.openmicroscopy.org: Name or service not known

Try to use the minio data available above...

Install goofys on idr-ftp to copy data there.

$ cd
$ sudo wget https://github.com/kahing/goofys/releases/latest/download/goofys
$ sudo chmod +x ./goofys 

$ sudo mkdir ./minio
$ sudo ~/goofys --endpoint https://minio-dev.openmicroscopy.org/ -o allow_other idr0012 ./minio
2023/07/12 12:15:51.233904 main.FATAL Unable to mount file system, see syslog for details

will-moore · 2023-07-12T14:24:20Z

Downloaded 7 zips to idr-ftp machine with....

$ wget https://minio-dev.openmicroscopy.org/idr/idr0016/24294.ome.zarr.zip
etc...
...
$ ls -lh
total 42G
-rw-rw-r--. 1 wmoore wmoore 6.1G Jul 12 02:28 24280.ome.zarr.zip
-rw-rw-r--. 1 wmoore wmoore 6.1G Jul 12 02:53 24293.ome.zarr.zip
-rw-rw-r--. 1 wmoore wmoore 6.1G Jul 12 03:17 24294.ome.zarr.zip
-rw-rw-r--. 1 wmoore wmoore 6.0G Jul 12 03:42 24295.ome.zarr.zip
-rw-rw-r--. 1 wmoore wmoore 6.0G Jul 12 04:10 24296.ome.zarr.zip
-rw-rw-r--. 1 wmoore wmoore 6.1G Jul 12 04:40 24297.ome.zarr.zip
-rw-rw-r--. 1 wmoore wmoore 6.0G Jul 12 05:29 24300.ome.zarr.zip

Upload to BioStudies...

sudo /root/.aspera/cli/bin/ascp -P33001 -i /root/.aspera/cli/etc/asperaweb_id_dsa.openssh -d /data/ngff/idr0016/idr0016/ [email protected]:5f/xxxxxx

will-moore · 2023-07-13T08:45:19Z

Tried to install p7zip on ome-zarr-dev1 without success...

$ sudo yum install p7zip
Loaded plugins: langpacks, product-id, rhnplugin, search-disabled-repos, subscription-manager
The SSL certificate failed verification.

will-moore · 2023-07-13T09:11:13Z

@dominikl
Plates are taking about 3 hours to export.
We have exported about 27 / 413 Plates (from idr-testing.openmicroscopy.org).

(386 * 3) / 24 = 48 days
This is too long, so we need to speed this up and run on multiple machines, exporting from multiple servers. E.g. idr-testing and idr.openmicroscopy.org.

The first 10 Plates (done) and 100 (now running) leave 303 Plates to follow (or export at the same time elsewhere).

These 303 IDs are:

 4991 4992 4993 4994 4995 4996 4997 4998 4999 5001 5000 5002 5004 5003 5005 5006 5007 5008 5010 5009 5011 5012 5014 5013 5015 5017 5016 5019 5018 5020 5021 5023 5022 5024 5025 5026 5029 5027 5028 5032 5031 5030 5033 5035 5034 5036 5037 5038 5039 5040 5041 5042 5044 5043 5047 5046 5045 5050 5048 5049 5052 5051 5053 5054 5056 5055 5059 5058 5057 5062 5060 5061 5063 5065 5064 5066 5068 5067 5069 5071 5070 5072 5074 5073 5075 5076 5077 5080 5079 5078 5081 5082 5083 5084 5085 5086 5087 5088 5089 5091 5090 5092 5094 5093 5095 5096 5097 5098 5101 5100 5099 5102 5103 5104 5105 5106 5107 5151 5152 5153 5154 5155 5156 5157 5158 5159 5160 5161 5162 5163 5164 5165 5166 5167 5168 5169 5170 5171 5172 5173 5174 5175 5176 5177 5178 5179 5180 5181 5182 5183 5184 5185 5186 5187 5188 5189 5190 5191 5192 5193 5194 5195 5196 5197 5198 5199 5200 5201 5202 5203 5204 5205 5206 5207 5208 5209 5210 5211 5212 5213 5214 5215 5216 5217 5218 5219 5220 5221 5222 5223 5224 5225 5226 5227 5228 5229 5230 5231 5232 5233 5234 5235 5236 5237 5238 5239 5240 5241 5242 5243 5244 5245 5246 5247 5248 5249 5251 5250 5252 5253 5254 5255 5256 5257 5259 5258 5260 5261 5262 5263 5264 5265 5266 5267 5268 5269 5270 5271 5272 5273 5274 5275 5276 5277 5278 5279 5280 5281 5282 5283 5284 5285 5286 5287 5288 5289 5290 5291 5292 5293 5294 5295 5296 5297 5298 5299 5300 5301 5302 5351 5303 5304 5305 5306 5307 5308 5380 5353 5354 5355 5356 5357 5358 5359 5360 5361 5362 5363 5364 5365 5366 5367 5368 5369 5370 5371 5372 5373 5374 5375 5376 5377 5378

NB: when creating conda env for exporting, use pip install git+https://github.com/will-moore/omero-cli-zarr.git@name_option and run with omero zarr export Plate:$id --name_by name; as described above.

Current export is running at

$ ssh -A ome-zarr-dev1.openmicroscopy.org
(base) [wmoore@ome-zarr-dev1 ~]$ ls -lh /lifesci/groups/jrs/wmoore/idr0016/batch2
total 4.0K
drwxr-xr-x 18 wmoore lsd  180 Jul 13 04:47 24320.ome.zarr
drwxr-xr-x 18 wmoore lsd  180 Jul 13 07:21 24321.ome.zarr
drwxr-xr-x 18 wmoore lsd  180 Jul 13 09:50 24352.ome.zarr
drwxr-xr-x  3 wmoore lsd   45 Jul 13 10:00 24357.ome.zarr
drwxr-xr-x 10 wmoore lsd 4.0K Jul 13 09:43 idr0016

Getting data off that machine is hard as aspera times-out badly and I can't install p7zip (more reasons to run other batches elsewhere).
But the exported data can sit there till I'm back. Should be enough space for over 100 Plates (6.6 GB each)

(base) [wmoore@ome-zarr-dev1 ~]$ df -h ./
Filesystem                   Size  Used Avail Use% Mounted on
/dev/mapper/VolGroup00-root  994G  170G  824G  18% /

will-moore · 2023-09-22T08:21:26Z

@sbesson

On that system, OMERO is running but not doing anything too demanding - just looking up Filesets. However, it seems that I've mostly been seeing these failures if I'm running omero mkngff sql in more than 1 terminal/screen at a time. So, I'm going to stick with just 1 at a time now
The recursive walking is much reduced since we don't walk into .zarray dirs with Dont walk arrays omero-mkngff#11 , but yes - still lots of recursive walking on Plates
It would be good to have a round of testing new NGFF Filesets on idr-testing, and see how it performs

will-moore · 2023-09-22T13:02:42Z

Cancelled mkngff sql on idr-testing just now as I realised there's a bug that omits .zarray files.
Stopped after Fileset 21510 (233 / 413) in the idr0016.csv

Fixed in IDR/omero-mkngff@cac303d

Updated...

(venv3) [root@test120-omeroreadwrite wmoore]# pip install git+https://github.com/will-moore/omero-mkngff.git@dont_walk_arrays
...
  Resolved https://github.com/will-moore/omero-mkngff.git to commit 08db883c54410265783d5f5a4cf5f6b31d2dd5e3

will-moore · 2023-09-22T21:33:15Z

Start from scratch on idr0138-pilot as regular wmoore user...

wget https://raw.githubusercontent.com/IDR/idr-utils/ebbb0b9dc6ed548db9bbe298c062a14885411097/scripts/ngff_filesets/idr0016.csv

(venv3) (base) [wmoore@pilot-idr0138-omeroreadwrite ~]$ for r in $(cat $IDRID.csv); do
>   biapath=$(echo $r | cut -d',' -f2)
>   uuid=$(echo $biapath | cut -d'/' -f2)
>   fsid=$(echo $r | cut -d',' -f3)
>   omero mkngff sql $fsid "/bia-integrator-data/$biapath/$uuid.zarr" >> "$IDRID/$fsid.sql"
> done
Using session for demo@localhost:4064. Idle timeout: 10 min. Current group: Public
Found prefix: demo_2/2016-06/24/05-33-04.817 for fileset: 21405
...

will-moore · 2023-09-25T05:45:16Z

The last successful sql generated above was number 72/413:

idr0016/26595.ome.zarr,S-BIAD851/2632c5cd-86ec-434a-9da7-5277ab002250,21570

It seems that at that point this failed, and is current status

$ ls /bia-integrator-data 
ls: cannot access /bia-integrator-data: Transport endpoint is not connected

will-moore · 2023-09-25T12:03:43Z

Re-mounted goofys /bia-integrator-data and restarted server...
Edited idr0016.csv to remove all lines before idr0016/26595.ome.zarr,S-BIAD851/2632c5cd-86ec-434a-9da7-5277ab002250,21570
(maybe should have removed that line too?)...
And re-ran omero mkngff sql as above...

will-moore · 2023-09-27T05:35:59Z

restart, after 116 filesets processed since last restart...

(venv3) (base) [wmoore@pilot-idr0138-omeroreadwrite ~]$ for r in $(cat $IDRID.csv); do   biapath=$(echo $r | cut -d',' -f2);   uuid=$(echo $biapath | cut -d'/' -f2);   fsid=$(echo $r | cut -d',' -f3);   omero mkngff sql $fsid "/bia-integrator-data/$biapath/$uuid.zarr" > "$IDRID/$fsid.sql"; done
Using session for [email protected]:4064. Idle timeout: 10 min. Current group: Public
Found prefix: demo_2/2016-06/25/06-09-19.476 for fileset: 21471
...

will-moore · 2023-09-27T21:46:03Z

All done...
A couple of files are 0 bytes:

idr0016/21453.sql
idr0016/21256.sql

will-moore · 2023-11-15T12:42:44Z

Need to re-convert Plate named 24667 since previous NGFF conversion is missing some files from N10 field 1:

https://ome.github.io/ome-ngff-validator/?source=https%3A%2F%2Fuk1s3.embassy.ebi.ac.uk%2Fbia-integrator-data%2FS-BIAD851%2F2c49b893-ec6d-4329-9cc3-569b820075f2%2F2c49b893-ec6d-4329-9cc3-569b820075f2.zarr&well=all
and
https://ome.github.io/ome-ngff-validator/?source=https://uk1s3.embassy.ebi.ac.uk/bia-integrator-data/S-BIAD851/2c49b893-ec6d-4329-9cc3-569b820075f2/2c49b893-ec6d-4329-9cc3-569b820075f2.zarr/N/10/

On zarr1-dev-pilot...

conda activate bioformats2raw2
~/bioformats2raw-0.6.0-24/bin/bioformats2raw --memo-directory ../memo  /uod/idr/metadata/idr0016-wawer-bioactivecompoundprofiling/screens/24667.screen 24667.ome.zarr

EDIT - this failed! Forgot that we're using omero-cli-zarr for idr0016 exports...

conda activate omero-zarr-export
pip install -U git+https://github.com/will-moore/omero-cli-zarr.git@name_option

$ omero zarr export Plate:6202 --name_by name

Error loading: /home/wmoore/miniconda3/envs/omero_zarr_export/lib/python3.9/site-packages/omero/plugins/zarr.py
Traceback (most recent call last):
  File "/home/wmoore/miniconda3/envs/omero_zarr_export/lib/python3.9/site-packages/omero/cli.py", line 1690, in loadpath
    execfile(str(pathobj), loc)
  File "/home/wmoore/miniconda3/envs/omero_zarr_export/lib/python3.9/site-packages/past/builtins/misc.py", line 87, in execfile
    exec_(code, myglobals, mylocals)
  File "/home/wmoore/miniconda3/envs/omero_zarr_export/lib/python3.9/site-packages/omero/plugins/zarr.py", line 1, in <module>
    from omero_zarr.cli import HELP, ZarrControl
  File "/home/wmoore/miniconda3/envs/omero_zarr_export/lib/python3.9/site-packages/omero_zarr/__init__.py", line 21, in <module>
    from ._version import version as __version__
ModuleNotFoundError: No module named 'omero_zarr._version'
usage: /home/wmoore/miniconda3/envs/omero_zarr_export/bin/omero
       [-h] [-v] [-d DEBUG] [--path PATH] [-C] [-s SERVER] [-p PORT]
       [-g GROUP] [-u USER] [-w PASSWORD] [-k KEY] [--sudo ADMINUSER] [-q]
       <subcommand> ...
/home/wmoore/miniconda3/envs/omero_zarr_export/bin/omero: error: argument <subcommand>: invalid choice: 'zarr'

will-moore · 2023-11-15T15:38:18Z

Use idr-ftp as above:

$ conda activate omero_zarr_export
(omero_zarr_export) [wmoore@idrftp-ftp idr0016]$ pip freeze | grep zarr
ome-zarr==0.8.0
omero-cli-zarr @ git+https://github.com/will-moore/omero-cli-zarr.git@c73d40046536f8b5cc62908ebdaa86d097a30d0b
zarr==2.16.1

omero zarr export Plate:6202 --name_by name

Check that plate isn't missing Well N/10/1 as above...

(omero_zarr_export) [wmoore@idrftp-ftp idr0016]$ ls -alh 24667.ome.zarr/N/10/
total 12K
drwxrwxr-x.  8 wmoore wmoore  126 Nov 15 16:25 .
drwxrwxr-x. 26 wmoore wmoore 4.0K Nov 15 16:27 ..
drwxrwxr-x.  6 wmoore wmoore  100 Nov 15 16:25 0
drwxrwxr-x.  6 wmoore wmoore  100 Nov 15 16:25 1
drwxrwxr-x.  6 wmoore wmoore  100 Nov 15 16:25 2
drwxrwxr-x.  6 wmoore wmoore  100 Nov 15 16:25 3
drwxrwxr-x.  6 wmoore wmoore  100 Nov 15 16:25 4
drwxrwxr-x.  6 wmoore wmoore  100 Nov 15 16:25 5
-rw-rw-r--.  1 wmoore wmoore  420 Nov 15 16:25 .zattrs
-rw-rw-r--.  1 wmoore wmoore   24 Nov 15 16:25 .zgroup
(omero_zarr_export) [wmoore@idrftp-ftp idr0016]$ ls -alh 24667.ome.zarr/N/10/1
total 12K
drwxrwxr-x. 6 wmoore wmoore  100 Nov 15 16:25 .
drwxrwxr-x. 8 wmoore wmoore  126 Nov 15 16:25 ..
drwxrwxr-x. 7 wmoore wmoore   94 Nov 15 16:25 0
drwxrwxr-x. 7 wmoore wmoore   94 Nov 15 16:25 1
drwxrwxr-x. 7 wmoore wmoore   94 Nov 15 16:25 2
drwxrwxr-x. 7 wmoore wmoore   94 Nov 15 16:25 3
-rw-rw-r--. 1 wmoore wmoore 4.5K Nov 15 16:25 .zattrs
-rw-rw-r--. 1 wmoore wmoore   24 Nov 15 16:25 .zgroup

$ zip -r 24667.ome.zarr.zip 24667.ome.zarr

Delete 24667.ome.zarr.zip from https://www.ebi.ac.uk/biostudies/submissions/files?path=%2Fuser%2Fidr0016 and reupload...

sudo /root/.aspera/cli/bin/ascp -P33001 -i /root/.aspera/cli/etc/asperaweb_id_dsa.openssh -d /data/idr0016/idr0016/ [email protected]:/5f/13xxxxxxx

24667.ome.zarr.zip              100% 5461MB  454Mb/s    01:34    
Completed: 5592294K bytes transferred in 95 seconds
 (480827K bits/sec), in 1 file, 1 directory.

will-moore · 2023-12-04T22:40:50Z

idr0016 plates (Names) that are not yet viewable in idr-testing:

24634
25585
25592
25675
25740
25909

Lets run sql etc on clean idr0125-pilot data...

Update SECRET in sql... as wmoore

$ cd idr-util/scripts/ngff_filesets/idr0016
$ for i in $(ls); do sudo sed -i 's/SECRETUUID/c6b02bb7-2c22-4c45-be8d-30484c380a9c/g' $i; done

as omero-server user...

$ cd ngff_filesets/
$ export IDRID=idr0016

(venv3) (base) bash-4.2$ for r in $(cat $IDRID.csv); do
>   biapath=$(echo $r | cut -d',' -f2)
>   uuid=$(echo $biapath | cut -d'/' -f2)
>   fsid=$(echo $r | cut -d',' -f3 | tr -d '[:space:]')
>   psql -U omero -d idr -h $DBHOST -f "$IDRID/$fsid.sql"
>   omero mkngff symlink /data/OMERO/ManagedRepository $fsid "/bia-integrator-data/$biapath/$uuid.zarr" --bfoptions
> done
UPDATE 2304
BEGIN
 mkngff_fileset 
----------------
        5288754
(1 row)

COMMIT
Using session for demo@localhost:4064. Idle timeout: 10 min. Current group: Public
Checking for prefix_dir /data/OMERO/ManagedRepository/demo_2/2016-06/24/05-33-04.817
Creating dir at /data/OMERO/ManagedRepository/demo_2/2016-06/24/05-33-04.817_mkngff
Creating symlink /data/OMERO/ManagedRepository/demo_2/2016-06/24/05-33-04.817_mkngff/000f81bf-a7b2-4610-99c3-47dc5fec8c92.zarr -> /bia-integrator-data/S-BIAD851/000f81bf-a7b2-4610-99c3-47dc5fec8c92/000f81bf-a7b2-4610-99c3-47dc5fec8c92.zarr
Checking for prefix_dir /data/OMERO/ManagedRepository/demo_2/2016-06/24/05-33-04.817
write bfoptions to: /data/OMERO/ManagedRepository/demo_2/2016-06/24/05-33-04.817_mkngff/000f81bf-a7b2-4610-99c3-47dc5fec8c92.zarr.bfoptions
UPDATE 2304
BEGIN
 mkngff_fileset 
----------------
        5288755
(1 row)

COMMIT
Using session for demo@localhost:4064. Idle timeout: 10 min. Current group: Public
Checking for prefix_dir /data/OMERO/ManagedRepository/demo_2/2017-08/17/02-13-40.469
Creating dir at /data/OMERO/ManagedRepository/demo_2/2017-08/17/02-13-40.469_mkngff
...

Using session for demo@localhost:4064. Idle timeout: 10 min. Current group: Public
Checking for prefix_dir /data/OMERO/ManagedRepository/demo_2/2016-06/19/23-32-50.888
Creating dir at /data/OMERO/ManagedRepository/demo_2/2016-06/19/23-32-50.888_mkngff
Creating symlink /data/OMERO/ManagedRepository/demo_2/2016-06/19/23-32-50.888_mkngff/fd822d4b-3060-46e9-8178-982510009c93.zarr -> /bia-integrator-data/S-BIAD851/fd822d4b-3060-46e9-8178-982510009c93/fd822d4b-3060-46e9-8178-982510009c93.zarr
Checking for prefix_dir /data/OMERO/ManagedRepository/demo_2/2016-06/19/23-32-50.888
write bfoptions to: /data/OMERO/ManagedRepository/demo_2/2016-06/19/23-32-50.888_mkngff/fd822d4b-3060-46e9-8178-982510009c93.zarr.bfoptions
UPDATE 2304
BEGIN
 mkngff_fileset
----------------
        5289164
(1 row)

COMMIT
Using session for demo@localhost:4064. Idle timeout: 10 min. Current group: Public
Checking for prefix_dir /data/OMERO/ManagedRepository/demo_2/2016-06/27/02-33-37.895
Creating dir at /data/OMERO/ManagedRepository/demo_2/2016-06/27/02-33-37.895_mkngff
Creating symlink /data/OMERO/ManagedRepository/demo_2/2016-06/27/02-33-37.895_mkngff/fdf51c56-0ecf-4e1c-8b47-c35cafd78a2c.zarr -> /bia-integrator-data/S-BIAD851/fdf51c56-0ecf-4e1c-8b47-c35cafd78a2c/fdf51c56-0ecf-4e1c-8b47-c35cafd78a2c.zarr
Checking for prefix_dir /data/OMERO/ManagedRepository/demo_2/2016-06/27/02-33-37.895
write bfoptions to: /data/OMERO/ManagedRepository/demo_2/2016-06/27/02-33-37.895_mkngff/fdf51c56-0ecf-4e1c-8b47-c35cafd78a2c.zarr.bfoptions
UPDATE 2304
BEGIN
 mkngff_fileset
----------------
        5289165
(1 row)

COMMIT
Using session for demo@localhost:4064. Idle timeout: 10 min. Current group: Public
Checking for prefix_dir /data/OMERO/ManagedRepository/demo_2/2016-06/20/22-41-20.985
Creating dir at /data/OMERO/ManagedRepository/demo_2/2016-06/20/22-41-20.985_mkngff
Creating symlink /data/OMERO/ManagedRepository/demo_2/2016-06/20/22-41-20.985_mkngff/feea9b2d-dd05-428a-a04e-5ebd45048401.zarr -> /bia-integrator-data/S-BIAD851/feea9b2d-dd05-428a-a04e-5ebd45048401/feea9b2d-dd05-428a-a04e-5ebd45048401.zarr
Checking for prefix_dir /data/OMERO/ManagedRepository/demo_2/2016-06/20/22-41-20.985
write bfoptions to: /data/OMERO/ManagedRepository/demo_2/2016-06/20/22-41-20.985_mkngff/feea9b2d-dd05-428a-a04e-5ebd45048401.zarr.bfoptions
UPDATE 2304
BEGIN
 mkngff_fileset
----------------
        5289166
(1 row)

COMMIT
Using session for demo@localhost:4064. Idle timeout: 10 min. Current group: Public
Checking for prefix_dir /data/OMERO/ManagedRepository/demo_2/2016-06/27/12-57-28.592
Creating dir at /data/OMERO/ManagedRepository/demo_2/2016-06/27/12-57-28.592_mkngff
Creating symlink /data/OMERO/ManagedRepository/demo_2/2016-06/27/12-57-28.592_mkngff/ff85e5f2-258a-46ad-bdd0-d4f296aec28e.zarr -> /bia-integrator-data/S-BIAD851/ff85e5f2-258a-46ad-bdd0-d4f296aec28e/ff85e5f2-258a-46ad-bdd0-d4f296aec28e.zarr
Checking for prefix_dir /data/OMERO/ManagedRepository/demo_2/2016-06/27/12-57-28.592
write bfoptions to: /data/OMERO/ManagedRepository/demo_2/2016-06/27/12-57-28.592_mkngff/ff85e5f2-258a-46ad-bdd0-d4f296aec28e.zarr.bfoptions

will-moore · 2023-12-05T00:26:09Z

Viewing images (first 2 viewed, waiting...)
http://localhost:1040/webclient/?show=image-2330212
http://localhost:1040/webclient/?show=image-2340843
This one got an error
http://localhost:1040/webclient/?show=image-2376573

    serverExceptionClass = ome.conditions.ResourceError
    message = Error instantiating pixel buffer: /data/OMERO/ManagedRepository/demo_2/2016-06/21/01-46-55.560_mkngff/5aec8bec-8573-44ec-9e9e-24fb81623fbe.zarr/B/1/.zattrs

Done but not tried viewing yet
http://localhost:1040/webclient/?show=image-2486279
http://localhost:1040/webclient/?show=image-2131185

not done yet
http://localhost:1040/webclient/?show=image-2435591

will-moore · 2023-12-05T11:58:45Z

Looking at the last Fileset generated above 5289166, Find Image ID via psql...
Fileset doesn't have clientpath set:

last row of idr0016.csv:

idr0016/26110.ome.zarr,S-BIAD851/ff85e5f2-258a-46ad-bdd0-d4f296aec28e,21526

looking at 21526.sql...

UPDATE pixels SET name = '.zattrs', path = 'demo_2/2016-06/27/12-57-28.592_mkngff/ff85e5f2-258a-46ad-bdd0-d4f296aec28e.zarr' where image in (select id from Image where fileset = 21526);

begin;
    select mkngff_fileset(
      21526,
      'c6b02bb7-2c22-4c45-be8d-30484c380a9c',
      'cdf35825-def1-4580-8d0b-9c349b8f78d6',
      'demo_2/2016-06/27/12-57-28.592_mkngff/',
      array[
          ['demo_2/2016-06/27/12-57-28.592_mkngff/ff85e5f2-258a-46ad-bdd0-d4f296aec28e.zarr/', '.zattrs', 'application/octet-stream', 'https://uk1s3.embassy.ebi.ac.uk/bia-integrator-data/S-BIAD851/ff85e5f2-258a-46ad-bdd0-d4f296aec28e/ff85e5f2-258a-46ad-bdd0-d4f296aec28e.zarr/.zattrs'],
          ['demo_2/2016-06/27/12-57-28.592_mkngff/ff85e5f2-258a-46ad-bdd0-d4f296aec28e.zarr/', '.zgroup', 'application/octet-stream', 'https://uk1s3.embassy.ebi.ac.uk/bia-integrator-data/S-BIAD851/ff85e5f2-258a-46ad-bdd0-d4f296aec28e/ff85e5f2-258a-46ad-bdd0-d4f296aec28e.zarr/.zgroup'],
...

Ah!!! - I forgot to update and run setup.sql which creates the mkngff_fileset() sql function!

will-moore · 2023-12-05T13:42:21Z

checking error above for http://localhost:1040/webclient/?show=image-2376573
Fileset ID: 5288909 (new)
Plate name: 25675

serverExceptionClass = ome.conditions.ResourceError
    message = Error instantiating pixel buffer: /data/OMERO/ManagedRepository/demo_2/2016-06/21/01-46-55.560_mkngff/5aec8bec-8573-44ec-9e9e-24fb81623fbe.zarr/B/1/.zattrs

Other Images on the same plate have same path/name

idr=> select path, name from pixels where image=2376573;
                                        path                                         |  name   
-------------------------------------------------------------------------------------+---------
 demo_2/2016-06/21/01-46-55.560_mkngff/5aec8bec-8573-44ec-9e9e-24fb81623fbe.zarr/B/1 | .zattrs
(1 row)

idr=> select path, name from pixels where image=2378563;
                                        path                                         |  name   
-------------------------------------------------------------------------------------+---------
 demo_2/2016-06/21/01-46-55.560_mkngff/5aec8bec-8573-44ec-9e9e-24fb81623fbe.zarr/B/1 | .zattrs
(1 row)

idr0016.csv row:

idr0016/25675.ome.zarr,S-BIAD851/5aec8bec-8573-44ec-9e9e-24fb81623fbe,21385

21385.sql is missing ...zarr/.zattrs and all A/1 etc:

UPDATE pixels SET name = '.zattrs', path = 'demo_2/2016-06/21/01-46-55.560_mkngff/5aec8bec-8573-44ec-9e9e-24fb81623fbe.zarr/B/1' where image in (select id from Image where fileset = 21385);

begin;
    select mkngff_fileset(
      21385,
      'c6b02bb7-2c22-4c45-be8d-30484c380a9c',
      'cdf35825-def1-4580-8d0b-9c349b8f78d6',
      'demo_2/2016-06/21/01-46-55.560_mkngff/',
      array[
          ['demo_2/2016-06/21/01-46-55.560_mkngff/5aec8bec-8573-44ec-9e9e-24fb81623fbe.zarr/B/', '.zgroup', 'application/octet-stream', 'https://uk1s3.embassy.ebi.ac.uk/bia-integrator-data/S-BIAD851/5aec8bec-8573-44ec-9e9e-24fb81623fbe/5aec8bec-8573-44ec-9e9e-24fb81623fbe.zarr/B/.zgroup'],
          ['demo_2/2016-06/21/01-46-55.560_mkngff/5aec8bec-8573-44ec-9e9e-24fb81623fbe.zarr/B/1/', '.zattrs', 'application/octet-stream', 'https://uk1s3.embassy.ebi.ac.uk/bia-integrator-data/S-BIAD851/5aec8bec-8573-44ec-9e9e-24fb81623fbe/5aec8bec-8573-44ec-9e9e-24fb81623fbe.zarr/B/1/.zattrs'],

Invalid: https://ome.github.io/ome-ngff-validator/?source=https://uk1s3.embassy.ebi.ac.uk/bia-integrator-data/S-BIAD851/5aec8bec-8573-44ec-9e9e-24fb81623fbe/5aec8bec-8573-44ec-9e9e-24fb81623fbe.zarr

Check for other plates that have fewer files than expected:...

$ grep -c https ./*.sql
./21206.sql:14610
./21207.sql:14610
./21208.sql:14610
./21209.sql:14610

./21272.sql:5906
./21334.sql:14572
./21363.sql:14607
./21365.sql:14602
./21369.sql:14603
./21385.sql:13695
./21451.sql:1802
./21476.sql:14602
./21548.sql:14607
./21551.sql:14154

./21552.sql:14420
./21555.sql:14344
./21558.sql:574
./21559.sql:2672
./21560.sql:2101
./21561.sql:3319
./21562.sql:3242
./21563.sql:3965
./21564.sql:8643
./21565.sql:10544
./21566.sql:12672
./21567.sql:14268
./21569.sql:14268
./21570.sql:14534
./21571.sql:13774
./21572.sql:13926
./21573.sql:14306
./21574.sql:14572

./21582.sql:14344
./23603.sql:14603

will-moore · 2023-12-05T15:28:54Z

Checking http://localhost:1040/webclient/?show=image-2330212
on idr0125-pilot...
Finally gave a webclient error:

    serverExceptionClass = ome.conditions.ResourceError
    message = Error instantiating pixel buffer: /data/OMERO/ManagedRepository/demo_2/2016-06/20/05-04-12.866_mkngff/1b63ff26-391a-4b44-995c-83b7cbf5372f.zarr/.zattrs

$ grep -A 5 -B 5 "995c-83b7cbf5372f" /opt/omero/server/OMERO.server/var/log/Blitz-0.log
...
2023-12-05 14:02:34,787 INFO  [ome.services.sessions.state.SessionCache] (2-thread-5) Synchronization took 71 ms.
2023-12-05 14:02:36,183 DEBUG [                   loci.formats.Memoizer] (l.Server-1) start[1701784193425] time[762757] tag[loci.formats.Memoizer.setId]
2023-12-05 14:02:36,185 ERROR [         ome.io.bioformats.BfPixelBuffer] (l.Server-1) Failed to instantiate BfPixelsWrapper with /data/OMERO/ManagedRepository/demo_2/2016-06/20/05-04-12.866_mkngff/1b63ff26-391a-4b44-995c-83b7cbf5372f.zarr/.zattrs
2023-12-05 14:02:36,187 ERROR [                ome.io.nio.PixelsService] (l.Server-1) Error instantiating pixel buffer: /data/OMERO/ManagedRepository/demo_2/2016-06/20/05-04-12.866_mkngff/1b63ff26-391a-4b44-995c-83b7cbf5372f.zarr/.zattrs
java.lang.RuntimeException: java.io.IOException: '.zgroup' expected but is not readable or missing in store.
	at ome.io.bioformats.BfPixelBuffer.reader(BfPixelBuffer.java:79)
	at ome.io.bioformats.BfPixelBuffer.setSeries(BfPixelBuffer.java:124)
	at ome.io.nio.PixelsService.createBfPixelBuffer(PixelsService.java:898)
	at ome.io.nio.PixelsService._getPixelBuffer(PixelsService.java:653)

Checking ALL Wells at https://ome.github.io/ome-ngff-validator/?source=https%3A%2F%2Fuk1s3.embassy.ebi.ac.uk%2Fbia-integrator-data%2FS-BIAD851%2F1b63ff26-391a-4b44-995c-83b7cbf5372f%2F1b63ff26-391a-4b44-995c-83b7cbf5372f.zarr&well=all
Finds 2 Images invalid:
https://ome.github.io/ome-ngff-validator/?source=https://uk1s3.embassy.ebi.ac.uk/bia-integrator-data/S-BIAD851/1b63ff26-391a-4b44-995c-83b7cbf5372f/1b63ff26-391a-4b44-995c-83b7cbf5372f.zarr/K/24/4/
missing...zarray
Error Loading https://uk1s3.embassy.ebi.ac.uk/bia-integrator-data/S-BIAD851/1b63ff26-391a-4b44-995c-83b7cbf5372f/1b63ff26-391a-4b44-995c-83b7cbf5372f.zarr/K/24/4/2/.zarray: Not Found

Missing .zattrs at

https://ome.github.io/ome-ngff-validator/?source=https://uk1s3.embassy.ebi.ac.uk/bia-integrator-data/S-BIAD851/1b63ff26-391a-4b44-995c-83b7cbf5372f/1b63ff26-391a-4b44-995c-83b7cbf5372f.zarr/K/24/5/

https://uk1s3.embassy.ebi.ac.uk/bia-integrator-data/S-BIAD851/1b63ff26-391a-4b44-995c-83b7cbf5372f/1b63ff26-391a-4b44-995c-83b7cbf5372f.zarr/K/24/5/.zattrs: Not Found```

will-moore · 2023-12-05T15:46:24Z

http://localhost:1040/webclient/?show=image-2340843

idr0016.csv

idr0016/25592.ome.zarr,S-BIAD851/481a1ae0-a492-451f-bf7d-fa2c106c7cf5,21369

(base) [wmoore@pilot-idr0125-omeroreadwrite ~]$ grep -A 5 -B 5 "fa2c106c7cf5" /opt/omero/server/OMERO.server/var/log/Blitz-0.log

--
2023-12-05 12:50:45,581 INFO  [         ome.security.basic.EventHandler] (l.Server-1)  Auth:	user=52,group=3,event=null(User),sess=3e1880d1-c38b-46b1-8e4c-460886f220fb
2023-12-05 12:50:45,586 INFO  [                 org.perf4j.TimingLogger] (l.Server-1) start[1701780645579] time[6] tag[omero.call.success.ome.logic.AdminImpl.lookupGroups]
2023-12-05 12:50:45,586 INFO  [        ome.services.util.ServiceHandler] (l.Server-1)  Rslt:	(ome.model.meta.ExperimenterGroup:Id_0, ome.model.meta.ExperimenterGroup:Id_1, ome.model.meta.ExperimenterGroup:Id_2, ... 3 more)
2023-12-05 12:50:55,948 DEBUG [                   loci.formats.Memoizer] (l.Server-6) start[1701780110386] time[545562] tag[loci.formats.Memoizer.setId]
2023-12-05 12:50:55,949 DEBUG [                   loci.formats.Memoizer] (l.Server-0) start[1701780107265] time[548684] tag[loci.formats.Memoizer.setId]
2023-12-05 12:50:55,950 ERROR [         ome.io.bioformats.BfPixelBuffer] (l.Server-6) Failed to instantiate BfPixelsWrapper with /data/OMERO/ManagedRepository/demo_2/2016-06/20/10-44-15.650_mkngff/481a1ae0-a492-451f-bf7d-fa2c106c7cf5.zarr/.zattrs
2023-12-05 12:50:55,950 DEBUG [                   loci.formats.Memoizer] (l.Server-8) start[1701779997653] time[658297] tag[loci.formats.Memoizer.setId]
2023-12-05 12:50:55,950 DEBUG [                   loci.formats.Memoizer] (l.Server-7) start[1701780089348] time[566601] tag[loci.formats.Memoizer.setId]
2023-12-05 12:50:55,950 ERROR [         ome.io.bioformats.BfPixelBuffer] (l.Server-0) Failed to instantiate BfPixelsWrapper with /data/OMERO/ManagedRepository/demo_2/2016-06/20/10-44-15.650_mkngff/481a1ae0-a492-451f-bf7d-fa2c106c7cf5.zarr/.zattrs
2023-12-05 12:50:55,950 ERROR [         ome.io.bioformats.BfPixelBuffer] (l.Server-8) Failed to instantiate BfPixelsWrapper with /data/OMERO/ManagedRepository/demo_2/2016-06/20/10-44-15.650_mkngff/481a1ae0-a492-451f-bf7d-fa2c106c7cf5.zarr/.zattrs
2023-12-05 12:50:55,950 ERROR [         ome.io.bioformats.BfPixelBuffer] (l.Server-7) Failed to instantiate BfPixelsWrapper with /data/OMERO/ManagedRepository/demo_2/2016-06/20/10-44-15.650_mkngff/481a1ae0-a492-451f-bf7d-fa2c106c7cf5.zarr/.zattrs
2023-12-05 12:50:55,951 ERROR [                ome.io.nio.PixelsService] (l.Server-6) Error instantiating pixel buffer: /data/OMERO/ManagedRepository/demo_2/2016-06/20/10-44-15.650_mkngff/481a1ae0-a492-451f-bf7d-fa2c106c7cf5.zarr/.zattrs
java.lang.RuntimeException: java.io.IOException: '.zgroup' expected but is not readable or missing in store.
	at ome.io.bioformats.BfPixelBuffer.reader(BfPixelBuffer.java:79)
	at ome.io.bioformats.BfPixelBuffer.setSeries(BfPixelBuffer.java:124)
	at ome.io.nio.PixelsService.createBfPixelBuffer(PixelsService.java:898)
	at ome.io.nio.PixelsService._getPixelBuffer(PixelsService.java:653)
--

1 Well from the whole plate has errors:
2 Files missing at
https://ome.github.io/ome-ngff-validator/?source=https://uk1s3.embassy.ebi.ac.uk/bia-integrator-data/S-BIAD851/481a1ae0-a492-451f-bf7d-fa2c106c7cf5/481a1ae0-a492-451f-bf7d-fa2c106c7cf5.zarr/J/12/4/

1 at
https://ome.github.io/ome-ngff-validator/?source=https://uk1s3.embassy.ebi.ac.uk/bia-integrator-data/S-BIAD851/481a1ae0-a492-451f-bf7d-fa2c106c7cf5/481a1ae0-a492-451f-bf7d-fa2c106c7cf5.zarr/J/12/5/

will-moore · 2023-12-05T16:11:51Z

http://localhost:1040/webclient/?show=image-2486279

idr0016.csv

idr0016/25909.ome.zarr,S-BIAD851/382abcd4-404b-4796-bbe2-b02fedcbb015,21476

Found 1 Well with errors:

Missing 3 files from
https://ome.github.io/ome-ngff-validator/?source=https://uk1s3.embassy.ebi.ac.uk/bia-integrator-data/S-BIAD851/382abcd4-404b-4796-bbe2-b02fedcbb015/382abcd4-404b-4796-bbe2-b02fedcbb015.zarr/A/5/2/

and at least 1 from
https://ome.github.io/ome-ngff-validator/?source=https://uk1s3.embassy.ebi.ac.uk/bia-integrator-data/S-BIAD851/382abcd4-404b-4796-bbe2-b02fedcbb015/382abcd4-404b-4796-bbe2-b02fedcbb015.zarr/A/5/3/

will-moore · 2023-12-05T16:21:36Z

http://localhost:1040/webclient/?show=image-2131185

idr0016/24634.ome.zarr,S-BIAD851/22803be2-9732-41ab-b9bf-fcd37c3b3b84,21278

One Well invalid:
https://ome.github.io/ome-ngff-validator/?source=https://uk1s3.embassy.ebi.ac.uk/bia-integrator-data/S-BIAD851/22803be2-9732-41ab-b9bf-fcd37c3b3b84/22803be2-9732-41ab-b9bf-fcd37c3b3b84.zarr/K/19/

2 missing files:
https://ome.github.io/ome-ngff-validator/?source=https://uk1s3.embassy.ebi.ac.uk/bia-integrator-data/S-BIAD851/22803be2-9732-41ab-b9bf-fcd37c3b3b84/22803be2-9732-41ab-b9bf-fcd37c3b3b84.zarr/K/19/4/

1 missing .zattrs
https://ome.github.io/ome-ngff-validator/?source=https://uk1s3.embassy.ebi.ac.uk/bia-integrator-data/S-BIAD851/22803be2-9732-41ab-b9bf-fcd37c3b3b84/22803be2-9732-41ab-b9bf-fcd37c3b3b84.zarr/K/19/5/

will-moore · 2023-12-05T16:27:52Z

http://localhost:1040/webclient/?show=image-2435591

idr0016/25740.ome.zarr,S-BIAD851/f8583882-cfa6-4409-b452-50b849b1397e,21454

1 Well invalid:

https://ome.github.io/ome-ngff-validator/?source=https://uk1s3.embassy.ebi.ac.uk/bia-integrator-data/S-BIAD851/f8583882-cfa6-4409-b452-50b849b1397e/f8583882-cfa6-4409-b452-50b849b1397e.zarr/L/10/

3 Files missing:
https://ome.github.io/ome-ngff-validator/?source=https://uk1s3.embassy.ebi.ac.uk/bia-integrator-data/S-BIAD851/f8583882-cfa6-4409-b452-50b849b1397e/f8583882-cfa6-4409-b452-50b849b1397e.zarr/L/10/3/

.zattrs missing
https://ome.github.io/ome-ngff-validator/?source=https://uk1s3.embassy.ebi.ac.uk/bia-integrator-data/S-BIAD851/f8583882-cfa6-4409-b452-50b849b1397e/f8583882-cfa6-4409-b452-50b849b1397e.zarr/L/10/4/

will-moore · 2023-12-05T21:45:49Z

Need to re-export broken plates...

Plate name: 25585 ID: 5065
Plate name: 25592 ID: 5069
Plate name: 25675 ID: 5085
Plate name: 25909 ID: 5176
Plate name: 24634 ID: 4978
Plate name: 25740 ID: 5154

for id in 5065 5069 5085 5176 4978 5154; do
  echo $id;
  omero zarr export Plate:$id --name_by name;
done

EDIT: 6th Dec. zipping 6 plates...

for i in */; do zip -r "${i%/}.zip" "$i"; done

will-moore · 2023-12-06T14:30:13Z

(base) [wmoore@idrftp-ftp idr0004]$ sudo /root/.aspera/cli/bin/ascp -P33001 -i /root/.aspera/cli/etc/asperaweb_id_dsa.openssh -d /data/idr0016/idr0016/ [email protected]:5f/1xxxxxxxxxx      
24634.ome.zarr.zip                                                                      100% 5683MB 94.0Mb/s    06:44
25585.ome.zarr.zip                                                                      100% 5986MB  162Mb/s    13:45
25592.ome.zarr.zip                                                                      100% 6090MB  162Mb/s    20:41
25675.ome.zarr.zip                                                                       100% 5759MB 67.7Mb/s    27:13
25740.ome.zarr.zip                                                                       100% 5695MB  147Mb/s    33:29    
25909.ome.zarr.zip                                                                      100% 5949MB  244Mb/s    38:58    
Completed: 36009422K bytes transferred in 2338 seconds
 (126141K bits/sec), in 6 files, 1 directory.

will-moore · 2023-12-19T12:38:16Z

Checking https://uk1s3.embassy.ebi.ac.uk/bia-integrator-data/pages/S-BIAD851.html
Comparing with rows removed from idr0016.csv at IDR/idr-utils@125c4e5

Unchanged:
idr0016/24634.ome.zarr K19 still missing:
https://ome.github.io/ome-ngff-validator/?source=https%3A%2F%2Fuk1s3.embassy.ebi.ac.uk%2Fbia-integrator-data%2FS-BIAD851%2F22803be2-9732-41ab-b9bf-fcd37c3b3b84%2F22803be2-9732-41ab-b9bf-fcd37c3b3b84.zarr&well=all

idr0016/25585 https://ome.github.io/ome-ngff-validator/?source=https://uk1s3.embassy.ebi.ac.uk/bia-integrator-data/S-BIAD851/1b63ff26-391a-4b44-995c-83b7cbf5372f/1b63ff26-391a-4b44-995c-83b7cbf5372f.zarr

...

will-moore · 2024-01-04T14:07:31Z

Let's upload 7 plates to uk1s3 for testing and mkngff...
First command timed-out on last 2 plates - re-ran those 2...

(base) [wmoore@idrftp-ftp ~]$ ls /data/idr0016/idr0016
24634.ome.zarr  24667.ome.zarr  25585.ome.zarr  25592.ome.zarr  25675.ome.zarr  25740.ome.zarr  25909.ome.zarr

./mc cp -r /data/idr0016/idr0016/ uk1s3/idr0016/zarr
.../idr0016/25675.ome.zarr/K/3/5/2/2/0/0: 29.50 GiB / 30.50 GiB ━━━━━━━━━━━━ 18.09 MiB/s

(base) [wmoore@idrftp-ftp ~]$ ./mc cp -r /data/idr0016/idr0016/25740.ome.zarr/ uk1s3/idr0016/zarr/25740.ome.zarr
...6/25740.ome.zarr/P/9/5/3/4/0/0: 6.27 GiB / 6.27 GiB ━━━━━━━━━━━━━━━━ 19.02 MiB/s 5m37s

(base) [wmoore@idrftp-ftp ~]$ ./mc cp -r /data/idr0016/idr0016/25909.ome.zarr/ uk1s3/idr0016/zarr/25909.ome.zarr
...6/25909.ome.zarr/P/9/5/3/4/0/0: 6.42 GiB / 6.42 GiB ━━━━━━━━━━━━━ 16.89 MiB/s 6m29s

on idr0125-pilot..

sudo mkdir /idr0016 && sudo /opt/goofys --endpoint https://uk1s3.embassy.ebi.ac.uk/ -o allow_other idr0016 /idr0016

(base) [wmoore@pilot-idr0125-omeroreadwrite ~]$ ls /idr0016/zarr/
24279.ome.zarr  24634.ome.zarr  24667.ome.zarr  25585.ome.zarr  25592.ome.zarr  25675.ome.zarr  25740.ome.zarr

Validated all Wells in the 7 Plates, e.g. https://ome.github.io/ome-ngff-validator/?source=https%3A%2F%2Fuk1s3.embassy.ebi.ac.uk%2Fidr0016%2Fzarr%2F25909.ome.zarr&well=all

will-moore · 2024-01-04T15:31:23Z

idr0016.csv based on IDR/idr-utils@125c4e5

idr0016/25585.ome.zarr,25585.ome.zarr,21365
idr0016/24634.ome.zarr,24634.ome.zarr,21278
idr0016/24667.ome.zarr,24667.ome.zarr,23603
idr0016/25909.ome.zarr,25909.ome.zarr,21476
idr0016/25592.ome.zarr,25592.ome.zarr,21369
idr0016/25675.ome.zarr,25675.ome.zarr,21385
idr0016/25740.ome.zarr,25740.ome.zarr,21454

On idr0125-pilot, logged-in to idr.openmicroscopy.org...

for r in $(cat $IDRID.csv); do
  platename=$(echo $r | cut -d',' -f2)
  fsid=$(echo $r | cut -d',' -f3 | tr -d '[:space:]')
  omero mkngff sql $fsid --clientpath="https://uk1s3.embassy.ebi.ac.uk/idr0016/zarr/$platename" "/idr0016/zarr/$platename" > "$IDRID/$fsid.sql"
done

(venv3) (base) [wmoore@pilot-idr0125-omeroreadwrite ~]$ for i in $(ls idr0016); do grep -c ".zattrs" idr0016/$i; done
2690
2690
2690
2690
2690
2690
2690

for i in $(ls); do sed -i 's/SECRETUUID/9630ba1e-ed3a-42e3-9296-59ccf23a7039/g' $i; done

for r in $(cat $IDRID.csv); do
  platename=$(echo $r | cut -d',' -f2)
  fsid=$(echo $r | cut -d',' -f3 | tr -d '[:space:]')
  psql -U omero -d idr -h $DBHOST -f "$IDRID/$fsid.sql"
  omero mkngff symlink /data/OMERO/ManagedRepository $fsid "/idr0016/zarr/$platename" --bfoptions
done

Updated in IDR/mkngff_upgrade_scripts@0ba84be

We can't test mkngff on idr0125-pilot as Filesets have already been updated and IDs no longer valid.
Use idr0138-pilot instead...

will-moore · 2024-01-04T16:04:14Z

On idr0138-pilot...

As root, update...

pip install 'omero-mkngff @ git+https://github.com/will-moore/omero-mkngff@bfoptions'

Get sql and update...

cd /uod/idr/metadata
sudo -Es git clone https://github.com/IDR/mkngff_upgrade_scripts.git
cd mkngff_upgrade_scripts/ngff_filesets/idr0016
for i in $(ls); do sudo sed -i 's/SECRETUUID/de86d158-d945-4256-a8c0-de34d689ca0c/g' $i; done

# csv content from above...
sudo vi idr0016_7.csv

As omero-server...

omero login
omero mkngff setup > setup.sql
psql -U omero -d idr -h $DBHOST -f setup.sql

cd /uod/idr/metadata/mkngff_upgrade_scripts/ngff_filesets/
for r in $(cat idr0016_7.csv); do
  platename=$(echo $r | cut -d',' -f2)
  fsid=$(echo $r | cut -d',' -f3 | tr -d '[:space:]')
  psql -U omero -d idr -h $DBHOST -f "$IDRID/$fsid.sql"
  omero mkngff symlink /data/OMERO/ManagedRepository $fsid "/idr0016/zarr/$platename" --bfoptions
done

UPDATE 2304
BEGIN
 mkngff_fileset 
----------------
        5815205
(1 row)

COMMIT
Using session for demo@localhost:4064. Idle timeout: 10 min. Current group: Public
Checking for prefix_dir /data/OMERO/ManagedRepository/demo_2/2016-06/20/05-04-12.866
Creating dir at /data/OMERO/ManagedRepository/demo_2/2016-06/20/05-04-12.866_mkngff
Creating symlink /data/OMERO/ManagedRepository/demo_2/2016-06/20/05-04-12.866_mkngff/25585.ome.zarr -> /idr0016/zarr/25585.ome.zarr
Checking for prefix_dir /data/OMERO/ManagedRepository/demo_2/2016-06/20/05-04-12.866
write bfoptions to: /data/OMERO/ManagedRepository/demo_2/2016-06/20/05-04-12.866_mkngff/25585.ome.zarr.bfoptions
UPDATE 2304
BEGIN
...

will-moore · 2024-01-16T22:54:59Z

On idr-testing...
idr0016.csv:

idr0016/25585.ome.zarr,S-BIAD851/da7853ca-2f99-4b46-b159-00371ffbe4c3,6313495
idr0016/24634.ome.zarr,S-BIAD851/cd887d08-ea47-489f-ab2d-039a64af938b,6313510
idr0016/24667.ome.zarr,S-BIAD851/adbbfaf5-5ac4-48f1-acf7-3180b3a6445d,6313525
idr0016/25909.ome.zarr,S-BIAD851/076d5d53-e59f-40cd-bae9-c914c7a1ae26,6313540
idr0016/25592.ome.zarr,S-BIAD851/539f798f-99c9-4351-a4ca-35c6ad4f2d22,6313570
idr0016/25675.ome.zarr,S-BIAD851/312f9dbd-eaec-4993-b0c5-95c9db5a0248,6313600
idr0016/25740.ome.zarr,S-BIAD851/759823ab-291d-460b-bfca-9f8903e50631,6313840

export IDRID=idr0016
for r in $(cat $IDRID.csv); do
  biapath=$(echo $r | cut -d',' -f2)
  uuid=$(echo $biapath | cut -d'/' -f2)
  fsid=$(echo $r | cut -d',' -f3 | tr -d '[:space:]')
  omero mkngff sql $fsid --fs_suffix=None --clientpath="https://uk1s3.embassy.ebi.ac.uk/bia-integrator-data/$biapath/$uuid.zarr" "/bia-integrator-data/$biapath/$uuid.zarr" > "$IDRID/$fsid.sql"
done

for i in $(ls); do sed -i 's/SECRETUUID/f464e059-16b5-4013-b9a2-417e5976371c/g' $i; done

for r in $(cat $IDRID.csv); do
  biapath=$(echo $r | cut -d',' -f2)
  uuid=$(echo $biapath | cut -d'/' -f2)
  fsid=$(echo $r | cut -d',' -f3 | tr -d '[:space:]')
  psql -U omero -d idr -h $DBHOST -f "$IDRID/$fsid.sql"
  omero mkngff symlink /data/OMERO/ManagedRepository $fsid "/bia-integrator-data/$biapath/$uuid.zarr" --fs_suffix=None --bfoptions
done
...

UPDATE 2304
BEGIN
 mkngff_fileset 
----------------
        6314910
(1 row)
COMMIT
Using session for demo@localhost:4064. Idle timeout: 10 min. Current group: Public
Checking for prefix_dir /data/OMERO/ManagedRepository/demo_2/2016-06/24/16-32-36.710_mkngff
Creating dir at /data/OMERO/ManagedRepository/demo_2/2016-06/24/16-32-36.710_mkngff
Creating symlink /data/OMERO/ManagedRepository/demo_2/2016-06/24/16-32-36.710_mkngff/759823ab-291d-460b-bfca-9f8903e50631.zarr -> /bia-integrator-data/S-BIAD851/759823ab-291d-460b-bfca-9f8903e50631/759823ab-291d-460b-bfca-9f8903e50631.zarr
Checking for prefix_dir /data/OMERO/ManagedRepository/demo_2/2016-06/24/16-32-36.710_mkngff
write bfoptions to: /data/OMERO/ManagedRepository/demo_2/2016-06/24/16-32-36.710_mkngff/759823ab-291d-460b-bfca-9f8903e50631.zarr.bfoptions

dominikl moved this to test convert in NGFF conversion Feb 22, 2023

dominikl added this to NGFF conversion Feb 22, 2023

dominikl mentioned this issue Feb 27, 2023

idr0016-wawer-bioactivecompoundprofiling to NGFF #646

Closed

dominikl added the bug label Mar 6, 2023

sbesson self-assigned this Apr 29, 2023

sbesson mentioned this issue May 18, 2023

idr0036-gustafsdottir-cellpainting S-BIAD855 #640

Open

sbesson removed their assignment Jun 12, 2023

will-moore added the question label Jun 16, 2023

will-moore removed bug question labels Jul 10, 2023

will-moore self-assigned this Jul 10, 2023

will-moore moved this from test convert to convert all data to NGFF in NGFF conversion Jul 10, 2023

will-moore moved this from check_pixels to check_pixels in progress in NGFF conversion Dec 4, 2023

will-moore moved this from check_pixels in progress to check_pixels in NGFF conversion Dec 6, 2023

will-moore added the resubmitted to EBI label Dec 6, 2023

will-moore removed the resubmitted to EBI label Jan 16, 2024

will-moore moved this from check_pixels to NGFF studies in NGFF conversion May 21, 2024

idr0016-wawer-bioactivecompoundprofiling S-BIAD851 #638

idr0016-wawer-bioactivecompoundprofiling S-BIAD851 #638

Comments

dominikl commented Feb 22, 2023

sbesson commented May 29, 2023

will-moore commented Jul 10, 2023

will-moore commented Jul 10, 2023 • edited Loading

will-moore commented Jul 10, 2023

will-moore commented Jul 11, 2023

will-moore commented Jul 11, 2023

will-moore commented Jul 11, 2023

will-moore commented Jul 11, 2023

will-moore commented Jul 11, 2023

will-moore commented Jul 11, 2023 • edited Loading

will-moore commented Jul 12, 2023

will-moore commented Jul 12, 2023

will-moore commented Jul 12, 2023

will-moore commented Jul 12, 2023 • edited Loading

will-moore commented Jul 12, 2023

will-moore commented Jul 12, 2023

will-moore commented Jul 13, 2023

will-moore commented Jul 13, 2023

will-moore commented Sep 22, 2023

will-moore commented Sep 22, 2023 • edited Loading

will-moore commented Sep 22, 2023 • edited Loading

will-moore commented Sep 25, 2023 • edited Loading

will-moore commented Sep 25, 2023

will-moore commented Sep 27, 2023 • edited Loading

will-moore commented Sep 27, 2023

will-moore commented Nov 15, 2023 • edited Loading

will-moore commented Nov 15, 2023 • edited Loading

will-moore commented Dec 4, 2023 • edited Loading

will-moore commented Dec 5, 2023

will-moore commented Dec 5, 2023

will-moore commented Dec 5, 2023

will-moore commented Dec 5, 2023 • edited Loading

will-moore commented Dec 5, 2023

will-moore commented Dec 5, 2023

will-moore commented Dec 5, 2023

will-moore commented Dec 5, 2023

will-moore commented Dec 5, 2023 • edited Loading

will-moore commented Dec 6, 2023 • edited Loading

will-moore commented Dec 19, 2023

will-moore commented Jan 4, 2024 • edited Loading

will-moore commented Jan 4, 2024

will-moore commented Jan 4, 2024

will-moore commented Jan 16, 2024 • edited Loading

will-moore commented Jul 10, 2023 •

edited

Loading

will-moore commented Jul 11, 2023 •

edited

Loading

will-moore commented Jul 12, 2023 •

edited

Loading

will-moore commented Sep 22, 2023 •

edited

Loading

will-moore commented Sep 22, 2023 •

edited

Loading

will-moore commented Sep 25, 2023 •

edited

Loading

will-moore commented Sep 27, 2023 •

edited

Loading

will-moore commented Nov 15, 2023 •

edited

Loading

will-moore commented Nov 15, 2023 •

edited

Loading

will-moore commented Dec 4, 2023 •

edited

Loading

will-moore commented Dec 5, 2023 •

edited

Loading

will-moore commented Dec 5, 2023 •

edited

Loading

will-moore commented Dec 6, 2023 •

edited

Loading

will-moore commented Jan 4, 2024 •

edited

Loading

will-moore commented Jan 16, 2024 •

edited

Loading