idr0033-rohban-pathways S-BIAD848 #649
Comments
|
Reimport still in progress on pilot-idrtesting (since 28th Feb...) |
|
Incidentally, I came across this conversion as part of a separate body of work. With the help of @melissalinkert, a workaround was found to inject additional plate metadata, let this data be recognized via the MetaXpress reader and use On the pilot with SSD drive If that's useful for the ongoing work, happy to upload the NGFF dataset for the first plate into a temporary |
|
https://ome.github.io/ome-ngff-validator/?source=https://uk1s3.embassy.ebi.ac.uk/idr0033/zarr/41744.zarr - currently partially uploaded |
|
For reference, this is the script that was used for symlinking the source TIFFs and renaming them according to the pattern expected by the reader import os
import re
pattern=r'(.*)(_[a-p]\d\d)(.*)'
sourcedir='/uod/idr/filesets/idr0033-rohban-pathways/20170214-original/'
targetdir='/data/idr0033/sources/'
plates = ['41744', '41749', '41754', '41755', '41756', '41757']
types = ['images', 'images_illum_corrected']
def replace(s):
return s.groups()[0]+ s.groups()[1].upper()+s.groups()[2]
for type in types:
for plate in plates:
source_plate = os.path.join(sourcedir, type, plate)
target_plate = os.path.join(targetdir, type, plate)
tiffs = os.listdir(source_plate)
for tiff in tiffs:
new_tiff = re.sub(pattern, replace, tiff)
os.symlink(os.path.join(source_plate, tiff), os.path.join(target_plate, new_tiff))and the content of the metadata file which must be saved as With the above, Bio-Formats correctly detects the plate as MetaXpress and the |
|
Started import of metadata-only plate on idr0125-pilot - uploading nearly 4 hours... - Import took > 8 hours. |
Plate Images are viewable on pilot-idr0125: http://localhost:1080/webclient/?show=plate-10388 |
|
@sbesson - you're currently converting the rest of the plates and uploading to s3 bucket? I'll update the status of this issue... |
will-moore
moved this from re-import test image
to convert all data to NGFF
in NGFF conversion
|
Yes converting and updating the checklist as a new dataset is uploaded to the bucket |
sbesson
moved this from convert all data to NGFF
to upload data to s3
in NGFF conversion
will-moore
moved this from upload data to s3
to create new Fileset to replace original Fileset
in NGFF conversion
|
Checked all plates on s3 with ome-ngff-validator 👍 |
will-moore
moved this from create new Fileset to replace original Fileset
to Zip and upload to BioStudies
in NGFF conversion
will-moore
moved this from Zip and upload to BioStudies
to upload some data to s3 and test
in NGFF conversion
will-moore
moved this from upload some data to s3 and test
to Zip and upload to BioStudies
in NGFF conversion
|
@sbesson Where are the |
|
On Ready for the next phase and the upload |
will-moore
moved this from Zip and upload to BioStudies
to BioStudies Submission
in NGFF conversion
francesw
moved this from BioStudies Submission
to Data on Embassy s3
in NGFF conversion
francesw
changed the title
francesw
moved this from Data on Embassy s3
to BioStudies Submission
in NGFF conversion
francesw
moved this from BioStudies Submission
to Data on Embassy s3
in NGFF conversion
francesw
moved this from Data on Embassy s3
to BioStudies Submission
in NGFF conversion
will-moore
moved this from BioStudies Submission
to Data on Embassy s3
in NGFF conversion
francesw
moved this from Data on Embassy s3
to BioStudies Submission
in NGFF conversion
francesw
moved this from BioStudies Submission
to Data on Embassy s3
in NGFF conversion
|
Testing |
|
After about 50 mins, only the first 2 Filesets are done, 3rd in progress (lots of files to walk in these Plates)... Took nearly 2 hours in total - about 20 mins per Fileset. |
But this certainly has the correct secret: |
|
As @sbesson noted, I have the wrong host in that psql query for secret! |
|
https://uk1s3.embassy.ebi.ac.uk/bia-integrator-data/pages/S-BIAD848.html forgot to update UPDATE: after "19 hours" images are still not visible (checking plate |
|
Still checking http://localhost:1080/webclient/?show=image-3201293 and no image showing yet after > 24 hours |
will-moore
moved this from Data on Embassy s3
to create new Filesets in idr-next
in NGFF conversion
|
Testing on idr-testing:omeroreadwrite with todays OMEZarrReader.jar including ome/ZarrReader#64 Started mkngff on 12 Plates (see IDR/idr-utils@504b9d9) at 13:09... |
|
All sql generated overnight... last one completed at |
|
After sql update, viewing Image in first plate to init memo file... http://localhost:1080/webclient/?show=image-3191231 |
Memo file creation (black images) took |
will-moore
moved this from check_pixels in progress
to pixels validated
in NGFF conversion
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Conversion
The text was updated successfully, but these errors were encountered: