Adding New Acquisition Software, ThorImage

[Paschas]

ThorImage: New Element-Calcium Acquisition Software

Problem

Analyze 2-photon calcium imaging data that were produced by ThorImage of ThorsLab. Until now its not an available option.

Requirements

Some functions and make methods would need to change.

Justification

This new feature will be a great addition to the element and expand on lab-to-lab compatibility and will motivate labs that use thorsimage software to adopt Datajoint's Element-calcium pipeline

Alternative Considerations

So far i try to expand the make method of the ScanInfo table (located in scan.py)

@schema
class ScanInfo(dj.Imported):

         class Field(dj.Part):....

         class ScanFile(dj.Part):....

    def make(self, key):
        """Populate the ScanInfo with the information parsed from image files."""

        acq_software = (Scan & key).fetch1("acq_software")
        scan_filepaths = get_calcium_imaging_files(key, acq_software)

        if acq_software == "ScanImage": ....
        elif acq_software == "Scanbox": .....
        elif acq_software == "NIS": .....
        elif acq_software == "PrairieView": ....
        elif acq_software == "ThorImage":
            import utils2p
            import os
            # Find the xml file of the given folder
            relative_path = os.path.dirname(scan_filepaths[0])
            meta = utils2p.find_metadata_file(relative_path)
            # Class for managing ThorImage metadata.
            metadata = utils2p.Metadata(meta)
            is_multiROI = False  # MultiROI to be implemented later

            self.insert1(
                dict(
                    key,
                    nfields= 1, # metadata.get_metadata_value('ZStage','frames') ,
                    nchannels = metadata.get_n_channels(),
                    # Number of scanning depths = (planes)
                    ndepths= metadata.get_metadata_value('ZStage','zStreamFrames'), # PVScan_info["num_planes"],
                    nframes=metadata.get_metadata_value('Streaming','frames'),
                    nrois= 1,
                    x= metadata.get_metadata_value('Sample','initialStageLocationX'), #PVScan_info["x_pos"],
                    y= metadata.get_metadata_value('Sample','initialStageLocationY'), #PVScan_info["y_pos"],
                    fps = metadata.get_metadata_value("LSM","frameRate"),
                    bidirectional= False,
                    #usecs_per_line= metadata.get_dwell_time(), # PVScan_info["usecs_per_line"],
                    scan_datetime =  metadata.get_date_time(),
                    scan_duration = float(metadata.get_metadata_value('Streaming','frames'))/float(metadata.get_metadata_value('LSM','frameRate')),
                )
            )

            self.Field.insert(
                dict(
                    key,
                    field_idx= 1,
                    px_height = metadata.get_num_y_pixels(), 
                    px_width = metadata.get_num_x_pixels(),
                    um_height= metadata.get_metadata_value('LSM','heightUM'), #PVScan_info["height_in_um"]
                    um_width = metadata.get_width(), # PVScan_info["width_in_um"]
                    field_x= metadata.get_metadata_value('LSM','fineOffsetX'), # PVScan_info["fieldX"], 
                    field_y= metadata.get_metadata_value('LSM','fineOffsetY'), # PVScan_info["fieldY"],

        # Insert file(s)
        root_dir = find_root_directory(get_imaging_root_data_dir(), scan_filepaths[0])

        scan_files = [
            pathlib.Path(f).relative_to(root_dir).as_posix() for f in scan_filepaths
        ]
        self.ScanFile.insert([{**key, "file_path": f} for f in scan_files])

To get the ThorImage metadata i use the module utils2p.

I also altered the get_calcium_imaging_files(scan_key, acq_software: str) from tutirial_pipeline.py

def get_calcium_imaging_files(scan_key, acq_software: str):
    print("So far so good")
    """Retrieve the list of absolute paths of the calcium imaging files associated with a given Scan and a given acquisition software (e.g. "ScanImage", "PrairieView", etc.)."""
    # Folder structure: root / subject / session / .tif or .sbx or .nd2
    session_dir = element_interface.utils.find_full_path(
        get_imaging_root_data_dir(),
        (session.SessionDirectory & scan_key).fetch1("session_dir"),
    )

    if acq_software == "ScanImage":
        filepaths = [fp.as_posix() for fp in session_dir.glob("*.tif")]
    elif acq_software == "Scanbox":
        filepaths = [fp.as_posix() for fp in session_dir.glob("*.sbx")]
    elif acq_software == "NIS":
        filepaths = [fp.as_posix() for fp in session_dir.glob("*.nd2")]
    elif acq_software == "PrairieView":
        filepaths = [fp.as_posix() for fp in session_dir.glob("*.tif")]
    elif acq_software == "ThorImage":
        filepaths = [fp.as_posix() for fp in session_dir.glob("*0.tif")]
    else:
        raise NotImplementedError(f"{acq_software} is not implemented")

    if not filepaths:
        raise FileNotFoundError(f"No {acq_software} file found in {session_dir}")
    return filepaths

Disclaimer: My alterations are not based on efficiency, instead i try to alter the original code as less as possible

Additional Research and Context

utils2p (1) :

"This module provides utility functions necessary for loading and processing 2-photon imaging data acquired with ThorLabs microscopes. It includes function to read files generated by ThorImage and ThorSync."