23.2.0a3: `template_iterator_wf.select_xfm` failing with `MNIPediatricAsym` when cohort specified #407

smeisler · 2023-12-15T13:53:12Z

What happened?

select_xfm is failing on

What command did you use?

singularity run -e --containall -B ${scratch},${cache_dir}\
${fmriprep_img} $scratch/data $scratch/data/derivatives/fmriprep participant \
--participant_label ${subject:4} \
-w $scratch \
--fs-license-file ${scratch}/data/code/license.txt \
--fs-subjects-dir $scratch/data/derivatives/freesurfer/ \
--cifti-output 91k \
--skip-bids-validation \
--stop-on-first-crash \
--nprocs $nprocs \
--omp-nthreads $omp \
--notrack \
--mem-mb ${mem_mb} \
--output-spaces MNIPediatricAsym:cohort-4:res-2 \
--fd-spike-threshold 0.3 --dvars-spike-threshold 0.5 \
--longitudinal

What version of fMRIPrep are you running?

23.2.0a3

How are you running fMRIPrep?

Singularity

Is your data BIDS valid?

No

Are you reusing any previously computed results?

No

Please copy and paste any relevant log output.

Node select_xfm.a0 failed to run on host node101.
231215-01:23:22,460 nipype.workflow ERROR:
	 Saving crash info to /om2/scratch/tmp/oozernov/READER_fitlins/sub-reader403/data/derivatives/fmriprep/sub-reader403/log/20231214-180959_e0b6ad78-2005-43ba-be80-e8a7f8f4d25e/crash-20231215-012322-oozernov-select_xfm.a0-dc6b4a8f-510b-48f6-ae9a-5aa433545778.txt
Traceback (most recent call last):
  File "/opt/conda/envs/fmriprep/lib/python3.10/site-packages/nipype/pipeline/plugins/multiproc.py", line 344, in _send_procs_to_workers
    self.procs[jobid].run(updatehash=updatehash)
  File "/opt/conda/envs/fmriprep/lib/python3.10/site-packages/nipype/pipeline/engine/nodes.py", line 527, in run
    result = self._run_interface(execute=True)
  File "/opt/conda/envs/fmriprep/lib/python3.10/site-packages/nipype/pipeline/engine/nodes.py", line 645, in _run_interface
    return self._run_command(execute)
  File "/opt/conda/envs/fmriprep/lib/python3.10/site-packages/nipype/pipeline/engine/nodes.py", line 771, in _run_command
    raise NodeExecutionError(msg)
nipype.pipeline.engine.nodes.NodeExecutionError: Exception raised while executing Node select_xfm.

Traceback:
	Traceback (most recent call last):
	  File "/opt/conda/envs/fmriprep/lib/python3.10/site-packages/nipype/interfaces/base/core.py", line 400, in run
	    outputs = self.aggregate_outputs(runtime)
	  File "/opt/conda/envs/fmriprep/lib/python3.10/site-packages/nipype/interfaces/base/core.py", line 429, in aggregate_outputs
	    predicted_outputs = self._list_outputs()  # Predictions from _list_outputs
	  File "/opt/conda/envs/fmriprep/lib/python3.10/site-packages/niworkflows/interfaces/utility.py", line 217, in _list_outputs
	    index = self.inputs.keys.index(self.inputs.key)
	ValueError: 'MNIPediatricAsym:cohort-4' is not in list


231215-01:23:22,469 nipype.workflow CRITICAL:
	 fMRIPrep failed: Traceback (most recent call last):
  File "/opt/conda/envs/fmriprep/lib/python3.10/site-packages/nipype/pipeline/plugins/multiproc.py", line 344, in _send_procs_to_workers
    self.procs[jobid].run(updatehash=updatehash)
  File "/opt/conda/envs/fmriprep/lib/python3.10/site-packages/nipype/pipeline/engine/nodes.py", line 527, in run
    result = self._run_interface(execute=True)
  File "/opt/conda/envs/fmriprep/lib/python3.10/site-packages/nipype/pipeline/engine/nodes.py", line 645, in _run_interface
    return self._run_command(execute)
  File "/opt/conda/envs/fmriprep/lib/python3.10/site-packages/nipype/pipeline/engine/nodes.py", line 771, in _run_command
    raise NodeExecutionError(msg)
nipype.pipeline.engine.nodes.NodeExecutionError: Exception raised while executing Node select_xfm.

Traceback:
	Traceback (most recent call last):
	  File "/opt/conda/envs/fmriprep/lib/python3.10/site-packages/nipype/interfaces/base/core.py", line 400, in run
	    outputs = self.aggregate_outputs(runtime)
	  File "/opt/conda/envs/fmriprep/lib/python3.10/site-packages/nipype/interfaces/base/core.py", line 429, in aggregate_outputs
	    predicted_outputs = self._list_outputs()  # Predictions from _list_outputs
	  File "/opt/conda/envs/fmriprep/lib/python3.10/site-packages/niworkflows/interfaces/utility.py", line 217, in _list_outputs
	    index = self.inputs.keys.index(self.inputs.key)
	ValueError: 'MNIPediatricAsym:cohort-4' is not in list

Additional information / screenshots

BIDS Validator outputs below. I don't think any of these errors would be related to the present error.

[email protected]
(node:34612) Warning: Closing directory handle on garbage collection
(Use `node --trace-warnings ...` to show where the warning was created)
	1: [ERR] Invalid JSON file. The file is not formatted according the schema. (code: 55 - JSON_SCHEMA_VALIDATION_ERROR)
		./sub-reader031/ses-pre/func/sub-reader031_ses-pre_task-listening_run-1_bold.json
			Evidence: .CogAtlasID should match format "uri"
		./sub-reader031/ses-pre/func/sub-reader031_ses-pre_task-listening_run-2_bold.json
			Evidence: .CogAtlasID should match format "uri"
		./sub-reader031/ses-pre/func/sub-reader031_ses-pre_task-listening_run-3_bold.json
			Evidence: .CogAtlasID should match format "uri"
		./sub-reader031/ses-pre/func/sub-reader031_ses-pre_task-reading_run-1_bold.json
			Evidence: .CogAtlasID should match format "uri"
		./sub-reader031/ses-pre/func/sub-reader031_ses-pre_task-reading_run-2_bold.json
			Evidence: .CogAtlasID should match format "uri"
		./sub-reader031/ses-pre/func/sub-reader031_ses-pre_task-rest_run-1_bold.json
			Evidence: .CogAtlasID should match format "uri"
		./task-listening_bold.json
			Evidence: .CogAtlasID should match format "uri"
		./task-reading_bold.json
			Evidence: .CogAtlasID should match format "uri"
		./task-rest_bold.json
			Evidence: .CogAtlasID should match format "uri"

	Please visit https://neurostars.org/search?q=JSON_SCHEMA_VALIDATION_ERROR for existing conversations about this issue.

	2: [ERR] The value of (EchoTime2 - EchoTime1) should be within the range of 0.0001 - 0.01. (code: 83 - ECHOTIME1_2_DIFFERENCE_UNREASONABLE)
		./sub-reader031/ses-pre/fmap/sub-reader031_ses-pre_acq-a_run-2_phasediff.nii.gz

	Please visit https://neurostars.org/search?q=ECHOTIME1_2_DIFFERENCE_UNREASONABLE for existing conversations about this issue.

	1: [WARN] Tabular file contains custom columns not described in a data dictionary (code: 82 - CUSTOM_COLUMN_WITHOUT_DESCRIPTION)
		./sub-reader031/ses-pre/func/sub-reader031_ses-pre_task-rest_run-1_events.tsv
			Evidence: Columns: TODO -- fill in rows and add more tab-separated columns if desired not defined, please define in: /events.json, /task-rest_events.json,/run-1_events.json,/task-rest_run-1_events.json,/sub-reader031/sub-reader031_events.json,/sub-reader031/sub-reader031_task-rest_events.json,/sub-reader031/sub-reader031_run-1_events.json,/sub-reader031/sub-reader031_task-rest_run-1_events.json,/sub-reader031/ses-pre/sub-reader031_ses-pre_events.json,/sub-reader031/ses-pre/sub-reader031_ses-pre_task-rest_events.json,/sub-reader031/ses-pre/sub-reader031_ses-pre_run-1_events.json,/sub-reader031/ses-pre/sub-reader031_ses-pre_task-rest_run-1_events.json,/sub-reader031/ses-pre/func/sub-reader031_ses-pre_events.json,/sub-reader031/ses-pre/func/sub-reader031_ses-pre_task-rest_events.json,/sub-reader031/ses-pre/func/sub-reader031_ses-pre_run-1_events.json,/sub-reader031/ses-pre/func/sub-reader031_ses-pre_task-rest_run-1_events.json
		./sub-reader031/ses-pre/sub-reader031_ses-pre_scans.tsv
			Evidence: Columns: operator, randstr not defined, please define in: /scans.json, /sub-reader031/sub-reader031_scans.json,/sub-reader031/ses-pre/sub-reader031_ses-pre_scans.json

	Please visit https://neurostars.org/search?q=CUSTOM_COLUMN_WITHOUT_DESCRIPTION for existing conversations about this issue.

	2: [WARN] The recommended file /README is missing. See Section 03 (Modality agnostic files) of the BIDS specification. (code: 101 - README_FILE_MISSING)

	Please visit https://neurostars.org/search?q=README_FILE_MISSING for existing conversations about this issue.

        Summary:                  Available Tasks:                          Available Modalities: 
        53 Files, 229.24MB        listening                                 MRI                   
        1 - Subject               reading                                                         
        1 - Session               rest                                                            
                                  TODO: full task name for reading                                
                                  TODO: full task name for listening                              
                                  TODO: full task name for rest                                   


	If you have any questions, please post on https://neurostars.org/tags/bids.

The text was updated successfully, but these errors were encountered:

effigies · 2023-12-15T14:12:03Z

Interesting. Seems to be the cohort/res combo. Can you share sub_reader403_wf/template_iterator_wf/select_xfm/_report/report.rst?

smeisler · 2023-12-15T14:14:52Z

Node: sub_reader403_wf (template_iterator_wf (select_xfm (utility)
==================================================================


 Hierarchy : fmriprep_23_2_wf.sub_reader403_wf.template_iterator_wf.select_xfm
 Exec ID : select_xfm.a0


Original Inputs
---------------


* anat2std_xfm : ['/om2/scratch/tmp/oozernov/READER_fitlins/sub-reader403/data/derivatives/fmriprep/sub-reader403/anat/sub-reader403_run-1_from-T1w_to-MNIPediatricAsym_mode-image_xfm.h5', '/om2/scratch/tmp/oozernov/READER_fitlins/sub-reader403/data/derivatives/fmriprep/sub-reader403/anat/sub-reader403_run-1_from-T1w_to-MNI152NLin2009cAsym_mode-image_xfm.h5', '/om2/scratch/tmp/oozernov/READER_fitlins/sub-reader403/data/derivatives/fmriprep/sub-reader403/anat/sub-reader403_run-1_from-T1w_to-MNI152NLin6Asym_mode-image_xfm.h5']
* key : MNIPediatricAsym:cohort-4
* keys : ['MNIPediatricAsym', 'MNI152NLin2009cAsym', 'MNI152NLin6Asym']

effigies · 2023-12-15T15:06:33Z

I see. So the issue is that we currently pass around template as if it's meaningful on its own, because resolutions are not considered distinct templates (you can register to 1 and resample to 2). But cohorts should be considered distinct templates, so we need to pass around cohort information.

Right now we pass template and template_spec out of register_template_wf:

smriprep/smriprep/workflows/fit/registration.py

Lines 205 to 208 in 8830bf8

    
           (split_desc, outputnode, [ 
        
               ('name', 'template'), 
        
               ('spec', 'template_spec'), 
        
           ]),

And this needs to be combined into {template}[:cohort-<cohort>]. Either that can be done in that workflow or in the fit workflow

smriprep/smriprep/workflows/anatomical.py

Lines 971 to 1013 in 8830bf8

    
           # Stage 4: Normalization 
        
           templates = [] 
        
           found_xfms = {} 
        
           for template in spaces.get_spaces(nonstandard=False, dim=(3,)): 
        
               xfms = precomputed.get('transforms', {}).get(template, {}) 
        
               if set(xfms) != {'forward', 'reverse'}: 
        
                   templates.append(template) 
        
               else: 
        
                   found_xfms[template] = xfms 
        
           template_buffer.inputs.in1 = list(found_xfms) 
        
           anat2std_buffer.inputs.in1 = [xfm['forward'] for xfm in found_xfms.values()] 
        
           std2anat_buffer.inputs.in1 = [xfm['reverse'] for xfm in found_xfms.values()] 
        
           if templates: 
        
               LOGGER.info(f'ANAT Stage 4: Preparing normalization workflow for {templates}') 
        
               register_template_wf = init_register_template_wf( 
        
                   sloppy=sloppy, 
        
                   omp_nthreads=omp_nthreads, 
        
                   templates=templates, 
        
               ) 
        
               ds_template_registration_wf = init_ds_template_registration_wf(output_dir=output_dir) 
        
               # fmt:off 
        
               workflow.connect([ 
        
                   (inputnode, register_template_wf, [('roi', 'inputnode.lesion_mask')]), 
        
                   (t1w_buffer, register_template_wf, [('t1w_preproc', 'inputnode.moving_image')]), 
        
                   (refined_buffer, register_template_wf, [('t1w_mask', 'inputnode.moving_mask')]), 
        
                   (sourcefile_buffer, ds_template_registration_wf, [ 
        
                       ('source_files', 'inputnode.source_files') 
        
                   ]), 
        
                   (register_template_wf, ds_template_registration_wf, [ 
        
                       ('outputnode.template', 'inputnode.template'), 
        
                       ('outputnode.anat2std_xfm', 'inputnode.anat2std_xfm'), 
        
                       ('outputnode.std2anat_xfm', 'inputnode.std2anat_xfm'), 
        
                   ]), 
        
                   (register_template_wf, template_buffer, [('outputnode.template', 'in2')]), 
        
                   (ds_template_registration_wf, std2anat_buffer, [('outputnode.std2anat_xfm', 'in2')]), 
        
                   (ds_template_registration_wf, anat2std_buffer, [('outputnode.anat2std_xfm', 'in2')]), 
        
               ]) 
        
               # fmt:on 
        
           if found_xfms: 
        
               LOGGER.info(f'ANAT Stage 4: Found pre-computed registrations for {found_xfms}')

Another approach is to pass around template_spec everywhere we pass around template and decide what to do with the collection of keys.

One thing that seems clear to me is that we'll need to add cohort-<label> to the transform file, so that we can distinguish pre-computed transforms.

That will require updates to the path patterns in niworkflows: https://github.com/nipreps/niworkflows/blob/a0af2636521580d84b6b4eb1c32b0cfc4e1bff85/niworkflows/data/nipreps.json#L154-L184

As well as adding the cohort to the how we look up precomputed transforms in:

smriprep/smriprep/utils/bids.py

Lines 53 to 63 in 8830bf8

    
           transforms = derivs_cache.setdefault('transforms', {}) 
        
           for space in std_spaces: 
        
               for k, q in spec['transforms'].items(): 
        
                   q = q.copy() 
        
                   q['subject'] = subject_id 
        
                   q['from'] = q['from'] or space 
        
                   q['to'] = q['to'] or space 
        
                   item = layout.get(return_type='filename', **q) 
        
                   if not item: 
        
                       continue 
        
                   transforms.setdefault(space, {})[k] = item[0] if len(item) == 1 else item

Which is called here:

smriprep/smriprep/workflows/base.py

Lines 365 to 369 in 8830bf8

    
           deriv_cache = {} 
        
           std_spaces = spaces.get_spaces(nonstandard=False, dim=(3,)) 
        
           std_spaces.append('fsnative') 
        
           for deriv_dir in derivatives: 
        
               deriv_cache.update(collect_derivatives(deriv_dir, subject_id, std_spaces))

If any changes are needed to the last part, they'll need to be propagated to fMRIPrep here:

https://github.com/nipreps/fmriprep/blob/7f4b302ab916de4d6257cd76f9eba7ef0931c948/fmriprep/workflows/base.py#L250-L263

effigies · 2023-12-15T15:08:19Z

@smeisler Do you have any time or interest in working into this? Would you need help getting set up with a dev environment? I'm working on another issue today, so I would be glad of any effort you're able to put at this.

cc also @mgxd as this will affect nibabies significantly.

smeisler · 2023-12-15T15:27:28Z

I can give it a shot. I shouldn't have trouble setting up a dev environment (changes will be happening in https://github.com/smeisler/smriprep/tree/fix/cohort_res) but will update here if I run into issues.

smeisler · 2023-12-15T15:39:10Z

Actually before I do too much, I should make sure the dev environment was set up correctly. Steps I took:

Fork the repo smeisler/smriprep
Start a new branch smeisler/smriprep/fix/cohort_res
Clone it to machine
Switch to my working branch:cd smriprep, git checkout fix/cohort_res
Create new conda environement mamba create -n smriprep_dev pip; mamba activate smriprep_dev
Install smriprep in that environment from the cloned repo pip install -e .

Anything I'm missing? I realize it's not containerized, so there might be some oddities with software versions I have on my computer, but hopefully I can at least make the changes to the workflow.

effigies · 2023-12-15T15:54:26Z

You're going to need to work with niworkflows and you'll at least want to test with fmriprep, so I think you'll want to install all of these:

for REPO in nipreps/{fmriprep,smriprep,niworkflows}; do
    git clone https://github.com/$REPO
done
mamba env create -f fmriprep/env.yml  # Can use -n to override the default name of fmriprep
mamba env config vars set -n fmriprep FSLDIR=$CONDA_PREFIX  # Make sure FSL tools can find each other
mamba activate fmriprep
pip install --no-deps -e ./fmriprep -e ./smriprep -e ./niworkflows

It's not hard to add additional repos after the fact.

You'll also need FreeSurfer, AFNI and Connectome Workbench installed in your PATH, but otherwise you should be good to go.

Edit: Note that I reordered one command. You want to set the environment variable before activating to make sure it gets loaded into your shell.

effigies · 2023-12-19T19:10:26Z

@smeisler Just checking in about how this is going for you.

smeisler · 2023-12-20T02:17:39Z

Haven't had much time to work on this yet, hope to get to it tomorrow.

miltoncamachocamacho · 2024-03-21T00:36:19Z

Hello @smeisler have you had any luck? I am facing the same issue but using MNIInfant:cohort-2. ValueError: 'MNIInfant:cohort-2' is not in list

WangYunHong98 · 2024-04-19T13:45:18Z

Hi @effigies

I have the same issue: ValueError: 'MNIPediatricAsym:cohort-1' is not in list. Has this issue been solved or can I use other versions released before fmriprep 23.2.1.

Best,
Yunhong

fphsFischmeister · 2024-05-10T09:14:34Z

Hi @effigies, hi @smeisler,
is there any progress on that issue or workaround? We have the same issues with ValueError: 'MNIPediatricAsym:cohort-6' is not in list but would like to proceed with fmriprep 23.2 instead switching to an older version.
Thanks in advance,
Florian

effigies · 2024-05-10T11:15:02Z

If you use the :unstable tag, you can test the next version prior to release.

fphsFischmeister · 2024-05-12T18:53:14Z

Hi @effigies, I tested the current _: unstable _ version with _ MNIPediatricAsym: res-2:cohert-6 _. Everything worked fine for me and did not produce any errors, so I'm looking forward to the 22.2.3 release.

Thanks for all the effort,

Florian

effigies · 2024-05-12T18:59:38Z

The release that will be included in will be 24.0.0.

smeisler added the bug Something isn't working label Dec 15, 2023

effigies transferred this issue from nipreps/fmriprep Dec 15, 2023

smeisler changed the title ~~23.2.0a3: select_xfm failing with MNIPediatricAsym~~ 23.2.0a3: select_xfm failing with MNIPediatricAsym when resolution specified Dec 15, 2023

effigies changed the title ~~23.2.0a3: select_xfm failing with MNIPediatricAsym when resolution specified~~ 23.2.0a3: select_xfm failing with MNIPediatricAsym when cohort specified Dec 15, 2023

effigies changed the title ~~23.2.0a3: select_xfm failing with MNIPediatricAsym when cohort specified~~ 23.2.0a3: template_iterator_wf.select_xfm failing with MNIPediatricAsym when cohort specified Dec 15, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

23.2.0a3: `template_iterator_wf.select_xfm` failing with `MNIPediatricAsym` when cohort specified #407

23.2.0a3: `template_iterator_wf.select_xfm` failing with `MNIPediatricAsym` when cohort specified #407

smeisler commented Dec 15, 2023

effigies commented Dec 15, 2023

smeisler commented Dec 15, 2023

effigies commented Dec 15, 2023

effigies commented Dec 15, 2023

smeisler commented Dec 15, 2023

smeisler commented Dec 15, 2023 •

edited

Loading

effigies commented Dec 15, 2023 •

edited

Loading

effigies commented Dec 19, 2023

smeisler commented Dec 20, 2023

miltoncamachocamacho commented Mar 21, 2024

WangYunHong98 commented Apr 19, 2024 •

edited

Loading

fphsFischmeister commented May 10, 2024

effigies commented May 10, 2024

fphsFischmeister commented May 12, 2024

effigies commented May 12, 2024

23.2.0a3: template_iterator_wf.select_xfm failing with MNIPediatricAsym when cohort specified #407

23.2.0a3: template_iterator_wf.select_xfm failing with MNIPediatricAsym when cohort specified #407

Comments

smeisler commented Dec 15, 2023

What happened?

What command did you use?

What version of fMRIPrep are you running?

How are you running fMRIPrep?

Is your data BIDS valid?

Are you reusing any previously computed results?

Please copy and paste any relevant log output.

Additional information / screenshots

effigies commented Dec 15, 2023

smeisler commented Dec 15, 2023

effigies commented Dec 15, 2023

effigies commented Dec 15, 2023

smeisler commented Dec 15, 2023

smeisler commented Dec 15, 2023 • edited Loading

effigies commented Dec 15, 2023 • edited Loading

effigies commented Dec 19, 2023

smeisler commented Dec 20, 2023

miltoncamachocamacho commented Mar 21, 2024

WangYunHong98 commented Apr 19, 2024 • edited Loading

fphsFischmeister commented May 10, 2024

effigies commented May 10, 2024

fphsFischmeister commented May 12, 2024

effigies commented May 12, 2024

23.2.0a3: `template_iterator_wf.select_xfm` failing with `MNIPediatricAsym` when cohort specified #407

23.2.0a3: `template_iterator_wf.select_xfm` failing with `MNIPediatricAsym` when cohort specified #407

smeisler commented Dec 15, 2023 •

edited

Loading

effigies commented Dec 15, 2023 •

edited

Loading

WangYunHong98 commented Apr 19, 2024 •

edited

Loading