🐛 pipeline does not produce with/without ICAAROMA or two nuisance pipelines specified.

[nimzodisaster]

Describe the bug

My pipeline YAML: yml_orig.txt
The minimized YAML inthe log directory: yml_min.txt
Expected Outputs:
`anat:

• sub-101376100*_ses-4*_desc-brain*_mask*
• sub-101376100*_ses-4*_desc-dsegAxial*_quality*
• sub-101376100*_ses-4*_desc-dsegSagittal*_quality*
• sub-101376100*_ses-4*_desc-head*_T1w*
• sub-101376100*_ses-4*_desc-preproc*_T1w*
• sub-101376100*_ses-4*_from-MNI152NLin6ASym*_to-T1w*_mode-image*_desc-linear*_xfm*
• sub-101376100*_ses-4*_from-MNI152NLin6ASym*_to-T1w*_mode-image*_desc-nonlinear*_xfm*
• sub-101376100*_ses-4*_from-MNI152NLin6ASym*_to-T1w*_mode-image*_xfm*
• sub-101376100*_ses-4*_from-MNI152NLin6Sym*_to-T1w*_mode-image*_desc-linear*_xfm*
• sub-101376100*_ses-4*_from-MNI152NLin6Sym*_to-T1w*_mode-image*_desc-nonlinear*_xfm*
• sub-101376100*_ses-4*_from-MNI152NLin6Sym*_to-T1w*_mode-image*_xfm*
• sub-101376100*_ses-4*_from-T1w*_to-MNI152NLin6ASym*_mode-image*_desc-linear*_xfm*
• sub-101376100*_ses-4*_from-T1w*_to-MNI152NLin6ASym*_mode-image*_desc-nonlinear*_xfm*
• sub-101376100*_ses-4*_from-T1w*_to-MNI152NLin6ASym*_mode-image*_xfm*
• sub-101376100*_ses-4*_from-T1w*_to-MNI152NLin6Sym*_mode-image*_desc-linear*_xfm*
• sub-101376100*_ses-4*_from-T1w*_to-MNI152NLin6Sym*_mode-image*_desc-nonlinear*_xfm*
• sub-101376100*_ses-4*_from-T1w*_to-MNI152NLin6Sym*_mode-image*_xfm*
• sub-101376100*_ses-4*_label-CSF*_desc-preproc*_mask*
• sub-101376100*_ses-4*_label-CSF*_mask*
• sub-101376100*_ses-4*_label-CSF*_probseg*
• sub-101376100*_ses-4*_label-GM*_desc-preproc*_mask*
• sub-101376100*_ses-4*_label-GM*_mask*
• sub-101376100*_ses-4*_label-GM*_probseg*
• sub-101376100*_ses-4*_label-WM*_desc-preproc*_mask*
• sub-101376100*_ses-4*_label-WM*_mask*
• sub-101376100*_ses-4*_label-WM*_probseg*
• sub-101376100*_ses-4*_space-MNI152NLin6ASym*_desc-brain*_mask*
• sub-101376100*_ses-4*_space-MNI152NLin6ASym*_desc-head*_T1w*
• sub-101376100*_ses-4*_space-MNI152NLin6ASym*_desc-preproc*_T1w*
• sub-101376100*_ses-4*_space-MNI152NLin6ASym*_label-CSF*_mask*
• sub-101376100*_ses-4*_space-MNI152NLin6ASym*_label-GM*_mask*
• sub-101376100*_ses-4*_space-MNI152NLin6ASym*_label-WM*_mask*
  func:
• sub-101376100*_ses-4*_desc-MeanSCA*_timeseries*
• sub-101376100*_ses-4*_desc-MultReg*_statmap*
• sub-101376100*_ses-4*_desc-SpatReg*_timeseries*
• sub-101376100*_ses-4*_desc-boldSnr*_quality*
• sub-101376100*_ses-4*_desc-boldSnrAxial*_quality*
• sub-101376100*_ses-4*_desc-boldSnrHist*_quality*
• sub-101376100*_ses-4*_desc-boldSnrSagittal*_quality*
• sub-101376100*_ses-4*_desc-brain*_mask*
• sub-101376100*_ses-4*_desc-confounds*_timeseries*
• sub-101376100*_ses-4*_desc-framewiseDisplacementJenkinsonPlot*_quality*
• sub-101376100*_ses-4*_desc-mean*_bold*
• sub-101376100*_ses-4*_desc-movementParameters*_motion*
• sub-101376100*_ses-4*_desc-movementParametersRot*_quality*
• sub-101376100*_ses-4*_desc-movementParametersTrans*_quality*
• sub-101376100*_ses-4*_desc-preproc*_bold*
• sub-101376100*_ses-4*_desc-summary*_motion*
• sub-101376100*_ses-4*_from-MNI152NLin6ASym*_to-bold*_mode-image*_xfm*
• sub-101376100*_ses-4*_from-MNI152NLin6Sym*_to-bold*_mode-image*_xfm*
• sub-101376100*_ses-4*_from-bold*_to-MNI152NLin6ASym*_mode-image*_xfm*
• sub-101376100*_ses-4*_from-bold*_to-MNI152NLin6Sym*_mode-image*_xfm*
• sub-101376100*_ses-4*_from-bold*_to-T1w*_mode-image*_desc-linear*_xfm*
• sub-101376100*_ses-4*_motion*
• sub-101376100*_ses-4*_sbref*
• sub-101376100*_ses-4*_space-MNI152NLin6ASym*_dcb*
• sub-101376100*_ses-4*_space-MNI152NLin6ASym*_dcw*
• sub-101376100*_ses-4*_space-MNI152NLin6ASym*_desc-Mean*_timeseries*
• sub-101376100*_ses-4*_space-MNI152NLin6ASym*_desc-MeanSCA*_correlations*
• sub-101376100*_ses-4*_space-MNI152NLin6ASym*_desc-MultReg*_correlations*
• sub-101376100*_ses-4*_space-MNI152NLin6ASym*_desc-PartialNilearn*_correlations*
• sub-101376100*_ses-4*_space-MNI152NLin6ASym*_desc-PearsonNilearn*_correlations*
• sub-101376100*_ses-4*_space-MNI152NLin6ASym*_desc-bold*_mask*
• sub-101376100*_ses-4*_space-MNI152NLin6ASym*_desc-preproc*_bold*
• sub-101376100*_ses-4*_space-MNI152NLin6ASym*_desc-sm*_dcb*
• sub-101376100*_ses-4*_space-MNI152NLin6ASym*_desc-sm*_dcw*
• sub-101376100*_ses-4*_space-MNI152NLin6ASym*_desc-sm*_ecb*
• sub-101376100*_ses-4*_space-MNI152NLin6ASym*_desc-sm*_ecw*
• sub-101376100*_ses-4*_space-MNI152NLin6ASym*_desc-sm*_lfcdb*
• sub-101376100*_ses-4*_space-MNI152NLin6ASym*_desc-sm*_lfcdw*
• sub-101376100*_ses-4*_space-MNI152NLin6ASym*_desc-smZstd*_dcb*
• sub-101376100*_ses-4*_space-MNI152NLin6ASym*_desc-smZstd*_dcw*
• sub-101376100*_ses-4*_space-MNI152NLin6ASym*_desc-smZstd*_ecb*
• sub-101376100*_ses-4*_space-MNI152NLin6ASym*_desc-smZstd*_ecw*
• sub-101376100*_ses-4*_space-MNI152NLin6ASym*_desc-smZstd*_lfcdb*
• sub-101376100*_ses-4*_space-MNI152NLin6ASym*_desc-smZstd*_lfcdw*
• sub-101376100*_ses-4*_space-MNI152NLin6ASym*_desc-zstd*_dcb*
• sub-101376100*_ses-4*_space-MNI152NLin6ASym*_desc-zstd*_dcw*
• sub-101376100*_ses-4*_space-MNI152NLin6ASym*_desc-zstd*_ecb*
• sub-101376100*_ses-4*_space-MNI152NLin6ASym*_desc-zstd*_ecw*
• sub-101376100*_ses-4*_space-MNI152NLin6ASym*_desc-zstd*_lfcdb*
• sub-101376100*_ses-4*_space-MNI152NLin6ASym*_desc-zstd*_lfcdw*
• sub-101376100*_ses-4*_space-MNI152NLin6ASym*_ecb*
• sub-101376100*_ses-4*_space-MNI152NLin6ASym*_ecw*
• sub-101376100*_ses-4*_space-MNI152NLin6ASym*_lfcdb*
• sub-101376100*_ses-4*_space-MNI152NLin6ASym*_lfcdw*
• sub-101376100*_ses-4*_space-MNI152NLin6ASym*_res-4mm*_desc-bold*_mask*
• sub-101376100*_ses-4*_space-MNI152NLin6ASym*_sbref*
• sub-101376100*_ses-4*_space-T1w*_sbref*
• sub-101376100*_ses-4*_vmhc*`

To reproduce

No response

Preconfig

• default
• abcd-options
• anat-only
• blank
• ccs-options
• fmriprep-options
• fx-options
• monkey
• monkey-ABCD
• ndmg
• nhp-macaque
• preproc
• rbc-options
• rodent

Custom pipeline configuration

No response

Run command

No response

Expected behavior

produce the full set of derivatives.

Acceptance criteria

Produce full set of outputs

Screenshots

No response

C-PAC version

i.8.7.dev1

Container platform

No response

Docker and/or Singularity version(s)

No response

Additional context

No response

[nimzodisaster]

If this is an issue with my yaml, apologies, but I'll confess, I'm pretty frustrated.

[tamsinrogers]

Hi @nimzodisaster, thanks for reaching out!

It looks like the run is forking at the ICA-AROMA fork point, but there’s a naming collision bug when getting the outputs from the working directory to the output directory - only one of the forks is making it to the output directory.

If you still have the working directory, you can look for pairs of directories with the same name. For example, in our team's test runs, we have working/pipeline_APP0Base/cpac_sub-NDARAB348EWR_ses-1/_scan_movieTP/nii_desc-preproc_bold_265 and working/pipeline_APP0Base/cpac_sub-NDARAB348EWR_ses-1/_scan_movieTP/nii_desc-preproc_bold_269 directories with the final preprocessed BOLD images with and without ICA-AROMA, but only the with ICA-AROMA was copied over to the output directory. Until we resolve this issue, unfortunately the simplest course of action is to run two configs, one with and one without ICA-AROMA.

The attached pipeline config only has one nuisance regression strategy:

Regressors:
      - Name: yGSRySCRUB
        Motion:
          include_delayed: On
          include_squared: On
          include_delayed_squared: On
        aCompCor:
          summary:
            method: DetrendPC
            components: 5
          tissues:
          - WhiteMatter
          - CerebrospinalFluid
          extraction_resolution: 2
        GlobalSignal:
          summary: Mean
        CerebrospinalFluid:
          summary: Mean
          extraction_resolution: 2
          erode_mask: On
        Censor:
          method: Kill
          number_of_previous_trs_to_censor: 0
          number_of_subsequent_trs_to_censor: 0
          thresholds:
          - type: FD_J
            value: 0.25
        PolyOrt:
          degree: 2
        Bandpass:
          bottom_frequency: 0.01
          top_frequency: 0.1
          method: default

so there’s nothing to fork over there. When we tried adding a second:

Regressors:
      - Name: yGSRySCRUB
        Motion:
          include_delayed: On
          include_squared: On
          include_delayed_squared: On
        aCompCor:
          summary:
            method: DetrendPC
            components: 5
          tissues:
          - WhiteMatter
          - CerebrospinalFluid
          extraction_resolution: 2
        GlobalSignal:
          summary: Mean
        CerebrospinalFluid:
          summary: Mean
          extraction_resolution: 2
          erode_mask: On
        Censor:
          method: Kill
          number_of_previous_trs_to_censor: 0
          number_of_subsequent_trs_to_censor: 0
          thresholds:
          - type: FD_J
            value: 0.25
        PolyOrt:
          degree: 2
        Bandpass:
          bottom_frequency: 0.01
          top_frequency: 0.1
          method: default
      - Name: nGSRnSCRUB
        Motion:
          include_delayed: On
          include_squared: On
          include_delayed_squared: On
        aCompCor:
          summary:
            method: DetrendPC
            components: 5
          tissues:
          - WhiteMatter
          - CerebrospinalFluid
          extraction_resolution: 2
        CerebrospinalFluid:
          summary: Mean
          extraction_resolution: 2
          erode_mask: On
        PolyOrt:
          degree: 2
        Bandpass:
          bottom_frequency: 0.01
          top_frequency: 0.1
          method: default

we did see outputs generated for both nuisance forks - e.g.:

- sub-NDARAB348EWR*_ses-1*_reg-nGSRnSCRUB*_desc-MeanSCA*_timeseries*
- sub-NDARAB348EWR*_ses-1*_reg-nGSRnSCRUB*_desc-MultReg*_statmap*
- sub-NDARAB348EWR*_ses-1*_reg-nGSRnSCRUB*_desc-boldSnr*_quality*
- sub-NDARAB348EWR*_ses-1*_reg-nGSRnSCRUB*_desc-boldSnrAxial*_quality*
- sub-NDARAB348EWR*_ses-1*_reg-nGSRnSCRUB*_desc-boldSnrHist*_quality*
- sub-NDARAB348EWR*_ses-1*_reg-nGSRnSCRUB*_desc-boldSnrSagittal*_quality*
- sub-NDARAB348EWR*_ses-1*_reg-nGSRnSCRUB*_desc-confounds*_timeseries*
- sub-NDARAB348EWR*_ses-1*_reg-nGSRnSCRUB*_desc-preproc*_bold*
- sub-NDARAB348EWR*_ses-1*_reg-nGSRnSCRUB*_vmhc*
- sub-NDARAB348EWR*_ses-1*_reg-yGSRySCRUB*_desc-MeanSCA*_timeseries*
- sub-NDARAB348EWR*_ses-1*_reg-yGSRySCRUB*_desc-MultReg*_statmap*
- sub-NDARAB348EWR*_ses-1*_reg-yGSRySCRUB*_desc-boldSnr*_quality*
- sub-NDARAB348EWR*_ses-1*_reg-yGSRySCRUB*_desc-boldSnrAxial*_quality*
- sub-NDARAB348EWR*_ses-1*_reg-yGSRySCRUB*_desc-boldSnrHist*_quality*
- sub-NDARAB348EWR*_ses-1*_reg-yGSRySCRUB*_desc-boldSnrSagittal*_quality*
- sub-NDARAB348EWR*_ses-1*_reg-yGSRySCRUB*_desc-confounds*_timeseries*
- sub-NDARAB348EWR*_ses-1*_reg-yGSRySCRUB*_desc-preproc*_bold*
- sub-NDARAB348EWR*_ses-1*_reg-yGSRySCRUB*_vmhc*

Please let me know if that helps and if there are any additional questions I can answer for you!

[nimzodisaster]

Thanks for getting back to me. Unfortunately I do not have the working directories anymore as I was working on a workstation with limited disk space, and was auto removing the work directory. I have moved my processing to another workstation with lots of disk space and so will be keeping the working directory in the future.

I noticed a few other things about outputs:
My config included smoothed and unsmoothed,, z-scored and raw, but the outputted files only contained one set.
e.g:

OUTPUTS AND DERIVATIVES

-----------------------

post_processing:
spatial_smoothing:
run: On
output: [smoothed, nonsmoothed]

z-scoring:
run: On
output: [z-scored, raw]

I think my default will be to make each config as direct as possible and to keep the working directory. I do hope this is only an issue with the outputting process, and not mixing them up in the working directory.

[tamsinrogers]

Hi @nimzodisaster,

It’s pretty likely that the naming collision is also source of the error with these outputs. To alleviate your concerns about mixing up the configs in the working directory, when C-PAC forks, each fork is unaware of the other forks until copying over to the output directory at the end, so there’s no chance of an internal mix-up.