Continuous testing of neuroimaging results across pipelines and datasets

[jacobsanz97]

Title

Continuous testing of neuroimaging results across pipelines and datasets

Leaders

Jacob Sanz-Robinson (Mattermost: @jacobsanz).

Collaborators

Jacob Sanz-Robinson (Mattermost: @jacobsanz).

Brainhack Global 2021 Event

Brainhack Montreal

Project Description

• What are you doing, for whom, and why?
  Neuroimaging results are sensitive to variations in processing pipelines, contributing to scientific result reproducibility issues. In general, there is no ground truth and it is unclear which pipeline should be applied to data when they yield different results. This project is a software framework capable of helping neuroscientists test whether neuroimaging results are robust across pipelines and replicable across datasets.

• What makes your project special and exciting?
  It is, to the best of my knowledge, the first attempt at a generalized framework for performing distributed computations in a Continuous Integration setting. This will help scientists systematically evaluate result variability in their experiments. Users can pinpoint biases and discrepancies caused by processing methods and datasets, and quantify their impact on results. In areas where result reproducibility is a concern, the uptake of this novel tool could aid in consolidating knowledge and explaining the uncertainty in the field.

• How to get started?
  Have a gander at the project repo!
  Any additional functionality is welcome in a Pull Request. If you are familiar with any neuroimaging pipelines, building Boutiques descriptors for them is extremely useful. Functions to visualize the data or retrieve/process summary statistics are also welcome!

• Where to find key resources?
  The project repo (readme) is a good place to start. Otherwise feel free to message me by e-mail, Mattermost, or homing pigeon (if it's clean).

Link to project repository/sources

https://github.com/jacobsanz97/NDR-CI

Goals for Brainhack Global

My personal efforts will be directed at a mechanism to repopulate the cache files by querying the CBRAIN distributed computation system in case they become corrupted or lost.
If there is additional time I will work on making prettier data visualizations and including statistical summaries.
If other people join, then hopefully we can containerize a pipeline or two, and make more progress on the visualizations.

Good first issues

1. Issue one: Create a Boutiques descriptors for any neuroimaging pipeline of your choice!

2. Issue two: Create visualizations that could help us compare results across pipelines and datasets.

3. Issue three: Any additional functionality you may deem useful!

4. Issue four: cache file repopulation/retrieval (this is what I will be working on).

Communication channels

https://mattermost.brainhack.org/brainhack/channels/neuro-repro-ci

Skills

• Python
• Command line/bash
• Any neuroimaging pipelines! (I have been dealing with FSL and FreeSurfer so far...If you are familiar with ANTs, MINC tools, CIVET, or SPM, you will be extremely useful!)
• Containers/Boutiques.
• Data visualization (You don't need to be advanced! Some basic experience in matplotlib/seaborn/plotly or the desire to learn these is more than enough!)

Onboarding documentation

See readme: https://github.com/jacobsanz97/NDR-CI

What will participants learn?

If you wish to learn about containerization, running neuroimaging pipelines, or data visualization...This is a good project to pick up the basics of one or more of these :) .

Data to use

No response

Number of collaborators

more

Credit to collaborators

Contributors will be listed in the project ReadMe.

Image

Type

coding_methods

Development status

1_basic structure

Topic

reproducible_scientific_methods

Tools

AFNI, ANTs, Datalad, Freesurfer, FSL, other

Programming language

Python

Modalities

other, not_applicable

Git skills

2_branches_PRs

Anything else?

For Tools: Any and all tools and/or neuroimaging pipelines are useful to integrate into the project! At the same time, none are strictly necessary if you wish to focus your development efforts on another aspect of the framework.

Things to do after the project is submitted and ready to review.

• Add a comment below the main post of your issue saying: Hi @brainhackorg/project-monitors my project is ready!
• Twitter-sized summary of your project pitch.

[Remi-Gau]

@jacobsanz97

Will put this on the website.

One small suggestion would be to create actual issues on your repo to provide a "forum" to help guide new comers.
The 2 first issues you listed here seem quite good candidates for that.