✨ Automate Regression Testing

[amygutierrez]

Related problem

Automate regression testing so that a regression test can be done whenever developer marks a pull request ready for review. This will allow for any bugs to be detected early and will give developers more time for debugging 👩‍💻

Proposed feature

1. Create CLI tool called reg-suite. Have subcommands for cpac run and correlations. CLI tool will be portable enough to use locally and through git actions.
2. Create lite and full version of the regression test. Lite version will run downsampled 5mm data (for faster runs) and full version will run complete datasets on all the pipelines.
3. lite version will get triggered when pull request is ready for review. full version will get triggered when ready to merge to main

Acceptance criteria

automated regression test through git is more important.

Alternatives

No response

Additional context

No response

[sgiavasis]

Updates:

• The lite regression test automation is now operational, and will be triggered anytime a PR is ready for review.

Next:

• Setup the lite regression test so that the results can be pulled easily.
• Automation of the inputs into the correlation scripts.
• Full regression test: structure needs to be changed (re: when it will run and which branch).
  • Has to run on ACCESS, so Singularity instead of Docker.
  • Different script compared to lite version.
• Report generation.

[sgiavasis]

Reminder here so I don't forget, and this should be a new issue/project - but it would be great if we set up some runs from fmriprep, ABCD, and CCS pipelines, save those outputs, and also automate correlations against the CPAC preconfigs for these pipelines.

[shnizzedy]

While I'm still working on simplifying all the necessary variables and secrets passing among the various repos for this feature , here's a

preview of the first-draft of a comment the CI will push when the lite regression tests are run

• The Theodore bot account will post the comments. The example is me because I pushed it manually with my PAT while testing the final step.
• The heatmap image in the comment links to an interactive version like this.
• The upside to having all the various repos involved in this process is when issues like 🧑‍💻 Redesign boxplot setup CPAC_regtest_pack#10 are resolved, those improvements will be reflected in these automated comments.
• I'm waiting until I have this flow working end-to-end to make all the various PRs ready-for-review, because I anticipate having to make changes in each of them before the whole thing works.
• I'm doing a small amount of scaling up (adding a few subjects from multiple sites, changing the x-axis from site to subject) in this first draft.