Bin QC Improvements #707

dialvarezs · 2024-10-27T21:31:08Z

This PR adds:

CheckM2 as an alternative for bin qc
Updates CheckM and GUNC modules
A new BIN_QC subworkflow, integrating CheckM, CheckM2, BUSCO and GUNC

Closes #607.

PR checklist

- Update modules - Update integration in mag and with other tools (bin_summary, gtdb-tk) - Update test - Update schema

jfy133 · 2024-10-27T21:40:34Z

@nf-core-bot fix linting

jfy133 · 2024-10-27T21:42:43Z

Before you continue (sorry this is a bit late):

I generally don't like to deprecate old version of tools for a while, but rather keep them as alternative tools.

In some cases people want to stick with the original version for compatibility with previous runs

Could you 'revert' (or reinstall) the old checkm module and wrap it in an if/else statement (but within the subworkflow :) )

@muabnezor did a similar thing when adding porechop_ABI hree: #674

dialvarezs · 2024-10-27T21:49:06Z

@jfy133 that makes sense, I will revert the CheckM removal. Bad for me for not asking before 😅.

dialvarezs · 2024-10-28T12:15:01Z

Sorry, I didn't catch your last comment about including both tools in a single workflow. With that in mind, would make sense to include BUSCO as well, and just make a "bin_qc" subworkflow?

Also, simplify bin_summary regarding bin qc

jfy133 · 2024-10-29T09:47:00Z

Sorry, I didn't catch your last comment about including both tools in a single workflow. With that in mind, would make sense to include BUSCO as well, and just make a "bin_qc" subworkflow?

Yes that would be perfect! We need to subworkflow the sh*t out of this monster 😅 thank you!!!

prototaxites · 2024-11-11T15:18:39Z

I think this looks good, though I don't have the latitude to run any manual tests of it myself at the moment - so probably wait for someone else to sign off on it!

My only question is whether it might be good to move all the binqc database preparation steps (e.g. CHECKM2_DATABASEDOWNLOAD, and the bits that initialise the files/channels) inside the BINQC workflow where they are consumed? I think keeping conceptually-related code together will probably help with maintenance down the line, and the main mag.nf workflow is already over-full. Plus, there's less subworkflow inputs and outputs to keep track of. Might be a job for another PR though!

dialvarezs · 2024-11-12T06:50:59Z

@prototaxites Absolutely, that makes sense to me. That would help to simplify the mag workflow a bit. What do you think about this @jfy133 ?

jfy133 · 2024-11-12T07:45:46Z

I've pondered that a few times, however many of these download steps take a very long time, and thus from a user PoV I think it makes sense to have it triggered right at the beginning of the pipeline so by the time assembly and binning is done it's ready to go rather than getting all the way to binning and then having to wait the same length of time again before you can start the binning QC.

In my mind to clear up the code I would rather have a dB download subworkflow for all DB Downloadsto make the code clearer. Maybe from a related module PoV it's not as efficient but functionally they are related

Any counter arguments?

dialvarezs · 2024-11-12T07:55:14Z

In most cases, database downloads don’t depend on anything, so, if I'm not wrong, the processes should start immediately regardless of where they are in placed in the code. Or am I missing something?

prototaxites · 2024-11-12T09:24:31Z

In most cases, database downloads don’t depend on anything, so, if I'm not wrong, the processes should start immediately regardless of where they are in placed in the code. Or am I missing something?

Yes, that's my understanding - Nextflow processes kick off as soon as any inputs are available, so processes beginning with file/URL input from parameters should start as soon as the pipeline begins, no matter how many subworkflows deep they are (there might be a latency hit if you go 10,000 subworkflows deep...).

Also not opposed to a "database download" subWF - but that seems a little more limited in scope. In particular I'm thinking about all the steps that are like "if(params.some_db) { db = Channel.value(file(path)) } else { db = Channel.empty }" - there are a lot of these input databases, often just a zip or fasta file, and building these in a single WF would make something with a lot of outputs. Hence why I think it may be more maintainable to keep that code near to where the outputs are consumed (set the PhiX fasta in the "short read preprocessing" subWF, lambda in the "long read preprocessing", etc.). But either way cleaning up the main WF should help - this all just may be beyond the scope of this PR.

jfy133 · 2024-11-13T12:41:10Z

In most cases, database downloads don’t depend on anything, so, if I'm not wrong, the processes should start immediately regardless of where they are in placed in the code. Or am I missing something?

Yes, that's my understanding - Nextflow processes kick off as soon as any inputs are available, so processes beginning with file/URL input from parameters should start as soon as the pipeline begins, no matter how many subworkflows deep they are (there might be a latency hit if you go 10,000 subworkflows deep...).

Hmm fair. I might be feeling over defensive due to the huge number of conditions mag has... so maybe this is indeed the case.

Also not opposed to a "database download" subWF - but that seems a little more limited in scope. In particular I'm thinking about all the steps that are like "if(params.some_db) { db = Channel.value(file(path)) } else { db = Channel.empty }" - there are a lot of these input databases, often just a zip or fasta file, and building these in a single WF would make something with a lot of outputs. Hence why I think it may be more maintainable to keep that code near to where the outputs are consumed (set the PhiX fasta in the "short read preprocessing" subWF, lambda in the "long read preprocessing", etc.). But either way cleaning up the main WF should help - this all just may be beyond the scope of this PR.

Fair point.

Ok - if you're feeling to up to it @dialvarezs go ahead and move the relevant database downloads to the BINQC subworklow(s) :)

Note you don't need a separate CheckM download CI check -> I would rather you just include an UNTAR step in the pipeline isntead.

I added the seaprte CheckM CI tests originally due to instable downloads when connecting to the servers in Australia, but as these databases are now on Zenodo this should rarely happen.

dialvarezs · 2024-11-14T08:12:38Z

@jfy133 I'll get back to this on Friday or Saturday.

Ok - if you're feeling to up to it @dialvarezs go ahead and move the relevant database downloads to the BINQC subworklow(s) :)

I think this is a step in the right direction to improve the modularity of mag, so I will.

Note you don't need a separate CheckM download CI check -> I would rather you just include an UNTAR step in the pipeline isntead.

Got it, I will remove the checkm2 ci test I added. Should I remove the checkm ci test as well?

jfy133 · 2024-11-14T08:18:09Z

@jfy133 I'll get back to this on Friday or Saturday.

Ok - if you're feeling to up to it @dialvarezs go ahead and move the relevant database downloads to the BINQC subworklow(s) :)

I think this is a step in the right direction to improve the modularity of mag, so I will.

Note you don't need a separate CheckM download CI check -> I would rather you just include an UNTAR step in the pipeline isntead.

Got it, I will remove the checkm2 ci test I added. Should I remove the checkm ci test as well?

Yes you probably can! Thank you!

jfy133

This is looking good to me!

I just completed a comparison test between checkm/checkm2 and just need to compare the results.

Then I'll run with BUSCO/GUNC next week (have to stay home with the kiddo tomorow as well unfortunately) to make sure they aren't broken and then I think we are good to go.

Do you want to include the moving of the database downloading in this PR @dialvarezs or in a follow up one?

conf/modules.config

docs/output.md

modules/local/bin_summary.nf

nextflow.config

nextflow_schema.json

subworkflows/local/bin_qc.nf

jfy133 · 2024-11-21T14:38:54Z

subworkflows/local/bin_qc.nf

+
+        BUSCO(ch_input_bins_for_qc, ch_db_for_busco)
+
+        BUSCO_SUMMARY(


Missing versions file being mixed into ch_versions for this process, the SAVE_DOWNLOAD doesn't have one, although given that process is entirely empty, shouldn't that be replaced by a publishDir specification in modules.conf 🤔

(I know you didn't write this, but might be worth fixing this now :) )

By the comment on that module, the problem with simply uisng publishDir is that the same files would get written multiple times. I couldn't think of a more elegant solution, so I will add the version, unless you have a better idea.

subworkflows/local/bin_qc.nf

jfy133 · 2024-11-21T14:57:29Z

Note to self: Ah crap I didn't update the directory name, will have to run the two checkm tests again 🤦

  nextflow run ../main.nf -profile test,docker --outdir ./results_checkm --binqc_tool checkm
  nextflow run dialvarezs/mag -r dev-checkm2 -profile test,docker --outdir ./results_checkm --binqc_tool checkm2

dialvarezs · 2024-11-22T13:23:17Z

Hi @jfy133, sorry for the delay, I didn't get enough time on last weekend for this. But I will address your comments shortly, and I plan to include the database downloads in this PR.

jfy133 · 2024-11-22T13:51:25Z

No worries!

No stress if you can't make it ATM, but I think we are close to a release once this is in!

dialvarezs · 2024-11-22T13:55:16Z

If you have time, can you review this PR? nf-core/modules#6920
I also plan to update the GUNC/CHEKM modules in this PR, and also the GUNC PR should solve the problem with gunc outputs.

jfy133 · 2024-11-25T07:54:43Z

If you have time, can you review this PR? nf-core/modules#6920 I also plan to update the GUNC/CHEKM modules in this PR, and also the GUNC PR should solve the problem with gunc outputs.

Done!

jfy133 · 2024-11-25T07:55:40Z

My kid is sick again 🙄 but if you can update GUNC in this PR before then, then I will be working on mag Thursday so can finish reviewing this PR again :) (but no stress!)

dialvarezs · 2024-11-27T03:24:15Z

I made all the important updates. I will run a complete test with every QC tool in out HPC to check if everything is alright, and with that done it should be ready.

dialvarezs · 2024-11-28T19:12:57Z

It should be ready now. There is a last minor issue that should be solved by this PR: nf-core/modules#7119

feat: Migrate from CheckM to CheckM2

53864d2

- Update modules - Update integration in mag and with other tools (bin_summary, gtdb-tk) - Update test - Update schema

dialvarezs marked this pull request as draft October 27, 2024 21:31

dialvarezs added 3 commits October 27, 2024 19:05

fix: Bring back CheckM

ca2f97b

fix: One more thing

05e3393

fix: Linting

3e49dbf

dialvarezs changed the title ~~feat: Replace from CheckM to CheckM2~~ feat: Add CheckM2 Oct 27, 2024

fix: Option and checks in bin summary script

b1b6518

dialvarezs force-pushed the dev-checkm2 branch from c63e084 to b1b6518 Compare October 27, 2024 22:40

dialvarezs added 2 commits October 27, 2024 19:43

Fix: missing import

73b5794

docs: Output, changelog and citation for CheckM2

325310b

dialvarezs marked this pull request as ready for review October 28, 2024 10:30

dialvarezs added 3 commits October 28, 2024 07:32

docs: readme

b88302a

Merge remote-tracking branch 'upstream/dev' into dev-checkm2

d62a671

docs: Update readme

3da7441

dialvarezs added 2 commits October 28, 2024 09:57

refactor: Merge checkm and checkm2 subworkflows in a single one

3d98415

Also, simplify bin_summary regarding bin qc

fix: Restore mistakenly deleted code

8b4fcc7

dialvarezs added 4 commits October 31, 2024 17:26

Bin QC workflow

76dac5c

Merge remote-tracking branch 'upstream/dev' into dev-checkm2

0ebab07

Cleanup

38ce756

Final touches

3702329

dialvarezs force-pushed the dev-checkm2 branch 2 times, most recently from a4f42ef to da52285 Compare November 1, 2024 00:27

Integrate GUNC in BIN_QC subworkflow

0eb167a

dialvarezs force-pushed the dev-checkm2 branch from 4007932 to 0eb167a Compare November 1, 2024 04:13

Code style improvements

f0a6999

jfy133 reviewed Nov 21, 2024

View reviewed changes

jfy133 and others added 7 commits November 25, 2024 08:56

Merge branch 'dev' into dev-checkm2

7e21b62

Address several review comments

3158c52

Update modules

6152696

Make binqc_tool an input for BIN_SUMMARY

ef1828c

Move bin qc database setup to the subworkflow

bcae7f9

Fix

158ee12

Remove another it

5738215

dialvarezs and others added 6 commits November 27, 2024 00:30

Remove checkm/checkm2 ci tests

49ebb71

gtdbtk: add check when params.busco_db is not defined

4ecd36f

Don't flatten bins for GUNC

1cf1da7

Update changelog

ea9075f

Emit bash version on busco_save_download

a079596

Fix declaration

80641b0

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Bin QC Improvements #707

Bin QC Improvements #707

dialvarezs commented Oct 27, 2024 •

edited

Loading

jfy133 commented Oct 27, 2024

jfy133 commented Oct 27, 2024

dialvarezs commented Oct 27, 2024

dialvarezs commented Oct 28, 2024 •

edited

Loading

jfy133 commented Oct 29, 2024

prototaxites commented Nov 11, 2024

dialvarezs commented Nov 12, 2024

jfy133 commented Nov 12, 2024 •

edited

Loading

dialvarezs commented Nov 12, 2024

prototaxites commented Nov 12, 2024 •

edited

Loading

jfy133 commented Nov 13, 2024

dialvarezs commented Nov 14, 2024

jfy133 commented Nov 14, 2024

jfy133 left a comment

jfy133 Nov 21, 2024

dialvarezs Nov 27, 2024 •

edited

Loading

jfy133 commented Nov 21, 2024

dialvarezs commented Nov 22, 2024

jfy133 commented Nov 22, 2024

dialvarezs commented Nov 22, 2024

jfy133 commented Nov 25, 2024

jfy133 commented Nov 25, 2024

dialvarezs commented Nov 27, 2024

dialvarezs commented Nov 28, 2024

Bin QC Improvements #707

Are you sure you want to change the base?

Bin QC Improvements #707

Conversation

dialvarezs commented Oct 27, 2024 • edited Loading

PR checklist

jfy133 commented Oct 27, 2024

jfy133 commented Oct 27, 2024

dialvarezs commented Oct 27, 2024

dialvarezs commented Oct 28, 2024 • edited Loading

jfy133 commented Oct 29, 2024

prototaxites commented Nov 11, 2024

dialvarezs commented Nov 12, 2024

jfy133 commented Nov 12, 2024 • edited Loading

dialvarezs commented Nov 12, 2024

prototaxites commented Nov 12, 2024 • edited Loading

jfy133 commented Nov 13, 2024

dialvarezs commented Nov 14, 2024

jfy133 commented Nov 14, 2024

jfy133 left a comment

Choose a reason for hiding this comment

jfy133 Nov 21, 2024

Choose a reason for hiding this comment

dialvarezs Nov 27, 2024 • edited Loading

Choose a reason for hiding this comment

jfy133 commented Nov 21, 2024

dialvarezs commented Nov 22, 2024

jfy133 commented Nov 22, 2024

dialvarezs commented Nov 22, 2024

jfy133 commented Nov 25, 2024

jfy133 commented Nov 25, 2024

dialvarezs commented Nov 27, 2024

dialvarezs commented Nov 28, 2024

dialvarezs commented Oct 27, 2024 •

edited

Loading

dialvarezs commented Oct 28, 2024 •

edited

Loading

jfy133 commented Nov 12, 2024 •

edited

Loading

prototaxites commented Nov 12, 2024 •

edited

Loading

dialvarezs Nov 27, 2024 •

edited

Loading