v0.5.0

This update introduces several quality-of-life improvements aimed at streamlining the user experience with the Dandelion class. Again, it doesn't affect the preprocessing pipeline so the singularity image will not be updated.

Simpler way to modify sequence and cell ids
Previously, modifying sequence and cell IDs required manual updates and could become confusing, especially when managing IDs across different slots. This update introduces a non-destructive approach to modify IDs, with the option to reset them to the original initialized values. However, this functionality currently supports only prefix or suffix changes, not both.

• Cell and Sequence ID Renaming: The process of renaming cell and sequence IDs has been simplified for easier management. this can now be modified directly from the Dandelion class with add_{sequence/cell}_{prefix/suffix} methods, which modifies the IDs in place.
• Gene Annotations: Gene annotations can now be simplified so that there's only 1 call (without allelic info) per cell. For example, a call of IGHV1-2**01,IGHV1-2**02 will just become IGHV1-2. This behaviour is not switch on by default and users have to manually call .simplify().

Deprecation of ddl.pp.filter_contigs
As part of ongoing code maintenance, the ddl.pp.filter_contigs function has been deprecated. Maintaining this function became increasingly difficult, especially compared to the more robust ddl.pp.check_contigs function. Moving forward, ddl.pp.check_contigs will be the primary tool for contig quality control. This function provides a clearer and more flexible approach, allowing users to adjust their QC criteria based on their needs.

• Improved QC Process: check_contigs allows for easy customization e.g. deciding whether to filter extra contigs, keeping only 1 or 2 VDJ/VJ chains, offering options for both strict and relaxed filtering based on user requirements. The current default setting strikes a balance between aggressive filtering and more lenient QC.
• Inclusion of Consensus Count into QC logic - now, when there's ties in the umi count (especially for low umi count contigs), the consensus count can actually be used to break ties and keep the most representative contig. Thanks @HongjianSun for alerting me to this
• Flexible 'exceptions' - Previously hard-coded extra pair exceptions in QC were confusing. Now, this behavior can be toggled off, and the logic behind identifying exceptions is more transparent. Additionally, a new assertion has been added for IgM/IgD BCRs, requiring that V and J genes match to avoid treating mismatches as exceptions.

Support for reading other formats
There's now read_airr/read_parse_airr/read_bd_airr to support the other formats that are emerging.

What's Changed

• Update tests_weekly.yml by @zktuong in #433
• make the check for folder names simpler and more relaxed by @zktuong in #435
• Update formatting.yaml by @zktuong in #437
• Index syncing by @zktuong in #440
• Refactor contig QC process to consider consensus counts to break ties. by @zktuong in #442
• toggle keep extra properly in check_contigs. by @zktuong in #443
• Add airr readers by @zktuong in #444
• update api doc by @zktuong in #445

Dependency updates

• pip dev(deps-dev): update sphinx-rtd-theme requirement from <=3.0.1 to <=3.0.2 by @dependabot in #441
• pip dev(deps-dev): update scirpy requirement from <=0.18.0 to <=0.19.0 by @dependabot in #439
• pip dev(deps-dev): update sphinx-autodoc-typehints requirement from <=2.4.4 to <=2.5.0 by @dependabot in #438
• pip dev(deps-dev): update sphinx-rtd-theme requirement from <=2.0.0 to <=3.0.1 by @dependabot in #436

Full Changelog: v0.4.0...v0.5.0

v0.4.0

New features! This doesn't change anything in the preprocessing/singularity workflow.

1. new write_10x function with Dandelion class so that it can now format AIRR-formatted tables to 10x's input files (e.g. all_contig_annotations.csv and all_contig.fasta)
2. new saving mode - makes no difference to users but the new format is different in the backend as i have dropped the dependency on pandas - now it directly saves with h5py, which means that the way the files can be accessed is different. files saves previously should still be possible to be read in - if resaved, it will overwrite with the new format.
3. downsampling in generate_network now can also downsample orphan VJ chains (previously it relied on VDJ chains primarily), and also works on alternative clone_key
4. dandelion is now compatible with python 3.12. Just in time before python 3.13 😩
5. compatibility with AnnData is fixed. @grst, you should be able to unpin dandelion in scirpy!
6. revamped the testing suite with micromamba with new caching options.

What's Changed

• Fix anndata compat by @zktuong in #408
• fix clone id key by @zktuong in #414
• new saving mode by @zktuong in #420
• fix downsampling logic in generate network by @zktuong in #424
• fix clone_key not respected properly if non-default clone_id by @zktuong in #429
• prepare for python 3.12 by @zktuong in #363

dependency updates

• pip prod(deps): update pandas requirement from <=2.2.2,>=1.0.3 to >=1.0.3,<=2.2.3 by @dependabot in #431
• pip dev(deps-dev): update sphinx-autodoc-typehints requirement from <=2.2.3 to <=2.4.4 by @dependabot in #430
• pip dev(deps-dev): update scirpy requirement from <=0.17.2 to <=0.18.0 by @dependabot in #426

Full Changelog: v0.3.8...v0.4.0

v0.3.8

What's Changed

Bug fixes mostly on the container build and functions to reduce warning messages and deal with processing failures and actually allow R to start up properly.

• fix isotype parsing by @zktuong in #397
• Clean up warnings by @zktuong in #402
• Add a return step to gently deal with preprocessing failures during the tigger step. by @zktuong in #403
• assert sample names as strings during preprocessing by @zktuong in #404

Dependabot updates

• pip dev(deps-dev): update sphinx-autodoc-typehints requirement from <=2.1.1 to <=2.2.2 by @dependabot in #394
• pip dev(deps-dev): update scirpy requirement from <=0.17.0 to <=0.17.2 by @dependabot in #401
• pip dev(deps-dev): update sphinx-autodoc-typehints requirement from <=2.2.2 to <=2.2.3 by @dependabot in #405

Full Changelog: v0.3.7...v0.3.8

v0.3.7

What's Changed

Bug fixes for plotting and container script

• fix matplotlib compatibility by @zktuong in #384
• adjust manual_threshold to skip running findThreshold by @zktuong in #386
• fix MakeDb_gentle.py file name by @zktuong in #391

Dependabot updates

• pip dev(deps-dev): update scirpy requirement from <=0.16.1 to <=0.17.0 by @dependabot in #385
• pip dev(deps-dev): update sphinx-autodoc-typehints requirement from <=2.1.0 to <=2.1.1 by @dependabot in #388

Full Changelog: v0.3.6...v0.3.7

v0.3.6.post1

What's Changed

Thanks to @grst for alerting me to incompatibility to latest matplotlib.
This small release just fixes that as per:

• fix matplotlib compatibility by @zktuong in #384

Full Changelog: v0.3.6...v0.3.6.post1

v0.3.6

What's Changed

New features

• add option to supply prefix or suffix with io functions by @zktuong in #367
• disregard non productive contigs when reporting chain/locus status by @zktuong in #366

Bug fixes

• fix unbound variable in check contigs by @zktuong in #370
• add fix for mouse tra annotation problem by @zktuong in #377

Other

• remove testing on macos-latest because there's no blast/igblast available by @zktuong in #380

Documentation updates

• Fix dead link in 1_dandelion_preprocessing-10x_data.ipynb by @tnieuwe in #361
• Update README.rst by @zktuong in #374

Dependabot updates

• pip dev(deps-dev): update scirpy requirement from <=0.15.0 to <=0.16.0 by @dependabot in #359
• pip dev(deps-dev): update sphinx-autodoc-typehints requirement from <=1.25.3 to <=2.0.0 by @dependabot in #360
• pip prod(deps): update pandas requirement from <=2.2.0,>=1.0.3 to >=1.0.3,<=2.2.1 by @dependabot in #362
• pip prod(deps): update pandas requirement from <=2.2.1,>=1.0.3 to >=1.0.3,<=2.2.2 by @dependabot in #371
• pip dev(deps-dev): update sphinx-autodoc-typehints requirement from <=2.0.0 to <=2.0.1 by @dependabot in #372
• pip dev(deps-dev): update scirpy requirement from <=0.16.0 to <=0.16.1 by @dependabot in #373
• pip dev(deps-dev): update sphinx-autodoc-typehints requirement from <=2.0.1 to <=2.1.0 by @dependabot in #375

Full Changelog: v0.3.5...v0.3.6

v0.3.5

What's Changed

• fix container script bug by @zktuong in #350
• reorder content table on docs by @zktuong in #352
• fix entry of anndata with NaN 'sequence_id' values by @amoschoomy in #351
• add dependabot dependency review for PR by @zktuong in #353
• add else statement to check contigs when there's no sequence by @MeganS92 in #354
• pip prod(deps): update pandas requirement from <=2.1.4,>=1.0.3 to >=1.0.3,<=2.2.0 by @dependabot in #355
• pip dev(deps-dev): update sphinx-autodoc-typehints requirement from <=1.25.2 to <=1.25.3 by @dependabot in #356
• pip dev(deps-dev): update scirpy requirement from <=0.14 to <=0.15.0 by @dependabot in #357
• convert to use umi_count by @zktuong in #358

New Contributors

• @MeganS92 made their first contribution in #354

Full Changelog: v0.3.4...v0.3.5

v0.3.4

Summary

• Speed up network generation in generate_network
• Add soft filtering and normalisation to vdj_psuedobulk functions - @ktpolanski
• Created a new column in .data (extra) to flag if contig is considered extra.
• New clone id definition to insert VDJ and VJ to the id to reduce ambiguity - need to check if it does it properly for cells with no clone ids. This also means that now clone ids can be created for orphan chains.
• New to_scirpy/from_scirpy functions that will now convert them to the new scverse airr formats - @amoschoomy
• Container build is now simplified and uses mamba to manage all the dependencies.
• New option to run preprocessing with ogrdb references in both the base package and the container.
• New reference download function in the container folder to ensure the latest references are pulled for every new iteration of the container.
• Deprecate support for python3.7 tests.

What's Changed

• create a remove_malformed option by @zktuong in #315
• fix future warning for multimappers by @zktuong in #316
• fix cleaning up step in the singularity image by @zktuong in #317
• Speed up network generation by @zktuong in #319
• remove d gene VJ columns by @zktuong in #320
• Running palantir by @zktuong in #321
• update workflow dependencies by @zktuong in #323
• Fix #325 by @zktuong in #326
• add concat in tutorial by @zktuong in #327
• Update 3_dandelion_findingclones-10x_data.ipynb by @tnieuwe in #328
• disable 3.7 tests by @zktuong in #330
• add logic to deal with contigs with no clone id during network generation step by @zktuong in #335
• Soft filtering in setup_vdj_pseudobulk() by @ktpolanski in #334
• attempting to reduce code bloat for find_clones by @zktuong in #329
• fix bug with check contigs by @zktuong in #341
• add an option to the singularity container to skip tigger step. by @zktuong in #343
• add new option to check contigs by @zktuong in #344
• simplify build container by @zktuong in #346
• Awkward to pandas by @amoschoomy in #342
• Fix yml by @zktuong in #349
• fix container build definition file by @zktuong in #348

dependabot updates

• pip prod(deps): update pandas requirement from <=2.1.0,>=1.0.3 to >=1.0.3,<=2.1.1 by @dependabot in #314
• pip dev(deps-dev): update readthedocs-sphinx-ext requirement from <=2.2.2 to <=2.2.3 by @dependabot in #318
• pip prod(deps): update pandas requirement from <=2.1.1,>=1.0.3 to >=1.0.3,<=2.1.2 by @dependabot in #324
• pip dev(deps-dev): update sphinx-autodoc-typehints requirement from <=1.24.0 to <=1.24.1 by @dependabot in #331
• pip prod(deps): update pandas requirement from <=2.1.2,>=1.0.3 to >=1.0.3,<=2.1.3 by @dependabot in #332
• pip dev(deps-dev): update sphinx-autodoc-typehints requirement from <=1.24.1 to <=1.25.2 by @dependabot in #333
• pip dev(deps-dev): update sphinx-rtd-theme requirement from <=1.2.2 to <=2.0.0 by @dependabot in #338
• pip prod(deps): update pandas requirement from <=2.1.3,>=1.0.3 to >=1.0.3,<=2.1.4 by @dependabot in #339
• pip dev(deps-dev): update readthedocs-sphinx-ext requirement from <=2.2.3 to <=2.2.4 by @dependabot in #340
• pip dev(deps-dev): update readthedocs-sphinx-ext requirement from <=2.2.4 to <=2.2.5 by @dependabot in #345
• Bump tj-actions/changed-files from 35 to 41 in /.github/workflows by @dependabot in #347

New Contributors

• @tnieuwe made their first contribution in #328
• @amoschoomy made their first contribution in #342

Full Changelog: v0.3.3...v0.3.4

v0.3.3

What's Changed

• Mainly updates and bug fixes to tl.clone_overlap and pl.clone_overlap.
• simplified pre-processing functions to call command line arguments instead of running within the code.

Detailed notes:

• Update docs for clone overlap by @zktuong in #276
• Allow additional arguments in define_clones by @zktuong in #280
• add an if statement to check if actor is dependabot by @zktuong in #289
• pip dev(deps-dev): update sphinx-autodoc-typehints requirement from <=1.23.0 to <=1.23.3 by @dependabot in #284
• pip dev(deps-dev): update sphinx-rtd-theme requirement from <=1.2.0 to <=1.2.2 by @dependabot in #285
• pip dev(deps-dev): update readthedocs-sphinx-ext requirement from <=2.2.0 to <=2.2.2 by @dependabot in #286
• pip dev(deps-dev): update nbsphinx requirement from <=0.9.1 to <=0.9.2 by @dependabot in #287
• enable auto-merge for dependabot by @zktuong in #290
• refactoring how external scripts and locations are called by @zktuong in #288
• fix reassign_alleles by @zktuong in #293
• remove deprecated function from docs by @zktuong in #297
• pip dev(deps-dev): update sphinx-autodoc-typehints requirement from <=1.23.3 to <=1.24.0 by @dependabot in #296
• fix weekly tests by @zktuong in #301
• pip prod(deps): update mizani requirement from <0.10.0 to <0.11.0 by @dependabot in #302
• add options to plotting clone overlap by @zktuong in #307
• add requirements.txt by @zktuong in #309
• should be cartesian product instead of combination by @zktuong in #312

Full Changelog: v0.3.2...v0.3.3

v0.3.2

What's Changed

Mainly to fix compatibility with dependencies.

• minor wording/renaming tweaks in tutorial by @ktpolanski in #252
• ensure additional column names are present in strict mode by @zktuong in #253
• fix weekly tests by @zktuong in #255
• fix singularity preprocessing for org by @zktuong in #256
• Easy docs2 by @zktuong in #260
• Update _network.py by @zktuong in #258
• fix requirements by @zktuong in #262
• Create dependabot.yml by @zktuong in #263
• pip prod(deps): update pandas requirement from <1.5.0,>=1.0.3 to >=1.0.3,<2.1.0 by @dependabot in #264
• getting rid of CI warnings by @zktuong in #265
• update actions by @zktuong in #266
• barplot bug by @zktuong in #267
• fix query behaviour for merge by @zktuong in #268
• Update colab doc by @zktuong in #271

New Contributors

• @dependabot made their first contribution in #264

Full Changelog: v0.3.1...v0.3.2