Repository navigation
Release 1.2.0 - #228
Merged
Merged
Release 1.2.0#228
Conversation
* Ignore igenomes in config * CHANGELOG.md
* Better documentation of CheckM(2) and GTDB-Tk input * Make tidyverse/joinmetadata handle a combination of CheckM and CheckM2 files * Setup test case for #187 * Add CLAUDE.md * Implement --species_preference local to drop duplicate-species remote genomes When a user-provided local genome and a Sourmash-selected remote genome represent the same GTDB species, prefer the local one and skip fetching the remote duplicate. Species are resolved from --gtdbtk_metadata (local) and --gtdb_metadata (remote) in a new TIDYVERSE_SELECTGENOMESPECIES module, called before WGET_GENOME so duplicates are never downloaded. The 'completeness' and 'gtdb' preference modes remain unimplemented and now error explicitly instead of silently no-op'ing. Also fixes stale tRNA/rRNA assertions and thresholds in sourmash_mix_dupl_species.nf.test left over from before this feature existed, and updates its snapshot. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * Add test profile for prefer local in combination with sample-based genomesets * Add per-sample genome index manifest and fix single-file CheckM bug - Add --bbmap_save_genome_list (auto-on for --genomeset_mode sample) to publish which genome accessions went into each BBMap index, giving a read-independent way to verify per-sample genome selection. Threaded through CREATE_BBMAP_INDEX and asserted on in sourmash_mix_dupl_species_sampleset.nf.test. - Fix modules/local/tidyverse/joinmetadata: combining CheckM/CheckM2 files broke with exactly one --checkm_metadata file (unbalanced parens from a fragile string-joining trick), producing an R syntax error. Rebuilt with purrr::map + reduce(union), which works uniformly for any number of files. - Add two non-deterministic MultiQC files (multiqc_featurecounts.txt, featureCounts_assignment_plot.txt) to tests/.nftignore; their content can vary run-to-run due to BBMap's multi-threaded ambiguous-mapping resolution. - Update snapshots for metadata, sourmash_mix_dupl_species and sourmash_mix_dupl_species_sampleset (verified passing). The other test files sharing tests/.nftignore still need their snapshots regenerated on next run. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * Update snapshots after nftignore additions and more * Temporarily set test dataset url to my fork * Update two snapshots * Temporarily point test data at fork; validate per-sample genome selection - Move the fork test-data override from tests/nextflow.config to nextflow.config's own pipelines_testdata_base_path default. The conf/test_*.config files eagerly concatenate this value while being parsed as part of nextflow.config's profiles block, which happens before tests/nextflow.config (a later -c file) is ever merged, so overriding it there had no effect. Revert once the upstream nf-core/test-datasets PR merges. - Update sourmash_mix_dupl_species(.nf.test/.snap) and its sampleset variant against the updated fork data (one local genome removed, surfacing a previously-species-duplicated remote genome). - Rewrite the sampleset test's per-sample genome-list assertions: they previously assumed all samples get an identical genome set, which only held because the old test data had no local/remote divergence. With the new data, arc00/arc02 correctly pick up the surfaced remote genome via their own sourmash matches while arc01 doesn't -- this now genuinely exercises species_preference=local interacting with genomeset_mode=sample instead of being indistinguishable from joint mode. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * Implement --species_preference completeness and gtdb Extends TIDYVERSE_SELECTGENOMESPECIES (previously local-only) to also score genomes by CheckM completeness ('completeness') or GTDB's completeness - 5*contamination criterion ('gtdb'), comparing local and remote candidates for the same species and keeping whichever wins -- unlike 'local', these two modes can drop a local genome in favor of a better-scoring remote one. Ties are resolved in favor of the local genome. subworkflows/local/sourmash/main.nf is restructured so the local genome channel can actually be filtered post-hoc, which no mode needed before now. Adds dedicated test profiles/configs for completeness, gtdb, and completeness combined with genomeset_mode=sample, alongside the existing local-mode ones; updates the two existing dupl_species snapshots for the module's new (always-reported) purrr version entry. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * Consolidate overlapping Sourmash test profiles into two files Replaces 10 near-duplicate conf/test_*.config + tests/*.nf.test pairs (sourmash, sourmash_mix{,_sampleset}, sourmash_10genomes_{joint,sample}, sourmash_mix_dupl_species{,_sampleset,_completeness,_gtdb, _completeness_genomeset_sample}) with two files, each a single base config plus multiple nf-test cases that only override the params that differ: - tests/species_preference.nf.test: the full species_preference (local/completeness/gtdb) x genomeset_mode (joint/sample) matrix. - tests/sourmash_genome_selection.nf.test: species_preference='all', a remote-only scenario (no --genomeinfo), and multiple comma-separated --indexes files -- covering what the retired sourmash/sourmash_mix/10genomes tests checked, on the same archaeal dataset instead of three separate ones (e_coli, 24genomes, 10genomes). Also reverts pipelines_testdata_base_path in nextflow.config back to the official nf-core/test-datasets repo now that the updated archaeal test data (one local species removed, needed to actually exercise genomeset_mode=sample divergence) has been merged upstream. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * Linting fixes * Update CHANGELOG.md, usage.md and output.md * Add genome-selection summary to MultiQC report and unify genome-list manifest Summarize local vs. remote genome counts per genome set (per-sample in 'sample' mode, a single row in 'joint' mode) as a MultiQC custom-content table, plus a full per-genome breakdown in summary_tables/. The bbmap/*.genomes.txt manifest is now always written for both genomeset modes, making --bbmap_save_genome_list redundant, so it's removed. Also fix the sourmash subworkflow's meta.yml (previously undocumented/ mismatched outputs) and ignore Sourmash's *.sbt.zip index in nf-test snapshots, since its content isn't reproducible across runs. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * Make forgotten version bump to 1.2.0.dev * Stop publishing Sourmash index/signature files by default *.sbt.zip (genome index), *.sig (signatures) and *.sig.zip (gather matches/prefetch/unassigned) are large, non-reproducible intermediate Sourmash artifacts with little standalone value; only the *.csv.gz gather results remain gated behind --sourmash_save_sourmash. Update the one affected test snapshot to match, and refresh the "Workflow" version string (v1.1.0 -> v1.2.0dev) in every snapshot left stale by the recent version bump. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * Delayed version bump * Move CLAUDE.md content to AGENTS.md, matching other nf-core repos CLAUDE.md is now a thin pointer to AGENTS.md, which follows the same structure used in nf-core-ampliseq (What this is / Commands / Architecture / Conventions / Gotchas, plus the critical remote nf-core/agents fetch instruction). Also refreshes stale test file references from earlier test-suite consolidation and adds a few gotchas learned this session (empty withName blocks, CI's 4-cpu runners, oversized base.config memory labels for test genomes). Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * Speed up Prokka in tests and restore .sbt.zip publishing Give Prokka its own tight memory reservation in every test profile (2GB vs. its process_low label's 12GB default, which was never actually clamped by the resourceLimits ceiling) so more Prokka jobs run concurrently instead of serializing in pairs -- Prokka dominates test wall-clock time (~90%+ in local profiling) and only uses ~450MB in practice on these small genomes. Also fix an empty withName: GENOME_INDEX {} block (introduced when I first stopped publishing *.sbt.zip) that Nextflow 25.10.4 rejects with "Unknown config attribute", and restore *.sbt.zip publishing behind --sourmash_save_sourmash after reconsidering: it's a real genome index worth keeping on request, unlike *.sig/*.sig.zip. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * Clarify usage docs a bit * Remove CLAUDE.md, per nf-core consensus to use AGENTS.md only Repo-specific instructions now live entirely in AGENTS.md; the global CLAUDE.md at the user level already redirects to AGENTS.md for any repo that has one, so a per-repo CLAUDE.md is redundant and against current nf-core core-team consensus on avoiding coding-agent file clutter in pipeline repos. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * Switch GTDB metadata URL to a working mirror data.gtdb.aau.ecogenomic.org has an expired TLS certificate, breaking CI (certificate_expired PKIX validation failure staging ar53_metadata_r226.tsv.gz). data.gtdb.ecogenomic.org mirrors the same release and works. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * Add missing COLLECT_GENOMESELECTION snapshot coverage: metadata This test was never updated for the COLLECT_GENOMESELECTION MultiQC summary feature added earlier -- it runs unconditionally for every genomeset_mode=joint profile but only the sourmash-related snapshots were updated at the time. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * Add missing COLLECT_GENOMESELECTION snapshot coverage: default Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * Add missing COLLECT_GENOMESELECTION snapshot coverage: single Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * Add missing COLLECT_GENOMESELECTION snapshot coverage: duplicates Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * Add missing COLLECT_GENOMESELECTION snapshot coverage: skip_preprocessing_steps Last of the 5 affected snapshot files. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * Bump minimum Nextflow version to 26.04.0 Nextflow 25.10.4 has a version-topic-aggregation bug that drops the PROKKAGFF2TSV versions.yml block whenever that process runs more than once per pipeline execution, breaking snapshot comparisons uniformly across every test block with multiple genomes lacking a GFF. 26.04.6 and 26.07.0-edge both aggregate correctly, so raise the declared minimum instead of working around the bug. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * Regenerate ro-crate-metadata.json for Nextflow 26.04.0 minimum Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * Update sourmash_genome_selection.nf.test.snap for Nextflow 26.04.0 This snapshot was recorded while Nextflow 25.10.4's version-topic aggregation bug was silently dropping PROKKAGFF2TSV from the versions map whenever it ran more than once per execution. Now that the declared minimum is 26.04.0, the versions map is correctly populated and the snapshot needs the same block added to match. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * Clearer test names * Manual addition of PROKKAGFF2TSV block in snapshot * Record Nextflow minimum change in CHANGELOG.md --------- Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
…nd subworkflows (#215) * Better documentation of CheckM(2) and GTDB-Tk input * Make tidyverse/joinmetadata handle a combination of CheckM and CheckM2 files * Setup test case for #187 * Add CLAUDE.md * Implement --species_preference local to drop duplicate-species remote genomes When a user-provided local genome and a Sourmash-selected remote genome represent the same GTDB species, prefer the local one and skip fetching the remote duplicate. Species are resolved from --gtdbtk_metadata (local) and --gtdb_metadata (remote) in a new TIDYVERSE_SELECTGENOMESPECIES module, called before WGET_GENOME so duplicates are never downloaded. The 'completeness' and 'gtdb' preference modes remain unimplemented and now error explicitly instead of silently no-op'ing. Also fixes stale tRNA/rRNA assertions and thresholds in sourmash_mix_dupl_species.nf.test left over from before this feature existed, and updates its snapshot. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * Add test profile for prefer local in combination with sample-based genomesets * Add per-sample genome index manifest and fix single-file CheckM bug - Add --bbmap_save_genome_list (auto-on for --genomeset_mode sample) to publish which genome accessions went into each BBMap index, giving a read-independent way to verify per-sample genome selection. Threaded through CREATE_BBMAP_INDEX and asserted on in sourmash_mix_dupl_species_sampleset.nf.test. - Fix modules/local/tidyverse/joinmetadata: combining CheckM/CheckM2 files broke with exactly one --checkm_metadata file (unbalanced parens from a fragile string-joining trick), producing an R syntax error. Rebuilt with purrr::map + reduce(union), which works uniformly for any number of files. - Add two non-deterministic MultiQC files (multiqc_featurecounts.txt, featureCounts_assignment_plot.txt) to tests/.nftignore; their content can vary run-to-run due to BBMap's multi-threaded ambiguous-mapping resolution. - Update snapshots for metadata, sourmash_mix_dupl_species and sourmash_mix_dupl_species_sampleset (verified passing). The other test files sharing tests/.nftignore still need their snapshots regenerated on next run. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * Update snapshots after nftignore additions and more * Temporarily set test dataset url to my fork * Update two snapshots * Temporarily point test data at fork; validate per-sample genome selection - Move the fork test-data override from tests/nextflow.config to nextflow.config's own pipelines_testdata_base_path default. The conf/test_*.config files eagerly concatenate this value while being parsed as part of nextflow.config's profiles block, which happens before tests/nextflow.config (a later -c file) is ever merged, so overriding it there had no effect. Revert once the upstream nf-core/test-datasets PR merges. - Update sourmash_mix_dupl_species(.nf.test/.snap) and its sampleset variant against the updated fork data (one local genome removed, surfacing a previously-species-duplicated remote genome). - Rewrite the sampleset test's per-sample genome-list assertions: they previously assumed all samples get an identical genome set, which only held because the old test data had no local/remote divergence. With the new data, arc00/arc02 correctly pick up the surfaced remote genome via their own sourmash matches while arc01 doesn't -- this now genuinely exercises species_preference=local interacting with genomeset_mode=sample instead of being indistinguishable from joint mode. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * Implement --species_preference completeness and gtdb Extends TIDYVERSE_SELECTGENOMESPECIES (previously local-only) to also score genomes by CheckM completeness ('completeness') or GTDB's completeness - 5*contamination criterion ('gtdb'), comparing local and remote candidates for the same species and keeping whichever wins -- unlike 'local', these two modes can drop a local genome in favor of a better-scoring remote one. Ties are resolved in favor of the local genome. subworkflows/local/sourmash/main.nf is restructured so the local genome channel can actually be filtered post-hoc, which no mode needed before now. Adds dedicated test profiles/configs for completeness, gtdb, and completeness combined with genomeset_mode=sample, alongside the existing local-mode ones; updates the two existing dupl_species snapshots for the module's new (always-reported) purrr version entry. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * Consolidate overlapping Sourmash test profiles into two files Replaces 10 near-duplicate conf/test_*.config + tests/*.nf.test pairs (sourmash, sourmash_mix{,_sampleset}, sourmash_10genomes_{joint,sample}, sourmash_mix_dupl_species{,_sampleset,_completeness,_gtdb, _completeness_genomeset_sample}) with two files, each a single base config plus multiple nf-test cases that only override the params that differ: - tests/species_preference.nf.test: the full species_preference (local/completeness/gtdb) x genomeset_mode (joint/sample) matrix. - tests/sourmash_genome_selection.nf.test: species_preference='all', a remote-only scenario (no --genomeinfo), and multiple comma-separated --indexes files -- covering what the retired sourmash/sourmash_mix/10genomes tests checked, on the same archaeal dataset instead of three separate ones (e_coli, 24genomes, 10genomes). Also reverts pipelines_testdata_base_path in nextflow.config back to the official nf-core/test-datasets repo now that the updated archaeal test data (one local species removed, needed to actually exercise genomeset_mode=sample divergence) has been merged upstream. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * Linting fixes * Update CHANGELOG.md, usage.md and output.md * Add genome-selection summary to MultiQC report and unify genome-list manifest Summarize local vs. remote genome counts per genome set (per-sample in 'sample' mode, a single row in 'joint' mode) as a MultiQC custom-content table, plus a full per-genome breakdown in summary_tables/. The bbmap/*.genomes.txt manifest is now always written for both genomeset modes, making --bbmap_save_genome_list redundant, so it's removed. Also fix the sourmash subworkflow's meta.yml (previously undocumented/ mismatched outputs) and ignore Sourmash's *.sbt.zip index in nf-test snapshots, since its content isn't reproducible across runs. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * Make forgotten version bump to 1.2.0.dev * Stop publishing Sourmash index/signature files by default *.sbt.zip (genome index), *.sig (signatures) and *.sig.zip (gather matches/prefetch/unassigned) are large, non-reproducible intermediate Sourmash artifacts with little standalone value; only the *.csv.gz gather results remain gated behind --sourmash_save_sourmash. Update the one affected test snapshot to match, and refresh the "Workflow" version string (v1.1.0 -> v1.2.0dev) in every snapshot left stale by the recent version bump. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * Delayed version bump * Move CLAUDE.md content to AGENTS.md, matching other nf-core repos CLAUDE.md is now a thin pointer to AGENTS.md, which follows the same structure used in nf-core-ampliseq (What this is / Commands / Architecture / Conventions / Gotchas, plus the critical remote nf-core/agents fetch instruction). Also refreshes stale test file references from earlier test-suite consolidation and adds a few gotchas learned this session (empty withName blocks, CI's 4-cpu runners, oversized base.config memory labels for test genomes). Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * Template update for nf-core/tools version 4.0.3 * Speed up Prokka in tests and restore .sbt.zip publishing Give Prokka its own tight memory reservation in every test profile (2GB vs. its process_low label's 12GB default, which was never actually clamped by the resourceLimits ceiling) so more Prokka jobs run concurrently instead of serializing in pairs -- Prokka dominates test wall-clock time (~90%+ in local profiling) and only uses ~450MB in practice on these small genomes. Also fix an empty withName: GENOME_INDEX {} block (introduced when I first stopped publishing *.sbt.zip) that Nextflow 25.10.4 rejects with "Unknown config attribute", and restore *.sbt.zip publishing behind --sourmash_save_sourmash after reconsidering: it's a real genome index worth keeping on request, unlike *.sig/*.sig.zip. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * Clarify usage docs a bit * Remove CLAUDE.md, per nf-core consensus to use AGENTS.md only Repo-specific instructions now live entirely in AGENTS.md; the global CLAUDE.md at the user level already redirects to AGENTS.md for any repo that has one, so a per-repo CLAUDE.md is redundant and against current nf-core core-team consensus on avoiding coding-agent file clutter in pipeline repos. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * Ignore local genomes/ dir and .Rproj file These are local artifacts (documented in AGENTS.md's Gotchas), not pipeline source -- gitignoring them alongside the existing magmap_genomes/magmap_prokka entries also unblocks `nf-core pipelines lint --fix`, which refuses to run with any uncommitted/untracked changes present. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * Regenerate ro-crate-metadata.json for template 4.0.3 nf-core pipelines lint --fix rocrate_readme_sync, resolving the merge conflict from the template sync properly instead of just keeping the stale (pre-sync) content. * Update nf-core modules and subworkflows nf-core modules update --all + nf-core subworkflows update --all. Notable: wget's local patch (2-arg [meta, url] interface instead of the vendor's 3-arg [meta, url, suffix], matching magmap's own WGET_GENOME call site) failed to auto-reapply after the module's version bump, so it was reapplied by hand and the patch file regenerated via `nf-core modules patch wget`. Everything else (bam_sort_stats_samtools/bam_stats_samtools's take/emit interface, multiqc, samtools/*, trimgalore, cat/cat, custom/dumpsoftwareversions, utils_nfschema_plugin) updated without needing manual intervention. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * Regenerate container config files for updated modules nf-core pipelines lint --fix container_configs, picking up MultiQC's 1.34 -> 1.35 bump from the module update. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * Fill in CHANGELOG for template sync and module/subworkflow updates PR number left as a placeholder (#NN) until the PR exists. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * Pre-update tool version strings in nf-test snapshots Anticipates the module updates: samtools 1.23.1 -> 1.24 and trimgalore 2.1.0 -> 2.3.0, the only two tool versions that changed and are actually captured in snapshot content. Avoids a wasted first nf-test run that would fail solely on these known, predictable version-string diffs. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * Fix nf-test snapshots: correct version quoting, add missing genome-selection coverage Two issues surfaced by a real nf-test run after the module updates: - My earlier version-string pre-patch quoted samtools's new version ("1.24") the same way as the old one ("1.23.1"), but 1.24 parses as a JSON number and gets serialized unquoted -- fixed across all 7 affected snapshot files. - default/metadata/single/duplicates/skip_preprocessing_steps.nf.test were never updated for the COLLECT_GENOMESELECTION MultiQC summary feature added earlier this session -- I'd only updated the sourmash-related snapshots at the time, missing that this process runs unconditionally for every genomeset_mode=joint profile. Regenerated properly via --update-snapshot (real pipeline runs, not hand-edits) since the exact md5s/content aren't predictable. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * Switch GTDB metadata URL to a working mirror data.gtdb.aau.ecogenomic.org has an expired TLS certificate, breaking CI (certificate_expired PKIX validation failure staging ar53_metadata_r226.tsv.gz). data.gtdb.ecogenomic.org mirrors the same release and works. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * Bump minimum Nextflow version to 26.04.0 Nextflow 25.10.4 has a version-topic-aggregation bug that drops the PROKKAGFF2TSV versions.yml block whenever that process runs more than once per pipeline execution, breaking snapshot comparisons uniformly across every test block with multiple genomes lacking a GFF. 26.04.6 and 26.07.0-edge both aggregate correctly, so raise the declared minimum instead of working around the bug. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * Record Nextflow 26.04.0 minimum in CHANGELOG.md Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * Regenerate ro-crate-metadata.json for Nextflow 26.04.0 minimum Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * Manually add PROKKAGFF2TSV block to snapshots for Nextflow 26.04.0 Nextflow 25.10.4's version-topic aggregation bug silently dropped the PROKKAGFF2TSV versions.yml block whenever that process ran more than once per execution (see nextflow.config bump to 26.04.0). Recorded snapshots predate the bump, so the block is missing wherever GENOME_SKETCH indicates multiple genomes were processed. Added by hand using the same version strings already present in the other snapshot files in this branch (metadata/default/single/duplicates/skip_preprocessing_steps.nf.test.snap). Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * Regenerate ro-crate-metadata.json after merging upstream/dev Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * Remove version output channel from wget to allow storeDir * Revert faulty eWaterCycle/setup-apptainer digest from the 4.0.3 template sync nf-core/tools 4.0.3's template bumped this action to a digest (4bb22c52d4f63406c49e94c804632975787312b3, tagged v2.0.0) that's known to cause a namespace error in some pipelines' Singularity/Apptainer CI jobs. Revert to the previous pin (3f706d898c9db585b1d741b4692e66755f3a1b40, v2) until upstream fixes the release. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * Update CHANGELOG.md Co-authored-by: Diego Alvarez S. <dialvarezs@gmail.com> * Update CHANGELOG.md Co-authored-by: Diego Alvarez S. <dialvarezs@gmail.com> --------- Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com> Co-authored-by: nf-core-bot <core@nf-co.re> Co-authored-by: Diego Alvarez S. <dialvarezs@gmail.com>
Co-authored-by: Danilo Di Leo <78909156+danilodileo@users.noreply.github.com> Co-authored-by: nf-core-bot <core@nf-co.re> Co-authored-by: Daniel Lundin <erik.rikard.daniel@gmail.com>
* Better documentation of CheckM(2) and GTDB-Tk input * Make tidyverse/joinmetadata handle a combination of CheckM and CheckM2 files * Setup test case for #187 * Add CLAUDE.md * Implement --species_preference local to drop duplicate-species remote genomes When a user-provided local genome and a Sourmash-selected remote genome represent the same GTDB species, prefer the local one and skip fetching the remote duplicate. Species are resolved from --gtdbtk_metadata (local) and --gtdb_metadata (remote) in a new TIDYVERSE_SELECTGENOMESPECIES module, called before WGET_GENOME so duplicates are never downloaded. The 'completeness' and 'gtdb' preference modes remain unimplemented and now error explicitly instead of silently no-op'ing. Also fixes stale tRNA/rRNA assertions and thresholds in sourmash_mix_dupl_species.nf.test left over from before this feature existed, and updates its snapshot. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * Add test profile for prefer local in combination with sample-based genomesets * Add per-sample genome index manifest and fix single-file CheckM bug - Add --bbmap_save_genome_list (auto-on for --genomeset_mode sample) to publish which genome accessions went into each BBMap index, giving a read-independent way to verify per-sample genome selection. Threaded through CREATE_BBMAP_INDEX and asserted on in sourmash_mix_dupl_species_sampleset.nf.test. - Fix modules/local/tidyverse/joinmetadata: combining CheckM/CheckM2 files broke with exactly one --checkm_metadata file (unbalanced parens from a fragile string-joining trick), producing an R syntax error. Rebuilt with purrr::map + reduce(union), which works uniformly for any number of files. - Add two non-deterministic MultiQC files (multiqc_featurecounts.txt, featureCounts_assignment_plot.txt) to tests/.nftignore; their content can vary run-to-run due to BBMap's multi-threaded ambiguous-mapping resolution. - Update snapshots for metadata, sourmash_mix_dupl_species and sourmash_mix_dupl_species_sampleset (verified passing). The other test files sharing tests/.nftignore still need their snapshots regenerated on next run. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * Update snapshots after nftignore additions and more * Temporarily set test dataset url to my fork * Update two snapshots * Temporarily point test data at fork; validate per-sample genome selection - Move the fork test-data override from tests/nextflow.config to nextflow.config's own pipelines_testdata_base_path default. The conf/test_*.config files eagerly concatenate this value while being parsed as part of nextflow.config's profiles block, which happens before tests/nextflow.config (a later -c file) is ever merged, so overriding it there had no effect. Revert once the upstream nf-core/test-datasets PR merges. - Update sourmash_mix_dupl_species(.nf.test/.snap) and its sampleset variant against the updated fork data (one local genome removed, surfacing a previously-species-duplicated remote genome). - Rewrite the sampleset test's per-sample genome-list assertions: they previously assumed all samples get an identical genome set, which only held because the old test data had no local/remote divergence. With the new data, arc00/arc02 correctly pick up the surfaced remote genome via their own sourmash matches while arc01 doesn't -- this now genuinely exercises species_preference=local interacting with genomeset_mode=sample instead of being indistinguishable from joint mode. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * Implement --species_preference completeness and gtdb Extends TIDYVERSE_SELECTGENOMESPECIES (previously local-only) to also score genomes by CheckM completeness ('completeness') or GTDB's completeness - 5*contamination criterion ('gtdb'), comparing local and remote candidates for the same species and keeping whichever wins -- unlike 'local', these two modes can drop a local genome in favor of a better-scoring remote one. Ties are resolved in favor of the local genome. subworkflows/local/sourmash/main.nf is restructured so the local genome channel can actually be filtered post-hoc, which no mode needed before now. Adds dedicated test profiles/configs for completeness, gtdb, and completeness combined with genomeset_mode=sample, alongside the existing local-mode ones; updates the two existing dupl_species snapshots for the module's new (always-reported) purrr version entry. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * Consolidate overlapping Sourmash test profiles into two files Replaces 10 near-duplicate conf/test_*.config + tests/*.nf.test pairs (sourmash, sourmash_mix{,_sampleset}, sourmash_10genomes_{joint,sample}, sourmash_mix_dupl_species{,_sampleset,_completeness,_gtdb, _completeness_genomeset_sample}) with two files, each a single base config plus multiple nf-test cases that only override the params that differ: - tests/species_preference.nf.test: the full species_preference (local/completeness/gtdb) x genomeset_mode (joint/sample) matrix. - tests/sourmash_genome_selection.nf.test: species_preference='all', a remote-only scenario (no --genomeinfo), and multiple comma-separated --indexes files -- covering what the retired sourmash/sourmash_mix/10genomes tests checked, on the same archaeal dataset instead of three separate ones (e_coli, 24genomes, 10genomes). Also reverts pipelines_testdata_base_path in nextflow.config back to the official nf-core/test-datasets repo now that the updated archaeal test data (one local species removed, needed to actually exercise genomeset_mode=sample divergence) has been merged upstream. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * Linting fixes * Update CHANGELOG.md, usage.md and output.md * Add genome-selection summary to MultiQC report and unify genome-list manifest Summarize local vs. remote genome counts per genome set (per-sample in 'sample' mode, a single row in 'joint' mode) as a MultiQC custom-content table, plus a full per-genome breakdown in summary_tables/. The bbmap/*.genomes.txt manifest is now always written for both genomeset modes, making --bbmap_save_genome_list redundant, so it's removed. Also fix the sourmash subworkflow's meta.yml (previously undocumented/ mismatched outputs) and ignore Sourmash's *.sbt.zip index in nf-test snapshots, since its content isn't reproducible across runs. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * Make forgotten version bump to 1.2.0.dev * Stop publishing Sourmash index/signature files by default *.sbt.zip (genome index), *.sig (signatures) and *.sig.zip (gather matches/prefetch/unassigned) are large, non-reproducible intermediate Sourmash artifacts with little standalone value; only the *.csv.gz gather results remain gated behind --sourmash_save_sourmash. Update the one affected test snapshot to match, and refresh the "Workflow" version string (v1.1.0 -> v1.2.0dev) in every snapshot left stale by the recent version bump. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * Delayed version bump * Move CLAUDE.md content to AGENTS.md, matching other nf-core repos CLAUDE.md is now a thin pointer to AGENTS.md, which follows the same structure used in nf-core-ampliseq (What this is / Commands / Architecture / Conventions / Gotchas, plus the critical remote nf-core/agents fetch instruction). Also refreshes stale test file references from earlier test-suite consolidation and adds a few gotchas learned this session (empty withName blocks, CI's 4-cpu runners, oversized base.config memory labels for test genomes). Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * Template update for nf-core/tools version 4.0.3 * Speed up Prokka in tests and restore .sbt.zip publishing Give Prokka its own tight memory reservation in every test profile (2GB vs. its process_low label's 12GB default, which was never actually clamped by the resourceLimits ceiling) so more Prokka jobs run concurrently instead of serializing in pairs -- Prokka dominates test wall-clock time (~90%+ in local profiling) and only uses ~450MB in practice on these small genomes. Also fix an empty withName: GENOME_INDEX {} block (introduced when I first stopped publishing *.sbt.zip) that Nextflow 25.10.4 rejects with "Unknown config attribute", and restore *.sbt.zip publishing behind --sourmash_save_sourmash after reconsidering: it's a real genome index worth keeping on request, unlike *.sig/*.sig.zip. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * Clarify usage docs a bit * Remove CLAUDE.md, per nf-core consensus to use AGENTS.md only Repo-specific instructions now live entirely in AGENTS.md; the global CLAUDE.md at the user level already redirects to AGENTS.md for any repo that has one, so a per-repo CLAUDE.md is redundant and against current nf-core core-team consensus on avoiding coding-agent file clutter in pipeline repos. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * Ignore local genomes/ dir and .Rproj file These are local artifacts (documented in AGENTS.md's Gotchas), not pipeline source -- gitignoring them alongside the existing magmap_genomes/magmap_prokka entries also unblocks `nf-core pipelines lint --fix`, which refuses to run with any uncommitted/untracked changes present. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * Regenerate ro-crate-metadata.json for template 4.0.3 nf-core pipelines lint --fix rocrate_readme_sync, resolving the merge conflict from the template sync properly instead of just keeping the stale (pre-sync) content. * Update nf-core modules and subworkflows nf-core modules update --all + nf-core subworkflows update --all. Notable: wget's local patch (2-arg [meta, url] interface instead of the vendor's 3-arg [meta, url, suffix], matching magmap's own WGET_GENOME call site) failed to auto-reapply after the module's version bump, so it was reapplied by hand and the patch file regenerated via `nf-core modules patch wget`. Everything else (bam_sort_stats_samtools/bam_stats_samtools's take/emit interface, multiqc, samtools/*, trimgalore, cat/cat, custom/dumpsoftwareversions, utils_nfschema_plugin) updated without needing manual intervention. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * Regenerate container config files for updated modules nf-core pipelines lint --fix container_configs, picking up MultiQC's 1.34 -> 1.35 bump from the module update. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * Fill in CHANGELOG for template sync and module/subworkflow updates PR number left as a placeholder (#NN) until the PR exists. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * Pre-update tool version strings in nf-test snapshots Anticipates the module updates: samtools 1.23.1 -> 1.24 and trimgalore 2.1.0 -> 2.3.0, the only two tool versions that changed and are actually captured in snapshot content. Avoids a wasted first nf-test run that would fail solely on these known, predictable version-string diffs. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * Fix nf-test snapshots: correct version quoting, add missing genome-selection coverage Two issues surfaced by a real nf-test run after the module updates: - My earlier version-string pre-patch quoted samtools's new version ("1.24") the same way as the old one ("1.23.1"), but 1.24 parses as a JSON number and gets serialized unquoted -- fixed across all 7 affected snapshot files. - default/metadata/single/duplicates/skip_preprocessing_steps.nf.test were never updated for the COLLECT_GENOMESELECTION MultiQC summary feature added earlier this session -- I'd only updated the sourmash-related snapshots at the time, missing that this process runs unconditionally for every genomeset_mode=joint profile. Regenerated properly via --update-snapshot (real pipeline runs, not hand-edits) since the exact md5s/content aren't predictable. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * Switch GTDB metadata URL to a working mirror data.gtdb.aau.ecogenomic.org has an expired TLS certificate, breaking CI (certificate_expired PKIX validation failure staging ar53_metadata_r226.tsv.gz). data.gtdb.ecogenomic.org mirrors the same release and works. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * Bump minimum Nextflow version to 26.04.0 Nextflow 25.10.4 has a version-topic-aggregation bug that drops the PROKKAGFF2TSV versions.yml block whenever that process runs more than once per pipeline execution, breaking snapshot comparisons uniformly across every test block with multiple genomes lacking a GFF. 26.04.6 and 26.07.0-edge both aggregate correctly, so raise the declared minimum instead of working around the bug. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * Record Nextflow 26.04.0 minimum in CHANGELOG.md Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * Regenerate ro-crate-metadata.json for Nextflow 26.04.0 minimum Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * Manually add PROKKAGFF2TSV block to snapshots for Nextflow 26.04.0 Nextflow 25.10.4's version-topic aggregation bug silently dropped the PROKKAGFF2TSV versions.yml block whenever that process ran more than once per execution (see nextflow.config bump to 26.04.0). Recorded snapshots predate the bump, so the block is missing wherever GENOME_SKETCH indicates multiple genomes were processed. Added by hand using the same version strings already present in the other snapshot files in this branch (metadata/default/single/duplicates/skip_preprocessing_steps.nf.test.snap). Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * Regenerate ro-crate-metadata.json after merging upstream/dev Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * Remove version output channel from wget to allow storeDir * Install and patch Bakta modules; drop Prokka's uncollected versions.yml output Install bakta/bakta and bakta/baktadbdownload from nf-core/modules. Both vendored modules emit their version via an eval() output on a topic: versions channel -- incompatible with storeDir (only val/path outputs are allowed there), which these processes will need for caching. Patch both to drop that line; a dedicated version-reporting process is added separately. Also drop Prokka's own versions.yml output entirely: it was never actually wired into the pipeline's version collation (a pre-existing bug), and since it's being replaced by the same dedicated version-reporting approach, there's no reason to keep declaring it -- doing so would otherwise require every existing user's prokka_store_dir cache to already contain that file, or force a re-run of already-cached genomes. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * Add PROKKA_VERSION/BAKTA_VERSION, decoupled from the storeDir-cached processes Both tools' versions don't vary per genome within a single pipeline run, so there's no need to report them per-genome (or at all, for cache-hit genomes). These are tiny, uncached, zero-input processes that just run `prokka --version`/`bakta --version` once per execution and report via the same eval() ... topic: versions one-liner GENOMES2ORFS already uses for sed -- since they have no storeDir, that pattern is safe here. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * Add TIDYVERSE_SELECTANNOTATOR and local Bakta subworkflow TIDYVERSE_SELECTANNOTATOR splits genomes lacking a GFF between Prokka and Bakta according to --annotator and each genome's GTDB domain (extracted from the gtdb_taxonomy column TIDYVERSE_JOINMETADATA already produces from --gtdb_metadata/--gtdbtk_metadata) -- no new metadata parsing needed. The local BAKTA subworkflow wraps BAKTA_BAKTADBDOWNLOAD -> BAKTA_BAKTA -> BAKTA_VERSION, mirroring the download-module + tool-module subworkflow pattern (e.g. metatdenovo's KOFAMSCAN/EGGNOG). BAKTA_BAKTA's on-disk output is patched to match Prokka's convention: gzipped files under a --output ${prefix} subdirectory, so a shared storeDir accumulates one subdirectory per genome the same way magmap_prokka does. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * Regenerate bakta-bakta.diff to include the --output/gzip patch The previous commit's patch file only reflected the version-line removal, not the later --output/gzip restructuring applied in the same commit -- a stale diff that would confuse nf-core modules update. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * Wire --annotator/--bakta_db/--bakta_store_dir into the pipeline Add --annotator (prokka/bakta_supported_only/bakta_all, default prokka) and route genomes lacking a GFF accordingly in workflows/magmap.nf: split via TIDYVERSE_SELECTANNOTATOR when not plain 'prokka', warn (once) about genomes bakta_supported_only couldn't classify by domain, and mix Bakta's annotated genomes back into ch_collected_genomes alongside Prokka's. Bakta's .gff3 goes through the existing PROKKAGFF2TSV module rather than Bakta's own .tsv, whose columns (Sequence Id/Type/Start/Stop/Strand/Locus Tag/Gene/Product/DbXrefs) don't match Prokka's -- reusing the generic GFF3 parser already used for user-supplied GFFs avoids a Bakta-specific TSV adapter entirely. Add --bakta_db (storeDir for BAKTA_BAKTADBDOWNLOAD, doubling as a bring-your- own-database path) and --bakta_store_dir (storeDir for BAKTA_BAKTA, mirroring --prokka_store_dir), including the same pre-create-directory handling in PIPELINE_INITIALISATION that genome_store_dir/prokka_store_dir already get. Also fix GENOMES2ORFS: it stripped only .gff/.gff.gz suffixes when deriving each genome's accno from its GFF filename, silently leaving Bakta's .gff3/ .gff3.gz extension in place and corrupting the accno column for every Bakta-annotated genome. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * Add test_bakta profile and nf-test coverage for --annotator Combines genomes from both existing test_metadata (bacterial, GCA_002688505.1) and test_species_preference (archaeal, the local_* genomes) via a new genomeinfo fixture that omits genome_gff, so both domains need annotation -- exercising the Bakta path (Bacteria) and the Prokka fallback path (Archaea, unsupported by Bakta) in the same run. Uses Bakta's light database (~1.3GB), the smallest official variant and what nf-core/modules' own bakta tests use too; there's no smaller custom test database available (see oschwengers/bakta#95). tests/bakta.nf.test covers both --annotator bakta_supported_only (the profile default) and bakta_all. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * Document --annotator/Bakta support and add Bakta citation Add a "Genome annotation" section to usage.md covering both Prokka and Bakta, including the --annotator table and the bakta_supported_only domain-fallback behaviour; extend the storage-directory docs with --bakta_db/--bakta_store_dir; add a matching "Bakta" section to output.md alongside Prokka's; add Bakta's citation to CITATIONS.md; record both the new feature and the Prokka-versions fix in CHANGELOG.md. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * Update existing test snapshots for PROKKA_VERSION Real nf-test run (--update-snapshot) against all 7 previously-existing profiles. Verified structurally (not just via raw diff, which is misleading here due to top-level key reordering across runs) that every change is exactly the new "PROKKA_VERSION": {"prokka": "1.14.6"} entry plus the matching +1 succeeded-task count -- nothing else differs. One case (sourmash_genome_selection.nf.test, 'local + remote, genomeset_mode: sample') failed on this run with a transient "Network is unreachable" error staging a test-datasets file, unrelated to any pipeline change; its snapshot is correctly untouched and will be retried separately. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * Update sourmash_genome_selection snapshot for the network-flake retry 'local + remote, genomeset_mode: sample' failed on the previous run with a transient network error unrelated to any pipeline change; retried and passed cleanly. Verified the diff is exactly the same PROKKA_VERSION addition as the other test cases in this file. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * Fix stale setup-apptainer digest missed by the dev merge Git's 3-way merge couldn't see this: bakta-annotator's copy went stale->faulty and was never touched again, while dev went stale->faulty->fixed (back to the same value as the merge-base). Since "theirs" ended up textually identical to the merge-base, git treated it as unchanged and silently kept our still-faulty side -- a real, if narrow, git merge blind spot, not something a conflict marker would have caught. Verified by diffing every file bakta-annotator had independently diverged on against upstream/dev post-merge; this was the only one affected. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * Simplify CHANGELOG entry for the setup-apptainer CI fix CHANGELOG entries are for end users, not internal CI plumbing detail like specific GitHub Action digests. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * Fix BAKTA_VERSION: set HOME so matplotlib doesn't corrupt the captured version BAKTA_BAKTA/BAKTA_BAKTADBDOWNLOAD both set HOME to a writable dir before running bakta; BAKTA_VERSION's empty script never did. Without a writable HOME, bakta's matplotlib/fontconfig cache setup fails and dumps multi-line warnings to stderr, which the eval() command's `2>&1` merges straight into the captured "version" string -- corrupting the collated versions.yml enough that MultiQC's YAML parser choked on it. Confirmed by a real nf-test run. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * Give each Bakta-annotated genome a unique --locus/--locus-tag Bakta's --locus (contig-name prefix) defaults to the literal "contig" for every genome, and --locus-tag (feature/gene ID prefix) defaults to an MD5 hash of the input sequences truncated to 6 characters -- both effectively the same across genomes, or collision-prone (the same class of fingerprint-based-prefix collision seen from Prokka's own algorithm before). Concatenating multiple Bakta-annotated genomes into one shared BBMap index broke on the resulting duplicate contig names; FEATURECOUNTS/GENOMES2ORFS would have hit the same problem via colliding GFF `ID=` values. Using the accno directly (as PROKKA's --prefix already does for its own contig naming) was tried and reverted: Bakta rejects locus prefixes over 20 characters or containing anything outside [A-Za-z0-9_-], and confirmed via a real nf-test run against a genome whose accno exceeded that. Local genome accnos are arbitrary user text with no such guarantee, unlike remote/NCBI ones. Hash the accno instead (SHA-256, first 20 hex chars, uppercase) -- deterministic and always a valid prefix regardless of the accno's length or characters. This gives up being able to eyeball which genome a contig/ORF name came from, which was a nice side effect of the accno-based scheme but never a requirement. Documented the resulting caveat in usage.md: since the hash is derived from accno, reusing an accno for a genuinely different local genome across runs still means the storeDir cache is keyed on the old (now-mismatched) identity -- the same class of caveat that already existed for --prokka_store_dir, now also true of Bakta's contig/feature naming. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * Add bakta.nf.test.snap: first successful --annotator test run Both bakta_supported_only and bakta_all pass cleanly against the accno-hash --locus/--locus-tag fix and the BAKTA_VERSION HOME fix. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * Remove TODO from CONTRIBUTING.md * Add mimetype to the --input schema property Fixes the nf-core lint schema_lint warning ("Input mimetype is missing or empty"). Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * Skip tests/bakta.nf.test in CI -- Bakta DB download exceeds job timeout Both cases in this file download the full Bakta database from scratch (no cache available on ephemeral CI runners), which regularly runs past 6h and gets the job cancelled. Exclude the file from nf-test's ignore list only when CI=true (set automatically by the GitHub Actions runner) so it's still discovered and runs normally locally, where the DB can be pre-populated via --bakta_db/--bakta_store_dir. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * Update CHANGELOG with PR #217 number Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com> Co-authored-by: nf-core-bot <core@nf-co.re>
* Add DUCKDB_TSV2PARQUET local module Converts a batch of gzipped/plain TSV files to Parquet using DuckDB's Python bindings (conda-forge's duckdb package ships the library + Python API, not a standalone CLI binary -- confirmed by trying to shell out to `duckdb` first and getting "command not found" despite the container pulling fine). Intended to be contributed to nf-core/modules once this branch's tests pass, per the project's convention for genuinely new, reusable functionality; kept local for now so it can be exercised end-to-end here first. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * Add --save_parquet to also write summary tables as Parquet Gated behind a new boolean param (default false); when set, DUCKDB_TSV2PARQUET runs once at the end of the workflow against the full set of already-produced summary_tables/*.tsv.gz files, writing a same-named *.parquet alongside each. Verified with a real nf-test run (test_duplicates + save_parquet: true) and spot-checked the resulting Parquet files' content/types directly with DuckDB. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * Migrate from local duckdb/tsv2parquet to vendored duckdb/table2parquet Now that the module has been merged upstream into nf-core/modules, replace the local batch-style module with the vendored per-file one. This changes DUCKDB_TABLE2PARQUET from a single task looping over all summary tables to one task per table, with the output basename now carried via meta.id (derived from each table's own filename) instead of being computed inside the script. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * Regenerate container config files for duckdb/table2parquet Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * Empty commit to restart CI The previous nf-test run's shard-discovery job was cancelled before its downstream jobs were created, leaving GitHub unable to rerun it directly. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * Add a line to trigger tests * Revert the test-trigger comment line The nf-test CI runner never picked up the last few pushes (jobs stuck in queued/pending with no runner provisioned, unrelated to nf-test's own changed-since test selection), so this didn't end up serving its intended purpose. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * github pr rerun checks --------- Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
…LECT_GENOMESELECTION (#222) * Document that overall_stats count columns share the same read units Verified empirically (real bbduk.sh/samtools/featureCounts runs, not just documentation) that n_post_trimming, n_non_contaminated, idxs_n_mapped/unmapped, and the featureCounts columns all count individual reads with both mates counted separately -- none of them count fragments/pairs as one unit, so the columns are comparable. The featureCounts case is the non-obvious one: SUBREAD_FEATURECOUNTS adds -p automatically for paired-end samples, but without --countReadPairs it still counts at the individual-read level, so it matches the other columns. Recorded this so a future "modernization" adding --countReadPairs doesn't silently break comparability. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * Define TPM in output.md * Fix brittle R code in COLLECT_STATS/COLLECT_GENOMESELECTION Addresses #218: - Both modules built R source text by directly interpolating user-controlled values (sample IDs, genome accessions -- documented as accepting arbitrary text via --genomeinfo) into single-quoted R string literals. A value containing a quote would break the generated R syntax; something more adversarial could inject R code. Added a top-level rq() helper that safely escapes a value as a single-quoted R literal, used everywhere such a value is embedded. (Defined as a top-level function rather than a script-local closure: `nextflow lint`'s strict-syntax parser rejects a closure captured from an outer scope being called from within a nested closure literal -- confirmed by writing a minimal reproduction and checking it against both forms.) - COLLECT_STATS also shelled out to grep/sed via pipe(sprintf(...)) to parse trimming reports, bbduk logs, and idxstats files, passing sample-derived filenames through a shell command string. Rewrote all three to read the files directly in R (readLines()/read_tsv() + regex, or a plain dplyr filter for idxstats) instead, removing the shell subprocess entirely. - COLLECT_STATS hardcoded FeatureCounts' column layout as a positional type-code string (col_types = 'ccciicicid'), which happened to mirror COLLECT_FEATURECOUNTS's column order but wasn't enforced to stay in sync. Switched to naming each column's type explicitly via cols(), so a future column reorder in COLLECT_FEATURECOUNTS can't silently misparse here. Verified the Groovy string-escaping fix with a standalone nf-test reproduction (tricky sample IDs containing quotes/backslashes, checked against both the working and broken closure forms), and the R rewrites against the real featureCounts-derived counts files pulled from an actual test run's work directory (exact same numbers as before: 340 CDS / 21 rRNA / 8 tRNA for sample s0). A full pipeline nf-test run could not be completed to confirm end-to-end: two consecutive attempts died to host-level OOM (Nextflow's own JVM orchestrator threads, not the R task) under memory pressure from unrelated processes on this machine, not from these changes -- still outstanding before this is considered fully proven. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * Try GitHub-hosted ubuntu-latest runners for nf-test (self-hosted runners stuck queued) * Add CHANGELOG entry for the #218 fix * Update CHANGELOG with PR #222 number, trim Fixed entry wording * Fix corrupted GString interpolation in COLLECT_STATS caused by a backtick in a comment Root cause of the two "column X not found" R failures seen in local nf-test runs and in CI (magmap#222, shard 5/7, docker|26.04.0): a comment introduced by the brittle-R-code fix contained a backtick and a `\*` escape (`` `grep -v '^\*'` ``). Inside the process's outer triple-quoted GString, that combination throws off the value-to-placeholder alignment for every interpolation from that point on -- confirmed by a controlled test: reverting just this comment's wording (no other change) took a run from reproducibly failing to passing, both locally and matching the exact corruption pattern seen on GitHub-hosted CI runners (never a host-memory or JVM-heap issue, despite earlier appearances -- two of the three failures happened to coincide with memory pressure, which was a red herring). Not a Groovy/Nextflow bug worth reporting upstream without a minimal standalone reproduction; a small isolated repro with the same nested-GString structure did not reproduce it, so whatever triggers this needs more specific conditions than have been isolated so far. Simplest fix: don't put a backtick + backslash-escape combination in a script comment. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * Fix intermittent ConcurrentModificationException in COLLECT_GENOMESELECTION CREATE_BBMAP_INDEX.out.genome_accnos is consumed by two independent channel subscribers: this collectFile (which calls accnos.sort()) and COLLECT_GENOMESELECTION (which iterates the same accnos list). Groovy's no-arg List.sort() sorts in place rather than returning a new list, so when both subscribers ran concurrently, one could be mutating the shared list while the other was mid-iteration over it -- seen as an intermittent ConcurrentModificationException in COLLECT_GENOMESELECTION on CI (magmap#222). Predates this branch (introduced in #212); found while chasing an intermittent nf-test CI failure on this PR. sort(false) returns a new sorted list instead of mutating the shared one, removing the race. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * Add CHANGELOG entry for the ConcurrentModificationException fix --------- Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
* Template update for nf-core/tools version 4.0.3 * Template update for nf-core/tools version 4.1.0 * Update nf-core modules and subworkflows for the 4.1.0 template sync Bumps prokka 1.14.6 -> 1.15.6 (openjdk 8.0.412 -> 8.0.472) and duckdb/table2parquet's test snapshot (no functional change -- upstream tidied redundant positional emit assertions). Removes a stale, unused "patch" reference for utils_nextflow_pipeline in modules.json (no .diff file ever existed for it). Pre-patches the prokka version string in pipeline-level test snapshots to avoid a wasted nf-test cycle. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * Regenerate container config files via nf-core pipelines lint --fix Fixes the container/conda directive mismatch left by the template regeneration (all fastqc/multiqc entries used the wrong 'container =' key instead of 'conda =' for conda-lock files) and restores the DUCKDB_TABLE2PARQUET entry that was missing from the regenerated files. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * Fix stale #NN placeholder for the save_parquet PR (#219) in CHANGELOG * Keep modules/local/prokka_version's environment in sync with modules/nf-core/prokka nf-core modules update only touches vendored modules/nf-core/ content, so the local prokka_version module (added in #217 to work around the vendored module never reporting its own version) still had environment.yml pinned to the OLD prokka=1.14.6/openjdk=8.0.412, silently reporting a wrong-but- plausible version after the vendored module's 1.15.6 bump in this sync -- caught only by comparing an actual nf-test run's real versions.yml against a pre-patched snapshot, not by any automated check. Bumped environment.yml to match the vendored module exactly (same prokka/openjdk/parallel package set) and switched the container reference to reuse modules/nf-core/prokka's own container verbatim, rather than building a near-duplicate Wave image for an environment that's now identical to the vendored one. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * Add missing Changed entry for the template sync, fix #NN placeholder The template-sync + module/subworkflow-update batch itself was missing its own CHANGELOG entry (should follow the same pattern as #215's 4.0.3 sync), and the prokka_version fix still had the #NN PR-number placeholder now that #223 is the real number. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * Reorder CHANGELOG entries descending by PR number within each section --------- Co-authored-by: nf-core-bot <core@nf-co.re> Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
Co-authored-by: Danilo Di Leo <78909156+danilodileo@users.noreply.github.com> Co-authored-by: nf-core-bot <core@nf-co.re> Co-authored-by: Matthias Hörtenhuber <mashehu@users.noreply.github.com>
Co-authored-by: Danilo Di Leo <78909156+danilodileo@users.noreply.github.com> Co-authored-by: nf-core-bot <core@nf-co.re> Co-authored-by: Matthias Hörtenhuber <mashehu@users.noreply.github.com>
# Conflicts: # .nf-core.yml
|
❌ nf-test failed with latest Nextflow versionNote Tests with Nextflow's latest version failed but it will not cause a CI workflow failure.
See the full run for details. |
erikrikarddaniel
marked this pull request as ready for review
August 4, 2026 15:04
…ir (#230) Genomes already present in --prokka_store_dir/--bakta_store_dir are skipped rather than re-annotated on later runs, so a genome library accumulated over time can end up with a mix of annotation-tool versions -- not reflected in the collated versions.yml/MultiQC report, which only shows the version used in the current run. Documenting this now (issue #229); actually scanning per-genome logs to report the real mix is a heavier follow-up, deferred to a later milestone pending a practical approach at genome-library scale (tens of thousands of genomes). Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
| The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/) | ||
| and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). | ||
|
|
||
| ## v1.2.0 - [YYYY-mm-dd] |
FriederikeHanssen
approved these changes
Aug 5, 2026
FriederikeHanssen
left a comment
FriederikeHanssen
left a comment
There was a problem hiding this comment.
Great work! There are a couple of smaller items, none are super big I think
- Add a DuckDB citation to CITATIONS.md (used by --save_parquet).
- Only create the Bakta database/store directories when --annotator
actually needs Bakta, not unconditionally regardless of annotator
choice. Threads params.annotator into PIPELINE_INITIALISATION to
make this possible (it wasn't passed there before).
- Add the missing wget component and Prokka/Bakta annotation step to
subworkflows/local/sourmash/meta.yml and README.md respectively.
- Rename modules/local/tidyverse/selectgenomespecies/environment.yml's
leftover scaffolding name ("r-test") to match the module. The same
leftover exists in six other local modules' environment.yml files,
left untouched here as out of scope for this PR.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
vagkaratzas
requested changes
Aug 6, 2026
vagkaratzas
left a comment
vagkaratzas
left a comment
There was a problem hiding this comment.
Left some comments, some of which were initiated by a bot review.
Make sure the nf-core p lint --release and nextflow lint * commands pass (with as few warnigns as possible).
I get a [✗] files_unchanged: .github/PULL_REQUEST_TEMPLATE.md does not match the template but I guess that's expected with the template sync madness going on lately. Just make sure
modules/local/collect/genomeselection/main.nf built an R tribble() literal with one line per (sample, accession) pair, interpolated directly into the generated script. In --genomeset_mode sample the row count is samples x genomes -- a run with hundreds of samples and thousands of genomes would generate a script with millions of lines. An empty genome set also produced a tribble() with no rows, which is an R syntax error rather than an empty table. Read accessions from the already-published bbmap/*.genomes.txt files instead (reusing the same collectFile channel already writing them, rather than duplicating the write), passed as a path input instead of a Groovy val. Fixes both the scaling issue and the empty-set syntax error as a side effect of no longer building the script from Groovy- interpolated text at all. Also drops the now-unnecessary accnos.sort(false) workaround in the collectFile that wrote bbmap/*.genomes.txt: it was specifically to avoid a race with COLLECT_GENOMESELECTION's other direct consumption of the same list, which this refactor removes. Verified in isolation (real accessions, a zero-genome sample, and the zero-genome-sets case) before running the full pipeline test. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
- Gate BAKTA_BAKTADBDOWNLOAD on ch_fasta actually having content, so the multi-GB database isn't downloaded when annotator requests Bakta but domain classification routed every genome to Prokka instead. - Move CHANGELOG entries to their correct release sections, fix a missing parenthesis, and add Bakta/DuckDB to the Dependencies table. - Remove a redundant "Prokka memory" comment copy-pasted across 7 test configs (YAGNI for comments). - Fix local-module names in bakta/sourmash meta.yml components lists to match nf-core lint's actual naming convention (derived from the process name, not the file path), clearing two lint warnings. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Matches each module's own meta.yml name field; same cleanup already applied to collect/genomeselection and tidyverse/selectgenomespecies earlier in this PR. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
- Pass species_preference into PIPELINE_INITIALISATION via take: instead of reading params.species_preference directly, matching this repo's params-flow-as-explicit-values convention. - Gitignore magmap_bakta/magmap_bakta_db/magmap_bakta_db_test, created in the launch directory. - Fold the single-param output_options schema category into the standard input_output_options category. - Apply two docs/output.md wording fixes and one docs/usage.md typo fix from the review's inline suggestions. - Fix comment alignment on the annotator take: parameter. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
vagkaratzas
reviewed
Aug 11, 2026
vagkaratzas
left a comment
vagkaratzas
left a comment
There was a problem hiding this comment.
Can you reply/resolve the addressed comments and leave the doubtful/remaining ones open so it's easier for me to follow what's been done and whatnot yet please? :D
Member
Author
I think I've replied to all yours now. I resolved some; don't know what's the best practice. And Rike's are all resolved except the update date reminder. |
vagkaratzas
self-requested a review
August 11, 2026 08:50
vagkaratzas
approved these changes
Aug 11, 2026
vagkaratzas
left a comment
vagkaratzas
left a comment
There was a problem hiding this comment.
Approved. Don't forget to add the release date on the CHANGELOG.
FYI, there were 12 more hidden review points that needed a reply :P
You have to expand those to see them on GitHub side.
Member
Author
|
Thanks @FriederikeHanssen and @vagkaratzas . Much better thanks to your efforts. |
erikrikarddaniel
added a commit
that referenced
this pull request
Aug 11, 2026
master and dev diverged when PR #228 (the v1.2.0 release) was squash-merged into master rather than merged with history preserved, and a follow-up date-fix commit (#232) was pushed directly to master outside the normal PR flow. This brings dev back in sync with master's actual released state. # Conflicts: # CHANGELOG.md
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
PR checklist
nf-core pipelines lint).nextflow run . -profile test,docker --outdir <OUTDIR>).nextflow run . -profile debug,test,docker --outdir <OUTDIR>).docs/usage.mdis updated.docs/output.mdis updated.CHANGELOG.mdis updated.README.mdis updated (including new tool citations and authors/contributors).Description
Release PR for v1.2.0.
Since the last release (v1.1.0, 2026-05-28):
COLLECT_STATS/COLLECT_GENOMESELECTION(#218)--save_parquetto also write summary tables as Parquet--species_preferenceparameter and genome-selection summary in MultiQCnf-core pipelines lint --releasepasses cleanly (0 failures).Note: the
CHANGELOG.mdrelease date (## v1.2.0 - [YYYY-mm-dd]) will be finalized with the actual merge date once this PR is approved, in a follow-up commit before merging.