Repository navigation
Merge master into dev (reconcile security patch #214) - #226
Merged
Merged
Conversation
Magmap release v1.1.0
Remove vulnerable PR-comment artifact pattern
# Conflicts: # .nf-core.yml
erikrikarddaniel
marked this pull request as ready for review
August 4, 2026 10:15
|
erikrikarddaniel
added a commit
that referenced
this pull request
Aug 11, 2026
* Update nf-schema and ignore igenomes in config (#210)
* Ignore igenomes in config
* CHANGELOG.md
* Add species-preference ability (#212)
* Better documentation of CheckM(2) and GTDB-Tk input
* Make tidyverse/joinmetadata handle a combination of CheckM and CheckM2 files
* Setup test case for #187
* Add CLAUDE.md
* Implement --species_preference local to drop duplicate-species remote genomes
When a user-provided local genome and a Sourmash-selected remote genome
represent the same GTDB species, prefer the local one and skip fetching
the remote duplicate. Species are resolved from --gtdbtk_metadata (local)
and --gtdb_metadata (remote) in a new TIDYVERSE_SELECTGENOMESPECIES module,
called before WGET_GENOME so duplicates are never downloaded. The
'completeness' and 'gtdb' preference modes remain unimplemented and now
error explicitly instead of silently no-op'ing.
Also fixes stale tRNA/rRNA assertions and thresholds in
sourmash_mix_dupl_species.nf.test left over from before this feature
existed, and updates its snapshot.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
* Add test profile for prefer local in combination with sample-based genomesets
* Add per-sample genome index manifest and fix single-file CheckM bug
- Add --bbmap_save_genome_list (auto-on for --genomeset_mode sample) to
publish which genome accessions went into each BBMap index, giving a
read-independent way to verify per-sample genome selection. Threaded
through CREATE_BBMAP_INDEX and asserted on in
sourmash_mix_dupl_species_sampleset.nf.test.
- Fix modules/local/tidyverse/joinmetadata: combining CheckM/CheckM2
files broke with exactly one --checkm_metadata file (unbalanced
parens from a fragile string-joining trick), producing an R syntax
error. Rebuilt with purrr::map + reduce(union), which works
uniformly for any number of files.
- Add two non-deterministic MultiQC files (multiqc_featurecounts.txt,
featureCounts_assignment_plot.txt) to tests/.nftignore; their content
can vary run-to-run due to BBMap's multi-threaded ambiguous-mapping
resolution.
- Update snapshots for metadata, sourmash_mix_dupl_species and
sourmash_mix_dupl_species_sampleset (verified passing). The other
test files sharing tests/.nftignore still need their snapshots
regenerated on next run.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
* Update snapshots after nftignore additions and more
* Temporarily set test dataset url to my fork
* Update two snapshots
* Temporarily point test data at fork; validate per-sample genome selection
- Move the fork test-data override from tests/nextflow.config to
nextflow.config's own pipelines_testdata_base_path default. The
conf/test_*.config files eagerly concatenate this value while being
parsed as part of nextflow.config's profiles block, which happens
before tests/nextflow.config (a later -c file) is ever merged, so
overriding it there had no effect. Revert once the upstream
nf-core/test-datasets PR merges.
- Update sourmash_mix_dupl_species(.nf.test/.snap) and its sampleset
variant against the updated fork data (one local genome removed,
surfacing a previously-species-duplicated remote genome).
- Rewrite the sampleset test's per-sample genome-list assertions: they
previously assumed all samples get an identical genome set, which
only held because the old test data had no local/remote divergence.
With the new data, arc00/arc02 correctly pick up the surfaced remote
genome via their own sourmash matches while arc01 doesn't -- this now
genuinely exercises species_preference=local interacting with
genomeset_mode=sample instead of being indistinguishable from joint
mode.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
* Implement --species_preference completeness and gtdb
Extends TIDYVERSE_SELECTGENOMESPECIES (previously local-only) to also
score genomes by CheckM completeness ('completeness') or GTDB's
completeness - 5*contamination criterion ('gtdb'), comparing local and
remote candidates for the same species and keeping whichever wins --
unlike 'local', these two modes can drop a local genome in favor of a
better-scoring remote one. Ties are resolved in favor of the local
genome. subworkflows/local/sourmash/main.nf is restructured so the
local genome channel can actually be filtered post-hoc, which no mode
needed before now.
Adds dedicated test profiles/configs for completeness, gtdb, and
completeness combined with genomeset_mode=sample, alongside the
existing local-mode ones; updates the two existing dupl_species
snapshots for the module's new (always-reported) purrr version entry.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
* Consolidate overlapping Sourmash test profiles into two files
Replaces 10 near-duplicate conf/test_*.config + tests/*.nf.test pairs
(sourmash, sourmash_mix{,_sampleset}, sourmash_10genomes_{joint,sample},
sourmash_mix_dupl_species{,_sampleset,_completeness,_gtdb,
_completeness_genomeset_sample}) with two files, each a single base
config plus multiple nf-test cases that only override the params that
differ:
- tests/species_preference.nf.test: the full species_preference
(local/completeness/gtdb) x genomeset_mode (joint/sample) matrix.
- tests/sourmash_genome_selection.nf.test: species_preference='all',
a remote-only scenario (no --genomeinfo), and multiple
comma-separated --indexes files -- covering what the retired
sourmash/sourmash_mix/10genomes tests checked, on the same archaeal
dataset instead of three separate ones (e_coli, 24genomes,
10genomes).
Also reverts pipelines_testdata_base_path in nextflow.config back to
the official nf-core/test-datasets repo now that the updated archaeal
test data (one local species removed, needed to actually exercise
genomeset_mode=sample divergence) has been merged upstream.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
* Linting fixes
* Update CHANGELOG.md, usage.md and output.md
* Add genome-selection summary to MultiQC report and unify genome-list manifest
Summarize local vs. remote genome counts per genome set (per-sample in
'sample' mode, a single row in 'joint' mode) as a MultiQC custom-content
table, plus a full per-genome breakdown in summary_tables/. The
bbmap/*.genomes.txt manifest is now always written for both genomeset
modes, making --bbmap_save_genome_list redundant, so it's removed.
Also fix the sourmash subworkflow's meta.yml (previously undocumented/
mismatched outputs) and ignore Sourmash's *.sbt.zip index in nf-test
snapshots, since its content isn't reproducible across runs.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
* Make forgotten version bump to 1.2.0.dev
* Stop publishing Sourmash index/signature files by default
*.sbt.zip (genome index), *.sig (signatures) and *.sig.zip (gather
matches/prefetch/unassigned) are large, non-reproducible intermediate
Sourmash artifacts with little standalone value; only the *.csv.gz
gather results remain gated behind --sourmash_save_sourmash. Update
the one affected test snapshot to match, and refresh the "Workflow"
version string (v1.1.0 -> v1.2.0dev) in every snapshot left stale by
the recent version bump.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
* Delayed version bump
* Move CLAUDE.md content to AGENTS.md, matching other nf-core repos
CLAUDE.md is now a thin pointer to AGENTS.md, which follows the same
structure used in nf-core-ampliseq (What this is / Commands / Architecture
/ Conventions / Gotchas, plus the critical remote nf-core/agents fetch
instruction). Also refreshes stale test file references from earlier
test-suite consolidation and adds a few gotchas learned this session
(empty withName blocks, CI's 4-cpu runners, oversized base.config memory
labels for test genomes).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
* Speed up Prokka in tests and restore .sbt.zip publishing
Give Prokka its own tight memory reservation in every test profile
(2GB vs. its process_low label's 12GB default, which was never actually
clamped by the resourceLimits ceiling) so more Prokka jobs run
concurrently instead of serializing in pairs -- Prokka dominates test
wall-clock time (~90%+ in local profiling) and only uses ~450MB in
practice on these small genomes.
Also fix an empty withName: GENOME_INDEX {} block (introduced when I
first stopped publishing *.sbt.zip) that Nextflow 25.10.4 rejects with
"Unknown config attribute", and restore *.sbt.zip publishing behind
--sourmash_save_sourmash after reconsidering: it's a real genome index
worth keeping on request, unlike *.sig/*.sig.zip.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
* Clarify usage docs a bit
* Remove CLAUDE.md, per nf-core consensus to use AGENTS.md only
Repo-specific instructions now live entirely in AGENTS.md; the global
CLAUDE.md at the user level already redirects to AGENTS.md for any repo
that has one, so a per-repo CLAUDE.md is redundant and against current
nf-core core-team consensus on avoiding coding-agent file clutter in
pipeline repos.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
* Switch GTDB metadata URL to a working mirror
data.gtdb.aau.ecogenomic.org has an expired TLS certificate, breaking
CI (certificate_expired PKIX validation failure staging
ar53_metadata_r226.tsv.gz). data.gtdb.ecogenomic.org mirrors the same
release and works.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
* Add missing COLLECT_GENOMESELECTION snapshot coverage: metadata
This test was never updated for the COLLECT_GENOMESELECTION MultiQC
summary feature added earlier -- it runs unconditionally for every
genomeset_mode=joint profile but only the sourmash-related snapshots
were updated at the time.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
* Add missing COLLECT_GENOMESELECTION snapshot coverage: default
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
* Add missing COLLECT_GENOMESELECTION snapshot coverage: single
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
* Add missing COLLECT_GENOMESELECTION snapshot coverage: duplicates
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
* Add missing COLLECT_GENOMESELECTION snapshot coverage: skip_preprocessing_steps
Last of the 5 affected snapshot files.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
* Bump minimum Nextflow version to 26.04.0
Nextflow 25.10.4 has a version-topic-aggregation bug that drops the
PROKKAGFF2TSV versions.yml block whenever that process runs more than
once per pipeline execution, breaking snapshot comparisons uniformly
across every test block with multiple genomes lacking a GFF. 26.04.6
and 26.07.0-edge both aggregate correctly, so raise the declared
minimum instead of working around the bug.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
* Regenerate ro-crate-metadata.json for Nextflow 26.04.0 minimum
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
* Update sourmash_genome_selection.nf.test.snap for Nextflow 26.04.0
This snapshot was recorded while Nextflow 25.10.4's version-topic
aggregation bug was silently dropping PROKKAGFF2TSV from the versions
map whenever it ran more than once per execution. Now that the
declared minimum is 26.04.0, the versions map is correctly populated
and the snapshot needs the same block added to match.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
* Clearer test names
* Manual addition of PROKKAGFF2TSV block in snapshot
* Record Nextflow minimum change in CHANGELOG.md
---------
Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
* Template update for nf-core/tools 4.0.3, and update nf-core modules and subworkflows (#215)
* Better documentation of CheckM(2) and GTDB-Tk input
* Make tidyverse/joinmetadata handle a combination of CheckM and CheckM2 files
* Setup test case for #187
* Add CLAUDE.md
* Implement --species_preference local to drop duplicate-species remote genomes
When a user-provided local genome and a Sourmash-selected remote genome
represent the same GTDB species, prefer the local one and skip fetching
the remote duplicate. Species are resolved from --gtdbtk_metadata (local)
and --gtdb_metadata (remote) in a new TIDYVERSE_SELECTGENOMESPECIES module,
called before WGET_GENOME so duplicates are never downloaded. The
'completeness' and 'gtdb' preference modes remain unimplemented and now
error explicitly instead of silently no-op'ing.
Also fixes stale tRNA/rRNA assertions and thresholds in
sourmash_mix_dupl_species.nf.test left over from before this feature
existed, and updates its snapshot.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
* Add test profile for prefer local in combination with sample-based genomesets
* Add per-sample genome index manifest and fix single-file CheckM bug
- Add --bbmap_save_genome_list (auto-on for --genomeset_mode sample) to
publish which genome accessions went into each BBMap index, giving a
read-independent way to verify per-sample genome selection. Threaded
through CREATE_BBMAP_INDEX and asserted on in
sourmash_mix_dupl_species_sampleset.nf.test.
- Fix modules/local/tidyverse/joinmetadata: combining CheckM/CheckM2
files broke with exactly one --checkm_metadata file (unbalanced
parens from a fragile string-joining trick), producing an R syntax
error. Rebuilt with purrr::map + reduce(union), which works
uniformly for any number of files.
- Add two non-deterministic MultiQC files (multiqc_featurecounts.txt,
featureCounts_assignment_plot.txt) to tests/.nftignore; their content
can vary run-to-run due to BBMap's multi-threaded ambiguous-mapping
resolution.
- Update snapshots for metadata, sourmash_mix_dupl_species and
sourmash_mix_dupl_species_sampleset (verified passing). The other
test files sharing tests/.nftignore still need their snapshots
regenerated on next run.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
* Update snapshots after nftignore additions and more
* Temporarily set test dataset url to my fork
* Update two snapshots
* Temporarily point test data at fork; validate per-sample genome selection
- Move the fork test-data override from tests/nextflow.config to
nextflow.config's own pipelines_testdata_base_path default. The
conf/test_*.config files eagerly concatenate this value while being
parsed as part of nextflow.config's profiles block, which happens
before tests/nextflow.config (a later -c file) is ever merged, so
overriding it there had no effect. Revert once the upstream
nf-core/test-datasets PR merges.
- Update sourmash_mix_dupl_species(.nf.test/.snap) and its sampleset
variant against the updated fork data (one local genome removed,
surfacing a previously-species-duplicated remote genome).
- Rewrite the sampleset test's per-sample genome-list assertions: they
previously assumed all samples get an identical genome set, which
only held because the old test data had no local/remote divergence.
With the new data, arc00/arc02 correctly pick up the surfaced remote
genome via their own sourmash matches while arc01 doesn't -- this now
genuinely exercises species_preference=local interacting with
genomeset_mode=sample instead of being indistinguishable from joint
mode.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
* Implement --species_preference completeness and gtdb
Extends TIDYVERSE_SELECTGENOMESPECIES (previously local-only) to also
score genomes by CheckM completeness ('completeness') or GTDB's
completeness - 5*contamination criterion ('gtdb'), comparing local and
remote candidates for the same species and keeping whichever wins --
unlike 'local', these two modes can drop a local genome in favor of a
better-scoring remote one. Ties are resolved in favor of the local
genome. subworkflows/local/sourmash/main.nf is restructured so the
local genome channel can actually be filtered post-hoc, which no mode
needed before now.
Adds dedicated test profiles/configs for completeness, gtdb, and
completeness combined with genomeset_mode=sample, alongside the
existing local-mode ones; updates the two existing dupl_species
snapshots for the module's new (always-reported) purrr version entry.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
* Consolidate overlapping Sourmash test profiles into two files
Replaces 10 near-duplicate conf/test_*.config + tests/*.nf.test pairs
(sourmash, sourmash_mix{,_sampleset}, sourmash_10genomes_{joint,sample},
sourmash_mix_dupl_species{,_sampleset,_completeness,_gtdb,
_completeness_genomeset_sample}) with two files, each a single base
config plus multiple nf-test cases that only override the params that
differ:
- tests/species_preference.nf.test: the full species_preference
(local/completeness/gtdb) x genomeset_mode (joint/sample) matrix.
- tests/sourmash_genome_selection.nf.test: species_preference='all',
a remote-only scenario (no --genomeinfo), and multiple
comma-separated --indexes files -- covering what the retired
sourmash/sourmash_mix/10genomes tests checked, on the same archaeal
dataset instead of three separate ones (e_coli, 24genomes,
10genomes).
Also reverts pipelines_testdata_base_path in nextflow.config back to
the official nf-core/test-datasets repo now that the updated archaeal
test data (one local species removed, needed to actually exercise
genomeset_mode=sample divergence) has been merged upstream.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
* Linting fixes
* Update CHANGELOG.md, usage.md and output.md
* Add genome-selection summary to MultiQC report and unify genome-list manifest
Summarize local vs. remote genome counts per genome set (per-sample in
'sample' mode, a single row in 'joint' mode) as a MultiQC custom-content
table, plus a full per-genome breakdown in summary_tables/. The
bbmap/*.genomes.txt manifest is now always written for both genomeset
modes, making --bbmap_save_genome_list redundant, so it's removed.
Also fix the sourmash subworkflow's meta.yml (previously undocumented/
mismatched outputs) and ignore Sourmash's *.sbt.zip index in nf-test
snapshots, since its content isn't reproducible across runs.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
* Make forgotten version bump to 1.2.0.dev
* Stop publishing Sourmash index/signature files by default
*.sbt.zip (genome index), *.sig (signatures) and *.sig.zip (gather
matches/prefetch/unassigned) are large, non-reproducible intermediate
Sourmash artifacts with little standalone value; only the *.csv.gz
gather results remain gated behind --sourmash_save_sourmash. Update
the one affected test snapshot to match, and refresh the "Workflow"
version string (v1.1.0 -> v1.2.0dev) in every snapshot left stale by
the recent version bump.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
* Delayed version bump
* Move CLAUDE.md content to AGENTS.md, matching other nf-core repos
CLAUDE.md is now a thin pointer to AGENTS.md, which follows the same
structure used in nf-core-ampliseq (What this is / Commands / Architecture
/ Conventions / Gotchas, plus the critical remote nf-core/agents fetch
instruction). Also refreshes stale test file references from earlier
test-suite consolidation and adds a few gotchas learned this session
(empty withName blocks, CI's 4-cpu runners, oversized base.config memory
labels for test genomes).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
* Template update for nf-core/tools version 4.0.3
* Speed up Prokka in tests and restore .sbt.zip publishing
Give Prokka its own tight memory reservation in every test profile
(2GB vs. its process_low label's 12GB default, which was never actually
clamped by the resourceLimits ceiling) so more Prokka jobs run
concurrently instead of serializing in pairs -- Prokka dominates test
wall-clock time (~90%+ in local profiling) and only uses ~450MB in
practice on these small genomes.
Also fix an empty withName: GENOME_INDEX {} block (introduced when I
first stopped publishing *.sbt.zip) that Nextflow 25.10.4 rejects with
"Unknown config attribute", and restore *.sbt.zip publishing behind
--sourmash_save_sourmash after reconsidering: it's a real genome index
worth keeping on request, unlike *.sig/*.sig.zip.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
* Clarify usage docs a bit
* Remove CLAUDE.md, per nf-core consensus to use AGENTS.md only
Repo-specific instructions now live entirely in AGENTS.md; the global
CLAUDE.md at the user level already redirects to AGENTS.md for any repo
that has one, so a per-repo CLAUDE.md is redundant and against current
nf-core core-team consensus on avoiding coding-agent file clutter in
pipeline repos.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
* Ignore local genomes/ dir and .Rproj file
These are local artifacts (documented in AGENTS.md's Gotchas), not
pipeline source -- gitignoring them alongside the existing
magmap_genomes/magmap_prokka entries also unblocks
`nf-core pipelines lint --fix`, which refuses to run with any
uncommitted/untracked changes present.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
* Regenerate ro-crate-metadata.json for template 4.0.3
nf-core pipelines lint --fix rocrate_readme_sync, resolving the
merge conflict from the template sync properly instead of just
keeping the stale (pre-sync) content.
* Update nf-core modules and subworkflows
nf-core modules update --all + nf-core subworkflows update --all.
Notable: wget's local patch (2-arg [meta, url] interface instead of
the vendor's 3-arg [meta, url, suffix], matching magmap's own
WGET_GENOME call site) failed to auto-reapply after the module's
version bump, so it was reapplied by hand and the patch file
regenerated via `nf-core modules patch wget`. Everything else
(bam_sort_stats_samtools/bam_stats_samtools's take/emit interface,
multiqc, samtools/*, trimgalore, cat/cat, custom/dumpsoftwareversions,
utils_nfschema_plugin) updated without needing manual intervention.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
* Regenerate container config files for updated modules
nf-core pipelines lint --fix container_configs, picking up MultiQC's
1.34 -> 1.35 bump from the module update.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
* Fill in CHANGELOG for template sync and module/subworkflow updates
PR number left as a placeholder (#NN) until the PR exists.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
* Pre-update tool version strings in nf-test snapshots
Anticipates the module updates: samtools 1.23.1 -> 1.24 and
trimgalore 2.1.0 -> 2.3.0, the only two tool versions that changed
and are actually captured in snapshot content. Avoids a wasted first
nf-test run that would fail solely on these known, predictable
version-string diffs.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
* Fix nf-test snapshots: correct version quoting, add missing genome-selection coverage
Two issues surfaced by a real nf-test run after the module updates:
- My earlier version-string pre-patch quoted samtools's new version
("1.24") the same way as the old one ("1.23.1"), but 1.24 parses as
a JSON number and gets serialized unquoted -- fixed across all 7
affected snapshot files.
- default/metadata/single/duplicates/skip_preprocessing_steps.nf.test
were never updated for the COLLECT_GENOMESELECTION MultiQC summary
feature added earlier this session -- I'd only updated the
sourmash-related snapshots at the time, missing that this process
runs unconditionally for every genomeset_mode=joint profile.
Regenerated properly via --update-snapshot (real pipeline runs, not
hand-edits) since the exact md5s/content aren't predictable.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
* Switch GTDB metadata URL to a working mirror
data.gtdb.aau.ecogenomic.org has an expired TLS certificate, breaking
CI (certificate_expired PKIX validation failure staging
ar53_metadata_r226.tsv.gz). data.gtdb.ecogenomic.org mirrors the same
release and works.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
* Bump minimum Nextflow version to 26.04.0
Nextflow 25.10.4 has a version-topic-aggregation bug that drops the
PROKKAGFF2TSV versions.yml block whenever that process runs more than
once per pipeline execution, breaking snapshot comparisons uniformly
across every test block with multiple genomes lacking a GFF. 26.04.6
and 26.07.0-edge both aggregate correctly, so raise the declared
minimum instead of working around the bug.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
* Record Nextflow 26.04.0 minimum in CHANGELOG.md
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
* Regenerate ro-crate-metadata.json for Nextflow 26.04.0 minimum
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
* Manually add PROKKAGFF2TSV block to snapshots for Nextflow 26.04.0
Nextflow 25.10.4's version-topic aggregation bug silently dropped the
PROKKAGFF2TSV versions.yml block whenever that process ran more than
once per execution (see nextflow.config bump to 26.04.0). Recorded
snapshots predate the bump, so the block is missing wherever
GENOME_SKETCH indicates multiple genomes were processed. Added by hand
using the same version strings already present in the other snapshot
files in this branch (metadata/default/single/duplicates/skip_preprocessing_steps.nf.test.snap).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
* Regenerate ro-crate-metadata.json after merging upstream/dev
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
* Remove version output channel from wget to allow storeDir
* Revert faulty eWaterCycle/setup-apptainer digest from the 4.0.3 template sync
nf-core/tools 4.0.3's template bumped this action to a digest
(4bb22c52d4f63406c49e94c804632975787312b3, tagged v2.0.0) that's known to
cause a namespace error in some pipelines' Singularity/Apptainer CI jobs.
Revert to the previous pin (3f706d898c9db585b1d741b4692e66755f3a1b40, v2)
until upstream fixes the release.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
* Update CHANGELOG.md
Co-authored-by: Diego Alvarez S. <dialvarezs@gmail.com>
* Update CHANGELOG.md
Co-authored-by: Diego Alvarez S. <dialvarezs@gmail.com>
---------
Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
Co-authored-by: nf-core-bot <core@nf-co.re>
Co-authored-by: Diego Alvarez S. <dialvarezs@gmail.com>
* Remove vulnerable PR-comment artifact pattern (#216)
Co-authored-by: Danilo Di Leo <78909156+danilodileo@users.noreply.github.com>
Co-authored-by: nf-core-bot <core@nf-co.re>
Co-authored-by: Daniel Lundin <erik.rikard.daniel@gmail.com>
* Add Bakta as an alternative to Prokka for genome annotation (#217)
* Better documentation of CheckM(2) and GTDB-Tk input
* Make tidyverse/joinmetadata handle a combination of CheckM and CheckM2 files
* Setup test case for #187
* Add CLAUDE.md
* Implement --species_preference local to drop duplicate-species remote genomes
When a user-provided local genome and a Sourmash-selected remote genome
represent the same GTDB species, prefer the local one and skip fetching
the remote duplicate. Species are resolved from --gtdbtk_metadata (local)
and --gtdb_metadata (remote) in a new TIDYVERSE_SELECTGENOMESPECIES module,
called before WGET_GENOME so duplicates are never downloaded. The
'completeness' and 'gtdb' preference modes remain unimplemented and now
error explicitly instead of silently no-op'ing.
Also fixes stale tRNA/rRNA assertions and thresholds in
sourmash_mix_dupl_species.nf.test left over from before this feature
existed, and updates its snapshot.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
* Add test profile for prefer local in combination with sample-based genomesets
* Add per-sample genome index manifest and fix single-file CheckM bug
- Add --bbmap_save_genome_list (auto-on for --genomeset_mode sample) to
publish which genome accessions went into each BBMap index, giving a
read-independent way to verify per-sample genome selection. Threaded
through CREATE_BBMAP_INDEX and asserted on in
sourmash_mix_dupl_species_sampleset.nf.test.
- Fix modules/local/tidyverse/joinmetadata: combining CheckM/CheckM2
files broke with exactly one --checkm_metadata file (unbalanced
parens from a fragile string-joining trick), producing an R syntax
error. Rebuilt with purrr::map + reduce(union), which works
uniformly for any number of files.
- Add two non-deterministic MultiQC files (multiqc_featurecounts.txt,
featureCounts_assignment_plot.txt) to tests/.nftignore; their content
can vary run-to-run due to BBMap's multi-threaded ambiguous-mapping
resolution.
- Update snapshots for metadata, sourmash_mix_dupl_species and
sourmash_mix_dupl_species_sampleset (verified passing). The other
test files sharing tests/.nftignore still need their snapshots
regenerated on next run.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
* Update snapshots after nftignore additions and more
* Temporarily set test dataset url to my fork
* Update two snapshots
* Temporarily point test data at fork; validate per-sample genome selection
- Move the fork test-data override from tests/nextflow.config to
nextflow.config's own pipelines_testdata_base_path default. The
conf/test_*.config files eagerly concatenate this value while being
parsed as part of nextflow.config's profiles block, which happens
before tests/nextflow.config (a later -c file) is ever merged, so
overriding it there had no effect. Revert once the upstream
nf-core/test-datasets PR merges.
- Update sourmash_mix_dupl_species(.nf.test/.snap) and its sampleset
variant against the updated fork data (one local genome removed,
surfacing a previously-species-duplicated remote genome).
- Rewrite the sampleset test's per-sample genome-list assertions: they
previously assumed all samples get an identical genome set, which
only held because the old test data had no local/remote divergence.
With the new data, arc00/arc02 correctly pick up the surfaced remote
genome via their own sourmash matches while arc01 doesn't -- this now
genuinely exercises species_preference=local interacting with
genomeset_mode=sample instead of being indistinguishable from joint
mode.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
* Implement --species_preference completeness and gtdb
Extends TIDYVERSE_SELECTGENOMESPECIES (previously local-only) to also
score genomes by CheckM completeness ('completeness') or GTDB's
completeness - 5*contamination criterion ('gtdb'), comparing local and
remote candidates for the same species and keeping whichever wins --
unlike 'local', these two modes can drop a local genome in favor of a
better-scoring remote one. Ties are resolved in favor of the local
genome. subworkflows/local/sourmash/main.nf is restructured so the
local genome channel can actually be filtered post-hoc, which no mode
needed before now.
Adds dedicated test profiles/configs for completeness, gtdb, and
completeness combined with genomeset_mode=sample, alongside the
existing local-mode ones; updates the two existing dupl_species
snapshots for the module's new (always-reported) purrr version entry.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
* Consolidate overlapping Sourmash test profiles into two files
Replaces 10 near-duplicate conf/test_*.config + tests/*.nf.test pairs
(sourmash, sourmash_mix{,_sampleset}, sourmash_10genomes_{joint,sample},
sourmash_mix_dupl_species{,_sampleset,_completeness,_gtdb,
_completeness_genomeset_sample}) with two files, each a single base
config plus multiple nf-test cases that only override the params that
differ:
- tests/species_preference.nf.test: the full species_preference
(local/completeness/gtdb) x genomeset_mode (joint/sample) matrix.
- tests/sourmash_genome_selection.nf.test: species_preference='all',
a remote-only scenario (no --genomeinfo), and multiple
comma-separated --indexes files -- covering what the retired
sourmash/sourmash_mix/10genomes tests checked, on the same archaeal
dataset instead of three separate ones (e_coli, 24genomes,
10genomes).
Also reverts pipelines_testdata_base_path in nextflow.config back to
the official nf-core/test-datasets repo now that the updated archaeal
test data (one local species removed, needed to actually exercise
genomeset_mode=sample divergence) has been merged upstream.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
* Linting fixes
* Update CHANGELOG.md, usage.md and output.md
* Add genome-selection summary to MultiQC report and unify genome-list manifest
Summarize local vs. remote genome counts per genome set (per-sample in
'sample' mode, a single row in 'joint' mode) as a MultiQC custom-content
table, plus a full per-genome breakdown in summary_tables/. The
bbmap/*.genomes.txt manifest is now always written for both genomeset
modes, making --bbmap_save_genome_list redundant, so it's removed.
Also fix the sourmash subworkflow's meta.yml (previously undocumented/
mismatched outputs) and ignore Sourmash's *.sbt.zip index in nf-test
snapshots, since its content isn't reproducible across runs.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
* Make forgotten version bump to 1.2.0.dev
* Stop publishing Sourmash index/signature files by default
*.sbt.zip (genome index), *.sig (signatures) and *.sig.zip (gather
matches/prefetch/unassigned) are large, non-reproducible intermediate
Sourmash artifacts with little standalone value; only the *.csv.gz
gather results remain gated behind --sourmash_save_sourmash. Update
the one affected test snapshot to match, and refresh the "Workflow"
version string (v1.1.0 -> v1.2.0dev) in every snapshot left stale by
the recent version bump.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
* Delayed version bump
* Move CLAUDE.md content to AGENTS.md, matching other nf-core repos
CLAUDE.md is now a thin pointer to AGENTS.md, which follows the same
structure used in nf-core-ampliseq (What this is / Commands / Architecture
/ Conventions / Gotchas, plus the critical remote nf-core/agents fetch
instruction). Also refreshes stale test file references from earlier
test-suite consolidation and adds a few gotchas learned this session
(empty withName blocks, CI's 4-cpu runners, oversized base.config memory
labels for test genomes).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
* Template update for nf-core/tools version 4.0.3
* Speed up Prokka in tests and restore .sbt.zip publishing
Give Prokka its own tight memory reservation in every test profile
(2GB vs. its process_low label's 12GB default, which was never actually
clamped by the resourceLimits ceiling) so more Prokka jobs run
concurrently instead of serializing in pairs -- Prokka dominates test
wall-clock time (~90%+ in local profiling) and only uses ~450MB in
practice on these small genomes.
Also fix an empty withName: GENOME_INDEX {} block (introduced when I
first stopped publishing *.sbt.zip) that Nextflow 25.10.4 rejects with
"Unknown config attribute", and restore *.sbt.zip publishing behind
--sourmash_save_sourmash after reconsidering: it's a real genome index
worth keeping on request, unlike *.sig/*.sig.zip.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
* Clarify usage docs a bit
* Remove CLAUDE.md, per nf-core consensus to use AGENTS.md only
Repo-specific instructions now live entirely in AGENTS.md; the global
CLAUDE.md at the user level already redirects to AGENTS.md for any repo
that has one, so a per-repo CLAUDE.md is redundant and against current
nf-core core-team consensus on avoiding coding-agent file clutter in
pipeline repos.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
* Ignore local genomes/ dir and .Rproj file
These are local artifacts (documented in AGENTS.md's Gotchas), not
pipeline source -- gitignoring them alongside the existing
magmap_genomes/magmap_prokka entries also unblocks
`nf-core pipelines lint --fix`, which refuses to run with any
uncommitted/untracked changes present.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
* Regenerate ro-crate-metadata.json for template 4.0.3
nf-core pipelines lint --fix rocrate_readme_sync, resolving the
merge conflict from the template sync properly instead of just
keeping the stale (pre-sync) content.
* Update nf-core modules and subworkflows
nf-core modules update --all + nf-core subworkflows update --all.
Notable: wget's local patch (2-arg [meta, url] interface instead of
the vendor's 3-arg [meta, url, suffix], matching magmap's own
WGET_GENOME call site) failed to auto-reapply after the module's
version bump, so it was reapplied by hand and the patch file
regenerated via `nf-core modules patch wget`. Everything else
(bam_sort_stats_samtools/bam_stats_samtools's take/emit interface,
multiqc, samtools/*, trimgalore, cat/cat, custom/dumpsoftwareversions,
utils_nfschema_plugin) updated without needing manual intervention.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
* Regenerate container config files for updated modules
nf-core pipelines lint --fix container_configs, picking up MultiQC's
1.34 -> 1.35 bump from the module update.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
* Fill in CHANGELOG for template sync and module/subworkflow updates
PR number left as a placeholder (#NN) until the PR exists.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
* Pre-update tool version strings in nf-test snapshots
Anticipates the module updates: samtools 1.23.1 -> 1.24 and
trimgalore 2.1.0 -> 2.3.0, the only two tool versions that changed
and are actually captured in snapshot content. Avoids a wasted first
nf-test run that would fail solely on these known, predictable
version-string diffs.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
* Fix nf-test snapshots: correct version quoting, add missing genome-selection coverage
Two issues surfaced by a real nf-test run after the module updates:
- My earlier version-string pre-patch quoted samtools's new version
("1.24") the same way as the old one ("1.23.1"), but 1.24 parses as
a JSON number and gets serialized unquoted -- fixed across all 7
affected snapshot files.
- default/metadata/single/duplicates/skip_preprocessing_steps.nf.test
were never updated for the COLLECT_GENOMESELECTION MultiQC summary
feature added earlier this session -- I'd only updated the
sourmash-related snapshots at the time, missing that this process
runs unconditionally for every genomeset_mode=joint profile.
Regenerated properly via --update-snapshot (real pipeline runs, not
hand-edits) since the exact md5s/content aren't predictable.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
* Switch GTDB metadata URL to a working mirror
data.gtdb.aau.ecogenomic.org has an expired TLS certificate, breaking
CI (certificate_expired PKIX validation failure staging
ar53_metadata_r226.tsv.gz). data.gtdb.ecogenomic.org mirrors the same
release and works.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
* Bump minimum Nextflow version to 26.04.0
Nextflow 25.10.4 has a version-topic-aggregation bug that drops the
PROKKAGFF2TSV versions.yml block whenever that process runs more than
once per pipeline execution, breaking snapshot comparisons uniformly
across every test block with multiple genomes lacking a GFF. 26.04.6
and 26.07.0-edge both aggregate correctly, so raise the declared
minimum instead of working around the bug.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
* Record Nextflow 26.04.0 minimum in CHANGELOG.md
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
* Regenerate ro-crate-metadata.json for Nextflow 26.04.0 minimum
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
* Manually add PROKKAGFF2TSV block to snapshots for Nextflow 26.04.0
Nextflow 25.10.4's version-topic aggregation bug silently dropped the
PROKKAGFF2TSV versions.yml block whenever that process ran more than
once per execution (see nextflow.config bump to 26.04.0). Recorded
snapshots predate the bump, so the block is missing wherever
GENOME_SKETCH indicates multiple genomes were processed. Added by hand
using the same version strings already present in the other snapshot
files in this branch (metadata/default/single/duplicates/skip_preprocessing_steps.nf.test.snap).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
* Regenerate ro-crate-metadata.json after merging upstream/dev
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
* Remove version output channel from wget to allow storeDir
* Install and patch Bakta modules; drop Prokka's uncollected versions.yml output
Install bakta/bakta and bakta/baktadbdownload from nf-core/modules. Both
vendored modules emit their version via an eval() output on a topic:
versions channel -- incompatible with storeDir (only val/path outputs are
allowed there), which these processes will need for caching. Patch both to
drop that line; a dedicated version-reporting process is added separately.
Also drop Prokka's own versions.yml output entirely: it was never actually
wired into the pipeline's version collation (a pre-existing bug), and since
it's being replaced by the same dedicated version-reporting approach, there's
no reason to keep declaring it -- doing so would otherwise require every
existing user's prokka_store_dir cache to already contain that file, or force
a re-run of already-cached genomes.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
* Add PROKKA_VERSION/BAKTA_VERSION, decoupled from the storeDir-cached processes
Both tools' versions don't vary per genome within a single pipeline run, so
there's no need to report them per-genome (or at all, for cache-hit genomes).
These are tiny, uncached, zero-input processes that just run `prokka
--version`/`bakta --version` once per execution and report via the same
eval() ... topic: versions one-liner GENOMES2ORFS already uses for sed --
since they have no storeDir, that pattern is safe here.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
* Add TIDYVERSE_SELECTANNOTATOR and local Bakta subworkflow
TIDYVERSE_SELECTANNOTATOR splits genomes lacking a GFF between Prokka and
Bakta according to --annotator and each genome's GTDB domain (extracted from
the gtdb_taxonomy column TIDYVERSE_JOINMETADATA already produces from
--gtdb_metadata/--gtdbtk_metadata) -- no new metadata parsing needed.
The local BAKTA subworkflow wraps BAKTA_BAKTADBDOWNLOAD -> BAKTA_BAKTA ->
BAKTA_VERSION, mirroring the download-module + tool-module subworkflow
pattern (e.g. metatdenovo's KOFAMSCAN/EGGNOG). BAKTA_BAKTA's on-disk output
is patched to match Prokka's convention: gzipped files under a
--output ${prefix} subdirectory, so a shared storeDir accumulates one
subdirectory per genome the same way magmap_prokka does.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
* Regenerate bakta-bakta.diff to include the --output/gzip patch
The previous commit's patch file only reflected the version-line removal,
not the later --output/gzip restructuring applied in the same commit -- a
stale diff that would confuse nf-core modules update.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
* Wire --annotator/--bakta_db/--bakta_store_dir into the pipeline
Add --annotator (prokka/bakta_supported_only/bakta_all, default prokka) and
route genomes lacking a GFF accordingly in workflows/magmap.nf: split via
TIDYVERSE_SELECTANNOTATOR when not plain 'prokka', warn (once) about genomes
bakta_supported_only couldn't classify by domain, and mix Bakta's annotated
genomes back into ch_collected_genomes alongside Prokka's. Bakta's .gff3 goes
through the existing PROKKAGFF2TSV module rather than Bakta's own .tsv, whose
columns (Sequence Id/Type/Start/Stop/Strand/Locus Tag/Gene/Product/DbXrefs)
don't match Prokka's -- reusing the generic GFF3 parser already used for
user-supplied GFFs avoids a Bakta-specific TSV adapter entirely.
Add --bakta_db (storeDir for BAKTA_BAKTADBDOWNLOAD, doubling as a bring-your-
own-database path) and --bakta_store_dir (storeDir for BAKTA_BAKTA, mirroring
--prokka_store_dir), including the same pre-create-directory handling in
PIPELINE_INITIALISATION that genome_store_dir/prokka_store_dir already get.
Also fix GENOMES2ORFS: it stripped only .gff/.gff.gz suffixes when deriving
each genome's accno from its GFF filename, silently leaving Bakta's .gff3/
.gff3.gz extension in place and corrupting the accno column for every
Bakta-annotated genome.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
* Add test_bakta profile and nf-test coverage for --annotator
Combines genomes from both existing test_metadata (bacterial,
GCA_002688505.1) and test_species_preference (archaeal, the local_* genomes)
via a new genomeinfo fixture that omits genome_gff, so both domains need
annotation -- exercising the Bakta path (Bacteria) and the Prokka fallback
path (Archaea, unsupported by Bakta) in the same run. Uses Bakta's light
database (~1.3GB), the smallest official variant and what nf-core/modules'
own bakta tests use too; there's no smaller custom test database available
(see oschwengers/bakta#95).
tests/bakta.nf.test covers both --annotator bakta_supported_only (the
profile default) and bakta_all.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
* Document --annotator/Bakta support and add Bakta citation
Add a "Genome annotation" section to usage.md covering both Prokka and
Bakta, including the --annotator table and the bakta_supported_only
domain-fallback behaviour; extend the storage-directory docs with
--bakta_db/--bakta_store_dir; add a matching "Bakta" section to output.md
alongside Prokka's; add Bakta's citation to CITATIONS.md; record both the
new feature and the Prokka-versions fix in CHANGELOG.md.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
* Update existing test snapshots for PROKKA_VERSION
Real nf-test run (--update-snapshot) against all 7 previously-existing
profiles. Verified structurally (not just via raw diff, which is misleading
here due to top-level key reordering across runs) that every change is
exactly the new "PROKKA_VERSION": {"prokka": "1.14.6"} entry plus the
matching +1 succeeded-task count -- nothing else differs.
One case (sourmash_genome_selection.nf.test, 'local + remote, genomeset_mode:
sample') failed on this run with a transient "Network is unreachable" error
staging a test-datasets file, unrelated to any pipeline change; its snapshot
is correctly untouched and will be retried separately.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
* Update sourmash_genome_selection snapshot for the network-flake retry
'local + remote, genomeset_mode: sample' failed on the previous run with a
transient network error unrelated to any pipeline change; retried and passed
cleanly. Verified the diff is exactly the same PROKKA_VERSION addition as the
other test cases in this file.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
* Fix stale setup-apptainer digest missed by the dev merge
Git's 3-way merge couldn't see this: bakta-annotator's copy went
stale->faulty and was never touched again, while dev went
stale->faulty->fixed (back to the same value as the merge-base). Since
"theirs" ended up textually identical to the merge-base, git treated it as
unchanged and silently kept our still-faulty side -- a real, if narrow, git
merge blind spot, not something a conflict marker would have caught.
Verified by diffing every file bakta-annotator had independently diverged on
against upstream/dev post-merge; this was the only one affected.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
* Simplify CHANGELOG entry for the setup-apptainer CI fix
CHANGELOG entries are for end users, not internal CI plumbing detail like
specific GitHub Action digests.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
* Fix BAKTA_VERSION: set HOME so matplotlib doesn't corrupt the captured version
BAKTA_BAKTA/BAKTA_BAKTADBDOWNLOAD both set HOME to a writable dir before
running bakta; BAKTA_VERSION's empty script never did. Without a writable
HOME, bakta's matplotlib/fontconfig cache setup fails and dumps multi-line
warnings to stderr, which the eval() command's `2>&1` merges straight into
the captured "version" string -- corrupting the collated versions.yml enough
that MultiQC's YAML parser choked on it. Confirmed by a real nf-test run.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
* Give each Bakta-annotated genome a unique --locus/--locus-tag
Bakta's --locus (contig-name prefix) defaults to the literal "contig" for
every genome, and --locus-tag (feature/gene ID prefix) defaults to an MD5
hash of the input sequences truncated to 6 characters -- both effectively
the same across genomes, or collision-prone (the same class of
fingerprint-based-prefix collision seen from Prokka's own algorithm before).
Concatenating multiple Bakta-annotated genomes into one shared BBMap index
broke on the resulting duplicate contig names; FEATURECOUNTS/GENOMES2ORFS
would have hit the same problem via colliding GFF `ID=` values.
Using the accno directly (as PROKKA's --prefix already does for its own
contig naming) was tried and reverted: Bakta rejects locus prefixes over 20
characters or containing anything outside [A-Za-z0-9_-], and confirmed via a
real nf-test run against a genome whose accno exceeded that. Local genome
accnos are arbitrary user text with no such guarantee, unlike remote/NCBI
ones. Hash the accno instead (SHA-256, first 20 hex chars, uppercase) --
deterministic and always a valid prefix regardless of the accno's length or
characters. This gives up being able to eyeball which genome a contig/ORF
name came from, which was a nice side effect of the accno-based scheme but
never a requirement.
Documented the resulting caveat in usage.md: since the hash is derived from
accno, reusing an accno for a genuinely different local genome across runs
still means the storeDir cache is keyed on the old (now-mismatched) identity
-- the same class of caveat that already existed for --prokka_store_dir, now
also true of Bakta's contig/feature naming.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
* Add bakta.nf.test.snap: first successful --annotator test run
Both bakta_supported_only and bakta_all pass cleanly against the accno-hash
--locus/--locus-tag fix and the BAKTA_VERSION HOME fix.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
* Remove TODO from CONTRIBUTING.md
* Add mimetype to the --input schema property
Fixes the nf-core lint schema_lint warning ("Input mimetype is missing or
empty").
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
* Skip tests/bakta.nf.test in CI -- Bakta DB download exceeds job timeout
Both cases in this file download the full Bakta database from scratch
(no cache available on ephemeral CI runners), which regularly runs past
6h and gets the job cancelled. Exclude the file from nf-test's ignore
list only when CI=true (set automatically by the GitHub Actions
runner) so it's still discovered and runs normally locally, where the
DB can be pre-populated via --bakta_db/--bakta_store_dir.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
* Update CHANGELOG with PR #217 number
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
Co-authored-by: nf-core-bot <core@nf-co.re>
* Add --save_parquet to also write summary tables as Parquet (#219)
* Add DUCKDB_TSV2PARQUET local module
Converts a batch of gzipped/plain TSV files to Parquet using DuckDB's Python
bindings (conda-forge's duckdb package ships the library + Python API, not a
standalone CLI binary -- confirmed by trying to shell out to `duckdb` first
and getting "command not found" despite the container pulling fine).
Intended to be contributed to nf-core/modules once this branch's tests pass,
per the project's convention for genuinely new, reusable functionality; kept
local for now so it can be exercised end-to-end here first.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
* Add --save_parquet to also write summary tables as Parquet
Gated behind a new boolean param (default false); when set, DUCKDB_TSV2PARQUET
runs once at the end of the workflow against the full set of already-produced
summary_tables/*.tsv.gz files, writing a same-named *.parquet alongside each.
Verified with a real nf-test run (test_duplicates + save_parquet: true) and
spot-checked the resulting Parquet files' content/types directly with DuckDB.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
* Migrate from local duckdb/tsv2parquet to vendored duckdb/table2parquet
Now that the module has been merged upstream into nf-core/modules,
replace the local batch-style module with the vendored per-file one.
This changes DUCKDB_TABLE2PARQUET from a single task looping over all
summary tables to one task per table, with the output basename now
carried via meta.id (derived from each table's own filename) instead
of being computed inside the script.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
* Regenerate container config files for duckdb/table2parquet
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
* Empty commit to restart CI
The previous nf-test run's shard-discovery job was cancelled before its
downstream jobs were created, leaving GitHub unable to rerun it directly.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
* Add a line to trigger tests
* Revert the test-trigger comment line
The nf-test CI runner never picked up the last few pushes (jobs stuck
in queued/pending with no runner provisioned, unrelated to nf-test's
own changed-since test selection), so this didn't end up serving its
intended purpose.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
* github pr rerun checks
---------
Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
* Document read-count units and fix brittle R code in COLLECT_STATS/COLLECT_GENOMESELECTION (#222)
* Document that overall_stats count columns share the same read units
Verified empirically (real bbduk.sh/samtools/featureCounts runs, not
just documentation) that n_post_trimming, n_non_contaminated,
idxs_n_mapped/unmapped, and the featureCounts columns all count
individual reads with both mates counted separately -- none of them
count fragments/pairs as one unit, so the columns are comparable.
The featureCounts case is the non-obvious one: SUBREAD_FEATURECOUNTS
adds -p automatically for paired-end samples, but without
--countReadPairs it still counts at the individual-read level, so it
matches the other columns. Recorded this so a future "modernization"
adding --countReadPairs doesn't silently break comparability.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
* Define TPM in output.md
* Fix brittle R code in COLLECT_STATS/COLLECT_GENOMESELECTION
Addresses #218:
- Both modules built R source text by directly interpolating
user-controlled values (sample IDs, genome accessions -- documented
as accepting arbitrary text via --genomeinfo) into single-quoted R
string literals. A value containing a quote would break the
generated R syntax; something more adversarial could inject R code.
Added a top-level rq() helper that safely escapes a value as a
single-quoted R literal, used everywhere such a value is embedded.
(Defined as a top-level function rather than a script-local closure:
`nextflow lint`'s strict-syntax parser rejects a closure captured
from an outer scope being called from within a nested closure
literal -- confirmed by writing a minimal reproduction and checking
it against both forms.)
- COLLECT_STATS also shelled out to grep/sed via pipe(sprintf(...)) to
parse trimming reports, bbduk logs, and idxstats files, passing
sample-derived filenames through a shell command string. Rewrote all
three to read the files directly in R (readLines()/read_tsv() +
regex, or a plain dplyr filter for idxstats) instead, removing the
shell subprocess entirely.
- COLLECT_STATS hardcoded FeatureCounts' column layout as a positional
type-code string (col_types = 'ccciicicid'), which happened to mirror
COLLECT_FEATURECOUNTS's column order but wasn't enforced to stay in
sync. Switched to naming each column's type explicitly via cols(),
so a future column reorder in COLLECT_FEATURECOUNTS can't silently
misparse here.
Verified the Groovy string-escaping fix with a standalone nf-test
reproduction (tricky sample IDs containing quotes/backslashes,
checked against both the working and broken closure forms), and the R
rewrites against the real featureCounts-derived counts files pulled
from an actual test run's work directory (exact same numbers as
before: 340 CDS / 21 rRNA / 8 tRNA for sample s0). A full pipeline
nf-test run could not be completed to confirm end-to-end: two
consecutive attempts died to host-level OOM (Nextflow's own JVM
orchestrator threads, not the R task) under memory pressure from
unrelated processes on this machine, not from these changes -- still
outstanding before this is considered fully proven.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
* Try GitHub-hosted ubuntu-latest runners for nf-test (self-hosted runners stuck queued)
* Add CHANGELOG entry for the #218 fix
* Update CHANGELOG with PR #222 number, trim Fixed entry wording
* Fix corrupted GString interpolation in COLLECT_STATS caused by a backtick in a comment
Root cause of the two "column X not found" R failures seen in local nf-test runs
and in CI (magmap#222, shard 5/7, docker|26.04.0): a comment introduced by the
brittle-R-code fix contained a backtick and a `\*` escape
(`` `grep -v '^\*'` ``). Inside the process's outer triple-quoted GString, that
combination throws off the value-to-placeholder alignment for every
interpolation from that point on -- confirmed by a controlled test: reverting
just this comment's wording (no other change) took a run from reproducibly
failing to passing, both locally and matching the exact corruption pattern seen
on GitHub-hosted CI runners (never a host-memory or JVM-heap issue, despite
earlier appearances -- two of the three failures happened to coincide with
memory pressure, which was a red herring).
Not a Groovy/Nextflow bug worth reporting upstream without a minimal
standalone reproduction; a small isolated repro with the same nested-GString
structure did not reproduce it, so whatever triggers this needs more specific
conditions than have been isolated so far. Simplest fix: don't put a backtick
+ backslash-escape combination in a script comment.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
* Fix intermittent ConcurrentModificationException in COLLECT_GENOMESELECTION
CREATE_BBMAP_INDEX.out.genome_accnos is consumed by two independent channel
subscribers: this collectFile (which calls accnos.sort()) and
COLLECT_GENOMESELECTION (which iterates the same accnos list). Groovy's no-arg
List.sort() sorts in place rather than returning a new list, so when both
subscribers ran concurrently, one could be mutating the shared list while the
other was mid-iteration over it -- seen as an intermittent
ConcurrentModificationException in COLLECT_GENOMESELECTION on CI (magmap#222).
Predates this branch (introduced in #212); found while chasing an
intermittent nf-test CI failure on this PR. sort(false) returns a new sorted
list instead of mutating the shared one, removing the race.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
* Add CHANGELOG entry for the ConcurrentModificationException fix
---------
Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
* Template sync to nf-core/tools 4.1.0 and module updates (#223)
* Template update for nf-core/tools version 4.0.3
* Template update for nf-core/tools version 4.1.0
* Update nf-core modules and subworkflows for the 4.1.0 template sync
Bumps prokka 1.14.6 -> 1.15.6 (openjdk 8.0.412 -> 8.0.472) and
duckdb/table2parquet's test snapshot (no functional change -- upstream
tidied redundant positional emit assertions). Removes a stale, unused
"patch" reference for utils_nextflow_pipeline in modules.json (no .diff
file ever existed for it). Pre-patches the prokka version string in
pipeline-level test snapshots to avoid a wasted nf-test cycle.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
* Regenerate container config files via nf-core pipelines lint --fix
Fixes the container/conda directive mismatch left by the template
regeneration (all fastqc/multiqc entries used the wrong 'container ='
key instead of 'conda =' for conda-lock files) and restores the
DUCKDB_TABLE2PARQUET entry that was missing from the regenerated files.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
* Fix stale #NN placeholder for the save_parquet PR (#219) in CHANGELOG
* Keep modules/local/prokka_version's environment in sync with modules/nf-core/prokka
nf-core modules update only touches vendored modules/nf-core/ content, so
the local prokka_version module (added in #217 to work around the vendored
module never reporting its own version) still had environment.yml pinned to
the OLD prokka=1.14.6/openjdk=8.0.412, silently reporting a wrong-but-
plausible version after the vendored module's 1.15.6 bump in this sync --
caught only by comparing an actual nf-test run's real versions.yml against
a pre-patched snapshot, not by any automated check.
Bumped environment.yml to match the vendored module exactly (same
prokka/openjdk/parallel package set) and switched the container reference
to reuse modules/nf-core/prokka's own container verbatim, rather than
building a near-duplicate Wave image for an environment that's now
identical to the vendored one.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
* Add missing Changed entry for the template sync, fix #NN placeholder
The template-sync + module/subworkflow-update batch itself was missing
its own CHANGELOG entry (should follow the same pattern as #215's 4.0.3
sync), and the prokka_version fix still had the #NN PR-number
placeholder now that #223 is the real number.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
* Reorder CHANGELOG entries descending by PR number within each section
---------
Co-authored-by: nf-core-bot <core@nf-co.re>
Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
* Bump version: 1.2.0dev -> 1.2.0 (#224)
Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
* Remove vulnerable PR-comment artifact pattern (#226)
Co-authored-by: Danilo Di Leo <78909156+danilodileo@users.noreply.github.com>
Co-authored-by: nf-core-bot <core@nf-co.re>
Co-authored-by: Matthias Hörtenhuber <mashehu@users.noreply.github.com>
* Remove vulnerable PR-comment artifact pattern (#227)
Co-authored-by: Danilo Di Leo <78909156+danilodileo@users.noreply.github.com>
Co-authored-by: nf-core-bot <core@nf-co.re>
Co-authored-by: Matthias Hörtenhuber <mashehu@users.noreply.github.com>
* Document potential Prokka/Bakta version drift across a reused store dir (#230)
Genomes already present in --prokka_store_dir/--bakta_store_dir are
skipped rather than re-annotated on later runs, so a genome library
accumulated over time can end up with a mix of annotation-tool
versions -- not reflected in the collated versions.yml/MultiQC report,
which only shows the version used in the current run.
Documenting this now (issue #229); actually scanning per-genome logs
to report the real mix is a heavier follow-up, deferred to a later
milestone pending a practical approach at genome-library scale
(tens of thousands of genomes).
Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
* Update CHANGELOG with recent fixes and documentation
* Address review comments: DuckDB citation, Bakta dirs, docs gaps
- Add a DuckDB citation to CITATIONS.md (used by --save_parquet).
- Only create the Bakta database/store directories when --annotator
actually needs Bakta, not unconditionally regardless of annotator
choice. Threads params.annotator into PIPELINE_INITIALISATION to
make this possible (it wasn't passed there before).
- Add the missing wget component and Prokka/Bakta annotation step to
subworkflows/local/sourmash/meta.yml and README.md respectively.
- Rename modules/local/tidyverse/selectgenomespecies/environment.yml's
leftover scaffolding name ("r-test") to match the module. The same
leftover exists in six other local modules' environment.yml files,
left untouched here as out of scope for this PR.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
* Fix COLLECT_GENOMESELECTION not scaling to large genome collections
modules/local/collect/genomeselection/main.nf built an R tribble()
literal with one line per (sample, accession) pair, interpolated
directly into the generated script. In --genomeset_mode sample the row
count is samples x genomes -- a run with hundreds of samples and
thousands of genomes would generate a script with millions of lines.
An empty genome set also produced a tribble() with no rows, which is
an R syntax error rather than an empty table.
Read accessions from the already-published bbmap/*.genomes.txt files
instead (reusing the same collectFile channel already writing them,
rather than duplicating the write), passed as a path input instead of
a Groovy val. Fixes both the scaling issue and the empty-set syntax
error as a side effect of no longer building the script from Groovy-
interpolated text at all.
Also drops the …
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
PR checklist
nf-core pipelines lint).nextflow run . -profile test,docker --outdir <OUTDIR>).nextflow run . -profile debug,test,docker --outdir <OUTDIR>).docs/usage.mdis updated.docs/output.mdis updated.CHANGELOG.mdis updated.README.mdis updated (including new tool citations and authors/contributors).Description
Housekeeping ahead of the v1.2.0 release:
masterhad diverged fromdevsince the lastrelease via a direct security patch (#214, "Remove vulnerable PR-comment artifact
pattern", applied to
master2026-07-24) thatdevnever got merged back afterindependently getting its own equivalent fix via #216.
This PR merges
masterintodevto reconcile that (one real conflict in.nf-core.yml,resolved in favor of
dev's already-more-current post-4.1.0-template-sync state — theonly other difference was a stale
files_existoverride referencing a file the templatesync already renamed). No functional changes beyond that reconciliation.
Doing this as its own PR rather than folding it into the release PR, so the release PR's
diff stays limited to the actual release content.