Repository navigation
Mutmut all files - #2401
Mutmut all files#2401amilcarlucas wants to merge 4 commits into
Conversation
There was a problem hiding this comment.
🟡 Changes recommended
The reporter can misclassify missing shard manifests as fully checked empty shards, while test selection omits existing suites needed for the expanded source scope, making mutation results misleading.
2 open issues
What changed in this PR
Expands mutation testing across the application package and divides it into nine CI shards to reduce per-job runtime.
Changes:
- Broadens mutation targets and validates shard coverage, with up to three CI jobs running concurrently.
- Adds shard-specific saved state, timing measurements, and diagnostic reports for future workload balancing.
- Adds reporting regression tests and documents the updated workflow.
| File | Description |
|---|---|
tests/test_mutation_report.py |
Tests reporting for cached, partial, failed, and stale-scope runs. |
pyproject.toml |
Expands mutation targets to all non-excluded package Python files. |
CONTRIBUTING.md |
Documents sharding, state restoration, and diagnostic artifacts. |
.github/workflows/mutation.yml |
Adds the shard matrix, isolated caches, timing capture, and report uploads. |
.github/scripts/mutation_shard.py |
Validates target coverage and runs one shard while retaining completion logs. |
.github/scripts/mutation_report.py |
Exports mutation outcomes, timings, pending work, and CI summaries. |
Review effort: Balanced
| metadata = environment_metadata() | ||
| attempts = worker_timings(output_dir) | ||
| modules, test_seconds = module_measurements(state_dir, output_dir, attempts, warnings, metadata["mutmut_config"]) | ||
| empty_shard = bool(metadata["mutation_shard"]) and not metadata["active_mutation_targets"] |
| ] | ||
| # Mutate every Python source file. The shard runner validates that the complete | ||
| # package tree is assigned exactly once before starting Mutmut. | ||
| only_mutate = ["ardupilot_methodic_configurator/*.py"] |
304e68e to
4d62358
Compare
☂️ Code Coverage
Overall Coverage
New FilesNo new covered files... Modified FilesNo covered modified files...
|
Test Results 4 files - 1 4 suites - 1 51m 16s ⏱️ - 8m 3s For more details on these failures, see this check. Results for commit 5466f5c. ± Comparison against base commit 4393038. This pull request removes 67 and adds 253 tests. Note that renamed tests count towards both.This pull request skips 1 test.♻️ This comment has been updated with latest results. |
Previous review (2026-11-19)Automated review note — AI-generated (Claude+Codex), validated against the live diff. Please sanity-check before acting. Reviewed at head This PR shards Mutmut across the whole package, but it currently breaks the Pylint workflow, and the new diagnostics and test selection need fixes. Blocking
Should fix
Minor (1)
|
4d62358 to
6600b38
Compare
Previous review (2026-11-20)Automated review note — AI-generated (Claude+Codex), validated against the live diff. Please sanity-check before acting. Reviewed at head This PR shards Mutmut across the whole package. It has no blockers now, but the shard diagnostics and the test selection still need fixes. Since the review at Should fix
Minor (2)
The earlier CONTRIBUTING.md:199 wording note is withdrawn: lines 183-185 already explain the full rerun. Not fully reviewed: no real Mutmut shard run or live cache/artifact migration, and most CI for |
Add focused coverage for component imports, telemetry, log analysis and user workflows, with related maintenance and test fixture updates. Signed-off-by: Dr.-Ing. Amilcar do Carmo Lucas <amilcar.lucas@iav.de>
Capture mutation runtime data to support splitting the suite into balanced parallel jobs within GitHub Actions execution limits. - Enable CI-only forkserver logging to capture actual worker durations, avoiding Mutmut 3.8.1's zero-duration metadata - Record baseline and mutation elapsed time, exit codes, cutoff status, throughput, and process resource usage - Export per-module outcomes, pending work, test-cost estimates, worker timing samples, and test durations and associations - Record runner details, Python and Mutmut versions, configuration, dependency hashes, and restored-state provenance - Distinguish current-run measurements from cached verdicts and estimates - Handle missing logs, interrupted state writes, and stale cached targets - Expand the job summary and retain diagnostic artifacts for 90 days - Document the reports, their units, and timing limitations Validate with three regression tests, an end-to-end Mutmut smoke run, workflow shell checks, Pylint, Markdown linting, and pre-commit checks. Signed-off-by: Dr.-Ing. Amilcar do Carmo Lucas <amilcar.lucas@iav.de>
Split mutation testing into nine backend, data-model, and frontend shards across the main, log-analysis, and plugins packages. Assign battery_cell_voltages.py to the main data-model shard and imu_helpers.py to the plugin data-model shard. Keep the configured test selection intact for every shard. Give each shard its own mutation cache and report artifact, and migrate matching module state and test associations from the previous unsharded artifact. Retain complete worker logs by disabling Mutmut log rotation, and report empty shards explicitly. Document the shard workflow and report artifacts. Validate the partition against all configured mutation targets. Signed-off-by: Dr.-Ing. Amilcar do Carmo Lucas <amilcar.lucas@iav.de>
Expand Mutmut's source scope to every Python file under ardupilot_methodic_configurator/, excluding the files listed in .coveragerc's report omit setting. Assign unclassified package files to their area's backend shard. Validate that Mutmut exclusions match coverage and that every included source file is assigned to exactly one shard. Update contributor guidance to document the expanded scope and shard assignments. Signed-off-by: Dr.-Ing. Amilcar do Carmo Lucas <amilcar.lucas@iav.de>
6600b38 to
5466f5c
Compare
|
Automated review note — AI-generated (Claude+Codex), validated against the live diff. Please sanity-check before acting. Reviewed at head This PR shards Mutmut across the whole package; there are no blockers, but two should-fix issues in the shard diagnostics and the test selection remain. Since the review at Should fix
Minor (1)
Not fully reviewed: no real Mutmut shard run, live cache or artifact migration, or macOS/Windows run, and upstream CI for |
Coverage Report for CI Build 44972527310Coverage increased (+0.2%) to 96.238%Details
Uncovered ChangesNo uncovered changes found. Coverage RegressionsNo coverage regressions found. Coverage Stats💛 - Coveralls |

Description
Mutmut all files using 9 parallel CI jobs. So that each job hopefully takes less than the 6 hour github.com CI limit.
Checklist
git commit --signoff)Testing
Describe how you tested these changes: