Skip to content

Mutmut all files - #2401

Open
amilcarlucas wants to merge 4 commits into
masterfrom
mutmut_all_files
Open

amilcarlucas wants to merge 4 commits into
masterfrom
mutmut_all_files

Conversation

@amilcarlucas

Copy link
Copy Markdown
Collaborator

Description

Mutmut all files using 9 parallel CI jobs. So that each job hopefully takes less than the 6 hour github.com CI limit.

Checklist

  • Run pre-commit checks locally
  • Verified by a human programmer
  • All commits are signed off (use git commit --signoff)
  • Code follows our coding standards
  • Documentation updated if needed
  • No breaking changes or properly documented

Testing

Describe how you tested these changes:

  • Unit tests pass
  • Integration tests pass
  • Manual testing performed
  • Tested on flight controller hardware

Copilot AI balanced review requested due to automatic review settings November 19, 2026 11:57

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

The reporter can misclassify missing shard manifests as fully checked empty shards, while test selection omits existing suites needed for the expanded source scope, making mutation results misleading.

2 open issues
What changed in this PR

Expands mutation testing across the application package and divides it into nine CI shards to reduce per-job runtime.

Changes:

  • Broadens mutation targets and validates shard coverage, with up to three CI jobs running concurrently.
  • Adds shard-specific saved state, timing measurements, and diagnostic reports for future workload balancing.
  • Adds reporting regression tests and documents the updated workflow.
File Description
tests/​test_mutation_report.py Tests reporting for cached, partial, failed, and stale-scope runs.
pyproject.toml Expands mutation targets to all non-excluded package Python files.
CONTRIBUTING.md Documents sharding, state restoration, and diagnostic artifacts.
.github/​workflows/​mutation.yml Adds the shard matrix, isolated caches, timing capture, and report uploads.
.github/​scripts/​mutation_shard.py Validates target coverage and runs one shard while retaining completion logs.
.github/​scripts/​mutation_report.py Exports mutation outcomes, timings, pending work, and CI summaries.

Review effort: Balanced

metadata = environment_metadata()
attempts = worker_timings(output_dir)
modules, test_seconds = module_measurements(state_dir, output_dir, attempts, warnings, metadata["mutmut_config"])
empty_shard = bool(metadata["mutation_shard"]) and not metadata["active_mutation_targets"]
Comment thread pyproject.toml
]
# Mutate every Python source file. The shard runner validates that the complete
# package tree is assigned exactly once before starting Mutmut.
only_mutate = ["ardupilot_methodic_configurator/*.py"]
@github-actions

github-actions Bot commented Nov 19, 2026 •

Copy link
Copy Markdown
Contributor

☂️ Code Coverage

current status: ✅

Overall Coverage

Statements Covered Coverage Threshold Status
31775 30553 96% 89% 🟢

New Files

No new covered files...

Modified Files

No covered modified files...

updated for commit: 5466f5c by action🐍

@amilcarlucas amilcarlucas added the AIReview Request an automated AI review; picked up by the reviewprs sweep label Nov 19, 2026
@github-actions

github-actions Bot commented Nov 19, 2026 •

Copy link
Copy Markdown
Contributor

Test Results

     4 files   -     1       4 suites   - 1   51m 16s ⏱️ - 8m 3s
10 660 tests +  186  10 593 ✅ +  185   5 💤 + 1  62 ❌ ±0 
42 524 runs   - 9 537  42 351 ✅  - 9 513  74 💤  - 24  99 ❌ ±0 

For more details on these failures, see this check.

Results for commit 5466f5c. ± Comparison against base commit 4393038.

This pull request removes 67 and adds 253 tests. Note that renamed tests count towards both.
tests.test_backend_flightcontroller_connection.TestFakeSerialForTests ‑ test_close_does_not_raise
tests.test_backend_flightcontroller_connection.TestFakeSerialForTests ‑ test_device_attribute_is_stored
tests.test_backend_flightcontroller_connection.TestFakeSerialForTests ‑ test_in_waiting_returns_zero
tests.test_backend_flightcontroller_connection.TestFakeSerialForTests ‑ test_read_returns_empty_string
tests.test_backend_flightcontroller_connection.TestFakeSerialForTests ‑ test_write_always_raises_exception
tests.test_data_model_vehicle_components_base.TestComponentDataModelBase ‑ test_system_detects_when_model_has_components
tests.test_data_model_vehicle_components_base.TestComponentDataModelBase ‑ test_system_provides_multiple_component_access_methods
tests.test_data_model_vehicle_components_base.TestComponentDataModelBase ‑ test_system_validates_component_data_structure
tests.test_data_model_vehicle_components_base.TestComponentDataModelBase ‑ test_user_can_access_realistic_vehicle_components
tests.test_data_model_vehicle_components_base.TestComponentDataModelBase ‑ test_user_can_initialize_empty_model
…
tests.acceptance_template_import_from_params ‑ test_template_setup_prepares_exact_release_metadata[4.6.0-ArduCopter-4.6.3]
tests.acceptance_template_import_from_params ‑ test_template_setup_prepares_exact_release_metadata[4.6.0-ArduPlane-4.7.2]
tests.acceptance_template_import_from_params ‑ test_template_setup_prepares_exact_release_metadata[4.6.3-ArduCopter-4.6.3]
tests.acceptance_template_import_from_params ‑ test_template_setup_prepares_exact_release_metadata[4.6.3-ArduPlane-4.7.2]
tests.acceptance_template_import_from_params ‑ test_template_setup_prepares_exact_release_metadata[4.6.30-ArduCopter-4.6.3]
tests.acceptance_template_import_from_params ‑ test_template_setup_prepares_exact_release_metadata[4.6.30-ArduPlane-4.7.2]
tests.acceptance_template_import_from_params ‑ test_template_setup_prepares_exact_release_metadata[4.7.2-ArduCopter-4.6.3]
tests.acceptance_template_import_from_params ‑ test_template_setup_prepares_exact_release_metadata[4.7.2-ArduPlane-4.7.2]
tests.acceptance_template_import_from_params ‑ test_template_setup_prepares_exact_release_metadata[None-ArduCopter-4.6.3]
tests.acceptance_template_import_from_params ‑ test_template_setup_prepares_exact_release_metadata[None-ArduPlane-4.7.2]
…
This pull request skips 1 test.
tests.gui_frontend_tkinter_show.TestTooltipUserExperienceIntegration ‑ test_tooltip_positioning_across_multiple_monitors_windows

♻️ This comment has been updated with latest results.

@AP-Review

AP-Review commented Nov 19, 2026 •

Copy link
Copy Markdown

Deprecated — see below for the updated review.

Previous review (2026-11-19)

Automated review note — AI-generated (Claude+Codex), validated against the live diff. Please sanity-check before acting.
Verdict: REQUEST CHANGES

Reviewed at head 4d623589ee.
Full report: https://firmware.ardupilot.org/Tools/APReview/DevCallReviews/PRReviews/ardupilot/methodicconfigurator/2401/1.html#prMethodicConfigurator-2401

This PR shards Mutmut across the whole package, but it currently breaks the Pylint workflow, and the new diagnostics and test selection need fixes.

Blocking

  1. .github/scripts/mutation_shard.py:23: the module-level mutmut imports are unresolvable in the Pylint workflow, which doesn't install the mutation extra. ty check fails with 3 unresolved-import errors and pylint (3.15.2) goes red (green on base). Make mutmut resolvable there, e.g. add mutation to the Pylint workflow's uv pip install extras.

Should fix

  • .github/scripts/mutation_report.py:76: a missing or corrupt shard-targets.json (e.g. after a failed baseline) is treated as a validated empty shard. The empty_shard check at line 196 then reports inventory_fully_checked: true, drops the parse warning and says no run was needed. Treat absent or invalid manifests as unknown and only accept an explicit [].
  • pyproject.toml:196: only_mutate now covers the whole package, but pytest_add_cli_args_test_selection is unchanged. Suites such as tests/test_argparse_check_range.py and tests/test_backend_filesystem_migration.py never run against their newly mutated modules. Expand the selection along with the scope.
Minor (1)
  • CONTRIBUTING.md:199: the first sharded run does load the legacy v3 state, but changed test-selection and watched inputs (including pyproject.toml) reset it under on_dependency_change = "rerun". Say that results and associations are recollected rather than reused.

@AP-Review

AP-Review commented Nov 20, 2026 •

Copy link
Copy Markdown

Deprecated — see below for the updated review.

Previous review (2026-11-20)

Automated review note — AI-generated (Claude+Codex), validated against the live diff. Please sanity-check before acting.
Verdict: COMMENT

Reviewed at head 6600b389aa.
Full report: https://firmware.ardupilot.org/Tools/APReview/DevCallReviews/PRReviews/ardupilot/methodicconfigurator/2401/2.html#prMethodicConfigurator-2401

This PR shards Mutmut across the whole package. It has no blockers now, but the shard diagnostics and the test selection still need fixes.

Since the review at 4d623589, the head moved to 6600b389 and the Pylint/ty break from the missing mutation extra is fixed.

Should fix

  • .github/scripts/mutation_report.py:75: missing or corrupt shard-targets.json treated as an empty shard (still open). After a failed baseline the line 195 check reports inventory_fully_checked: true and 'no Mutmut run was needed'. Only an explicitly parsed [] should count as empty, and the parse warning should be kept.
  • pyproject.toml:196: test selection not grown with the scope (still open). tests/test_argparse_check_range.py, tests/test_backend_filesystem_migration.py and tests/test_frontend_tkinter_template_overview.py are still unselected, but their modules are now mutated.
Minor (2)
  • .github/scripts/mutation_shard.py:23: with mutmut installed (the documented .[dev,mutation] setup), the new suppressions fail pylint as useless-suppression (fail-on) and E1121 appears at line 170, and ty warns about unused ignores. This hits the language: system pre-commit pylint hook. Installing the extra in pylint.yml and dropping the suppressions, or importing mutmut lazily in main(), makes both environments agree.
  • tests/test_backend_flightcontroller_connection.py:1120: removing TestFakeSerialForTests leaves FakeSerialForTests (backend_flightcontroller_connection.py:111) unused in an already-mutated module. Delete it and its ARCHITECTURE_2 mention.

The earlier CONTRIBUTING.md:199 wording note is withdrawn: lines 183-185 already explain the full rerun.

Not fully reviewed: no real Mutmut shard run or live cache/artifact migration, and most CI for 6600b389 (Pylint, ruff, mypy, pyright, Docker, Ubuntu pytest) was still queued.

Add focused coverage for component imports, telemetry, log analysis and user workflows, with related maintenance and test fixture updates.

Signed-off-by: Dr.-Ing. Amilcar do Carmo Lucas <amilcar.lucas@iav.de>
Capture mutation runtime data to support splitting the suite into
balanced parallel jobs within GitHub Actions execution limits.

- Enable CI-only forkserver logging to capture actual worker durations,
  avoiding Mutmut 3.8.1's zero-duration metadata
- Record baseline and mutation elapsed time, exit codes, cutoff status,
  throughput, and process resource usage
- Export per-module outcomes, pending work, test-cost estimates, worker
  timing samples, and test durations and associations
- Record runner details, Python and Mutmut versions, configuration,
  dependency hashes, and restored-state provenance
- Distinguish current-run measurements from cached verdicts and estimates
- Handle missing logs, interrupted state writes, and stale cached targets
- Expand the job summary and retain diagnostic artifacts for 90 days
- Document the reports, their units, and timing limitations

Validate with three regression tests, an end-to-end Mutmut smoke run,
workflow shell checks, Pylint, Markdown linting, and pre-commit checks.

Signed-off-by: Dr.-Ing. Amilcar do Carmo Lucas <amilcar.lucas@iav.de>
Split mutation testing into nine backend, data-model, and frontend shards
across the main, log-analysis, and plugins packages. Assign
battery_cell_voltages.py to the main data-model shard and imu_helpers.py
to the plugin data-model shard.

Keep the configured test selection intact for every shard. Give each shard
its own mutation cache and report artifact, and migrate matching module
state and test associations from the previous unsharded artifact. Retain
complete worker logs by disabling Mutmut log rotation, and report empty
shards explicitly.

Document the shard workflow and report artifacts. Validate the partition
against all configured mutation targets.

Signed-off-by: Dr.-Ing. Amilcar do Carmo Lucas <amilcar.lucas@iav.de>
Expand Mutmut's source scope to every Python file under
ardupilot_methodic_configurator/, excluding the files listed in
.coveragerc's report omit setting.

Assign unclassified package files to their area's backend shard. Validate
that Mutmut exclusions match coverage and that every included source file
is assigned to exactly one shard.

Update contributor guidance to document the expanded scope and shard
assignments.

Signed-off-by: Dr.-Ing. Amilcar do Carmo Lucas <amilcar.lucas@iav.de>
@AP-Review

Copy link
Copy Markdown

Automated review note — AI-generated (Claude+Codex), validated against the live diff. Please sanity-check before acting.
Verdict: COMMENT

Reviewed at head 5466f5c820.
Full report: https://firmware.ardupilot.org/Tools/APReview/DevCallReviews/PRReviews/ardupilot/methodicconfigurator/2401/3.html#prMethodicConfigurator-2401

This PR shards Mutmut across the whole package; there are no blockers, but two should-fix issues in the shard diagnostics and the test selection remain.

Since the review at 6600b389, the head moved to 5466f5c8 (history squashed and rebased) and the unused FakeSerialForTests helper is gone; the mutation files themselves are unchanged.

Should fix

Minor (1)
  • .github/scripts/mutation_shard.py:23: the mutmut suppressions fail pylint and ty when the documented .[dev,mutation] extra is installed (still open). This affects the pre-push pylint hook, not CI.

Not fully reviewed: no real Mutmut shard run, live cache or artifact migration, or macOS/Windows run, and upstream CI for 5466f5c8 was still pending.

@coveralls

Copy link
Copy Markdown

Coverage Report for CI Build 44972527310

Coverage increased (+0.2%) to 96.238%

Details

  • Coverage increased (+0.2%) from the base build.
  • Patch coverage: No coverable lines changed in this PR.
  • No coverage regressions found.

Uncovered Changes

No uncovered changes found.

Coverage Regressions

No coverage regressions found.


Coverage Stats

Coverage Status
Relevant Lines: 32032
Covered Lines: 30827
Line Coverage: 96.24%
Relevant Branches: 9812
Covered Branches: 8965
Branch Coverage: 91.37%
Branches in Coverage %: No
Coverage Strength: 2.84 hits per line

💛 - Coveralls

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

AIReview Request an automated AI review; picked up by the reviewprs sweep

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants