Skip to content

feat: Coverage Matrix Generator service #45

Description

@miroslavpojer

Summary

Implement a new coverage_matrix service that cross-references doc-source.json with ui-tests.json to produce an AC-level test coverage matrix (coverage-matrix.json) per User Story.

Full specification: spec.md


Background

The PDF report generator needs a structured, pre-computed coverage view. Today that data must be assembled manually. This service automates the cross-reference so downstream tooling (PDF generator, dashboards) can consume a single, schema-versioned JSON file.


Acceptance Criteria

AC-01 — Loader

  • load_doc_input(path) reads a JSON array from disk; raises a descriptive error if file is missing or content is not an array.
  • load_tests_input(path) reads a JSON object from disk; raises a descriptive error if file is missing or the object lacks an items key.

AC-02 — US resolution

  • build_coverage_matrix() extracts the US numeric id from us.id (e.g. "absa-group/aul-ui/US-27""US-27").
  • Scenarios with us_id: null or a us_id that matches no US are collected into unlinked_tests.

AC-03 — AC resolution

  • An AC is covered when ≥ 1 scenario lists its id in ac_ids.
  • An AC is not_covered when no scenario references it.
  • An ac_id in a scenario that does not exist on the resolved US is added to stale_ac_refs.

AC-04 — Summary computation

  • coverage_pct is computed as covered_acs / active_acs * 100, rounded to 1 dp.
  • coverage_pct is null when active_acs == 0.
  • Deprecated ACs (state != "Active") are included in the matrix but excluded from active_acs and covered_acs.
  • Top-level summary and per-US summary follow the same fields.

AC-05 — Output schema

  • coverage-matrix-v1.0.0-schema.json (JSON Schema draft-07) is authored and committed as part of this implementation — no external definition exists.
  • Output is a valid JSON object conforming to that schema.
  • schema_version is "coverage-matrix-v1.0.0".
  • generated_at is an ISO-8601 UTC timestamp.

AC-06 — CLI

  • coverage-matrix --doc-input <path> --tests-input <path> --output <path> writes the matrix file.
  • --fail-under <N> exits with code 1 if coverage_pct < N; disabled by default.
  • All three required arguments are validated before any processing begins.

AC-07 — Error handling

  • Missing input files → log error, exit 1.
  • Invalid JSON structure (not array / missing items) → log error, exit 1.
  • US item missing id or acceptance_criteria → log warning, skip US, continue.
  • Output write failure → log error, exit 1.
  • --fail-under not met → log summary, exit 1.

AC-08 — Tests

  • Unit tests cover loader.py, matcher.py, and summary.py independently (no I/O in matcher/summary tests).
  • Coverage ≥ 80 % (pytest --cov).

Out of scope (v1)

  • Functional / unit test coverage sources
  • Historical trend tracking
  • HTML output
  • CLI filtering by tag, version, or AC state

Proposed code structure

packages/services/coverage_matrix/
    pyproject.toml
    README.md
    src/
        living_doc_service_coverage_matrix/
            __init__.py
            cli.py          # arg parsing, orchestration, exit codes
            loader.py       # load_doc_input(), load_tests_input() — I/O only
            matcher.py      # build_coverage_matrix() — pure, no I/O
            summary.py      # compute_summary(), compute_us_summary() — pure
            model/
                __init__.py
                coverage_item.py   # dataclasses for output schema
            schema/
                coverage-matrix-v1.0.0-schema.json
    tests/
        __init__.py
        test_loader.py
        test_matcher.py
        test_summary.py
        test_cli.py

Follows the same layout as packages/services/normalize_issues/.


Definition of Done

  • All ACs above are met and verified by automated tests.
  • pylint score ≥ 9.5/10 on tracked .py files.
  • mypy passes with no unresolved errors.
  • Coverage report generated to htmlcov/.
  • coverage-matrix-v1.0.0-schema.json is committed alongside the code.
  • Entry added to CHANGELOG.md.

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

No type

Fields

No fields configured for issues without a type.

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions