RFC: machine-readable `expected:` blocks in test YAMLs

[lambdabaa]

Problem

The L3 CQL Standard Operating Procedure (v1.0.0) states that "test cases shall exist and shall pass." Today, the YAML test scenarios at input/tests/plandefinition/<X>/<X>.yaml encode the expected outputs as free-text ### … comment lines at the top of each patient document — not in a machine-readable form. A Test Validation case-expression is generated per Logic library, but nothing in this repository executes it against the patient bundles. The field-level assertions that do run live in generated Karate .feature files at input/testing/features/plandefinition/<X>/, which require a live FHIR server (HAPI / CQF Ruler).

The result: the YAMLs encode authorial intent for what each Logic library should return per patient, but no in-repo tool can verify it. Any downstream consumer (the IG Publisher build, a Node test runner, the Karate flow) must hand-extract those expectations from comments.

Proposal

Add an optional expected: block per patient document, recording the expected per-define output values that the Logic library named in the sibling PlanDefinition should produce for that patient. Comments stay; the block is a machine-readable companion.

Schema (proposed)

expected:
  row: <L2 row identifier>           # optional human-readable cross-reference
  today: <YYYY-MM-DD>                # optional per-case anchor for the CQL `Today` parameter
  encounterId: <string>              # optional override for `EncounterId`
  defines:
    "<CQL define name>": <scalar>          # boolean/number/null/string — exact equality
    "<CQL define name>":
      equals: <value>                      # explicit equality (use for multi-line strings)
    # plus: matches: <regex>, isNull: true, isNotNull: true

Identifiers correspond to the published CQL define "<name>": declarations in the Logic library at input/cql/<X>.cql.

Compatibility

The proposal is additive.

• tools/node/makeExample.js accesses YAML keys by name (id, birth, patient, immunization, contraindication, medicationrequest) and ignores any others; the expected: block flows through untouched.
• tools/node/processDAK.js --existing re-emits the parsed YAML object verbatim into regenerated files, so blocks added by hand survive a future round-trip from the L2 Excel source.
• No change is required to the IG Publisher build, qa.html, or the Karate \$apply-based assertions.

What this enables

• An in-repo Node runner can execute the published Logic library against the existing patient bundles and diff per-define results against expected:. The pieces are already on disk — Logic libraries, patient bundles, the cql-execution JS engine — there is just no machine-readable expected-output specification today.
• The same blocks act as a self-describing reference for third-party conformance harnesses (we are building one externally — see the companion PR below).

First contribution

PR #137 adds expected: blocks to the existing Measles Low Transmission test YAML (input/tests/plandefinition/IMMZD2DTMeaslesLowTransmissionLogic/IMMZD2DTMeaslesLowTransmissionLogic.yaml) as a concrete example. Zero behaviour change to the build; one file modified; +72 lines.

Asks

1. Does the schema as proposed work for the working groups?
2. Is input/tests/plandefinition/<X>/<X>.yaml the right home for this convention, or would the L3 CQL SOP v2 be a more appropriate place to formalise it?
3. Are there existing in-repo or in-flight efforts that overlap (so this can align rather than duplicate)?

Happy to amend the schema, the doc location, or the PR scope based on feedback.