Working branch for updating new testing framework by forsyth2 · Pull Request #883 · E3SM-Project/zppy

forsyth2 · 2026-10-05T17:49:30Z

This PR should not be merged directly (I've made it a draft PR for easy review). It consists of my suggested updates to #876. Once Jill and I agree on any changes here, we can cherry-pick the relevant commits to that PR.

List of suggested features from #876:

Noted in #876 (comment):

(Addressed in 7a151c1) DependencyNeverSatisfied job handling -- e.g., cancelling if a job will literally never be able to run (and only looking at jobs related to this run)
Explicit documentation of commits made since last test (useful for knowing what changed to have caused errors)
Looks like some parameters might be mainly command-line flags; in general I find config files more user-friendly (and documentable).
I see you've removed legacy cfgs that are for all intents and purposes duplicates, but we still need to replace the 3.0.0 and 3.1.0 cfgs with 3.2.0 cfgs (that's for New post-v3.2.0 test cfgs and support EAMxx test cfgs. #837)
climo/ts/e3sm_to_cmip/tc_analysis aren't independently selectable. (Historically I've been focused on the image checker, but these non-plot producing tasks can also be the sources of errors. For example, [Bug]: e3sm_to_cmip failing due to srun error #875 is affecting e3sm_to_cmip, so we don't really need to run anything beyond that).

Noted in Test approach comparison 1: NCO error in e3sm_to_cmip:

For the baseline checking (python -m tests.complete_run.promote --machine chrysalis show), the sha field is useful, but perhaps a sha_date field would be good as well, so it's easy to immediately identify how old it is.
We should come up with more generic vocabulary. main_branch_testing, complete_run aren't always reflective of what we're actually testing. The test suite is for testing various configurations.
(Addressed in 7a151c1) The test isn't recognizing the DependencyNeverSatisfied jobs. It needs to review its list of job IDs and cancel any that have DependencyNeverSatisfied. In this case, the one remaining job that has only Dependency is because it's actually waiting on DependencyNeverSatisfied jobs. That is, it's a second-order dependency never satisified situation. So the test also has to keep cancelling jobs until none of the jobs it requested are in the queue. Then, it should continue on to print info about where to find output.
PR 871 also doesn't address this, but here we're really interested in NCO's use in e3sm_to_cmip, so there's not much of a need to run any of the plotting tasks. As mentioned above, "climo/ts/e3sm_to_cmip/tc_analysis aren't independently selectable"

Noted in #876 (comment):

enable automatic pinning of dependency versions. [...] What would be particularly useful is to be able to specify in the config any dependencies that should be pinned and have the test do this post-creation conda install itself automatically. Something like PIN_DEPENDENCIES: "nco<5.4.0," This also would have been helpful for addressing the issue we saw in August re: PMP's version (Integration testing strategy for `pcmdi_diags` #848). To clarify, I do not mean testing with frozen environments which we already decided against; I mean being able to temporarily pin an old version when we've already determined it's the dependency, and not our packages, causing the issue.

Noted in Test approach comparison 2: main branch testing using NCO 5.3.9:

In summary, suggestions for PR 876:

From #876 (comment):

/lcrc/group/e3sm/public_html/zppy_complete_run/runs has multiple runs, and it'd be very easy for two testers to accidentally choose the same test name. I think the username should be inserted into the path so it's something like runs/$USER/... to prevent this.
- Jill's preference is to use a date/timestamp instead of $USER to avoid name collisions.
It'd be great if we can make what expected results we're comparing against completely configurable. That is, for each task, for each cfg, there'd be an archive of expected results (up to a certain point in the past to avoid wasting space), and the tester could choose which set of expected results to use per task. The test report would then list which expected results set was used for each task. The cron job config would also specify which expected results it should work off of.

From #876 (comment):

PMP should also be part of "Packages that most often move results"

Newer features in the stack #887:

Improve test automation #871
- Add ability to specify Unified environment for zppy itself
- Add ability to specify a different Unified (e.g., a RC)
Use v3.2.0 cfg for legacy testing #886
- Replace older legacy cfgs with a v3.2.0 cfg
Enable partial updates of expected results for testing #852
- Ensure output exists to promote to expected results baseline (since a test clears its output upon successful completion, there might not be output available)

forsyth2 · 2026-10-05T17:50:05Z

Changes so far from #876:

Rebased off main
Added f54fdb6 to document how the testing framework works under the hood.

forsyth2 · 2026-10-05T18:07:55Z

@copilot Please implement the following fix directly on this PR’s branch.

Fix the DependencyNeverSatisfied hang in the new testing framework under tests/complete_run, on the current branch for PR #883 (branched from streamline-complete-run-test).

When upstream e3sm_to_cmip jobs fail, downstream PCMDI jobs remain queued:

1295284  PD  DependencyNeverSatisfied
1295285  PD  DependencyNeverSatisfied
1295286  PD  DependencyNeverSatisfied
1295287  PD  Dependency

The last job depends on the blocked PCMDI jobs. The framework keeps waiting until its four-hour timeout.

Update the existing polling loop to cancel this run’s jobs reporting DependencyNeverSatisfied, then refresh and repeat so downstream blocked jobs are also cancelled as their reasons update. Continue waiting for healthy jobs; do not cancel a job merely because it reports Dependency. Scope all cancellations to job IDs tracked by this run.

Once none of the run’s jobs remain queued or running, continue through failure reporting and print the output/log locations. The run should still report failure.

Keep the fix small and reuse existing scheduler helpers. Add focused mocked tests for direct cancellation, cascading cancellation, and preservation of healthy/unrelated jobs. Run the relevant tests and summarize the changes.

Copilot · 2026-10-05T18:12:18Z

@copilot Please implement the following fix directly on this PR’s branch.

Fix the DependencyNeverSatisfied hang in the new testing framework under tests/complete_run, on the current branch fo...

Implemented in 7a151c1. The polling now cancels only this run’s jobs with DependencyNeverSatisfied, refreshes to catch cascading blocks, waits for remaining run jobs, and logs report/output paths. Added focused mocked coverage. I did not run tests because repository instructions prohibit test execution; git diff --check passed. CodeQL found no alerts; automated code review could not run because its model was unavailable.

forsyth2 · 2026-10-05T18:26:02Z

@copilot Please make sure tests don't require external directories. It appears your latest commit, 7a151c1, has introduced a FAILED tests/test_complete_run_automation.py::test_wait_tracks_only_job_ids_from_submitted_cfgs - PermissionError: [Errno 13] Permission denied: '/lcrc' error in the GitHub Actions test

Copilot · 2026-10-05T18:27:13Z

@copilot Please make sure tests don't require external directories. It appears your latest commit, 7a151c1, has i...

Fixed in 4d1d66c. The test now uses a scratch root under tmp_path, so its status fixtures no longer write to /lcrc. git diff --check passed. Tests were not run locally per repository instructions.

forsyth2 · 2026-10-09T23:32:38Z

Saw a banner message on #886 pointing to https://docs.github.com/en/pull-requests/get-started/about-stacked-prs. Perhaps that might be useful for breaking this PR into reviewable chunks.

forsyth2 · 2026-10-10T00:52:48Z

I'm trying out the PR stack with #871 > #886 > #852 (Stack #887)

forsyth2 · 2026-10-13T16:28:16Z

@copilot Please implement the following fix directly on this PR’s branch.

In tests/complete_run, make configuration file-driven and add dependency pinning and Unified options.

Config file. Add --config PATH to automation.py. Format: INI via stdlib configparser, one
[complete_run] section. Keys are the long option names with underscores (machine, account,
cfg, task, branch, env_type, existing_env, nco_path, pin_dependencies, unified_load_script,
baseline, ...). List values are comma-separated; repo-keyed values are repo=value.
Precedence: CLI flag > config file > default. Unknown keys are an error naming the key.
Add tests/complete_run/complete-run.example.cfg documenting every key. Make the cron
controller and its env template pass --config instead of expanding many flags (keep only
controller-specific settings like LOCK_FILE, LOG_DIR, WEEK_PARITY there). Print the
effective config at startup and store it in status.json.
Pinning. Add --pin SPEC (repeatable) and config key pin_dependencies, e.g.
pin_dependencies = nco<5.4.0, e3sm_diags:xarray==2025.1.0. An optional repo: prefix limits
a spec to one repo's environment; with no prefix it applies to every dev, baseline, and
existing environment the run uses. After an environment is created (or reused), run
conda install -n <env> --yes <spec> through Environment.run_args/run_command. A failure
aborts prepare with a clear message. Pins are ignored with a warning for unified
environments. Pins must be applied BEFORE env_description.txt and environments/.yml are
written, so those reflect them. Record pins in manifest.json, status.json, and the report's
Environment section. Keep the env_description.txt format fixed except for appending a
Pinned: line after the existing fixed lines; update the provenance tests. Document that
this is temporary isolation of a dependency problem, not frozen-environment testing.
Unified options.
- Add --unified-load-script PATH / unified_load_script, overriding the machine profile's
  unified activation command everywhere Unified is used (unified env types, the image-checker
  job, livvkit/ilamb/climo/ts/tc_analysis). Fail at startup if the file doesn't exist.
  Record it in status.json, manifest.json, env descriptions, and the report.
- Support --env-type zppy=unified, which is currently rejected. Then the zppy CLI, the zppy
  unit tests, the integration pytests, and the image-checker job run in Unified (use the
  existing mechanism for unified environments); the zppy worktree is still created so the
  tests and cfg templates come from the tested commit; nothing is pip-installed into Unified;
  no zppy conda env is built; a zppy pin is ignored with a warning. Fail early with a clear
  message if pytest is not importable in Unified. Log zppy's executable, version, and package
  location at the start of prepare, and state in the report which zppy was tested (Unified
  version + path, and the commit the tests came from). Replace
  test_zppy_cannot_run_from_unified with tests of the new behavior.

Tests must go through the real CLI (automation.main([...])). Update
docs/source/dev_guide/tests/automated_test.rst and AGENTS.md (including its note about
configuration being passed as arguments).

Copilot · 2026-10-13T16:45:12Z

@copilot Please implement the following fix directly on this PR’s branch.

In tests/complete_run, make configuration file-driven and add dependency pinning and Unified options.
...

Implemented in f88183f: added INI configuration with CLI precedence, dependency pins, configurable Unified activation, and Unified-backed zppy execution. Updated the example config, cron controller, documentation, provenance/reporting, and focused tests. git diff --check and secret scanning passed; tests were not run because AGENTS.md prohibits running them. CodeQL found no alerts; automated code review was unavailable in this environment.

forsyth2 · 2026-10-13T16:53:18Z

The latest Copilot commit, f88183f, addresses these points:

Looks like some parameters might be mainly command-line flags; in general I find config files more user-friendly (and documentable).
enable automatic pinning of dependency versions. [...] What would be particularly useful is to be able to specify in the config any dependencies that should be pinned and have the test do this post-creation conda install itself automatically. Something like PIN_DEPENDENCIES: "nco<5.4.0," This also would have been helpful for addressing the issue we saw in August re: PMP's version (Integration testing strategy for `pcmdi_diags` #848). To clarify, I do not mean testing with frozen environments which we already decided against; I mean being able to temporarily pin an old version when we've already determined it's the dependency, and not our packages, causing the issue.
Add ability to specify Unified environment for zppy itself
Add ability to specify a different Unified (e.g., a RC)

forsyth2 · 2026-10-13T16:55:45Z

@copilot Please implement the following fix directly on this PR’s branch.

Make every zppy task independently selectable and visible in results, not only plot-producing ones.

Today --task only covers the image-checked plotting tasks. Extend --task (and the matching
config key) to accept: climo, ts, e3sm_to_cmip, tc_analysis, e3sm_diags, mpas_analysis,
global_time_series, ilamb, livvkit, pcmdi_diags.

Cfg generation (python -m tests.integration.utils --task ...) enables exactly the selected
tasks' sections plus any tasks they depend on, logging "also running X (required by Y)".
Define the dependency map once, next to WEEKLY_CFGS in tests/integration/weekly_cfgs.py, and
add a test that checks it against the templates.
Only the environments, unit tests, and pytest files relevant to the selected tasks are built
or run. Example: --task e3sm_to_cmip builds only e3sm_to_cmip's environment plus zppy's,
skips the image-checker job, and the report says "image check not applicable: no plotting
tasks selected" instead of failing or reporting zero images.
The validate status-file sweep covers only jobs of the selected tasks and their
dependencies.
The report and index.html "By package" table get a row for EVERY package whose task ran,
including zppy (climo/ts/tc_analysis) and e3sm_to_cmip, showing the environment or commit
used. For non-plotting packages the status comes from the job status-file sweep (OK / N failed
jobs) and the image columns show "n/a".
In envdiff.py, add PMP (pcmdi_metrics, also matching pcmdi-metrics) to
NOTABLE_PACKAGES so it is listed first as a package that often moves results, and add it to
UNIFIED_KEY_PACKAGES in provenance.py.

Add tests and update the docs and AGENTS.md.

forsyth2 · 2026-10-13T17:24:45Z

The latest Copilot commits (e4163a7, 314e34b, 060bafb) addresses these points:

climo/ts/e3sm_to_cmip/tc_analysis aren't independently selectable. (Historically I've been focused on the image checker, but these non-plot producing tasks can also be the sources of errors. For example, [Bug]: e3sm_to_cmip failing due to srun error #875 is affecting e3sm_to_cmip, so we don't really need to run anything beyond that).
PR 871 also doesn't address this, but here we're really interested in NCO's use in e3sm_to_cmip, so there's not much of a need to run any of the plotting tasks. As mentioned above, "climo/ts/e3sm_to_cmip/tc_analysis aren't independently selectable"
Indicate environments used for all tasks, not just the ones generating plots.
- (From Claude) zppy and e3sm_to_cmip are invisible in the "By package" results table. Only e3sm_diags, ILAMB, LIVVkit, MPAS-Analysis, and zppy-interfaces get a row. There's no result/status indicator at all for the two tasks that don't produce plots.
PMP should also be part of "Packages that most often move results"

forsyth2 · 2026-10-13T17:27:31Z

@copilot Please implement the following fix directly on this PR’s branch.

Make expected results configurable per cfg and per task, keeping the "a run is its own
baseline" model (never copy files into a baseline).

Model: a baseline channel is its default run (the existing baselines/ symlink) plus
optional overrides mapping (cfg, task) to another run tag, stored atomically in
baselines/.overrides.json. Add one function in layout.py, baseline_run_for(cfg, task),
and route everything that resolves a baseline through it: cfg generation (--baseline-dir),
image_checker, ZPPY_COMPLETE_RUN_BASELINE, the report, the viewers, and envdiff. With no
overrides file, behavior is exactly as today.
Partial promotion: keep promote run <tag> (whole run). Add repeatable --task and --cfg
to promote only those entries, writing overrides and leaving everything else pointing where it
did. The "passed / not a feature branch" safety checks apply only to the selected entries, so
a run that failed elsewhere can still promote a passing task without --allow-failed. Rolling
back is promoting the older tag for the same entries.
Choosing at test time: repeatable --baseline TASK=TAG and --baseline CFG:TASK=TAG, plus
config key baseline. The cron controller config supports it too. Default is the channel's
resolved baseline. For --env-type <repo>=baseline, use the baseline run that resolves for
that repo's tasks, and error clearly if they resolve to different runs.
Reporting: the report and index.html list, per (cfg, task), the baseline run used (tag, date,
commits), rather than one global line. The "Compared against" line summarizes when mixed.
promote show: print the default run, every override, and for each run the tag, per-repo
sha, a new sha_date (commit date, ISO) and the age in days.
Retention: prune --keep N and a new --older-than DAYS; never delete a run referenced by
any channel default or override.

Add tests and update update_expected_results.rst, automated_test.rst, and the AGENTS.md
promotion bullet.

forsyth2 · 2026-10-13T18:00:33Z

The latest Copilot commits (8fa4546, 1e060a0, 91bdf51) addresses these points:

For the baseline checking (python -m tests.complete_run.promote --machine chrysalis show), the sha field is useful, but perhaps a sha_date field would be good as well, so it's easy to immediately identify how old it is.
Allow for partial updates of the expected results (maybe even allow the user to set which expected results to use as baseline?)
- (From Claude) No support yet for staggered per-package baselines. promote --machine chrysalis show and the report (Compared against baseline 20260918_unified113) both show one global baseline used for every package except zppy's own dev env. PR 871's config exposes DIAGS_EXPECTED_RESULTS_DATE, MPAS_EXPECTED_RESULTS_DATE, etc. for exactly this reason. Until this is ported, none of the diff counts between the two approaches are apples-to-apples.
It'd be great if we can make what expected results we're comparing against completely configurable. That is, for each task, for each cfg, there'd be an archive of expected results (up to a certain point in the past to avoid wasting space), and the tester could choose which set of expected results to use per task. The test report would then list which expected results set was used for each task. The cron job config would also specify which expected results it should work off of.

Add Claude-generated rst file documenting the internal workings of the new testing framework.

Co-authored-by: forsyth2 <30700190+forsyth2@users.noreply.github.com>

forsyth2 · 2026-10-13T22:28:52Z

Rebased off the latest main; was based off of 9/24 main (after merging #815)

forsyth2 · 2026-10-13T22:45:46Z

Currently running this code configured to match the 2026-10-09 E3SM Unified 1.14.0rc3 Chrysalis test, for comparison.

forsyth2 · 2026-10-14T16:49:13Z

E3SM Unified 1.14.0rc3 Chrysalis test -- using refactored test framework

Trying to follow the 2026-10-09 E3SM Unified 1.14.0rc3 Chrysalis test as close as possible.

Set up the test

cd ~/ez/zppy
git status
# On branch streamline-complete-run-test-updated
# nothing to commit, working tree clean
git log --oneline | head -n 37
# b1797863 Mark report test as Unified-backed
# ...
# 80035514 Replace the bash complete run driver with tests/complete_run
# 64bffc9f Merge pull request #815 from E3SM-Project/zppy_pcmdi_enhancement

# Good, matches the 36 commits of https://github.com/E3SM-Project/zppy/pull/883/commits
# BUT we do need to rebase off the latest `main` to get to v3.3.0rc1
git fetch upstream main
git rebase upstream/main
git push -f upstream streamline-complete-run-test-updated # Update the branch on GitHub
# Good, GitHub Actions pass

# The launching env only needs Python and mache. The rc3 Unified env has both.
bash
source /lcrc/soft/climate/e3sm-unified/test_e3sm_unified_1.14.0rc3_chrysalis.sh

python -m tests.cfg_suite.promote --machine chrysalis show

{
  "channel": "main",
  "default_run": {
    "created_at": "",
    "exists": true,
    "generated": "2026-09-18T17:50:18.240674+00:00",
    "repos": {
      "e3sm_diags": {
        "age_days": null,
        "environment_type": "unified",
        "package": "e3sm_diags",
        "package_version": "3.2.0",
        "unified_version": "1.13.0"
      },
      "e3sm_to_cmip": {
        "age_days": null,
        "environment_type": "unified",
        "package": "e3sm_to_cmip",
        "package_version": "1.14.0",
        "unified_version": "1.13.0"
      },
      "mpas_analysis": {
        "age_days": null,
        "environment_type": "unified",
        "package": "mpas-analysis",
        "package_version": "1.15.0",
        "unified_version": "1.13.0"
      },
      "zppy": {
        "age_days": null,
        "branch": "streamline-complete-run-test",
        "environment": "test-zppy-streamline-complete-run-test-20260918_unified113",
        "environment_type": "dev",
        "remote": "local",
        "sha": "16969263de247311e1662d1c170e797b76196aee",
        "short_sha": "16969263",
        "source_repo": "/gpfs/fs1/home/ac.zhang40/zppy",
        "worktree": "/lcrc/globalscratch/ac.zhang40/zppy_complete_run/20260918_unified113/worktrees/zppy"
      },
      "zppy_interfaces": {
        "age_days": null,
        "environment_type": "unified",
        "package": "zppy-interfaces",
        "package_version": "0.2.1",
        "unified_version": "1.13.0"
      }
    },
    "run": "/lcrc/group/e3sm/public_html/zppy_complete_run/runs/20260918_unified113",
    "tag": "20260918_unified113",
    "username": ""
  },
  "exists": true,
  "generated": "2026-09-18T17:50:18.240674+00:00",
  "link": "/lcrc/group/e3sm/public_html/zppy_complete_run/baselines/latest-main",
  "machine": "chrysalis",
  "overrides": {},
  "repos": {
    "e3sm_diags": {
      "environment_type": "unified",
      "package": "e3sm_diags",
      "package_version": "3.2.0",
      "unified_version": "1.13.0"
    },
    "e3sm_to_cmip": {
      "environment_type": "unified",
      "package": "e3sm_to_cmip",
      "package_version": "1.14.0",
      "unified_version": "1.13.0"
    },
    "mpas_analysis": {
      "environment_type": "unified",
      "package": "mpas-analysis",
      "package_version": "1.15.0",
      "unified_version": "1.13.0"
    },
    "zppy": {
      "branch": "streamline-complete-run-test",
      "environment": "test-zppy-streamline-complete-run-test-20260918_unified113",
      "environment_type": "dev",
      "remote": "local",
      "sha": "16969263de247311e1662d1c170e797b76196aee",
      "short_sha": "16969263",
      "source_repo": "/gpfs/fs1/home/ac.zhang40/zppy",
      "worktree": "/lcrc/globalscratch/ac.zhang40/zppy_complete_run/20260918_unified113/worktrees/zppy"
    },
    "zppy_interfaces": {
      "environment_type": "unified",
      "package": "zppy-interfaces",
      "package_version": "0.2.1",
      "unified_version": "1.13.0"
    }
  },
  "run": "/lcrc/group/e3sm/public_html/zppy_complete_run/runs/20260918_unified113",
  "runs": {
    "20260918_unified113": {
      "created_at": "",
      "exists": true,
      "generated": "2026-09-18T17:50:18.240674+00:00",
      "repos": {
        "e3sm_diags": {
          "age_days": null,
          "environment_type": "unified",
          "package": "e3sm_diags",
          "package_version": "3.2.0",
          "unified_version": "1.13.0"
        },
        "e3sm_to_cmip": {
          "age_days": null,
          "environment_type": "unified",
          "package": "e3sm_to_cmip",
          "package_version": "1.14.0",
          "unified_version": "1.13.0"
        },
        "mpas_analysis": {
          "age_days": null,
          "environment_type": "unified",
          "package": "mpas-analysis",
          "package_version": "1.15.0",
          "unified_version": "1.13.0"
        },
        "zppy": {
          "age_days": null,
          "branch": "streamline-complete-run-test",
          "environment": "test-zppy-streamline-complete-run-test-20260918_unified113",
          "environment_type": "dev",
          "remote": "local",
          "sha": "16969263de247311e1662d1c170e797b76196aee",
          "short_sha": "16969263",
          "source_repo": "/gpfs/fs1/home/ac.zhang40/zppy",
          "worktree": "/lcrc/globalscratch/ac.zhang40/zppy_complete_run/20260918_unified113/worktrees/zppy"
        },
        "zppy_interfaces": {
          "age_days": null,
          "environment_type": "unified",
          "package": "zppy-interfaces",
          "package_version": "0.2.1",
          "unified_version": "1.13.0"
        }
      },
      "run": "/lcrc/group/e3sm/public_html/zppy_complete_run/runs/20260918_unified113",
      "tag": "20260918_unified113",
      "username": ""
    }
  },
  "tag": "20260918_unified113",
  "unified": {
    "load_script": "/lcrc/soft/climate/e3sm-unified/load_latest_e3sm_unified_chrysalis.sh",
    "packages": {
      "e3sm_diags": "3.2.0",
      "e3sm_to_cmip": "1.14.0",
      "livvkit": "3.3.1",
      "mpas-analysis": "1.15.0",
      "nco": "5.3.9",
      "numpy": "2.4.4",
      "python": "3.13.13",
      "xarray": "2026.2.0",
      "xcdat": "0.11.2",
      "zppy": "3.2.0",
      "zppy-interfaces": "0.2.1"
    },
    "prefix": "/lcrc/soft/climate/e3sm-unified/e3smu_1_13_0/chrysalis/pixi_login/.pixi/envs/default",
    "recorded": "backfilled 2026-09-18 from env_descriptions/e3sm_diags.txt, which the run wrote",
    "resolved_load_script": "/lcrc/soft/climate/e3sm-unified/load_e3sm_unified_1.13.0_chrysalis.sh",
    "version": "1.13.0"
  }
}

These expected results differ from what we used for the other test. We'd need to update the baseline at the same time for both to actually compare diffs properly. But for now, this will do.

cp ~/ez/zppy/tests/cfg_suite/cfg-suite.example.cfg ~/ez/zppy_cfg_suite_rc3.cfg
emacs ~/ez/zppy_cfg_suite_rc3.cfg
# Update the following:
# [cfg_suite]
# machine = chrysalis
# account = e3sm
# label = unified_rc3
# repo_root = /lcrc/group/e3sm/ac.forsyth2/zppy_pr876_test_dirs/
# branch = zppy=streamline-complete-run-test-updated
# env_type = zppy=unified,e3sm_diags=unified,e3sm_to_cmip=unified,mpas_analysis=unified,zppy_interfaces=unified
# unified_load_script = /lcrc/soft/climate/e3sm-unified/test_e3sm_unified_1.14.0rc3_chrysalis.sh
cd /lcrc/group/e3sm/ac.forsyth2/zppy_pr876_test_dirs/zppy
git status
# On branch streamline-complete-run-test-updated
# nothing to commit, working tree clean
git fetch upstream streamline-complete-run-test-updated
git reset --hard upstream/streamline-complete-run-test-updated

screen
ulimit -s unlimited # needed for MPAS-Analysis inside screen
cd /lcrc/group/e3sm/ac.forsyth2/zppy_pr876_test_dirs/zppy
cat ~/ez/zppy_cfg_suite_rc3.cfg # Looks good
time python -m tests.cfg_suite.automation --config ~/ez/zppy_cfg_suite_rc3.cfg \
    2>&1 | tee cfg_suite_20261013_run1.log
# Ctrl-A D to detach
screen -ls
# 3735093.pts-24.chrlogin1	(Detached)
tail -f cfg_suite_20261013_run1.log

# Picking back up the next day
cd /lcrc/group/e3sm/ac.forsyth2/zppy_pr876_test_dirs/zppy
tail cfg_suite_20261013_run1.log

2026-10-13 20:16:57,810 INFO    Wrote image review summary: /lcrc/group/e3sm/public_html/zppy_complete_run/runs/20261013_174427_unified_rc3/index.html
2026-10-13 20:16:57,814 INFO    Run reports: /lcrc/group/e3sm/public_html/zppy_complete_run/runs/20261013_174427_unified_rc3/cfg-suite-report.json and /lcrc/group/e3sm/public_html/zppy_complete_run/runs/20261013_174427_unified_rc3/cfg-suite-report.md
2026-10-13 20:16:57,814 INFO    Job scripts and logs: /lcrc/globalscratch/ac.forsyth2/zppy_complete_run/20261013_174427_unified_rc3/output
2026-10-13 20:16:57,867 INFO    Leaving 1 worktrees in place under /lcrc/globalscratch/ac.forsyth2/zppy_complete_run/20261013_174427_unified_rc3/worktrees

screen -R

2026-10-13 20:16:57,867 INFO    Leaving 1 worktrees in place under /lcrc/globalscratch/ac.forsyth2/zppy_complete_run/20261013_174427_unified_rc3/worktrees

real    152m41.548s
user    0m54.953s
sys     3m47.634s

exit # Exit screen
ls -t /lcrc/group/e3sm/public_html/zppy_complete_run/runs | head -n 3
# Tag is: 20261013_174427_unified_rc3
cat /lcrc/group/e3sm/public_html/zppy_complete_run/runs/20261013_174427_unified_rc3/cfg-suite-report.md
# Pasted below in the "Generated Markdown Report" section.
ls /lcrc/group/e3sm/public_html/zppy_complete_run/runs/20261013_174427_unified_rc3/index.html
# Good, the web page exists

Web page is visible at: https://web.lcrc.anl.gov/public/e3sm/zppy_complete_run/runs/20261013_174427_unified_rc3/

Generated Markdown Report

zppy cfg suite test -- 20261013_174427_unified_rc3

Status: tests_failed
Terminal stage: validation_failed
Machine: chrysalis
Baseline: 20260918_unified113 (produced 2026-09-18)

Validation found failures; see the report for details.

What was tested

Repository	Branch	Commit	Environment
`e3sm_diags`	—	—	E3SM-Unified ? (`e3sm_diags` unknown)
`e3sm_to_cmip`	—	—	E3SM-Unified ? (`e3sm_to_cmip` unknown)
`mpas_analysis`	—	—	E3SM-Unified ? (`mpas-analysis` unknown)
`zppy`	`streamline-complete-run-test-updated`	`2e8b63ae6756`	E3SM-Unified ? (`zppy` v3.3.0rc1; `/lcrc/soft/climate/e3sm-unified/e3smu_1_14_0rc3/chrysalis/pixi_login/.pixi/envs/default/lib/python3.14/site-packages/zppy/__init__.py`)
`zppy_interfaces`	—	—	E3SM-Unified ? (`zppy-interfaces` unknown)

Baseline by cfg/task

Cfg	Task	Run	Produced	Repository commits
`weekly_bundles`	`climo`	`20260918_unified113`	2026-09-18	`zppy` `1696926`
`weekly_bundles`	`e3sm_diags`	`20260918_unified113`	2026-09-18	`zppy` `1696926`
`weekly_bundles`	`e3sm_to_cmip`	`20260918_unified113`	2026-09-18	`zppy` `1696926`
`weekly_bundles`	`global_time_series`	`20260918_unified113`	2026-09-18	`zppy` `1696926`
`weekly_bundles`	`ilamb`	`20260918_unified113`	2026-09-18	`zppy` `1696926`
`weekly_bundles`	`ts`	`20260918_unified113`	2026-09-18	`zppy` `1696926`
`weekly_comprehensive_v2`	`climo`	`20260918_unified113`	2026-09-18	`zppy` `1696926`
`weekly_comprehensive_v2`	`e3sm_diags`	`20260918_unified113`	2026-09-18	`zppy` `1696926`
`weekly_comprehensive_v2`	`e3sm_to_cmip`	`20260918_unified113`	2026-09-18	`zppy` `1696926`
`weekly_comprehensive_v2`	`global_time_series`	`20260918_unified113`	2026-09-18	`zppy` `1696926`
`weekly_comprehensive_v2`	`ilamb`	`20260918_unified113`	2026-09-18	`zppy` `1696926`
`weekly_comprehensive_v2`	`mpas_analysis`	`20260918_unified113`	2026-09-18	`zppy` `1696926`
`weekly_comprehensive_v2`	`tc_analysis`	`20260918_unified113`	2026-09-18	`zppy` `1696926`
`weekly_comprehensive_v2`	`ts`	`20260918_unified113`	2026-09-18	`zppy` `1696926`
`weekly_comprehensive_v3`	`climo`	`20260918_unified113`	2026-09-18	`zppy` `1696926`
`weekly_comprehensive_v3`	`e3sm_diags`	`20260918_unified113`	2026-09-18	`zppy` `1696926`
`weekly_comprehensive_v3`	`e3sm_to_cmip`	`20260918_unified113`	2026-09-18	`zppy` `1696926`
`weekly_comprehensive_v3`	`global_time_series`	`20260918_unified113`	2026-09-18	`zppy` `1696926`
`weekly_comprehensive_v3`	`ilamb`	`20260918_unified113`	2026-09-18	`zppy` `1696926`
`weekly_comprehensive_v3`	`livvkit`	`20260918_unified113`	2026-09-18	`zppy` `1696926`
`weekly_comprehensive_v3`	`mpas_analysis`	`20260918_unified113`	2026-09-18	`zppy` `1696926`
`weekly_comprehensive_v3`	`pcmdi_diags`	`20260918_unified113`	2026-09-18	`zppy` `1696926`
`weekly_comprehensive_v3`	`ts`	`20260918_unified113`	2026-09-18	`zppy` `1696926`
`weekly_legacy_3.2.0_comprehensive_v3`	`climo`	new cfg: no baseline (not compared)	—	—
`weekly_legacy_3.2.0_comprehensive_v3`	`e3sm_diags`	new cfg: no baseline (not compared)	—	—
`weekly_legacy_3.2.0_comprehensive_v3`	`e3sm_to_cmip`	new cfg: no baseline (not compared)	—	—
`weekly_legacy_3.2.0_comprehensive_v3`	`global_time_series`	new cfg: no baseline (not compared)	—	—
`weekly_legacy_3.2.0_comprehensive_v3`	`ilamb`	new cfg: no baseline (not compared)	—	—
`weekly_legacy_3.2.0_comprehensive_v3`	`livvkit`	new cfg: no baseline (not compared)	—	—
`weekly_legacy_3.2.0_comprehensive_v3`	`mpas_analysis`	new cfg: no baseline (not compared)	—	—
`weekly_legacy_3.2.0_comprehensive_v3`	`pcmdi_diags`	new cfg: no baseline (not compared)	—	—
`weekly_legacy_3.2.0_comprehensive_v3`	`ts`	new cfg: no baseline (not compared)	—	—

Changes since the baseline

The listed zppy commits describe the worktree under test; they may not be included in the E3SM-Unified release used to run zppy.

`ilamb` (ilamb)

Baseline: 20260918_unified113 (produced 2026-09-18); commit date-based
Tested: E3SM-Unified @ E3SM-Unified
Changes: 0 (package-version)

Could not list changes: Could not find package version for ILAMB.

`ilamb` (ilamb)

Baseline: 20260918_unified113 (produced 2026-09-18); commit date-based
Tested: E3SM-Unified @ E3SM-Unified
Changes: 0 (package-version)

Could not list changes: Could not find package version for ILAMB.

`ilamb` (ilamb)

Baseline: 20260918_unified113 (produced 2026-09-18); commit date-based
Tested: E3SM-Unified @ E3SM-Unified
Changes: 0 (package-version)

Could not list changes: Could not find package version for ILAMB.

`livvkit` (livvkit)

Baseline: 20260918_unified113 (produced 2026-09-18); commit date-based
Tested: E3SM-Unified @ E3SM-Unified
Changes: 1 (package-version)
LIVVkit: 3.3.1 → unknown

`zppy` (climo, ts, tc_analysis)

Baseline: 20260918_unified113 (produced 2026-09-18); commit 16969263de24
Tested: streamline-complete-run-test-updated @ 2e8b63ae6756
Changes: 42 (merge-base)

The baseline commit is not an ancestor of the tested commit; the log starts at their merge-base.

2e8b63ae6756 (2026-10-13): Mark report test as Unified-backed
0130eb26c370 (2026-10-13): Select baseline task in regression test
f67251c77e98 (2026-10-13): Fix cfg-suite unit test failures
3565fabe3a76 (2026-10-13): Get pre-commit passing
1184ee05d23d (2026-10-13): Fix reported mypy errors
f8ed26db7931 (2026-10-13): Partial pre-commit fixes
803fb613e208 (2026-10-13): Fix cfg-suite flake8 findings
c9324894f02c (2026-10-13): Rename complete-run framework to cfg suite
035219b91654 (2026-10-13): Replace legacy comprehensive v3 configs
a0c9080a4768 (2026-10-13): Add baseline and image summary coverage
aa9a948a6475 (2026-10-13): Apply remaining changes
cff111bc9efd (2026-10-13): Add baseline change and image summary reporting
7783b4a71640 (2026-10-13): Refresh final environment reporting
5ec477f2e8ac (2026-10-13): Document configurable task baselines
09976d95eb26 (2026-10-13): Complete baseline feature tests
da8d4977a92d (2026-10-13): Apply remaining changes
510b97a7fbbb (2026-10-13): Complete scoped task reporting
ac194da7250f (2026-10-13): Test scoped task summaries
7522152a17c3 (2026-10-13): Scope complete-run tasks
36393b1bc451 (2026-10-13): Add INI config and Unified environment support
fa2df651e377 (2026-10-13): Keep run tracking test inside tmp_path
3894d1958a17 (2026-10-13): Fix pre-commit errors
b50795daf2f0 (2026-10-13): Cancel unsatisfiable run jobs while polling
915443e85c6f (2026-10-13): Document how the new testing framework works
59ab5a33c53b (2026-10-13): Stop advising a task-limited run as a partial baseline update
273fd5ec2e22 (2026-10-13): Mark notable packages with a bullet, not a stray "2"
4589fb06aa54 (2026-10-13): List only packages in this run's environment against E3SM-Unified
a84f44b1a912 (2026-10-13): Compare against an E3SM-Unified baseline through its package list
0fd642a7f5e7 (2026-10-13): Show images on the review pages; make package and task links visible
ec86daa4eb8f (2026-10-13): Let the image checker's own test run off an E3SM machine
344061f9b99b (2026-10-13): Resubmit the jobs a cancelled run left claiming to be queued
b7b99833496c (2026-10-13): Solve environments with mamba; give dev-spec.txt its channels
736fd428f48d (2026-10-13): Record which E3SM-Unified release a run used
c27410accebd (2026-10-13): Regenerate the settings baselines instead of collecting leftovers
bfeb5919dbb3 (2026-10-13): Activate conda from the run's profile in the image checker job
b7bec55e2cd9 (2026-10-13): Replace the bash complete run driver with tests/complete_run
cc780ec6ac03 (2026-10-02): Bump to 3.3.0rc1 (#882)
3c90170c312d (2026-10-02): Derive the EAMxx source vertical grid file at run time (#864)
19d7744ead93 (2026-10-02): Support ne256 in tc_analysis (#879)
b505ce7a56d8 (2026-10-01): Update pre-commit dependencies (#880)
7a051b71d939 (2026-09-29): Allow specifying SLURM --mail-user and --mail-type in config file (#878)
64bffc9f06e4 (2026-09-24): Merge pull request #815 from E3SM-Project/zppy_pcmdi_enhancement

Environment

Environment: WARNING 19 environments not compared

No dependency changed in the environments compared, but e3sm_diags, e3sm_diags, e3sm_diags, e3sm_to_cmip, e3sm_to_cmip, e3sm_to_cmip, mpas_analysis, mpas_analysis, zppy, zppy, zppy, zppy, zppy, zppy, zppy, zppy_interfaces, zppy_interfaces, zppy_interfaces, zppy_interfaces could not be compared, so image differences are not necessarily attributable to the code under test.

E3SM-Unified load script: /lcrc/soft/climate/e3sm-unified/test_e3sm_unified_1.14.0rc3_chrysalis.sh

Not compared for:

e3sm_diags: No environment export or package listing for this run; dependency changes could not be compared.
e3sm_diags: No environment export or package listing for this run; dependency changes could not be compared.
e3sm_diags: No environment export or package listing for this run; dependency changes could not be compared.
e3sm_to_cmip: No environment export or package listing for the baseline and this run; dependency changes could not be compared.
e3sm_to_cmip: No environment export or package listing for the baseline and this run; dependency changes could not be compared.
e3sm_to_cmip: No environment export or package listing for the baseline and this run; dependency changes could not be compared.
mpas_analysis: No environment export or package listing for this run; dependency changes could not be compared.
mpas_analysis: No environment export or package listing for this run; dependency changes could not be compared.
zppy: No environment export or package listing for this run; dependency changes could not be compared.
zppy: No environment export or package listing for this run; dependency changes could not be compared.
zppy: No environment export or package listing for this run; dependency changes could not be compared.
zppy: No environment export or package listing for this run; dependency changes could not be compared.
zppy: No environment export or package listing for this run; dependency changes could not be compared.
zppy: No environment export or package listing for this run; dependency changes could not be compared.
zppy: No environment export or package listing for this run; dependency changes could not be compared.
zppy_interfaces: No environment export or package listing for this run; dependency changes could not be compared.
zppy_interfaces: No environment export or package listing for this run; dependency changes could not be compared.
zppy_interfaces: No environment export or package listing for this run; dependency changes could not be compared.
zppy_interfaces: No environment export or package listing for this run; dependency changes could not be compared.

Job status files

All status files reported OK.

Integration tests

Test	Result
`tests/integration/test_last_year.py`	pass
`tests/integration/test_bash_generation.py`	FAIL
`tests/integration/test_campaign.py`	FAIL
`tests/integration/test_defaults.py`	FAIL
`tests/integration/test_bundles.py`	FAIL

Image checker

SLURM job: 1308020
Terminal state: FAILED
Exit code: 1:0
Summary source: final

Severity	Images	Reviewable
IDENTICAL	40	no
NEGLIGIBLE	14734	no
MINOR	1549	yes
MODERATE	529	yes
MAJOR	519	yes
STRUCTURAL	4	yes
MISSING	157	yes

2758 images need review.

Full summary: /lcrc/group/e3sm/public_html/zppy_complete_run/runs/20261013_174427_unified_rc3/test_images_summary_20261013_174427_unified_rc3.md

Review page, with a table across every package: /lcrc/group/e3sm/public_html/zppy_complete_run/runs/20261013_174427_unified_rc3/index.html

Next steps

Failures above are for human review. A cfg suite never promotes expected results; that remains the separate, explicit update_*_expected_files_<machine>.sh step.

Image check summary

cat /lcrc/group/e3sm/public_html/zppy_complete_run/runs/20261013_174427_unified_rc3/test_images_summary_20261013_174427_unified_rc3.md

Summary of test results

Test name	Total images	Correct images	Identical	Cosmetic only	Missing	Needs review (excluding missing)	Total needing attention	Severity
bundles_e3sm_diags	1762	1703	3	1700	0	59	59	28 moderate, 31 minor, 1700 negligible, 3 identical
bundles_global_time_series	3	3	0	3	0	0	0	3 negligible
bundles_ilamb	388	144	0	144	0	244	244	42 major, 44 moderate, 158 minor, 144 negligible
comprehensive_v2_e3sm_diags	3806	3697	3	3694	1	108	109	1 missing, 31 major, 56 moderate, 21 minor, 3694 negligible, 3 identical
comprehensive_v2_mpas_analysis	856	613	4	609	0	243	243	63 moderate, 180 minor, 609 negligible, 4 identical
comprehensive_v2_global_time_series	12	12	0	12	0	0	0	12 negligible
comprehensive_v2_ilamb	776	282	0	282	0	494	494	84 major, 88 moderate, 322 minor, 282 negligible
comprehensive_v3_e3sm_diags	5369	5056	3	5053	0	313	313	227 major, 53 moderate, 33 minor, 5053 negligible, 3 identical
comprehensive_v3_mpas_analysis	1280	903	6	897	0	377	377	71 moderate, 306 minor, 897 negligible, 6 identical
comprehensive_v3_global_time_series	1404	1398	0	1398	0	6	6	6 minor, 1398 negligible
comprehensive_v3_ilamb	837	348	12	336	0	489	489	99 major, 104 moderate, 286 minor, 336 negligible, 12 identical
comprehensive_v3_livvkit	392	303	8	295	0	89	89	7 moderate, 82 minor, 295 negligible, 8 identical
comprehensive_v3_pcmdi_diags	647	312	1	311	156	179	335	156 missing, 4 structural, 36 major, 15 moderate, 124 minor, 311 negligible, 1 identical

Legend: Missing means an expected image was not produced. Needs review excludes missing images. Total needing attention includes missing images and other reviewable differences.

forsyth2 · 2026-10-14T17:17:55Z

Test approach comparison 3: E3SM Unified 1.14.0rc3 testing

Current testing framework: E3SM Unified 1.14.0 testing #885 (comment)
Refactored testing framework: Working branch for updating new testing framework #883 (comment)

Below is Claude's analysis based on the output links above (and text pasted from https://web.lcrc.anl.gov/public/e3sm/zppy_complete_run/runs/20261013_174427_unified_rc3/). I need to spend more time reviewing this.

Comparison: current vs. experimental/refactored test framework (E3SM-Unified 1.14.0rc3, Chrysalis)

This compares the current test framework (the one actually in use) with the experimental/refactored test framework, both run against Unified 1.14.0rc3.

Known differences between the two runs

The current framework's output was produced before the legacy 3.0.0/3.1.0 cfgs were replaced with a legacy 3.2.0 cfg. That is why it doesn't show v3.2.0, and why the refactored framework shows no baseline for the v3.2.0 cfg ("new cfg: no baseline").
The two runs use different baselines, so their diffs aren't directly comparable.
- Current framework: the 9/28 test, promoted to expected results on 9/30. The current framework's testing notes say results from Unified 1.13.0 predate 5/20, so this baseline is not 1.13.0.
- Refactored framework: run 20260918_unified113, produced 2026-09-18 and promoted as the latest-main baseline. Here "main" is the baseline channel, not the branch that was tested. Per promote show, the baseline mixes:
  - released E3SM-Unified 1.13.0 for e3sm_diags (3.2.0), e3sm_to_cmip (1.14.0), mpas-analysis (1.15.0) and zppy-interfaces (0.2.1)
  - zppy from a dev env built from branch streamline-complete-run-test at 16969263, not zppy from Unified and not zppy main
- That commit is not an ancestor of the commit under test, so the branch has since been rebased or has diverged.
- The baseline run also used an earlier version of the framework: its output has complete-run-report.* files, while the rc3 run has cfg-suite-report.*.
In short, the current framework compares rc3 against a late-September pre-release. The refactored framework compares rc3 against Unified 1.13.0 tools plus a dev-branch zppy.

Things the current framework does that the refactored one doesn't

Dependency changes since the baseline: lists the PRs merged for e3sm_diags, e3sm_to_cmip, zppy-interfaces and zppy since the expected results were produced. The refactored framework only gives a zppy commit log and a package-version diff, and the version diff came back empty (see below).
Expected-results provenance: shows promotion dates per cfg/task, content-based "produced" dates, and *_LAST_TESTED_DATE for e3sm_to_cmip and zppy. The refactored framework only reports a baseline run tag and date.
Per-task env_description.txt files: lists them and copies them into each cfg's _www directory.
Unit-test status lines: reports zppy-interfaces, zppy and image-checker/report unit tests. The refactored framework's markdown only reports job status files and the five integration tests. It may run the others without reporting them.
Per-test links: the summary table has sample, list, grid and ranked links. The refactored framework's summary has none in the pasted text, though links may have been lost in the paste.
Legacy 3.0.0/3.1.0 cfgs: covered above.
Manual analysis steps: the rc2-vs-rc3 comparison script, and the grep confirming every bash script sources the rc3 load script. The refactored framework reports the load script but doesn't verify the generated scripts.

Things the refactored framework does that the current one doesn't

livvkit image checks: the current framework's summary table has no livvkit rows, even though livvkit has expected results. The reason is unclear.
Baseline tracking: baselines are tracked by run tag with a per-cfg/task commit, and a promote CLI manages them.
Automated change reporting: it lists the 42 zppy commits since the merge-base (with links) and attempts a package-version diff.
Environment warnings: it flags when image differences can't be attributed to code changes.
Review tables: by-package and by-check tables with MISSING and STRUCTURAL columns. They separate "missing" from "needs review" and record the 6 checks with no comparison.
Non-image tasks: e3sm_to_cmip and zppy tasks appear as OK/n/a.
Outputs and orchestration: it produces JSON, markdown and a web page that shows images. It runs from a single INI cfg (the current framework needs many hand-set date variables), supports task-scoped runs, and cancels unsatisfiable jobs and resubmits stale ones.
The v3.2.0 legacy cfg: covered above.

Discrepancies in how the two handle the same thing

Images needing review (current framework: "Needs review"; refactored: "Total needing attention", which includes missing images):

Test	Current (total / review)	Refactored (total / review)
bundles e3sm_diags	1843 / 30	1762 / 59
v2 e3sm_diags	3885 / 61	3806 / 109
v3 e3sm_diags	5464 / 295	5369 / 313
ilamb bundles	388 / 244	identical
ilamb v2	776 / 494	identical
ilamb v3	1860 / 1136	837 / 489
mpas_analysis v2	856 / 1	856 / 243
mpas_analysis v3	1280 / 3	1280 / 377
global_time_series v3	1404 / 0	1404 / 6
pcmdi_diags v3	724 / 0	647 / 335 (156 missing, 4 structural)
livvkit v3	not in table	392 / 89

Results that hold up despite different baselines are probably real. The v3 e3sm_diags major count is 227 in both, and ILAMB bundles and v2 match exactly.
mpas_analysis, pcmdi_diags and global_time_series differences are likely baseline-driven.
- The current framework's mpas results (1 and 3) were already confirmed acceptable by Xylar at rc2. The refactored framework's 243 and 377 compare against a Unified 1.13.0 baseline.
- The refactored framework's "identical" counts are tiny (3 to 6 where the current one has hundreds or thousands), which also fits comparing against an older environment.
- For pcmdi_diags, pcmdi_diags/mismatched_images.txt should show what the 156 missing images are.
Two results are odd and worth a look.
- The refactored framework's v3 ILAMB result (837 images; 99 major, 104 moderate, 286 minor, 336 negligible, 12 identical) exactly matches the current framework's legacy_3.0.0 row, not its v3 row (1860 images).
- The refactored framework's e3sm_diags totals are 79 to 95 images lower than the current framework's, for reasons that aren't clear.

"No environment export or package listing for this run; dependency changes could not be compared"

This is probably related to testing a Unified RC, but the reports alone don't confirm the cause.

The "What was tested" table shows E3SM-Unified ? and unknown package versions for everything except zppy, whose version was read from the import path (v3.3.0rc1). The framework apparently couldn't determine what's in the rc3 environment.
The 19 "not compared" entries are one per baseline cfg × task: 3 e3sm_diags, 3 e3sm_to_cmip, 2 mpas_analysis, 7 zppy and 4 zppy-interfaces. That is why the message repeats without saying which cfg it refers to.
It isn't purely an RC issue. The baseline's Unified package list was backfilled from e3sm_diags.txt alone, so e3sm_to_cmip has no listing on either side, as the "baseline and this run" wording indicates.
The baseline's package list has no ILAMB entry, which explains "Could not find package version for ILAMB". The livvkit change 3.3.1 → unknown is spurious, because the tested side is just unresolved.
The current framework did write env_description.txt files for this same rc3 run, so the information was obtainable.
The baseline directory has env_descriptions/ and environments/ subdirectories. I haven't yet checked what they contain, or whether the rc3 run directory has equivalents.
Suggested follow-ups:
- find where the framework emits this message;
- check how it turns the load script name (test_e3sm_unified_1.14.0rc3_chrysalis.sh) into a release version;
- compare the contents of the baseline's env_descriptions/ and environments/ against the rc3 run directory.

Does the refactored framework's web page match its markdown report?

Where they overlap, yes. The totals agree: 17,532 images, 157 missing, 2,601 needing review excluding missing, and 2,758 total needing attention. The per-test rows in test_images_summary also match the corresponding by-check rows on the web page.

But the web page is not just a prettier version of the markdown report, because each contains content the other doesn't:

Web page only: the by-package and by-check tables (cfg × task × baseline × severity) and the "11 of 19 checks need review" header.
Markdown report only: the what-was-tested table, per-cfg baselines, job status files, integration-test results, SLURM details and next steps.

The markdown report's image section only has aggregate severity totals, so the per-cfg/task breakdown exists only on the web page. The web page content was pasted as plain text, so links, images and styling weren't checked.

forsyth2 · 2026-10-14T17:21:16Z

livvkit image checks: the current framework's summary table has no livvkit rows, even though livvkit has expected results. The reason is unclear.

See #871 (comment)

forsyth2 self-assigned this Oct 5, 2026

forsyth2 added the Testing Files in `tests` modified label Oct 5, 2026

forsyth2 changed the title ~~Streamline complete run test updated~~ Working branch for updating new testing framework Oct 5, 2026

Copilot started work on behalf of forsyth2 October 5, 2026 18:08

View session

Copilot finished work on behalf of forsyth2 October 5, 2026 18:12

Copilot started work on behalf of forsyth2 October 5, 2026 18:26

View session

Copilot finished work on behalf of forsyth2 October 5, 2026 18:27

forsyth2 mentioned this pull request Oct 9, 2026

Use v3.2.0 cfg for legacy testing #886

Draft

1 of 7 tasks

Copilot started work on behalf of forsyth2 October 13, 2026 16:29

View session

Copilot finished work on behalf of forsyth2 October 13, 2026 16:45

Copilot started work on behalf of forsyth2 October 13, 2026 16:56

View session

Copilot finished work on behalf of forsyth2 October 13, 2026 17:17

Copilot started work on behalf of forsyth2 October 13, 2026 17:28

View session

Copilot finished work on behalf of forsyth2 October 13, 2026 17:41

Copilot started work on behalf of forsyth2 October 13, 2026 17:46

View session

Copilot finished work on behalf of forsyth2 October 13, 2026 17:52

forsyth2 and others added 24 commits October 13, 2026 17:27

Document how the new testing framework works

915443e

Add Claude-generated rst file documenting the internal workings of the new testing framework.

Cancel unsatisfiable run jobs while polling

b50795d

Co-authored-by: forsyth2 <30700190+forsyth2@users.noreply.github.com>

Fix pre-commit errors

3894d19

Keep run tracking test inside tmp_path

fa2df65

Co-authored-by: forsyth2 <30700190+forsyth2@users.noreply.github.com>

Add INI config and Unified environment support

36393b1

Co-authored-by: forsyth2 <30700190+forsyth2@users.noreply.github.com>

Scope complete-run tasks

7522152

Co-authored-by: forsyth2 <30700190+forsyth2@users.noreply.github.com>

Test scoped task summaries

ac194da

Co-authored-by: forsyth2 <30700190+forsyth2@users.noreply.github.com>

Complete scoped task reporting

510b97a

Co-authored-by: forsyth2 <30700190+forsyth2@users.noreply.github.com>

Apply remaining changes

da8d497

Co-authored-by: forsyth2 <30700190+forsyth2@users.noreply.github.com>

Complete baseline feature tests

09976d9

Co-authored-by: forsyth2 <30700190+forsyth2@users.noreply.github.com>

Document configurable task baselines

5ec477f

Co-authored-by: forsyth2 <30700190+forsyth2@users.noreply.github.com>

Refresh final environment reporting

7783b4a

Co-authored-by: forsyth2 <30700190+forsyth2@users.noreply.github.com>

Add baseline change and image summary reporting

cff111b

Co-authored-by: forsyth2 <30700190+forsyth2@users.noreply.github.com>

Apply remaining changes

aa9a948

Co-authored-by: forsyth2 <30700190+forsyth2@users.noreply.github.com>

Add baseline and image summary coverage

a0c9080

Co-authored-by: forsyth2 <30700190+forsyth2@users.noreply.github.com>

Replace legacy comprehensive v3 configs

035219b

Co-authored-by: forsyth2 <30700190+forsyth2@users.noreply.github.com>

Rename complete-run framework to cfg suite

c932489

Co-authored-by: forsyth2 <30700190+forsyth2@users.noreply.github.com>

Fix cfg-suite flake8 findings

803fb61

Co-authored-by: forsyth2 <30700190+forsyth2@users.noreply.github.com>

Partial pre-commit fixes

f8ed26d

Fix reported mypy errors

1184ee0

Co-authored-by: forsyth2 <30700190+forsyth2@users.noreply.github.com>

Get pre-commit passing

3565fab

Fix cfg-suite unit test failures

f67251c

Co-authored-by: forsyth2 <30700190+forsyth2@users.noreply.github.com>

Select baseline task in regression test

0130eb2

Co-authored-by: forsyth2 <30700190+forsyth2@users.noreply.github.com>

Mark report test as Unified-backed

2e8b63a

Co-authored-by: forsyth2 <30700190+forsyth2@users.noreply.github.com>

forsyth2 force-pushed the streamline-complete-run-test-updated branch from b179786 to 2e8b63a Compare October 13, 2026 22:28

Conversation

forsyth2 commented Oct 5, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

forsyth2 commented Oct 5, 2026

Uh oh!

forsyth2 commented Oct 5, 2026

Uh oh!

Copilot AI commented Oct 5, 2026

Uh oh!

forsyth2 commented Oct 5, 2026

Uh oh!

Copilot AI commented Oct 5, 2026

Uh oh!

forsyth2 commented Oct 9, 2026

Uh oh!

forsyth2 commented Oct 10, 2026

Uh oh!

forsyth2 commented Oct 13, 2026

Uh oh!

Copilot AI commented Oct 13, 2026

Uh oh!

forsyth2 commented Oct 13, 2026

Uh oh!

forsyth2 commented Oct 13, 2026

Uh oh!

forsyth2 commented Oct 13, 2026

Uh oh!

forsyth2 commented Oct 13, 2026

Uh oh!

forsyth2 commented Oct 13, 2026

Uh oh!

forsyth2 commented Oct 13, 2026

Uh oh!

forsyth2 commented Oct 13, 2026

Uh oh!

forsyth2 commented Oct 14, 2026

E3SM Unified 1.14.0rc3 Chrysalis test -- using refactored test framework

zppy cfg suite test -- 20261013_174427_unified_rc3

What was tested

Baseline by cfg/task

Changes since the baseline

ilamb (ilamb)

ilamb (ilamb)

ilamb (ilamb)

livvkit (livvkit)

zppy (climo, ts, tc_analysis)

Environment

Job status files

Integration tests

Image checker

Next steps

Summary of test results

Uh oh!

forsyth2 commented Oct 14, 2026

Test approach comparison 3: E3SM Unified 1.14.0rc3 testing

Comparison: current vs. experimental/refactored test framework (E3SM-Unified 1.14.0rc3, Chrysalis)

Known differences between the two runs

Things the current framework does that the refactored one doesn't

Things the refactored framework does that the current one doesn't

Discrepancies in how the two handle the same thing

"No environment export or package listing for this run; dependency changes could not be compared"

Does the refactored framework's web page match its markdown report?

Uh oh!

forsyth2 commented Oct 14, 2026

Uh oh!

This branch has not been deployed

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

forsyth2 commented Oct 5, 2026 •

edited

Loading

`ilamb` (ilamb)

`ilamb` (ilamb)

`ilamb` (ilamb)

`livvkit` (livvkit)

`zppy` (climo, ts, tc_analysis)