Repository navigation
Conversation
forsyth2
added
Documentation
Collaborator
Author
|
Updating expected results for the 8/12 test run Set up branchgit checkout partial-update-expectations
git fetch upstream main
git rebase upstream/main
git log --oneline | head -n 4
# b5d41f34 Add auto-generated update files
# ef72638c Change term from diags to tasks
# fffd2338 Support partial updates of weekly/campaign expected test results
# 4735ef4d Merge pull request #854 from E3SM-Project/update-pre-commit-deps
lcrc_conda
rm -rf build
conda clean --all --y
conda env create -f conda/dev.yml -n zppy-pr852-20260814
conda activate zppy-pr852-20260814
pre-commit run --all-files
python -m pip install .
cd docs
make html
cp -r _build/ /lcrc/group/e3sm/public_html/diagnostic_output/ac.forsyth2/zppy_docs_pr852
chmod -R 755 /lcrc/group/e3sm/public_html/diagnostic_output/ac.forsyth2/zppy_docs_pr852# First, let's copy over the old expected results #############################
cp -r /lcrc/group/e3sm/public_html/zppy_test_resources /lcrc/group/e3sm/public_html/zppy_test_resources_previous/expected_results_until_20260814 # < 30 min
# Second, update the expected results #########################################
# Let's update the simpler tests' results first:
cd /lcrc/group/e3sm/ac.forsyth2/zppy_main_branch_test_dirs/zppy
git status
# You might have changed branches since you ran the tests.
# Good, on correct branch: test_zppy_20260812_run1
conda activate test-zppy-main-20260812_run1
# Good, now on correct env
# Make sure the update script permissions are set up
chmod 755 tests/integration/generated/update_bash_generation_expected_files_chrysalis.sh
chmod 755 tests/integration/generated/update_campaign_expected_files_chrysalis.sh
chmod 755 tests/integration/generated/update_defaults_expected_files_chrysalis.sh
chmod 755 tests/integration/generated/update_weekly_expected_files_chrysalis.sh
# Specific to partial updates:
# Update the task list
emacs tests/integration/generated/update_weekly_expected_files_chrysalis.sh
# This doesn't have the necessary lines yet because PR #852 didn't exist when this was run!
git add -A
git commit -m "Changes made during testing"
# Remove the old templates and replace with the files from PR #852
rm tests/integration/template_update_campaign_expected_files.sh
rm tests/integration/template_update_weekly_expected_files.sh
emacs tests/integration/template_update_campaign_expected_files.sh
emacs tests/integration/template_update_weekly_expected_files.sh
git diff # Looks ok
git add -A
git commit -m "Copy changes from PR 852"
# Now, we need to re-render the generated files
python tests/integration/utils.py
git diff # Looks ok
chmod 755 tests/integration/generated/update_campaign_expected_files_chrysalis.sh
chmod 755 tests/integration/generated/update_weekly_expected_files_chrysalis.sh
# Double check we have a record of the original expected results
ls -lt /lcrc/group/e3sm/public_html/zppy_test_resources_previous/expected_results_until_20260814
# Update the task list now that it's available:
emacs tests/integration/generated/update_weekly_expected_files_chrysalis.sh
# These are the 3 tasks we know we can update the expected results for:
# tasks=("e3sm_diags" "global_time_series" "pcmdi_diags")
git add -A
git commit -m "Update the update scripts"
# These scripts update the expected results and re-run the tests:
./tests/integration/generated/update_bash_generation_expected_files_chrysalis.sh
# 1 passed in 2.84s
./tests/integration/generated/update_campaign_expected_files_chrysalis.sh
# 6 passed in 2.43s
./tests/integration/generated/update_defaults_expected_files_chrysalis.sh
# 1 passed in 0.72s
# This script only updates the expected results
./tests/integration/generated/update_weekly_expected_files_chrysalis.sh # ~30 minSkipped because ls /lcrc/group/e3sm/public_html/zppy_test_resources
# Confirmed there are expected results subdirs and image lists for each cfg
ls -lt /lcrc/group/e3sm/public_html/zppy_test_resources
# Confirmed everything was updated today 8/14
cd /lcrc/group/e3sm/ac.forsyth2/zppy_main_branch_test_dirs/zppy
pytest tests/integration/test_bundles.py
# 2 passed in 1.53s
salloc --nodes=1 --partition=debug --time=02:00:00 --account=e3sm
bash
source ~/.bashrc
lcrc_conda
conda activate test-zppy-main-20260812_run1
pytest tests/integration/test_images.py # ~40 min
exit # Exit bash shell
exit # Exit compute noteOutputOutput:
Failing image check tests:
Good, this table matches the |
Collaborator
Author
|
I think this PR may need to be merged independently of the environment tracking noted here . The expected-results-updater script just chooses which output directories to copy over. To actually populate those dirs with environment notes, we'd need to update the test script itself. And that's what #871 is doing. Specifically:
|
forsyth2
force-pushed
the
partial-update-expectations
branch
2 times, most recently
from
September 15, 2026 00:55
03ce4d2 to
7cfba4b
Compare
Collaborator
Author
|
Updating expected results for the 9/4 test run. We got confirmation that Set up branchcd ~/ez/zppy
git status
# nothing to commit, working tree clean
git checkout partial-update-expectations
git log --oneline | head -n 4
# b5d41f34 Add auto-generated update files
# ef72638c Change term from diags to tasks
# fffd2338 Support partial updates of weekly/campaign expected test results
# 4735ef4d Merge pull request #854 from E3SM-Project/update-pre-commit-deps
# Good, commit titles match https://github.com/E3SM-Project/zppy/pull/852/commits
# Hashes don't match though. We need to update the GitHub branch since we rebased
# on an earlier results update.
git push -f upstream partial-update-expectations
git fetch upstream main
git rebase upstream/main
git log --oneline | head -n 4
# 7cfba4b1 Add auto-generated update files
# 1ecb557d Change term from diags to tasks
# 9c6c5185 Support partial updates of weekly/campaign expected test results
# 5232fe30 Rank image check failures by severity (#865)
# Push the latest commit hashes
git push -f upstream partial-update-expectations
# Good, https://github.com/E3SM-Project/zppy/pull/852/commits is updated.
lcrc_conda
rm -rf build
conda clean --all --y
conda env create -f conda/dev.yml -n zppy-pr852-20260914
conda activate zppy-pr852-20260904
pre-commit run --all-files
python -m pip install .Copy over the old expected resultsls -lt /lcrc/group/e3sm/public_html/zppy_test_resources
# Updated 9/4
# To know when each specific task had its expected results updated, check:
ls -lt /lcrc/group/e3sm/public_html/zppy_test_resources/expected_comprehensive_v3/
# Sep 4 mpas_analysis
# Sep 4 e3sm_diags
# Aug 14 pcmdi_diags
# Aug 14 global_time_series
# May 20 livvkit
# May 20 ilamb
cp -r /lcrc/group/e3sm/public_html/zppy_test_resources /lcrc/group/e3sm/public_html/zppy_test_resources_previous/expected_results_until_20260914 # <30 minutesUpdate the expected results# Let's update the simpler tests' results first:
cd /lcrc/group/e3sm/ac.forsyth2/zppy_main_branch_test_dirs/zppy
git status
# Good, on correct branch: On branch test_zppy_20260904_run1
# Some changed files exist
git add -A
git commit -m "Testing" --no-verify
git log --oneline | head -n 20
# fdca17c7 Testing
# b72f1d03 Ignore hairlines when looking for changed values
# 4992994e Rank the cosmetic sample by how much actually differs
# 7fd3dbc6 Separate images that did not change from ones that look cosmetic
# a3eec64a Testing
# 35fb99ba Describe how images differ, not why
# ec3fe89f Do not claim a panel changed when only labels did
# da893007 Settle unchanged images from their file bytes
# ee8b0b03 Do not fail when every image is unchanged
# 3fb3d241 Testing
# 7775d42f Name the likely cause on each diff grid row
# 8588ffb2 Hide unused axes on the last page of the diff grid
# e13ee3e0 Label and order the diff grid PDF by severity
# 2dd4d83c Ignore scattered spots when looking for changed values
# ce369bd5 Document how to review image check results
# 3bbacefd Rank image check failures by severity instead of listing them flat
# 31f3d9cd Detect small isolated changes such as a changed number
# 5c346f72 Add image severity scorer
# d0aa0826 Testing
# 77f4ab66 Append try# suffix to image_check_failures dir instead of overwriting (#861)
# Let's drop these extra commits from testing https://github.com/E3SM-Project/zppy/pull/865
git checkout -b test_zppy_20260904_run1_backup20260914
git checkout test_zppy_20260904_run1
git rebase -i 77f4ab66 # Drop later commits
git log --oneline | head -n 2
# d0aa0826 Testing
# 77f4ab66 Append try# suffix to image_check_failures dir instead of overwriting (#861)
conda activate test-zppy-main-20260828_run1
# Good, now on correct env
# Make sure the update script permissions are set up
chmod 755 tests/integration/generated/update_bash_generation_expected_files_chrysalis.sh
chmod 755 tests/integration/generated/update_campaign_expected_files_chrysalis.sh
chmod 755 tests/integration/generated/update_defaults_expected_files_chrysalis.sh
chmod 755 tests/integration/generated/update_weekly_expected_files_chrysalis.sh
# Specific to partial updates:
# Update the task list
emacs tests/integration/generated/update_weekly_expected_files_chrysalis.sh
# This doesn't have the necessary lines yet because PR #852 didn't exist when this was run!
# Remove the old templates and replace with the files from PR #852
rm tests/integration/template_update_campaign_expected_files.sh
rm tests/integration/template_update_weekly_expected_files.sh
# Recall ~/ez/zppy is on the branch for PR #852
cp ~/ez/zppy/tests/integration/template_update_campaign_expected_files.sh tests/integration/template_update_campaign_expected_files.sh
cp ~/ez/zppy/tests/integration/template_update_weekly_expected_files.sh tests/integration/template_update_weekly_expected_files.sh
git diff # Looks ok
git add -A
git commit -m "Copy changes from PR 852"
# Now, we need to re-render the generated files
emacs tests/integration/utils.py # Confirm ALL cfgs_to_run and tasks_to_run are set to run
python tests/integration/utils.py
git status
# modified: tests/integration/generated/update_campaign_expected_files_chrysalis.sh
# modified: tests/integration/generated/update_weekly_expected_files_chrysalis.sh
chmod 755 tests/integration/generated/update_campaign_expected_files_chrysalis.sh
chmod 755 tests/integration/generated/update_weekly_expected_files_chrysalis.sh
# Double check we have a record of the original expected results
ls -lt /lcrc/group/e3sm/public_html/zppy_test_resources_previous/expected_results_until_20260914
# Update the task list now that it's available:
emacs tests/integration/generated/update_weekly_expected_files_chrysalis.sh
# What we know we can update the expected results for:
# tasks=("pcmdi_diags")
git add -A
git commit -m "Update the update scripts"
# All the non-image-checker integration tests passed, so we can skip these steps:
# These scripts update the expected results and re-run the tests:
# ./tests/integration/generated/update_bash_generation_expected_files_chrysalis.sh
# ./tests/integration/generated/update_campaign_expected_files_chrysalis.sh
# ./tests/integration/generated/update_defaults_expected_files_chrysalis.sh
# This script only updates the expected results
./tests/integration/generated/update_weekly_expected_files_chrysalis.sh # <10 minSkipped because ls /lcrc/group/e3sm/public_html/zppy_test_resources
# Confirmed there are expected results subdirs and image lists for each cfg
ls -lt /lcrc/group/e3sm/public_html/zppy_test_resources
# Confirmed the two pcmdi-diags-containing dirs were updated today 9/14.
# All image lists were updated today 9/14.
# Everything else was last updated 9/4.Rerun the tests against the new expected resultscd /lcrc/group/e3sm/ac.forsyth2/zppy_main_branch_test_dirs/zppy
git status
# On branch test_zppy_20260904_run1
# nothing to commit, working tree clean
pytest tests/integration/test_bundles.py
# 2 passed in 1.74s
salloc --nodes=1 --partition=debug --time=02:00:00 --account=e3sm
bash
source ~/.bashrc
lcrc_conda
conda activate test-zppy-main-20260828_run1
python -m pip install .
pytest tests/integration/test_images.py
exit # Exit bash shell
exit # Exit compute noteOutput:
The test output reduces correctly to the following, which correctly matches the 9/4 test run minus the
|
Collaborator
Author
|
We probably want to update the expected results for all tasks except failed tasks we explicitly don't have confirmation for yet. That is, we should update the expected results even if the image checker tests pass for a task. For example: ls -lt /lcrc/group/e3sm/public_html/zppy_test_resources/expected_comprehensive_v3currently shows: because |
Collaborator
Author
|
Promoting the 9/28 test run results to be the new expected results baseline. There, we saw that all diffs were to be expected. That means we can do an umbrella/blanket-update of the expected results. Set up branchcd ~/ez/zppy
git status
# On branch issue-869-improve-test-automation
# nothing to commit, working tree clean
git checkout partial-update-expectations
git log --oneline | head -n 4
# 7cfba4b1 Add auto-generated update files
# 1ecb557d Change term from diags to tasks
# 9c6c5185 Support partial updates of weekly/campaign expected test results
# 5232fe30 Rank image check failures by severity (#865)
# Good, has the 3 commits of https://github.com/E3SM-Project/zppy/pull/852/commits
# Titles and hashes match.
lcrc_conda
rm -rf build
conda clean --all --y
conda env create -f conda/dev.yml -n zppy-pr852-20260930
conda activate zppy-pr852-20260930
pre-commit run --all-files
python -m pip install .Copy over the old expected resultsls -lt /lcrc/group/e3sm/public_html/zppy_test_resources
# Updated 9/4, 9/14
# To know when each specific task had its expected results updated, check:
ls -lt /lcrc/group/e3sm/public_html/zppy_test_resources/expected_comprehensive_v3/
# Sep 4 mpas_analysis
# Sep 4 e3sm_diags
# Aug 14 pcmdi_diags
# Aug 14 global_time_series
# May 20 livvkit
# May 20 ilamb
cp -r /lcrc/group/e3sm/public_html/zppy_test_resources /lcrc/group/e3sm/public_html/zppy_test_resources_previous/expected_results_until_20260930 # <30 minutesUpdate the expected results# Let's update the simpler tests' results first:
cd /lcrc/group/e3sm/ac.forsyth2/zppy_main_branch_test_dirs/zppy
git status
# Good, on correct branch: On branch test_zppy_20260928_run6
# Some changed files exist
git add -A
git commit -m "Testing" --no-verify
git log --oneline | head -n 36
# db567d5c Testing
# c7dbac38 Guard apply_conda_install_line's conda install with set +u/-u
# ...
# 0b1dc8b8 Automate image checker, add per-task env descriptions, and auto-generate test report
# 64bffc9f Merge pull request #815 from E3SM-Project/zppy_pcmdi_enhancement
# Recall ~/ez/zppy is on the branch for PR #852
cp ~/ez/zppy/tests/integration/template_update_campaign_expected_files.sh tests/integration/template_update_campaign_expected_files.sh
cp ~/ez/zppy/tests/integration/template_update_weekly_expected_files.sh tests/integration/template_update_weekly_expected_files.sh
git diff # Looks ok
git add -A
git commit -m "Copy changes from PR 852"
# Now, we need to re-render the generated files
emacs tests/integration/utils.py # Confirm ALL cfgs_to_run and tasks_to_run are set to run
python tests/integration/utils.py
git status
# modified: tests/integration/generated/update_campaign_expected_files_chrysalis.sh
# modified: tests/integration/generated/update_weekly_expected_files_chrysalis.sh
# Make sure the update script permissions are set up
chmod 755 tests/integration/generated/update_bash_generation_expected_files_chrysalis.sh
chmod 755 tests/integration/generated/update_campaign_expected_files_chrysalis.sh
chmod 755 tests/integration/generated/update_defaults_expected_files_chrysalis.sh
chmod 755 tests/integration/generated/update_weekly_expected_files_chrysalis.sh
# Double check we have a record of the original expected results
ls -lt /lcrc/group/e3sm/public_html/zppy_test_resources_previous/expected_results_until_20260930
# Update the task list now that it's available:
emacs tests/integration/generated/update_weekly_expected_files_chrysalis.sh
# What we know we can update the expected results for:
# tasks=()
# We can actually leave it empty so that everything gets updated. 9/28 was a fully passing run (after accounting for expected diffs)
git add -A
git commit -m "Update the update scripts"
# All the non-image-checker integration tests passed,
# so the following steps are only necessary to update the modified-date.
# BUT we have to run each command twice,
# because the tests auto-delete output directories on passing runs, so initially there's nothing to transfer.
# These scripts update the expected results and re-run the tests:
./tests/integration/generated/update_bash_generation_expected_files_chrysalis.sh
# 1 passed in 0.61s
./tests/integration/generated/update_campaign_expected_files_chrysalis.sh
# 1 passed in 0.59s
./tests/integration/generated/update_defaults_expected_files_chrysalis.sh
# 1 passed in 0.68s
# This script only updates the expected results
./tests/integration/generated/update_weekly_expected_files_chrysalis.sh # <10 minMissing the legacy cfgs we deleted, of course. ls /lcrc/group/e3sm/public_html/zppy_test_resources
# Confirmed there are expected results subdirs and image lists for each cfg
ls -lt /lcrc/group/e3sm/public_html/zppy_test_resources
# The only thing not updated today 9/30 arte the image lists for the legacy cfgs we deleted.Rerun the tests against the new expected resultscd /lcrc/group/e3sm/ac.forsyth2/zppy_main_branch_test_dirs/zppy
git status
# On branch test_zppy_20260928_run6
# nothing to commit, working tree clean
pytest tests/integration/test_bundles.py
# 2 passed in 4.16s
salloc --nodes=1 --partition=debug --time=02:00:00 --account=e3sm
bash
source ~/.bashrc
lcrc_conda
conda activate test-zppy-issue-869-improve-test-automation-20260928_run6
python -m pip install .
pytest tests/integration/test_images.py
# 1 passed in 397.69s (0:06:37)
exit # Exit bash shell
exit # Exit compute noteGood, everything passes, as expected. |
forsyth2
commented
Oct 6, 2026
| # To update only a subset, comment out the line above and use e.g.: | ||
| # campaigns=("cryosphere") | ||
|
|
||
| for campaign in "${campaigns[@]}" |
Collaborator
Author
There was a problem hiding this comment.
The update here failed because there was no actual to copy over to expected (presumably because the test had passed before and therefore cleaned up its output space). Claude suggests adding a guard like the following
src=test_campaign_${campaign}_output/post/scripts
if ! ls ${src}/*.settings >/dev/null 2>&1; then
echo "No .settings files in ${src}; run the test once first. Skipping ${campaign}."
continue
fi
between echo ${campaign} and rm -rf below.
Collaborator
Author
There was a problem hiding this comment.
The commit titled "Ensure test output exists" (currently a30efc5) addresses this issue here and in similar places.
Collaborator
Author
There was a problem hiding this comment.
This is actually still not the best solution. The issue is that the tests (e.g., tests/integration/test_campaign.py) delete the output (e.g., os.system("rm -r test_campaign_water_cycle_override_output")) if successful, in an attempt to clean up. This rerun of the test is just relying on the test failing (because there's no baseline to compare against) and thus not cleaning up.
forsyth2
force-pushed
the
partial-update-expectations
branch
from
October 10, 2026 00:39
7cfba4b to
a30efc5
Compare
Collaborator
Author
forsyth2
marked this pull request as draft
October 10, 2026 00:46
forsyth2
force-pushed
the
partial-update-expectations
branch
from
October 10, 2026 00:51
a30efc5 to
9c1e077
Compare
26 of 27 tasks
Collaborator
Author
|
Promoting the 10/09 Unified rc3 Chrysalis test run results to be the new expected results baseline. There, we saw that all diffs were to be expected. That means we can do an umbrella/blanket-update of the expected results. Set up branchcd ~/ez/zppy
git status
# nothing to commit, working tree clean
git checkout partial-update-expectations
git log --oneline | head -n 5 # Based on top of #886, which is based on top of #871
# 9c1e0775 Ensure test output exists
# d9b32126 Add auto-generated update files
# 8d0441f8 Change term from diags to tasks
# 0de1c9c8 Support partial updates of weekly/campaign expected test results
# 14821d6a Replace legacy 3.1.0/3.0.0 with 3.2.0
# Good, has the 4 commits of https://github.com/E3SM-Project/zppy/pull/852/commits
# Titles and hashes match.
lcrc_conda
rm -rf build
conda clean --all --y
conda env create -f conda/dev.yml -n zppy-pr852-20261009
conda activate zppy-pr852-20261009
pre-commit run --all-files
python -m pip install .Copy over the old expected resultsls -lt /lcrc/group/e3sm/public_html/zppy_test_resources
# Updated 9/30, 9/14
# To know when each specific task had its expected results updated, check:
ls -lt /lcrc/group/e3sm/public_html/zppy_test_resources/expected_comprehensive_v3/
# All updated 9/30
cp -r /lcrc/group/e3sm/public_html/zppy_test_resources /lcrc/group/e3sm/public_html/zppy_test_resources_previous/expected_results_until_20261009 # <30 minutesUpdate the expected results# Let's update the simpler tests' results first:
cd /lcrc/group/e3sm/ac.forsyth2/zppy_main_branch_test_dirs/zppy
git status
# Good, on correct branch: On branch test_zppy_20261009_run1
# Some changed files exist
git add -A
git commit -m "Testing" --no-verify
git log --oneline | head -n 3
# 99411ab7 Testing
# 8bfc1cbd Add UNIFIED_LOAD_SCRIPT option to test a specific E3SM-Unified
# 79eb98c3 Allow testing zppy from E3SM-Unified in automated test script
# Recall ~/ez/zppy is on the branch for PR #852
cp ~/ez/zppy/tests/integration/template_update_campaign_expected_files.sh tests/integration/template_update_campaign_expected_files.sh
cp ~/ez/zppy/tests/integration/template_update_weekly_expected_files.sh tests/integration/template_update_weekly_expected_files.sh
cp ~/ez/zppy/tests/integration/template_update_bash_generation_expected_files.sh tests/integration/template_update_bash_generation_expected_files.sh
cp ~/ez/zppy/tests/integration/template_update_defaults_expected_files.sh tests/integration/template_update_defaults_expected_files.sh
git diff # Looks ok
git add -A
git commit -m "Copy changes from PR 852"
# Now, we need to re-render the generated files
emacs tests/integration/utils.py # Confirm ALL cfgs_to_run and tasks_to_run are set to run
# Remove these since we aren't testing them anymore
# "weekly_legacy_3.1.0_comprehensive_v3",
# "weekly_legacy_3.0.0_comprehensive_v3",
# Don't worry about adding the 3.2.0 cfg in since this test run never ran that cfg.
python tests/integration/utils.py
git status
# modified: tests/integration/generated/update_bash_generation_expected_files_chrysalis.sh
# modified: tests/integration/generated/update_campaign_expected_files_chrysalis.sh
# modified: tests/integration/generated/update_defaults_expected_files_chrysalis.sh
# modified: tests/integration/generated/update_weekly_expected_files_chrysalis.sh
# modified: tests/integration/utils.py
# Make sure the update script permissions are set up
chmod 755 tests/integration/generated/update_bash_generation_expected_files_chrysalis.sh
chmod 755 tests/integration/generated/update_campaign_expected_files_chrysalis.sh
chmod 755 tests/integration/generated/update_defaults_expected_files_chrysalis.sh
chmod 755 tests/integration/generated/update_weekly_expected_files_chrysalis.sh
# Double check we have a record of the original expected results
ls -lt /lcrc/group/e3sm/public_html/zppy_test_resources_previous/expected_results_until_20261009
# Update the task list now that it's available (via PR 852):
emacs tests/integration/generated/update_weekly_expected_files_chrysalis.sh
# What we know we can update the expected results for:
# tasks=()
# We can actually leave it empty so that everything gets updated. 10/09 was a fully passing run (after accounting for expected diffs)
git add -A
git commit -m "Update the update scripts"
# This script only updates the expected results:
./tests/integration/generated/update_weekly_expected_files_chrysalis.sh # <10 min 18:50Missing the legacy 3.2.0 cfgs since we didn't run that test. # We need to get into the Unified RC
source /lcrc/soft/climate/e3sm-unified/test_e3sm_unified_1.14.0rc3_chrysalis.sh
# These scripts update the expected results and re-run the tests:
./tests/integration/generated/update_bash_generation_expected_files_chrysalis.sh
# 1 passed in 4.07s
./tests/integration/generated/update_campaign_expected_files_chrysalis.sh
# 6 passed in 2.27s
./tests/integration/generated/update_defaults_expected_files_chrysalis.sh
# 1 passed in 0.58s
ls /lcrc/group/e3sm/public_html/zppy_test_resources
# Confirmed there are expected results subdirs and image lists for each cfg
ls -lt /lcrc/group/e3sm/public_html/zppy_test_resources
# Updates today 10/09 (also 9/30, 9/14, but only for obsolete cfgs)Rerun the tests against the new expected resultscd /lcrc/group/e3sm/ac.forsyth2/zppy_main_branch_test_dirs/zppy
git status
# On branch test_zppy_20261009_run1
# nothing to commit, working tree clean
pytest tests/integration/test_bundles.py
# 2 passed in 0.85s
salloc --nodes=1 --partition=debug --time=02:00:00 --account=e3sm
bash
source ~/.bashrc
# We need to get into the Unified RC
source /lcrc/soft/climate/e3sm-unified/test_e3sm_unified_1.14.0rc3_chrysalis.sh
pytest tests/integration/test_images.py
# 1 passed in 353.99s (0:05:53)
exit # Exit bash shell
exit # Exit compute noteGood, everything passes, as expected. |
forsyth2
force-pushed
the
partial-update-expectations
branch
from
October 14, 2026 17:52
9c1e077 to
3b24eb4
Compare
3 tasks done
forsyth2
force-pushed
the
partial-update-expectations
branch
from
October 16, 2026 18:19
3b24eb4 to
48ffaef
Compare
forsyth2
force-pushed
the
partial-update-expectations
branch
from
October 16, 2026 23:02
48ffaef to
a71f07d
Compare
Collaborator
Author
|
Promoting the 2026-10-16 Pre-E3SM Unified 1.14.0rc4 Chrysalis test results to be the new expected results baseline. There, we saw that all diffs were to be expected. That means we can do an umbrella/blanket-update of the expected results. Set up branchcd ~/ez/zppy
git status
# nothing to commit, working tree clean
git checkout partial-update-expectations
git log --oneline | head -n 5
# a71f07df Ensure test output exists
# b1bbb531 Add auto-generated update files
# f5f4b267 Change term from diags to tasks
# 92fe6799 Support partial updates of weekly/campaign expected test results
# 758bcbe5 Handle missing image baselines without crashing image tests
# Good, has the 4 commits of https://github.com/E3SM-Project/zppy/pull/852/commits
# Titles and hashes match.
# Based on top of #886, #871
lcrc_conda
conda activate zppy-pr871-improve-test
pre-commit run --all-files
python -m pip install .Copy over the old expected resultsls -lt /lcrc/group/e3sm/public_html/zppy_test_resources
# Updated 10/9 (also 9/30, 9/14 for obsolete tests)
# To know when each specific task had its expected results updated, check:
ls -lt /lcrc/group/e3sm/public_html/zppy_test_resources/expected_comprehensive_v3/
# All updated 10/9
cp -r /lcrc/group/e3sm/public_html/zppy_test_resources /lcrc/group/e3sm/public_html/zppy_test_resources_previous/expected_results_until_20261019 # <30 minutesUpdate the expected results# Let's update the simpler tests' results first:
cd /lcrc/group/e3sm/ac.forsyth2/zppy_main_branch_test_dirs/zppy
git status
# Good, on correct branch: On branch test_zppy_20261016_run1
# Some changed files exist
git add -A
git commit -m "Testing" --no-verify
git log --oneline | head -n 7
# 36148e35 Testing
# dc8ab27f Testing
# a71f07df Ensure test output exists
# b1bbb531 Add auto-generated update files
# f5f4b267 Change term from diags to tasks
# 92fe6799 Support partial updates of weekly/campaign expected test results
# 758bcbe5 Handle missing image baselines without crashing image tests
# This branch is already based off #852, #862, #871
# So, it has the updater scripts.
cat tests/integration/utils.py
# "cfgs_to_run": [
# "weekly_bundles",
# "weekly_comprehensive_v2",
# "weekly_comprehensive_v3",
# "weekly_legacy_3.2.0_comprehensive_v3",
# ],
python tests/integration/utils.py
git status
# As expected, nothing needed to be updated.
# Make sure the update script permissions are set up
chmod 750 tests/integration/generated/update_bash_generation_expected_files_chrysalis.sh
chmod 750 tests/integration/generated/update_campaign_expected_files_chrysalis.sh
chmod 750 tests/integration/generated/update_defaults_expected_files_chrysalis.sh
chmod 750 tests/integration/generated/update_weekly_expected_files_chrysalis.sh
# Double check we have a record of the original expected results
ls -lt /lcrc/group/e3sm/public_html/zppy_test_resources_previous/expected_results_until_20261019
# Update the task list now that it's available (via PR 852):
emacs tests/integration/generated/update_weekly_expected_files_chrysalis.sh
# What we know we can update the expected results for:
# tasks=()
# We can actually leave it empty so that everything gets updated. 10/19 was a fully passing run (after accounting for expected diffs)
# This script only updates the expected results:
./tests/integration/generated/update_weekly_expected_files_chrysalis.sh
# We need to install the conda env used during the test now
conda activate test-zppy-partial-update-expectations-20261016_run1
# These scripts update the expected results and re-run the tests:
./tests/integration/generated/update_bash_generation_expected_files_chrysalis.sh
# 1 passed in 4.04s
./tests/integration/generated/update_campaign_expected_files_chrysalis.sh
# 6 passed in 3.25s
./tests/integration/generated/update_defaults_expected_files_chrysalis.sh
# 1 passed in 1.41s
ls /lcrc/group/e3sm/public_html/zppy_test_resources
# Confirmed there are expected results subdirs and image lists for each cfg
ls -lt /lcrc/group/e3sm/public_html/zppy_test_resources
# Updates today 10/19 (also 9/30, 9/14, but only for obsolete cfgs)Rerun the tests against the new expected resultscd /lcrc/group/e3sm/ac.forsyth2/zppy_main_branch_test_dirs/zppy
git status
# On branch test_zppy_20261016_run1
# nothing to commit, working tree clean
pytest tests/integration/test_bundles.py
# 2 passed in 1.50s
salloc --nodes=1 --partition=debug --time=02:00:00 --account=e3sm
bash
source ~/.bashrc
# We need to get into the Unified RC
lcrc_conda
conda activate test-zppy-partial-update-expectations-20261016_run1
pytest tests/integration/test_images.py
# 1 passed in 508.07s (0:08:28)
exit # Exit bash shell
exit # Exit compute noteGood, everything passes, as expected. |
forsyth2
force-pushed
the
partial-update-expectations
branch
from
October 20, 2026 16:07
a71f07d to
a48dd33
Compare
Collaborator
Author
|
I noticed: |
forsyth2
force-pushed
the
partial-update-expectations
branch
from
October 30, 2026 17:32
a48dd33 to
5c768fd
Compare
Collaborator
Author
|
The latest commit, 3fa4066, addresses the comment above. I had noticed a couple issues: Issue 1: `image_check_failures` is showing upcd /lcrc/group/e3sm/public_html/zppy_test_resources
ls expected_comprehensive_v3/
# e3sm_diags mpas_analysis
# global_time_series pcmdi_diags
# ilamb provenance.20261016_191005_414768.cfg
# image_check_failures_comprehensive_v3_try2 provenance.20261016_191005_414768.settings
# livvkit
grep image_check_failures image_list_expected_* | wc -l
# 13,716 occurrencesIssue 2: legacy 3.0.0 and 3.1.0 are still showing up in expected results even though they've been replaced by legacy 3.2.0cd /lcrc/group/e3sm/public_html/zppy_test_resources
ls
# expected_bash_files
# expected_bundles
# expected_comprehensive_v2
# expected_comprehensive_v3
# expected_legacy_3.0.0_comprehensive_v3
# expected_legacy_3.1.0_comprehensive_v3
# expected_legacy_3.2.0_comprehensive_v3
# image_list_expected_bundles.txt
# image_list_expected_comprehensive_v2.txt
# image_list_expected_comprehensive_v3.txt
# image_list_expected_legacy_3.0.0_bundles.txt
# image_list_expected_legacy_3.0.0_comprehensive_v2.txt
# image_list_expected_legacy_3.0.0_comprehensive_v3.txt
# image_list_expected_legacy_3.1.0_bundles.txt
# image_list_expected_legacy_3.1.0_comprehensive_v2.txt
# image_list_expected_legacy_3.1.0_comprehensive_v3.txt
# image_list_expected_legacy_3.2.0_comprehensive_v3.txt
# test_campaign_cryosphere_expected_files
# test_campaign_cryosphere_override_expected_files
# test_campaign_high_res_v1_expected_files
# test_campaign_none_expected_files
# test_campaign_water_cycle_expected_files
# test_campaign_water_cycle_override_expected_files
# test_defaults_expected_files
ls *legacy_3.0.0*
# image_list_expected_legacy_3.0.0_bundles.txt
# image_list_expected_legacy_3.0.0_comprehensive_v2.txt
# image_list_expected_legacy_3.0.0_comprehensive_v3.txt
# expected_legacy_3.0.0_comprehensive_v3:
# e3sm_diags ilamb mpas_analysis provenance.20260929_003221_561557.cfg
# global_time_series livvkit pcmdi_diags provenance.20260929_003221_561557.settings
ls *legacy_3.1.0*
# image_list_expected_legacy_3.1.0_bundles.txt
# image_list_expected_legacy_3.1.0_comprehensive_v2.txt
# image_list_expected_legacy_3.1.0_comprehensive_v3.txt
# expected_legacy_3.1.0_comprehensive_v3:
# e3sm_diags ilamb mpas_analysis provenance.20260929_003209_423067.cfg
# global_time_series livvkit pcmdi_diags provenance.20260929_003209_423067.settingsClaude did agree that since |
Previously, update_weekly_expected_files_*.sh and
update_campaign_expected_files.sh always overwrote every diag
subdirectory (e.g. e3sm_diags, mpas_analysis, ilamb, ...) or every
campaign in one pass, making it impossible to accept new results for
one diagnostic while leaving another pinned to its old expected output.
- template_update_weekly_expected_files.sh: add a `diags` selector
array. Empty (default) preserves the old full-wipe-and-copy
behavior. Populated (e.g. diags=("e3sm_diags")), only those
subdirectories are refreshed; "bundle_files" is selectable the
same way for the bundle*.bash copy step. The image list is still
generated by walking the full directory, so it reflects both
updated and untouched diags.
- template_update_campaign_expected_files.sh: hardcoded campaign
list replaced with a `campaigns` array for the same kind of subset
updates.
- template_update_bash_generation_expected_files.sh and
template_update_defaults_expected_files.sh: unchanged -- each
produces a single expected-files dir with nothing to partially select.
- docs/source/dev_guide/tests/update_expected_results.rst: document
the new selectors and note the full backup step should still
always run first.
This diff was generated by Claude (Anthropic).
Add the `update_` files that `python tests/integration/utils.py` creates.
…lures - Prune expected_legacy_* dirs and image lists for legacy versions other than the current one, so replaced versions (e.g. 3.0.0, 3.1.0) no longer linger after a bump. Driven by a new `legacy_version` variable. - Remove image_check_failures_* by glob rather than exact name. The image checker appends _tryN to the directory name on reruns, so the exact-name rm missed it and the directory was copied into the expected files. - Exclude image_check_failures_* from the generated image lists. Generated with Claude.
forsyth2
force-pushed
the
partial-update-expectations
branch
from
October 30, 2026 23:43
3fa4066 to
b789c78
Compare
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Objectives:
Select one: This pull request is...
Small Change