Skip to content

Enable partial updates of expected results for testing #852

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
forsyth2 wants to merge 5 commits into from
Draft

Enable partial updates of expected results for testing #852

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
6a5e8b4
Support partial updates of weekly/campaign expected test results
forsyth2 Aug 13, 2026
b77d1af
Change term from diags to tasks
forsyth2 Aug 13, 2026
3c6ceea
Add auto-generated update files
forsyth2 Aug 13, 2026
a9dad16
Ensure test output exists
forsyth2 Oct 10, 2026
d282bfa
Fix weekly updater: prune stale legacy files, exclude image_check_fai…
forsyth2 Oct 30, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
35 changes: 35 additions & 0 deletions docs/source/dev_guide/tests/update_expected_results.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -114,3 +114,38 @@ Only do this part if you're updating the Official Results (i.e., the results we'
# Compy -- takes about 1h20m
ls ${expected_results_records_dir}/expected_results_for_${this_release}
# Confirm there are expected results subdirs and image lists for each cfg

Partial updates
~~~~~~~~~~~~~~~

Sometimes you only want to refresh results for one task (e.g. ``e3sm_diags``)
without touching others (e.g. ``mpas_analysis``) that aren't ready yet. Both the
weekly and campaign update scripts support this via a selector array near the top
of the generated script:

.. code-block:: bash

# In update_weekly_expected_files_${machine_name}.sh:
# Leave empty to update every task subdirectory.
tasks=("e3sm_diags")

# In update_campaign_expected_files_${machine_name}.sh:
# Restrict which campaigns get refreshed.
campaigns=("cryosphere")

Notes:

- ``tasks`` applies uniformly to every entry in ``tests`` -- e.g. if ``tests``
includes both ``comprehensive_v2`` and ``comprehensive_v3``, ``e3sm_diags`` is
refreshed in both.
- ``update_bash_generation_expected_files_${machine_name}.sh`` and
``update_defaults_expected_files_${machine_name}.sh`` each only produce a single
expected-files directory, so there's nothing to partially select within them --
simply skip running that script if you don't want to touch those results yet.
- Regardless of whether you're doing a full or partial update, **always** run the
full backup step first (``cp -r ${expected_results_dir} ${expected_results_records_dir}/expected_results_until_yyyymmdd``).
This keeps a complete snapshot to diff or revert against, even though the update
itself may only touch part of the tree.
- After a partial update, ``image_list_expected_${test_name}.txt`` is still
regenerated by walking the whole ``expected_${test_name}`` directory, so it
correctly reflects a mix of freshly-updated and previously-existing diag results.
8 changes: 7 additions & 1 deletion tests/integration/generated/update_campaign_expected_files_chrysalis.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,13 @@
# Run from the top level of the zppy repo
# Run as `./tests/integration/generated/update_campaign_expected_files.sh`

for campaign in "cryosphere" "cryosphere_override" "high_res_v1" "none" "water_cycle" "water_cycle_override"
# Update all campaigns
campaigns=("cryosphere" "cryosphere_override" "high_res_v1" "none" "water_cycle" "water_cycle_override")

# To update only a subset, comment out the line above and use e.g.:
# campaigns=("cryosphere")

for campaign in "${campaigns[@]}"

@forsyth2 forsyth2 Oct 6, 2026

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The update here failed because there was no actual to copy over to expected (presumably because the test had passed before and therefore cleaned up its output space). Claude suggests adding a guard like the following

src=test_campaign_${campaign}_output/post/scripts
if ! ls ${src}/*.settings >/dev/null 2>&1; then
    echo "No .settings files in ${src}; run the test once first. Skipping ${campaign}."
    continue
fi

between echo ${campaign} and rm -rf below.

@forsyth2 forsyth2 Oct 10, 2026

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The commit titled "Ensure test output exists" (currently a30efc5) addresses this issue here and in similar places.

@forsyth2 forsyth2 Oct 10, 2026

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is actually still not the best solution. The issue is that the tests (e.g., tests/integration/test_campaign.py) delete the output (e.g., os.system("rm -r test_campaign_water_cycle_override_output")) if successful, in an attempt to clean up. This rerun of the test is just relying on the test failing (because there's no baseline to compare against) and thus not cleaning up.

do
echo ${campaign}
rm -rf /lcrc/group/e3sm/public_html/zppy_test_resources/test_campaign_${campaign}_expected_files
Expand Down
60 changes: 50 additions & 10 deletions tests/integration/generated/update_weekly_expected_files_chrysalis.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,6 +17,19 @@ tests=("comprehensive_v2" "comprehensive_v3" "bundles" "legacy_3.1.0_comprehensi
# Update legacy 3.0.0 only
#tests=("legacy_3.0.0_comprehensive_v2" "legacy_3.0.0_comprehensive_v3" "legacy_3.0.0_bundles")

# ------------------------------------------------------------------------------
# Restrict updates to specific diagnostic subdirectories, e.g. e3sm_diags,
# mpas_analysis, global_time_series, ilamb, livvkit, pcmdi_diags.
# Leave empty (tasks=()) to update every task subdirectory (old/full behavior).
# Example -- only refresh e3sm_diags:
# tasks=("e3sm_diags")
# This applies uniformly across every entry in `tests` above.
# "bundle_files" (the bundle*.bash files, only relevant when "bundles" is in
# `tests`) is treated as its own selectable name -- include it in `tasks` if you
# want it refreshed during a partial update; it's always refreshed on a full one.
tasks=()
# ------------------------------------------------------------------------------

for test_name in "${tests[@]}"
do
# Example for Chrysalis:
Expand All @@ -31,32 +44,59 @@ do
#
# Each of those subdirectories has a corresponding image list of the form:
# `image_list_<subdir_name>.txt`
#
# Each of those subdirectories in turn contains task subdirectories,
# e.g.: e3sm_diags, global_time_series, ilamb, livvkit, mpas_analysis, pcmdi_diags

# Remove old expected files.
rm -rf /lcrc/group/e3sm/public_html/zppy_test_resources/expected_${test_name}
expected_dir=/lcrc/group/e3sm/public_html/zppy_test_resources/expected_${test_name}

# Your output will now become the new expectation.
# Copy output so you don't have to rerun zppy to generate the output.
if [[ "${test_name,,}" =~ "v2" ]]; then
# We need the v2 case name
cp -r /lcrc/group/e3sm/public_html/diagnostic_output/ac.forsyth2/zppy_weekly_${test_name}_www/unique_id/v2.LR.historical_0201 /lcrc/group/e3sm/public_html/zppy_test_resources/expected_${test_name}
output_case_dir=/lcrc/group/e3sm/public_html/diagnostic_output/ac.forsyth2/zppy_weekly_${test_name}_www/unique_id/v2.LR.historical_0201
else
# We need the v3 case name
cp -r /lcrc/group/e3sm/public_html/diagnostic_output/ac.forsyth2/zppy_weekly_${test_name}_www/unique_id/v3.LR.historical_0051 /lcrc/group/e3sm/public_html/zppy_test_resources/expected_${test_name}
output_case_dir=/lcrc/group/e3sm/public_html/diagnostic_output/ac.forsyth2/zppy_weekly_${test_name}_www/unique_id/v3.LR.historical_0051
fi

if [[ ${#tasks[@]} -eq 0 ]]; then
# Full update: remove old expected files, copy the entire output over.
rm -rf ${expected_dir}
# Your output will now become the new expectation.
# Copy output so you don't have to rerun zppy to generate the output.
cp -r ${output_case_dir} ${expected_dir}
else
# Partial update: only refresh the named task subdirectories, leaving the
# rest of expected_${test_name} untouched.
mkdir -p ${expected_dir}
for task_name in "${tasks[@]}"
do
if [[ "${task_name}" == "bundle_files" ]]; then
continue # handled below, alongside the "bundles" test_name check
fi
if [[ -d ${output_case_dir}/${task_name} ]]; then
rm -rf ${expected_dir}/${task_name}
cp -r ${output_case_dir}/${task_name} ${expected_dir}/${task_name}
else
echo "WARNING: ${output_case_dir}/${task_name} does not exist -- skipping (test_name=${test_name})."
fi
done
fi

# test_bundles.py also needs the bash files transferred.
# Note that for legacy cfgs, we're only testing test_images.py
if [[ "${test_name,,}" == "bundles" ]]; then
mkdir -p /lcrc/group/e3sm/public_html/zppy_test_resources/expected_bundles/bundle_files
cp -r /lcrc/group/e3sm/ac.forsyth2/zppy_weekly_bundles_output/unique_id/v3.LR.historical_0051/post/scripts/bundle*.bash /lcrc/group/e3sm/public_html/zppy_test_resources/expected_bundles/bundle_files
if [[ ${#tasks[@]} -eq 0 ]] || [[ " ${tasks[*]} " =~ " bundle_files " ]]; then
mkdir -p ${expected_dir}/bundle_files
cp -r /lcrc/group/e3sm/ac.forsyth2/zppy_weekly_bundles_output/unique_id/v3.LR.historical_0051/post/scripts/bundle*.bash ${expected_dir}/bundle_files
fi
fi

zppy_top_level=$(pwd)
cd /lcrc/group/e3sm/public_html/zppy_test_resources/expected_${test_name}
cd ${expected_dir}
# Remove the image check failures, so they don't end up in the expected files.
rm -rf image_check_failures_${test_name}
# This file will list all the expected images.
# This file will list all the expected images -- a mix of freshly-updated
# and previously-existing task subdirectories, if `tasks` restricted scope.
find . -type f -name '*.png' > ../image_list_expected_${test_name}.txt
cd ${zppy_top_level}
done
Expand Down
20 changes: 18 additions & 2 deletions tests/integration/template_update_bash_generation_expected_files.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,10 +2,26 @@
# Run from the top level of the zppy repo
# Run as `./tests/integration/generated/update_bash_generation_expected_files.sh`

src=test_bash_generation_output/post/scripts

# If the output is missing, run the test to generate it.
# (The test may fail against the old expected files; that's fine,
# we only need the output to be produced.)
if [ ! -d "${src}" ]; then
echo "No output in ${src}; running the test to generate it..."
pytest tests/integration/test_bash_generation.py || true
fi

# Re-check after (possibly) running the test
if [ ! -d "${src}" ]; then
echo "Still no output in ${src} after running the test. Exiting without changes."
exit 1
fi

rm -rf #expand expected_dir#expected_bash_files
# Your output will now become the new expectation.
# You can just move (i.e., not copy) the output since re-running this test will re-generate the output.
rm -rf test_bash_generation_output/post/scripts/provenance*
mv test_bash_generation_output/post/scripts #expand expected_dir#expected_bash_files
rm -rf ${src}/provenance*
mv ${src} #expand expected_dir#expected_bash_files
# Rerun test
pytest tests/integration/test_bash_generation.py
28 changes: 27 additions & 1 deletion tests/integration/template_update_campaign_expected_files.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,9 +2,35 @@
# Run from the top level of the zppy repo
# Run as `./tests/integration/generated/update_campaign_expected_files.sh`

for campaign in "cryosphere" "cryosphere_override" "high_res_v1" "none" "water_cycle" "water_cycle_override"
# Update all campaigns
campaigns=("cryosphere" "cryosphere_override" "high_res_v1" "none" "water_cycle" "water_cycle_override")

# To update only a subset, comment out the line above and use e.g.:
# campaigns=("cryosphere")

# Track whether we've already run the test to generate missing output
test_has_run=false

for campaign in "${campaigns[@]}"
do
echo ${campaign}
src=test_campaign_${campaign}_output/post/scripts

# If the output is missing, run the test once to generate it.
# (The test may fail against the old expected files; that's fine,
# we only need the output to be produced.)
if ! ls ${src}/*.settings >/dev/null 2>&1 && [ "${test_has_run}" = false ]; then
echo "No .settings files in ${src}; running the test to generate output..."
pytest tests/integration/test_campaign.py || true
test_has_run=true
fi

# Re-check after (possibly) running the test
if ! ls ${src}/*.settings >/dev/null 2>&1; then
echo "Still no .settings files in ${src} after running the test. Skipping ${campaign}."
continue
fi

rm -rf #expand expected_dir#test_campaign_${campaign}_expected_files
mkdir -p #expand expected_dir#test_campaign_${campaign}_expected_files
# Your output will now become the new expectation.
Expand Down
18 changes: 17 additions & 1 deletion tests/integration/template_update_defaults_expected_files.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,10 +2,26 @@
# Run from the top level of the zppy repo
# Run as `./tests/integration/generated/update_defaults_expected_files.sh`

src=test_defaults_output/post/scripts

# If the output is missing, run the test to generate it.
# (The test may fail against the old expected files; that's fine,
# we only need the output to be produced.)
if ! ls ${src}/*.settings >/dev/null 2>&1; then
echo "No .settings files in ${src}; running the test to generate them..."
pytest tests/integration/test_defaults.py || true
fi

# Re-check after (possibly) running the test
if ! ls ${src}/*.settings >/dev/null 2>&1; then
echo "Still no .settings files in ${src} after running the test. Exiting without changes."
exit 1
fi

rm -rf #expand expected_dir#test_defaults_expected_files
mkdir -p #expand expected_dir#test_defaults_expected_files
# Your output will now become the new expectation.
# You can just move (i.e., not copy) the output since re-running this test will re-generate the output.
mv test_defaults_output/post/scripts/*.settings #expand expected_dir#test_defaults_expected_files
mv ${src}/*.settings #expand expected_dir#test_defaults_expected_files
# Rerun test
pytest tests/integration/test_defaults.py
98 changes: 83 additions & 15 deletions tests/integration/template_update_weekly_expected_files.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,14 +5,53 @@

# NOTE: in `tests` below, do *not* include the `zppy_weekly_` prefix, as that is added later.

# Current legacy zppy version. Bump this when a new legacy version replaces the old one;
# expected files for every other legacy version are pruned below.
legacy_version="3.2.0"

# Update all
tests=("comprehensive_v2" "comprehensive_v3" "bundles" "legacy_3.2.0_comprehensive_v3")
tests=("comprehensive_v2" "comprehensive_v3" "bundles" "legacy_${legacy_version}_comprehensive_v3")

# Update regular only
#tests=("comprehensive_v2" "comprehensive_v3" "bundles")

# Update legacy 3.2.0 only
#tests=("legacy_3.2.0_comprehensive_v3")
# Update legacy only
#tests=("legacy_${legacy_version}_comprehensive_v3")

# ------------------------------------------------------------------------------
# Restrict updates to specific diagnostic subdirectories, e.g. e3sm_diags,
# mpas_analysis, global_time_series, ilamb, livvkit, pcmdi_diags.
# Leave empty (tasks=()) to update every task subdirectory (old/full behavior).
# Example -- only refresh e3sm_diags:
# tasks=("e3sm_diags")
# This applies uniformly across every entry in `tests` above.
# "bundle_files" (the bundle*.bash files, only relevant when "bundles" is in
# `tests`) is treated as its own selectable name -- include it in `tasks` if you
# want it refreshed during a partial update; it's always refreshed on a full one.
tasks=()
# ------------------------------------------------------------------------------

# Prune expected files for legacy versions other than ${legacy_version}.
# Only the current legacy version is tested, and older ones are never overwritten,
# so without this they linger. Safe in every mode (full or partial),
# since it never touches the current version.
resources_dir=#expand expected_dir#
if [[ -d "${resources_dir}" ]]; then
# Make globs that match nothing expand to nothing (instead of the literal pattern),
# so the loop below is skipped when there are no legacy files to prune.
shopt -s nullglob
for possibly_stale in ${resources_dir}expected_legacy_* ${resources_dir}image_list_expected_legacy_*
do
case "$(basename ${possibly_stale})" in
*legacy_${legacy_version}_*) ;;
*)
echo "Removing stale legacy expected file(s): ${possibly_stale}"
rm -rf ${possibly_stale}
;;
esac
done
shopt -u nullglob
fi

for test_name in "${tests[@]}"
do
Expand All @@ -27,33 +66,62 @@ do
#
# Each of those subdirectories has a corresponding image list of the form:
# `image_list_<subdir_name>.txt`
#
# Each of those subdirectories in turn contains task subdirectories,
# e.g.: e3sm_diags, global_time_series, ilamb, livvkit, mpas_analysis, pcmdi_diags

# Remove old expected files.
rm -rf #expand expected_dir#expected_${test_name}
expected_dir=#expand expected_dir#expected_${test_name}

# Your output will now become the new expectation.
# Copy output so you don't have to rerun zppy to generate the output.
if [[ "${test_name,,}" =~ "v2" ]]; then
# We need the v2 case name
cp -r #expand user_www#zppy_weekly_${test_name}_www/#expand unique_id#/#expand case_name_v2# #expand expected_dir#expected_${test_name}
output_case_dir=#expand user_www#zppy_weekly_${test_name}_www/#expand unique_id#/#expand case_name_v2#
else
# We need the v3 case name
cp -r #expand user_www#zppy_weekly_${test_name}_www/#expand unique_id#/#expand case_name# #expand expected_dir#expected_${test_name}
output_case_dir=#expand user_www#zppy_weekly_${test_name}_www/#expand unique_id#/#expand case_name#
fi

if [[ ${#tasks[@]} -eq 0 ]]; then
# Full update: remove old expected files, copy the entire output over.
rm -rf ${expected_dir}
# Your output will now become the new expectation.
# Copy output so you don't have to rerun zppy to generate the output.
cp -r ${output_case_dir} ${expected_dir}
else
# Partial update: only refresh the named task subdirectories, leaving the
# rest of expected_${test_name} untouched.
mkdir -p ${expected_dir}
for task_name in "${tasks[@]}"
do
if [[ "${task_name}" == "bundle_files" ]]; then
continue # handled below, alongside the "bundles" test_name check
fi
if [[ -d ${output_case_dir}/${task_name} ]]; then
rm -rf ${expected_dir}/${task_name}
cp -r ${output_case_dir}/${task_name} ${expected_dir}/${task_name}
else
echo "WARNING: ${output_case_dir}/${task_name} does not exist -- skipping (test_name=${test_name})."
fi
done
fi

# test_bundles.py also needs the bash files transferred.
# Note that for legacy cfgs, we're only testing test_images.py
if [[ "${test_name,,}" == "bundles" ]]; then
mkdir -p #expand expected_dir#expected_bundles/bundle_files
cp -r #expand user_output#zppy_weekly_bundles_output/#expand unique_id#/#expand case_name#/post/scripts/bundle*.bash #expand expected_dir#expected_bundles/bundle_files
if [[ ${#tasks[@]} -eq 0 ]] || [[ " ${tasks[*]} " =~ " bundle_files " ]]; then
mkdir -p ${expected_dir}/bundle_files
cp -r #expand user_output#zppy_weekly_bundles_output/#expand unique_id#/#expand case_name#/post/scripts/bundle*.bash ${expected_dir}/bundle_files
fi
fi

zppy_top_level=$(pwd)
cd #expand expected_dir#expected_${test_name}
cd ${expected_dir}
# Remove the image check failures, so they don't end up in the expected files.
rm -rf image_check_failures_${test_name}
# This file will list all the expected images.
find . -type f -name '*.png' > ../image_list_expected_${test_name}.txt
# The directory name can carry a suffix after the test name (e.g.
# image_check_failures_comprehensive_v3_try2), so match any suffix.
rm -rf image_check_failures_*
# This file will list all the expected images -- a mix of freshly-updated
# and previously-existing task subdirectories, if `tasks` restricted scope.
find . -type f -name '*.png' -not -path './image_check_failures_*' > ../image_list_expected_${test_name}.txt
cd ${zppy_top_level}
done

Expand Down
Loading