Skip to content

Use v3.2.0 cfg for legacy testing - #886

Draft
forsyth2 wants to merge 7 commits into
issue-869-improve-test-automationfrom
issue-837-legacy-3.2.0
Draft

forsyth2 wants to merge 7 commits into
issue-869-improve-test-automationfrom
issue-837-legacy-3.2.0

Conversation

@forsyth2

@forsyth2 forsyth2 commented Oct 9, 2026

Copy link
Copy Markdown
Collaborator

Summary

Objectives:

  • Add legacy 3.2.0 test cfg
  • Remove the legacy 3.1.0 and 3.0.0 tests and replace them with 3.2.0

Note: I've set this up to merge to #871, since it is based on top of its testing improvements. The changes of this PR will either enter main through that test improvement PR or through the large-scale refactor of #876/#883.

Issue resolution:

Select one: This pull request is...

  • a bug fix: increment the patch version
  • a small improvement: increment the minor version
  • a new feature: increment the minor version
  • an incompatible (non-backwards compatible) API change: increment the major version

Small Change

  • To merge, I will use "Squash and merge". That is, this change should be a single commit.
  • Logic: I have visually inspected the entire pull request myself.
  • Pre-commit checks: All the pre-commits checks have passed.

@forsyth2 forsyth2 self-assigned this Oct 9, 2026
@forsyth2 forsyth2 added the Testing Files in `tests` modified label Oct 9, 2026
@forsyth2
forsyth2 force-pushed the issue-837-legacy-3.2.0 branch from 6b8f866 to f926c1e Compare October 9, 2026 23:37
@forsyth2 forsyth2 changed the title Issue 837 legacy 3.2.0 Use v3.2.0 cfg for legacy testing Oct 9, 2026
@forsyth2
forsyth2 added this pull request to stack #887 October 10, 2026 00:46
@forsyth2
forsyth2 force-pushed the issue-837-legacy-3.2.0 branch 2 times, most recently from 14821d6 to 4c1a7b6 Compare October 14, 2026 17:28
@forsyth2

Copy link
Copy Markdown
Collaborator Author

@copilot d998c56 fixed a bug where livvkit wasn't being tested. Can you please make sure it's being tested on the v3.2.0 cfg too? Please implement the fix directly on this PR's branch.

Copilot AI commented Oct 14, 2026

Copy link
Copy Markdown
Contributor

@copilot d998c56 fixed a bug where livvkit wasn't being tested. Can you please make sure it's being tested on the...

Added livvkit to the legacy 3.2.0 image-check task list and extended the regression test. Commit: 7db028d.

@forsyth2

Copy link
Copy Markdown
Collaborator Author

Dropping Copilot commit because of changes made on #871

@forsyth2
forsyth2 force-pushed the issue-837-legacy-3.2.0 branch from 7db028d to 18e3683 Compare October 14, 2026 17:51
@forsyth2 forsyth2 mentioned this pull request Oct 14, 2026
3 tasks done
@forsyth2
forsyth2 force-pushed the issue-837-legacy-3.2.0 branch 2 times, most recently from 87ef5ca to 758bcbe Compare October 16, 2026 23:01
@forsyth2
forsyth2 force-pushed the issue-837-legacy-3.2.0 branch from 758bcbe to b20c45f Compare October 20, 2026 16:05
@forsyth2
forsyth2 force-pushed the issue-837-legacy-3.2.0 branch from b20c45f to 4472aee Compare October 30, 2026 17:29
Previously, image_checker raised FileNotFoundError when a cfg had no
expected image list (e.g. the new legacy_3.2.0 cfg), aborting the whole
test_images run and discarding results from other tests.

- Report "NO BASELINE" instead of crashing when the expected image list
  is missing or has no entries for a task. Nothing is compared, and the
  summary table flags it. Passes by default; set ZPPY_REQUIRE_BASELINES=1
  to fail.
- Record unexpected per-test exceptions and continue, so other tests'
  results are kept. The run still fails after the summary is written.
- Write run_test failure logs to images_logs/ like success logs.

Generated by Claude.
@forsyth2
forsyth2 force-pushed the issue-837-legacy-3.2.0 branch from 4472aee to 22475d8 Compare October 30, 2026 23:43
@forsyth2 forsyth2 mentioned this pull request Oct 30, 2026
1 of 17 tasks

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Testing Files in `tests` modified

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants