Skip to content

Reduce races from a few flaky unit tests - #6350

Open
pevogam wants to merge 1 commit into
avocado-framework:masterfrom
pevogam:flaky-ci-test-resilience
Open

pevogam wants to merge 1 commit into
avocado-framework:masterfrom
pevogam:flaky-ci-test-resilience

Conversation

@pevogam

@pevogam pevogam commented Sep 11, 2026

Copy link
Copy Markdown
Collaborator

Apply three targeted changes in avocado:

  • use a simulated clock, removing scheduler variability in a flaky unit test for the wait utility
  • measure monotonic time and allow one second of scheduling overhead in a functional test for the wait utility
  • run sequentially, preventing the deadline from interrupting the passing test’s startup in job timeout functional test

The last of the three is indicated by 9 = 8 | 1 status meaning both the job was interrupted and a test failed or errored (passtest ERROR). If we run in serial mode the deadline interrupts while passtest is still queued and is thus subsequently skipped.

Summary by CodeRabbit

  • Tests
    • Improved timeout test reliability under parallel execution and busy CI conditions.
    • Made timing-related tests deterministic by using a controlled monotonic clock.
    • Added verification that timeout waits return correctly without exceeding the permitted delay.

Apply three targeted changes in avocado:
- use a simulated clock, removing scheduler variability in a flaky
  unit test for the wait utility
- measure monotonic time and allow one second of scheduling overhead
  in a functional test for the wait utility
- run sequentially, preventing the deadline from interrupting the
  passing test’s startup in job timeout functional test

The last of the three is indicated by 9 = 8 | 1 status meaning both
the job was interrupted and a test failed or errored (passtest ERROR).
If we run in serial mode the deadline interrupts while passtest is
still queued and is thus subsequently skipped.

Signed-off-by: Plamen Dimitrov <plamen.dimitrov@intra2net.com>
@pevogam pevogam self-assigned this Sep 11, 2026
@mr-avocado mr-avocado Bot moved this to Review Requested in Default project Sep 11, 2026
@mr-avocado

mr-avocado Bot commented Sep 11, 2026

Copy link
Copy Markdown

You are changing one of the avocado utils which has already been migrated to AAutils project https://github.com/avocado-framework/aautils and this utility will be removed after the LTS release. Please make sure that all your proposed changes are already in AAutils and this PR is only backport.

For more information about AAutlis migration see https://avocado-framework.readthedocs.io/en/latest/blueprints/BP005.html

For a list of migrated utilities see https://avocado-framework.github.io/aautils.html

@coderabbitai

coderabbitai Bot commented Sep 11, 2026

Copy link
Copy Markdown

Review Change StackReview Change Stack

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Advanced

Run ID: b601c519-6643-40d4-98cb-7afc303747fb

📥 Commits

Reviewing files that changed from the base of the PR and between e3b0b2d and ab9ee24.

📒 Files selected for processing (3)
  • selftests/functional/job_timeout.py
  • selftests/functional/utils/wait.py
  • selftests/unit/utils/wait.py

Included review availability: Your plan provides up to 2 included reviews per hour; 1 remains after this review.


Walkthrough

The changes improve timeout test stability. The functional job timeout test limits parallel tasks to one. Functional wait timing uses time.monotonic() and allows scheduling delays. Unit wait timing uses a mocked monotonic clock and simulated sleep to verify deterministic timeout behavior.

Priority: ⬇️ Low

Estimated code review effort: 2 (Simple) | ~10 minutes

Merge Risk: ⚪ Minimal · up to ab9ee

The timeout tests now use deterministic scheduling and monotonic timing without introducing an identified merge-blocking risk.

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Docstring Coverage ⚠️ Warning Docstring coverage is 50.00% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 6 functions across 3 files. Write docstrings for the functions missing them to satisfy the coverage threshold.
✅ Passed checks (4 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title accurately describes the goal of reducing test flakiness and race conditions. It is slightly incomplete because the changes also affect functional tests, not only unit tests.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
  • Fix all pre-merge checks with AI

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@codecov

codecov Bot commented Sep 11, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 71.54%. Comparing base (e3b0b2d) to head (ab9ee24).

Additional details and impacted files
@@           Coverage Diff           @@
##           master    #6350   +/-   ##
=======================================
  Coverage   71.54%   71.54%           
=======================================
  Files         207      207           
  Lines       23799    23799           
=======================================
  Hits        17026    17026           
  Misses       6773     6773           

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@pevogam

pevogam commented Sep 11, 2026

Copy link
Copy Markdown
Collaborator Author

@harvey0100 @PraveenPenguin @maramsmurthy I know you have had to rerun jobs in order to deal with flaky tests and I collected three such with fixes proposed here to hopefully let us all have more stable CI jobs. Let me know if you have any other tests that you have observed to have races and occasionally fail.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

Status: Review Requested

Development

Successfully merging this pull request may close these issues.

1 participant