Skip to content

ci: fail-fast job timeouts + de-flake webkit scroll invariant-1#966

Merged
mremond merged 3 commits into
mainfrom
mr/ci-stuck-pr-963-21cc0e
Jul 10, 2026
Merged

ci: fail-fast job timeouts + de-flake webkit scroll invariant-1#966
mremond merged 3 commits into
mainfrom
mr/ci-stuck-pr-963-21cc0e

Conversation

@mremond

@mremond mremond commented Jul 10, 2026

Copy link
Copy Markdown
Member

CI jobs had no timeout-minutes, so a hung step could park a runner for GitHub's 6h default (as seen on #963, where a slow apt mirror stretched the Playwright OS-dep install to ~35min). Separately, scroll invariant-1 flaked intermittently on webkit under CI load, burning retries and printing scary red mid-log.

  • Fail-fast timeouts on all three jobs (test 15m, e2e-scroll 30m, rust 20m) so a genuine hang fails instead of sitting for hours.
  • Cache Playwright browser binaries and split the OS-dep (install-deps) step so a slow CDN can't add minutes to the browser download.
  • De-flake invariant-1: raise the demo-mount waitForSelector ceiling to 45s (webkit is slow to boot the demo bundle on a busy runner), and wait for the prepend restore to settle before the drift assertion instead of a single racy read. The settle-wait polls until drift stabilises without loosening PREPEND_DRIFT_PX, so a genuinely broken anchor still fails.

mremond added 3 commits July 10, 2026 15:26
The scroll e2e job could sit for GitHub's 6h default when a step hung, and
invariant-1 flaked intermittently on webkit under CI load, forcing retries.

- Add timeout-minutes to all three CI jobs (test 15, e2e-scroll 30, rust 20)
  so a genuine hang fails fast instead of parking a runner for hours.
- Cache the Playwright browser binaries and split OS-dep install so a slow
  CDN can't add minutes to the browser download.
- Harden invariant-1: raise the demo-mount waitForSelector ceiling to 45s
  (webkit is slow to boot the demo bundle on a busy runner), and wait for the
  prepend restore to settle before the drift assertion instead of a single
  racy read. The settle-wait polls until drift stabilises without loosening
  PREPEND_DRIFT_PX, so a genuinely broken anchor still fails.
apps/fluux/tsconfig.json pinned @fluux/sdk + /react /core /stores to the
workspace-relative SDK dist, but not the /demo and /cache subpaths the app
also imports. In a git worktree those two escaped the paths mapping and
resolved through node_modules to the *root* repo's dist, so demo.tsx pulled
XMPPClient/E2EEManager from two different copies of the SDK and tsc flagged
spurious type-identity mismatches (DemoClient not assignable to XMPPClient,
e2ee possibly null). Mapping every exported subpath to the same local dist
makes resolution consistent. No-op outside a worktree (single package).
…rance

The first de-flake pass reduced but didn't eliminate the webkit invariant-1
flake — CI still hit 2880px (settle-wait timed out mid-restore at 2500ms) and
28px (settled but over the 20px bound).

- Wait on scrollTop AND the anchor's virt offset both going quiet, not on their
  derived drift: during the re-assert loop the two move together so the drift
  reads ~0 mid-flight and falsely looks stable. Longer timeout (6s) so the read
  lands past the loop + trailing ResizeObserver even on a slow runner.
- Engine-specific tolerance: WebKit's settled residual runs ~28-40px (coarser
  height-measurement cadence) vs Chromium's <20px. Add PREPEND_DRIFT_WEBKIT_PX
  (48) and keep Chromium strict at 20. Still far below a real mis-anchor
  (~2880px, a dropped batch) and the LARGE_JUMP_PX oscillation gate.
@mremond mremond merged commit 78c0854 into main Jul 10, 2026
3 checks passed
@mremond mremond deleted the mr/ci-stuck-pr-963-21cc0e branch July 10, 2026 14:10
mremond added a commit that referenced this pull request Jul 13, 2026
… DOM (#968)

Follow-up to #966. That PR reduced but didn't eliminate the
retry-absorbed webkit flake in scroll `invariant-1`.

Root cause: Assertion B derived drift from `__fluuxGetVirtOffset`.
During the prepend re-assert loop that offset map can report a stale
pre-prepend value for a sustained window while `scrollTop` already
reflects the added batch, producing a ~2880–3034px phantom drift that
isn't visible on screen. A second trigger was the demo mount blowing the
60s per-test budget on a slow webkit runner.

- Measure the anchor row's own `getBoundingClientRect().top` vs its
captured before-position — the layout truth the user actually sees,
which can't go stale like the offset map. `waitForAnchorSettled` polls
that DOM offset until quiet.
- Raise the per-test timeout to 120s so slow webkit demo-boot proceeds
instead of failing the mount wait.
- Drop the now-unused virtualizer-offset path; keep the engine-specific
tolerance (chromium 20px, webkit 48px).
mremond added a commit that referenced this pull request Jul 13, 2026
The azure apt mirror can be very slow at certain times of day, which is
exactly the failure mode behind the earlier "stuck CI" episode (#966
added the timeout backstop). Steady-state the `install-deps` step is
only ~30–60s, but the download portion is the part that blows up on a
slow mirror.

This caches the downloaded `.deb` files rather than installed state, so
it stays robust: apt validates cached debs by hash and re-downloads only
what changed, so a stale cache degrades gracefully.

- **ci.yml (e2e-scroll):** restore-only cache of `~/.apt-debs`,
pre-seeded into `/var/cache/apt/archives` before `npx playwright
install-deps`. Keyed on the runner image version
(`$ImageOS-$ImageVersion`), since package versions roll with it
(~weekly); `restore-keys` falls back to the previous image's debs.
- **playwright-cache.yml:** new `seed-apt` job keeps the shared cache
warm from `main` (PR-created caches are PR-scoped, same reason as the
browser cache). Short-circuits when the cache already exists for the
current image version; warns if apt kept no debs so a silent no-op is
visible.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant