Skip to content

test: random differential tests, and expected failures for cycles, exceptions and writing effects - #131

Open
johnsoncodehk wants to merge 6 commits into
masterfrom
cursor/fuzz-tests-on-master-6a0e
Open

johnsoncodehk wants to merge 6 commits into
masterfrom
cursor/fuzz-tests-on-master-6a0e

Conversation

@johnsoncodehk

Copy link
Copy Markdown
Collaborator

Tests and measuring scripts only. src/ is not touched.

What is added

File What it does On master
tests/fuzz.spec.ts Random graphs with changing dependencies, batches and effects disposed and created again; after every step, what effects saw and what computeds return is compared with an evaluation from scratch passes (30,000 cases); the variant with effects that write is an expected failure
tests/chaos.spec.ts Effects disposed while running, nested effects, scopes, trigger; only asserts that everything agrees with an evaluation from scratch once it has settled passes (100,000 cases); the variant with getters, effects and cleanups that throw is an expected failure
tests/bound.spec.ts Counts getter calls and compares them with four reference models (re-run decided by version or by value; unwatched computed dropped or kept). A measurement, not a target passes; prints the report with BOUND_REPORT=1
tests/errors.spec.ts One small test per way an exception leaves a node that later changes no longer reach 2 pass, 6 expected failures
tests/cycle.spec.ts Computeds that read each other (#123), run against the build in a child process with a small heap, only asserting that the reads come back 1 passes, 4 expected failures
benchs/instructions.mjs, benchs/wallclock.mjs, benchs/workloads.mjs Compare builds by interpreted instructions (valgrind, Linux) and by time —

Expected failures use test.fails, so CI stays green and a fix shows up as a test that has to lose its .fails. The comment above each one states the cause. To see the actual failures:

sed -i 's/test\.fails(/test(/' tests/{errors,cycle,fuzz,chaos}.spec.ts
pnpm run build && pnpm vitest run --reporter=default
git checkout tests

What the expected failures are

  • Computed cycles (Edge case: infinite loop/out of memory in certain situations #123): a computed read while its own getter runs is handed out like one that has a value and becomes a dependency of its reader; the same happens one step removed, when a computed that depends on the running one is checked against it and found unchanged. Either way the graph gets a circle and checkDirty does not return.
  • Exceptions: a getter or effect that throws abandons flush, run or checkDirty half way. Left behind are an effect that is neither queued nor watching, effects dropped from the queue behind a computed that stays marked, a computed that stays marked while its reader is marked up to date, and a read that was never recorded as a dependency.
  • Effects that write: a write inside a running effect flushes the queue from inside that effect; an effect that runs in there and writes something the outer one has read is treated like the outer effect writing its own dependency.

None of these is fixed here. Every fix that was tried changes observable behaviour in some case, so they are left for a decision.

Reproducing the numbers

pnpm run build
FUZZ_CASES=30000 pnpm vitest run tests/fuzz.spec.ts
FUZZ_CASES=100000 pnpm vitest run tests/chaos.spec.ts
BOUND_REPORT=1 FUZZ_CASES=20000 pnpm vitest run tests/bound.spec.ts --reporter=default
node benchs/instructions.mjs esm
Open in Web Open in Cursor 

cursoragent and others added 6 commits November 12, 2026 14:41
Co-authored-by: Johnson Chu <johnsoncodehk@users.noreply.github.com>
Co-authored-by: Johnson Chu <johnsoncodehk@users.noreply.github.com>
Co-authored-by: Johnson Chu <johnsoncodehk@users.noreply.github.com>
Co-authored-by: Johnson Chu <johnsoncodehk@users.noreply.github.com>
Co-authored-by: Johnson Chu <johnsoncodehk@users.noreply.github.com>
…n last

Co-authored-by: Johnson Chu <johnsoncodehk@users.noreply.github.com>
@bolt-new-by-stackblitz

Copy link
Copy Markdown

Review PR in StackBlitz Codeflow Run & review this pull request in StackBlitz Codeflow.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants