Skip to content

optimizer and front end: lookibed's side-effect, inline, infer, typedecl, reinterpret and escape reports - #4480

Merged
borisbat merged 9 commits into
masterfrom
bbatkin/lookibed-optimizer-frontend
Nov 7, 2026
Merged

borisbat merged 9 commits into
masterfrom
bbatkin/lookibed-optimizer-frontend

Conversation

@borisbat

@borisbat borisbat commented Nov 7, 2026

Copy link
Copy Markdown
Collaborator

Behavior change: an assert over a field of an effectful call, or over an inserting table index, is now a compile error naming the side effect - write it as verify. Character and string literals take the C escape set, so '\a', '\v', '\?', '\xHH' and '\NNN' lex where they were errors.

Why. Seven of lookibed's optimizer and front-end reports: an unused call was dropped together with the addr(arr[i++]), computed index or field of a call result it carried; a constant-argument call into mutual recursion that reads a global was folded; a private function shaped while (true) { ...; while (true) { return } } looped forever once inlined; a property compound assignment on deref(p + k) crashed context relocation; tuple<void?> reused the tuple<int?> instance or reported two; reinterpret<int?>(f()) of a uint64 extern threw in the interpreter; the lexers lacked half the C escapes.

What changes.

  • A field, index or safe-index node carries the side effects of its operands, so an unused call keeps them and an assert over them reports.
  • Inferred side effects close over call cycles: a function in a cycle takes the cycle's union, not the partial flags the walk saw mid-cycle.
  • An inlined return out of a loop that ends a loop body leaves the outer loop too.
  • A property compound assignment gives its setter its own copy of the base; a with-scoped property copies the with value the same way.
  • An explicit type argument never takes the void? stand-in, so each pointer instance is its own.
  • A reinterpret across eval slots reads its operand through eval instead of the typed slot the extern does not answer.
  • '\0', '\a', '\v', '\"', '\?', '\xHH' and '\NNN' lex in both lexers through one escape rule per suffix; a string literal takes the same set and an embedded \0 ends it; the tree-sitter scanner reads the same set.

Observable behavior.

  • assert(f().x) with an effectful f, or assert(tab[k]) on a mutable table: silently kept or dropped -> compile error naming the side effect.
  • let r = mutual_a(3) where the cycle reads a global: folded to the wrong constant -> evaluated.
  • The nested-while private function: infinite loop under auto-inline -> returns.
  • obj.prop += 1 with obj = deref(p + k): crash in context relocation -> runs.
  • tuple<void?> beside tuple<int?>: wrong instance or multiple instances -> its own instance.
  • reinterpret<int?>(f()) of a uint64 extern: interpreter throw -> the value.
  • '\a', '\?', "\x41": lexer error -> the character.

Where to look. inheritEffects and closeCallCycles in src/ast/ast_unused.cpp; the escape rules in src/parser/ds2_lexer.lpp and ds_lexer.lpp; scan_escape in tree-sitter-daslang/src/scanner.c.

Fixes #4458, fixes #4460, fixes #4464, fixes #4449, fixes #4452, fixes #4457, fixes #4465.

Validation, claims, ledger

Validation

  • Full AOT sweep (preflight --only tests-aot, the test_aot build and sweep over tests/) passed at the final tip, rebased on dasLLVM: bool and pointer reinterprets, default<T[N]>, try/recover return, inline ptr_diff, speculatable lookups, --jit-remarks #4472's merge, 760 s; per-PR CI builds only the subset. Every other gate and lane passed at the same tip: lint, review-md, md-ascii, hash-refs, cpp-syntax, ast-verify, compile-sweep, tests-cpp, tests-interp, tests-jit, utils-tests.
  • Two codex review rounds, one at the audited tip and one at the final tip after the fix batch: no findings in either, every changed file in both traces.
  • OBSERVABLE_EFFECTS (src/ast/ast.cpp), the one citer of the changed ARCHITECTURE_EFFECTS.md section: the flag set it names is unchanged; the new paragraph documents the cycle closure beside it.
  • The local docs/digest gate is red on this tree for a reason outside the diff: the generator writes no function-* detail file for the core daslib modules on this box (algorithm, fio, ...), so the regenerated digest loses 113 one-line descriptions that the shipped digest took from stale, gitignored detail files; the main tree shows the same with master's binary. The shipped digest is not touched. docs/sphinx-html is the known sphinx_sitemap environment red.

Claims - stated, not tested

  • The noNativeSideEffects half of the effect inheritance has no test: its only consumer is the aot_order_side_effects lint, which defaults off and no test enables.
  • The tree-sitter scanner's octal digit limits and its rejection of '\x without a digit are not observable in the corpus (string content nodes are hidden); the corpus covers every accepted escape form.
  • The scanner's and the compiler's escape sets agree by construction (the scanner comment names CHAR_ESCAPE as its source); no test diffs the two.

Not done

  • The src/ast/REVIEW.md self-review items (the REVIEW.das rule's trailing sentence, the no_promo example list, the version-bump rule's trigger) are fixed on the JIT branch (dasLLVM: bool and pointer reinterprets, default<T[N]>, try/recover return, inline ptr_diff, speculatable lookups, --jit-remarks #4472), which edits the same file; this PR does not repeat them.
  • The regenerated lexer .cpp files carry the local flex build's #line preamble.
  • tests/REVIEW.md gains the daslang-child cache rule on the JIT branch (dasLLVM: bool and pointer reinterprets, default<T[N]>, try/recover return, inline ptr_diff, speculatable lookups, --jit-remarks #4472); this PR applies it (-no-module-cache on the spawned child) without restating it.
  • Checklist self-review items left as found, each its own restructure: tests/REVIEW.md's AOT trigger describes two sets and its module-cache routing and placement rules repeat one trigger; src/REVIEW.md's narrow-path rule restates what its gate checks and its opening routes by folder; daslib/REVIEW.md's pair-record duty covers only the daslib side and two of its rules describe the system; include/daScript/simulate/REVIEW.md's hot-path rule is keyed to a changed body and its ledger rule fuses two duties; include/daScript/ast/REVIEW.md's layout rule lists layout changes instead of naming the property; the root REVIEW.md's modules/<M> route keys on a folder name a require line never spells and its routed_paths sync duty runs one way.

#nightly - the diff touches src/ and include/ and regenerates the lexers.

borisbat and others added 2 commits November 7, 2026 06:45
A call dropped as unused no longer takes an addr(arr[i++]), a vector index
computed by a call, or a field of a call result with it. An assert over a
field of an effectful call, or over an inserting table index, now reports its
side effect like the plain forms already did; the in-tree sites move to
verify.

Fixes #4458

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
A function in a call cycle no longer keeps the partial flags the walk saw
mid-cycle, so a constant-argument call into mutual recursion that reads a
global is not folded.

Fixes #4460

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Copilot AI balanced review requested due to automatic review settings November 7, 2026 15:18

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Note

Copilot was unable to complete this review at the selected tier due to model policy or plan restrictions.

🔵 Needs a closer look

The change spans multiple high-risk core compiler subsystems (optimizer side-effects, const folding/inlining, interpreter eval slots, type identity, and lexer/tree-sitter/serializer behavior) with user-visible semantics, making it difficult to validate safely from the diff without maintainer review and broad CI/runtime verification.

No open issues 🎉

What changed in this PR

This pull request fixes multiple correctness issues across the optimizer, type/instance handling, interpreter simulation, and lexer/tooling parity, and expands character/string literal escaping to match the C escape set (compiler lexers + tree-sitter), with regressions and documentation updates.

Changes:

  • Preserve and properly close inferred side effects (including through field/index/safe-index nodes and across call cycles) to prevent incorrect dropping/folding and improve diagnostics (e.g., assert over effectful access).
  • Fix inlining control-flow for nested loop returns, property compound-assignment lowering stability, explicit-type generic instance separation (notably void? pointer-like shapes), and interpreter reinterpret handling across eval slots.
  • Expand and unify literal escape handling (\a, \v, \?, \xHH, \NNN, etc.) across compiler lexers and tree-sitter, with tests, canaries, docs, and changelog entries.
File Summary
tree-sitter-daslang/​test/​corpus/​literals.txt Adds corpus coverage for expanded C-style escapes in char/string literals.
tree-sitter-daslang/​src/​scanner.c Updates tree-sitter external scanner to recognize expanded escape forms (octal/hex + additional C escapes).
tree-sitter-daslang/​REVIEW.md Updates review/checklist guidance for scanner changes (requires corpus/canary coverage).
tests/​watchdog/​test_watchdog.das Switches watchdog timing/guard logic to shared child_deadline helper usage.
tests/​strings/​test_cpp_functions.das Adds/adjusts regression assertions around escape handling in safe_unescape.
tests/​README.md Registers new/updated tests and fixtures in the test index.
tests/​language/​test_reinterpret_extern_result.das Adds regression tests for reinterpret behavior on extern-return values.
tests/​language/​test_property_compound_assign_deref_offset.das Adds regression tests for property += with deref(p + k) and with bases.
tests/​language/​test_nested_while_private.das Adds regression coverage for nested while(true) private function return behavior under inlining.
tests/​language/​test_generic_tuple_void_ptr.das Adds tests ensuring tuple<void?> (and related) instantiates distinctly vs other pointer-like shapes.
tests/​language/​test_fold_mutual_recursion_global.das Adds regression preventing incorrect const-folding through call cycles that read globals.
tests/​language/​test_char_escapes.das Adds gen2 tests for expanded escape set and string NUL-termination behavior.
tests/​language/​test_char_escapes_gen1.das Adds gen1 tests for expanded escape set.
tests/​language/​test_addr_argument_side_effects.das Adds regression tests ensuring operand side effects are preserved through addr/index/field expressions.
tests/​language/​failed_assert_over_effectful_field.das Adds negative test validating assert over effectful field access is rejected with a side-effect-specific error.
tests/​language/​_fixture_nested_while_private.das Adds fixture program used by the nested-while private regression test.
tests/​fio/​test_process.das Switches process-test deadlines/guards to shared child_deadline helper usage.
tests/​fio/​fio_poll.das Switches poll-test deadlines/guards to shared child_deadline helper usage.
tests/​aot/​CMakeLists.txt Updates AOT subset module list (adds daslib/fio.das).
src/​simulate/​simulate_visit.cpp Extends simulate visitor support for new reinterpret-slot simulation node.
src/​simulate/​runtime_string.cpp Factors/extends shared character escape decoding and integrates it into runtime unescape paths.
src/​REVIEW.md Updates/tightens review guidance wording.
src/​parser/​REVIEW.md Updates parser review guidance to enforce compiler/tree-sitter parity expectations.
src/​parser/​lex2.yy.h Regenerated lexer header reflecting lexer changes.
src/​parser/​lex.yy.h Regenerated lexer header reflecting lexer changes.
src/​parser/​ds2_lexer.lpp Updates/extends lexer rules for expanded escapes (shared CHAR_ESCAPE approach).
src/​parser/​ds_lexer.lpp Updates/extends lexer rules for expanded escapes (shared CHAR_ESCAPE approach).
src/​ast/​ast_unused.cpp Fixes side-effect inference by closing over call cycles (union/fixpoint behavior).
src/​ast/​ast_typedecl.cpp Refines explicit type argument matching to avoid void? stand-in collisions for pointer-like instances.
src/​ast/​ast_simulate.cpp Adjusts simulation for reinterpret to read operands through eval when needed (cross-slot safety).
src/​ast/​ast_inline.cpp Fixes inline return/loop-exit interaction for nested-loop constructs.
src/​ast/​ast_infer_type_op.cpp Clones property base for compound assignment lowering to avoid relocation/crash scenarios.
src/​ast/​ast_infer_type_helper.cpp Clones with base value for property lowering to preserve stable evaluation semantics.
src/​ast/​ast_const_folding.cpp Ensures effect flags propagate through field/index/safe-index/swizzle reads and respects mutating indexes.
src/​ast/​ARCHITECTURE_EFFECTS.md Documents call-cycle closure behavior for side-effect inference.
skills/​daslang/​references/​types.md Updates language reference content to reflect the new escape set and literal rules.
modules/​dasLiveHost/​live/​live_watch.das Replaces assert with verify to match new assert restrictions around effectful expressions.
modules/​dasLiveHost/​live/​live_api.das Replaces assert with verify to match new assert restrictions around effectful expressions.
modules/​dasLiveHost/​live/​live_api_stdio.das Replaces assert with verify to match new assert restrictions around effectful expressions.
modules/​dasImgui/​tests/​test_grammar_canary.das Adds grammar canary coverage for expanded escape support.
include/​daScript/​simulate/​simulate_nodes.h Introduces SimNode_ReinterpretSlot node declaration/definition.
include/​daScript/​simulate/​runtime_string.h Exposes character escape decoding API/types for shared use.
include/​daScript/​simulate/​ARCHITECTURE.md Documents reinterpret-slot wrapper rationale and behavior.
include/​daScript/​ast/​ast_serializer.h Bumps serializer version to account for the explicit-type instance behavior change.
examples/​debugapi/​stack_walker.das Replaces assert with verify in debug-context checks.
examples/​debugapi/​hw_breakpoint.das Replaces assert with verify in debug-context checks.
examples/​debugapi/​allocation_tracking.das Replaces assert with verify in debug-context checks.
doc/​source/​reference/​language/​lexical_structure.rst Updates reference documentation for expanded escape set and literal behavior.
dastest/​testing_boost.das Adds shared child_deadline helper and related small testing-utility adjustments.
daslib/​profiler.das Replaces assert with verify in debug-agent install path.
daslib/​cpp_gen.das Replaces assert with verify for JSON shape checks per new assert restrictions.
CHANGELIST.md Adds changelog entries covering the behavior changes and fixes.

Review effort: Balanced

borisbat and others added 7 commits November 7, 2026 07:32
The auto-inlined body of a private function shaped
while (true) { ...; while (true) { return } } looped forever: the break guard
was skipped when the inner loop was the last statement.

Fixes #4464

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
The getter and the setter shared one subtree, and simulating it twice crashed
context relocation once the base was deref(p + k); a with-scoped property
shared the with value the same way.

Fixes #4449

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
A generic instance's arguments are explicit, so a tuple<void?> call no longer
reuses a tuple<int?> instance or finds two and reports multiple instances; it
instantiates its own.

Fixes #4452

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
An extern's typed eval answers only its own return kind, so
reinterpret<int?>(f()) of a uint64 extern, or reinterpret<uint64>(p + n) over
i_das_ptr_add, threw in the interpreter.

Fixes #4457

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
'\0', '\a', '\v', '\"', '\?', hex '\xHH' and octal '\NNN' lex in both the
gen2 and the gen1 lexer, through one escape rule per suffix; a string literal
takes \0, \a, \? and octal the same way, and an embedded \0 ends the string.
The tree-sitter scanner reads the same set.

Fixes #4465

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
The C escape set, the explicit-type instance rule and the unused-call side
effect rule are the three user-visible changes of the branch.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
The parser checklist's tree-sitter duty names the scanner for an external
token and the grammar for the rest; the tree-sitter checklist's two rules
trigger on what the grammar or the scanner accepts; the src checklist's
narrow-path row rule keeps its Windows ban in one sentence.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
@borisbat
borisbat force-pushed the bbatkin/lookibed-optimizer-frontend branch from 4aaf0c2 to 5d8a559 Compare November 7, 2026 15:32
Copilot AI balanced review requested due to automatic review settings November 7, 2026 15:32

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Note

Copilot was unable to complete this review at the selected tier due to model policy or plan restrictions.

🟡 Changes recommended

The change has critical active issues because both lexer definition files call decodeCharacterEscape without including the header that declares it, which is likely to break compilation of the generated lexers.

2 open issues

Review effort: Balanced

Comment thread src/parser/ds2_lexer.lpp Resolved
Comment thread src/parser/ds_lexer.lpp Resolved
@borisbat
borisbat merged commit 1404d5a into master Nov 7, 2026
56 of 71 checks passed
@borisbat
borisbat deleted the bbatkin/lookibed-optimizer-frontend branch November 7, 2026 16:10
borisbat added a commit to GaijinEntertainment/tree-sitter-daslang that referenced this pull request Nov 7, 2026
…ecl, reinterpret and escape reports (GaijinEntertainment/daScript#4480)

* optimizer: a field or index node carries its operands' side effects

A call dropped as unused no longer takes an addr(arr[i++]), a vector index
computed by a call, or a field of a call result with it. An assert over a
field of an effectful call, or over an inserting table index, now reports its
side effect like the plain forms already did; the in-tree sites move to
verify.

Fixes GaijinEntertainment/daScript#4458

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* optimizer: inferred side effects close over call cycles

A function in a call cycle no longer keeps the partial flags the walk saw
mid-cycle, so a constant-argument call into mutual recursion that reads a
global is not folded.

Fixes GaijinEntertainment/daScript#4460

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* inline: a return out of a loop that ends a loop body leaves both loops

The auto-inlined body of a private function shaped
while (true) { ...; while (true) { return } } looped forever: the break guard
was skipped when the inner loop was the last statement.

Fixes GaijinEntertainment/daScript#4464

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* infer: a property compound assignment copies the base for its setter

The getter and the setter shared one subtree, and simulating it twice crashed
context relocation once the base was deref(p + k); a with-scoped property
shared the with value the same way.

Fixes GaijinEntertainment/daScript#4449

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* typedecl: an explicit type takes no void-pointer stand-in

A generic instance's arguments are explicit, so a tuple<void?> call no longer
reuses a tuple<int?> instance or finds two and reports multiple instances; it
instantiates its own.

Fixes GaijinEntertainment/daScript#4452

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* simulate: a reinterpret across eval slots reads its operand through eval

An extern's typed eval answers only its own return kind, so
reinterpret<int?>(f()) of a uint64 extern, or reinterpret<uint64>(p + n) over
i_das_ptr_add, threw in the interpreter.

Fixes GaijinEntertainment/daScript#4457

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* lexer: character and string literals take the C escape set

'\0', '\a', '\v', '\"', '\?', hex '\xHH' and octal '\NNN' lex in both the
gen2 and the gen1 lexer, through one escape rule per suffix; a string literal
takes \0, \a, \? and octal the same way, and an embedded \0 ends the string.
The tree-sitter scanner reads the same set.

Fixes GaijinEntertainment/daScript#4465

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* CHANGELIST: the lookibed optimizer and front-end entries

The C escape set, the explicit-type instance rule and the unused-call side
effect rule are the three user-visible changes of the branch.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* review: three checklists name the files their triggers live in

The parser checklist's tree-sitter duty names the scanner for an external
token and the grammar for the rest; the tree-sitter checklist's two rules
trigger on what the grammar or the scanner accepts; the src checklist's
narrow-path row rule keeps its Windows ban in one sentence.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

---------

Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
daScript-Commit: 1404d5a6a95427d43ffb64fee0534f524bde4b74
This was referenced Nov 9, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

2 participants