Skip to content

dasLLVM: bool and pointer reinterprets, default<T[N]>, try/recover return, inline ptr_diff, speculatable lookups, --jit-remarks - #4472

Merged
borisbat merged 11 commits into
masterfrom
bbatkin/lookibed-jit
Nov 7, 2026
Merged

borisbat merged 11 commits into
masterfrom
bbatkin/lookibed-jit

Conversation

@borisbat

@borisbat borisbat commented Nov 7, 2026 •

Copy link
Copy Markdown
Collaborator

Behavior change: a return inside a try or recover arm is a compile error on every tier (error 30299) - set a variable in the arm and return after the pair. A closure's own return inside the arm stays allowed.

Why. Seven of lookibed's JIT reports: reinterpret<uint8>(true) read 0xff and failed the verifier when it fed a call; a return inside a try arm left the arm and the function answered wrong; a = default<int[4]> zeroed one element; reinterpret<uint>(p) emitted trunc ptr; p - q was a slot call with a run-time stride (0.32 s against 0.0045 s); a global read under a branch in a loop kept its lookup in the loop (0.405 s against 0.042 s); LLVM's optimization remarks had no switch.

What changes.

  • A bool-to-integer reinterpret zero-extends and integer-to-bool compares the low byte with zero. The test asserts a bool byte of 2 only under -jit: the interpreter and AOT read such a byte as a C++ bool, which a Debug build reads as its low bit.
  • The front end rejects a return inside a try/recover arm; the tests that pinned the interpreter's old return-from-try behavior keep their break and continue cases. AstSerializer::getVersion() moves to 246, so a cached module holding the shape is not served past the new check.
  • default<T[N]> zeroes the declared shape when no element is provided.
  • A pointer reinterprets to any integer width through ptrtoint.
  • i_das_ptr_diff is an intrinsic row: ptrtoint, sub, division by the constant stride.
  • jit_get_global_mnh and jit_get_shared_mnh are speculatable, so LLVM hoists a lookup out of a loop.
  • --jit-remarks=, --jit-remarks-missed= and --jit-remarks-analysis= print LLVM's remarks and run codegen past both caches.

Observable behavior.

  • print("{reinterpret<uint8>(true)}") under -jit: 0xff -> 0x1.
  • try { return 1 } recover { }; return 10: the interpreter returned 1 and the JIT 10 with a warning -> compile error on both.
  • a = default<int[4]> on [1,2,3,4] under -jit: [0,2,3,4] -> [0,0,0,0].
  • reinterpret<uint>(p) under -jit: Invalid bitcast -> the low 32 bits.
  • The pointer-difference loop: 0.32 s -> 0.0045 s; the branch-in-loop global read: 0.405 s -> 0.042 s.
  • daslang -jit x.das -- --jit-remarks-missed=loop-vectorize prints loop not vectorized: ... per loop.

Where to look. visitExprCast, preVisitExprMakeStruct and preVisitExprReturn in modules/dasLLVM/daslib/llvm_jit.das; the i_das_ptr_diff row in llvm_jit_intrin.das; the extern attributes in llvm_jit_common.das.

Validation, claims, ledger

Validation

  • The module-owned suite at the tip: bin/daslang dastest/dastest.das -- --test modules/dasLLVM/tests 170 passed, 5 skipped; tests/jit_tests under -jit 435 passed, 1 skipped.
  • Full AOT sweep: preflight --only tests-aot (the test_aot build and sweep over tests/) passed at the tip, 551 s; per-PR CI builds only the subset.
  • The three fixed emitter paths were negative-controlled: each test fails with its branch mutated and passes restored.
  • Codex review round at the emitter-only tip: no findings, every changed file in its trace. The front-end change came after it: tests/ interpreted 15588 passed, 9 skipped; tests/language under -jit 1981 passed.
  • The tree-sitter parser lane runs on ubuntu-24.04: on ubuntu-latest (26.04) the swift toolchain the test action installs cannot load libxml2.so.2, and the lane only runs when a PR touches tree-sitter-daslang/, which the example edit here does.
  • The sequence example's smoke clones dasCards main, which returned from inside a try arm; the restructure merged upstream, so the darwin15 lane goes green on a rerun.
  • Two docs sub-gates are red on this box for reasons outside the diff: docs/digest (master's shipped digest is stale, the regeneration adds the AstConstVisitor entries an earlier merge introduced) and docs/sphinx-html (the PATH sphinx-build lacks sphinx_sitemap).

Claims - stated, not tested

  • A remark without --jit-debug prints its location as <unknown>:0:0; the function name the issue asks for needs a custom diagnostic handler, which needs a C++ callback binding dasLLVM does not expose.

Not done

#nightly - the emitter set is pinned and the JIT sweep lanes are nightly.

Copilot AI balanced review requested due to automatic review settings November 7, 2026 06:05
@borisbat
borisbat force-pushed the bbatkin/lookibed-jit branch from f667b0b to 63938f2 Compare November 7, 2026 06:12

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Note

Copilot was unable to complete this review at the selected tier due to model policy or plan restrictions.

🔵 Needs a closer look

Because the changes alter JIT codegen semantics and add LLVM attributes like speculatable whose safety depends on subtle LLVM rules and cross-target runtime invariants, the review evidence calls for domain-expert sanity checks and broader validation before proceeding.

No open issues 🎉

What changed in this PR

This pull request updates dasLLVM’s JIT to fix several correctness edge cases (reinterpret lowering, fixed-array defaults, and return behavior in try/recover arms), improves performance for pointer-difference and global/shared-global lookup patterns, and adds --jit-remarks* flags to surface LLVM optimization remarks. It also adds/updates regression tests and documentation around these behaviors.

Changes:

  • Correct JIT lowering for reinterpret (bool↔integer semantics; pointer→integer via ptrtoint) and for default<T[N]> so fixed-size arrays zero the full declared shape.
  • Make return inside try/recover arms a hard JIT compile error (while preserving valid closure returns inside arms).
  • Add inline lowering for pointer difference (i_das_ptr_diff), enable remark printing via --jit-remarks*, and adjust the JIT pipeline to run past caches when remarks are requested; expand/refactor dasLLVM module tests and fixtures accordingly.

Approval Readiness

This pull request requires changes.

  • Moderate: speculatable on jit_get_global_mnh/jit_get_shared_mnh must be safe for speculative execution (no traps). The review notes the call can dereference ctx and may assert on miss; please make the preconditions explicit (e.g., add nonnull on ctx at least) to satisfy the attribute’s contract.
  • Moderate: lookups_inside_loop test logic appears potentially fragile because it only exempts .preheader blocks; LLVM can emit other canonical preheader names (e.g. .lr.ph*), which can cause false failures.
  • Nits (comment hygiene): multiple top-level multi-line //! doc comments won’t attach to the following def/[test] def and should be converted to // (or moved into the function body / made single-line contracts where appropriate).
File Summary
tests/​REVIEW.md Updates test/review guidance for spawned-child execution and cache/output isolation.
tests/​jit_tests/​return_in_try_recover.das Adds regression coverage asserting JIT rejects return inside try/recover arms (and validates allowed closure-return behavior).
tests/​jit_tests/​reinterpret_ptr_to_uint.das Adds regression coverage for pointer→integer reinterpret behavior under -jit.
tests/​jit_tests/​default_fixed_array_assign.das Adds regression coverage ensuring default<T[N]> assignment zeros the entire fixed-size array.
tests/​jit_tests/​bool_reinterpret_byte.das Adds regression coverage for bool↔integer reinterpret byte/zero-extension semantics under -jit.
tests/​jit_tests/​_fixture_return_in_try_recover.das Adds a child-run fixture used to reproduce and validate the return-in-try/recover JIT compile error.
modules/​dasLLVM/​tests/​llvm_jit_remarks.das Adds tests validating --jit-remarks* output and the “run past caches” contract.
modules/​dasLLVM/​tests/​llvm_jit_ptr_diff_inline.das Adds tests asserting i_das_ptr_diff lowers to inline arithmetic (not an indirect/slot call).
modules/​dasLLVM/​tests/​llvm_jit_pointer_aliasing.das Adds currently-skipped scaffolding for future typed-pointer aliasing coverage.
modules/​dasLLVM/​tests/​llvm_jit_never_fast_math.das Refactors tests to use shared IR-dump/helpers infrastructure.
modules/​dasLLVM/​tests/​llvm_jit_global_lookup.das Extends/refactors tests validating global/shared-global lookup hoisting behavior.
modules/​dasLLVM/​tests/​llvm_jit_debug_info.das Refactors tests to shared helpers and adjusts debug/symbol expectations.
modules/​dasLLVM/​tests/​_remarks_root.das Adds a fixture program intended to emit LLVM optimization remarks.
modules/​dasLLVM/​tests/​_ptr_diff_root.das Adds a fixture program for pointer-difference lowering inspection.
modules/​dasLLVM/​tests/​_pointer_aliasing_root.das Adds a fixture program for typed-pointer aliasing scenarios (future/placeholder coverage).
modules/​dasLLVM/​tests/​_global_lookup_root.das Extends fixture to include loop/branch global and shared-global lookup patterns.
modules/​dasLLVM/​tests/​_exe_test_common.das Introduces/updates shared helpers for JIT execution, IR dumping, and IR parsing in tests.
modules/​dasLLVM/​REVIEW.md Updates dasLLVM module review/checklist documentation.
modules/​dasLLVM/​daslib/​llvm_jit.das Implements core JIT lowering fixes (reinterpret behavior, fixed-array default zeroing, and return-in-try/recover refusal).
modules/​dasLLVM/​daslib/​llvm_jit_run.das Adjusts runtime behavior to ensure remark-enabled runs force recompilation (bypass caches).
modules/​dasLLVM/​daslib/​llvm_jit_plan.das Tracks remarks enablement and forces codegen to run past caches when requested.
modules/​dasLLVM/​daslib/​llvm_jit_link.das Routes remark-enabled runs through paths that bypass cached artifacts as intended.
modules/​dasLLVM/​daslib/​llvm_jit_intrin.das Adds intrinsic mapping/implementation for inline i_das_ptr_diff.
modules/​dasLLVM/​daslib/​llvm_jit_common.das Wires LLVM pass-remarks options and applies extern attributes (including speculatable) to selected calls.
modules/​dasLLVM/​daslib/​llvm_jit_cli.das Adds CLI flags: --jit-remarks=, --jit-remarks-missed=, --jit-remarks-analysis=.
modules/​dasLLVM/​CMakeLists.txt Registers newly added/updated dasLLVM tests in the module build configuration.
modules/​dasLLVM/​ARCHITECTURE.md Documents the remarks flags and updated JIT behavior/intent (including lookup hoisting behavior).

Review effort: Balanced

Copilot AI balanced review requested due to automatic review settings November 7, 2026 06:14

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Note

Copilot was unable to complete this review at the selected tier due to model policy or plan restrictions.

🔵 Needs a closer look

This change alters correctness/performance-sensitive LLVM JIT emitter IR/attribute semantics and related CLI/cache behavior in ways that require cross-target human validation beyond the coverage of the added tests and static diff review.

No open issues 🎉

Review effort: Balanced

@borisbat
borisbat force-pushed the bbatkin/lookibed-jit branch from 63938f2 to b2609cc Compare November 7, 2026 06:24
Copilot AI balanced review requested due to automatic review settings November 7, 2026 06:24

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Note

Copilot was unable to complete this review at the selected tier due to model policy or plan restrictions.

🟢 Approval recommended

All touched files were reviewed with only minor documentation-style nits noted and no unresolved functional concerns or blocking issues raised in the review evidence.

No open issues 🎉

Review effort: Balanced

@borisbat
borisbat force-pushed the bbatkin/lookibed-jit branch from b2609cc to 3390967 Compare November 7, 2026 07:05
Copilot AI balanced review requested due to automatic review settings November 7, 2026 07:05

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Note

Copilot was unable to complete this review at the selected tier due to model policy or plan restrictions.

🟡 Changes recommended

A moderate active issue remains because the implementation enforces the return-in-try/recover compile error across all compilation modes via the type inference tryDepth > 0 check, which conflicts with the PR description’s claim that it is a -jit-only behavior change and needs either description alignment or mode-specific gating.

1 open issue

Review effort: Balanced

Comment thread src/ast/ast_infer_type.cpp Resolved

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Note

Copilot was unable to complete this review at the selected tier due to model policy or plan restrictions.

🟡 Changes recommended

A documentation nit remains unresolved in ast_serializer.h where the serializer version comment mischaracterizes a behavior change (it’s now a hard compile error for return inside try/recover), which could confuse future cache-version audits.

2 open issues

Review effort: Balanced

Comment thread include/daScript/ast/ast_serializer.h Resolved
Copilot AI balanced review requested due to automatic review settings November 7, 2026 07:38

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Note

Copilot was unable to complete this review at the selected tier due to model policy or plan restrictions.

🔵 Needs a closer look

A remaining issue was identified in include/daScript/ast/ast_serializer.h where a version-note comment is misleading about the behavioral change, indicating the documentation needs correction before the change can be considered ready.

2 open issues

Review effort: Balanced

The cast emitted `bitcast i1 to i8`, which reads true as 0xff and fails the
verifier when it feeds a call. A bool source is zero-extended to the integer
width; an integer source compares its low byte with zero.

Fixes #4471

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Note

Copilot was unable to complete this review at the selected tier due to model policy or plan restrictions.

🔵 Needs a closer look

The changes span multiple cross-cutting compiler areas (type inference/control-flow semantics, AST cache/serialization versioning, and LLVM JIT codegen/CLI behavior), so correctness and performance require maintainer review and running the full multi-target test/JIT matrix beyond what static diff inspection can verify.

No open issues 🎉

2 resolved since last review

Review effort: Balanced

borisbat and others added 3 commits November 7, 2026 00:21
The JIT compiles the two arms as block functions called through
jit_try_recover, so a return in an arm left the arm and the function answered
wrong, while the interpreter returned from the function. There is no lowering
for the shape without a return slot threaded through the arm's captures, so
the front end now rejects it on every tier (error 30299, naming the fix: set a
variable in the arm and return after the pair). A closure's own return inside
the arm is the closure's and stays allowed. The flow test keeps its break and
continue cases; the return cases are gone with the semantics.

Fixes #4462

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
reinterpret<uint>(p) built `trunc ptr to i32`, which the verifier refuses;
only the 64-bit target went through ptrtoint. Every integer target does now,
and ptrtoint to a narrower width keeps the low bits, as the interpreter does.
The JIT half of #4108; the interpreter half is the reinterpret eval-slot fix
on the optimizer branch.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
The make-struct arm zeroed max(elements, 1) strides; for default<T[N]> there
are no elements and a stride is one element, so only the first element was
cleared and `a = default<int[4]>` left the rest. The arm now zeroes the
declared shape's size when no element is provided and reads the element type
through the fixed-array chain for the empty-tuple case, as the interpreter
does. Found by the regex rewrite: a set reset through default<> under -jit
kept its old bits.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
ubuntu-latest moved to 26.04, where the swift toolchain the parser test
action installs fails to load libxml2.so.2 and swift-build exits 127.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Note

Copilot was unable to complete this review at the selected tier due to model policy or plan restrictions.

🔵 Needs a closer look

The change affects high-impact core compiler semantics and LLVM JIT/codegen behaviors (including marking runtime helper globals as speculatable), which requires maintainer verification across supported platforms/configurations beyond what static diff review can safely confirm.

No open issues 🎉

Review effort: Balanced

@borisbat
borisbat merged commit 2131620 into master Nov 7, 2026
73 checks passed
@borisbat
borisbat deleted the bbatkin/lookibed-jit branch November 7, 2026 10:27
borisbat added a commit to GaijinEntertainment/tree-sitter-daslang that referenced this pull request Nov 7, 2026
…turn, inline ptr_diff, speculatable lookups, --jit-remarks (GaijinEntertainment/daScript#4472)

* jit: a reinterpret between bool and integer is the byte, not a bitcast

The cast emitted `bitcast i1 to i8`, which reads true as 0xff and fails the
verifier when it feeds a call. A bool source is zero-extended to the integer
width; an integer source compares its low byte with zero.

Fixes GaijinEntertainment/daScript#4471

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* infer: a return inside a try or recover arm is a compile error

The JIT compiles the two arms as block functions called through
jit_try_recover, so a return in an arm left the arm and the function answered
wrong, while the interpreter returned from the function. There is no lowering
for the shape without a return slot threaded through the arm's captures, so
the front end now rejects it on every tier (error 30299, naming the fix: set a
variable in the arm and return after the pair). A closure's own return inside
the arm is the closure's and stays allowed. The flow test keeps its break and
continue cases; the return cases are gone with the semantics.

Fixes GaijinEntertainment/daScript#4462

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* jit: a pointer reinterprets to any integer width through ptrtoint

reinterpret<uint>(p) built `trunc ptr to i32`, which the verifier refuses;
only the 64-bit target went through ptrtoint. Every integer target does now,
and ptrtoint to a narrower width keeps the low bits, as the interpreter does.
The JIT half of GaijinEntertainment/daScript#4108; the interpreter half is the reinterpret eval-slot fix
on the optimizer branch.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* jit: default<T[N]> assigned into a fixed array zeroes the whole array

The make-struct arm zeroed max(elements, 1) strides; for default<T[N]> there
are no elements and a stride is one element, so only the first element was
cleared and `a = default<int[4]>` left the rest. The arm now zeroes the
declared shape's size when no element is provided and reads the element type
through the fixed-array chain for the empty-tuple case, as the interpreter
does. Found by the regex rewrite: a set reset through default<> under -jit
kept its old bits.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* jit: a pointer difference is inline arithmetic, not a slot call

`p - q` called i_das_ptr_diff through the builtin's slot with the stride as a
run-time argument, so the division never folded; the intrinsic row emits
ptrtoint, sub and a division by the constant stride. The issue's loop goes
from 0.32 s to 0.0045 s.

Fixes GaijinEntertainment/daScript#4469

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* jit: a global lookup is speculatable and hoists out of a loop

jit_get_global_mnh depends on the name hash and the context alone and cannot
trap, so it carries `speculatable`; LLVM then hoists a lookup under a branch
inside a loop to the preheader. The issue's loop goes from 0.405 s to 0.042 s,
beside 0.035 s for the solid_context build.

Fixes GaijinEntertainment/daScript#4467

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* jit: --jit-remarks flags print LLVM's optimization remarks

--jit-remarks=, --jit-remarks-missed= and --jit-remarks-analysis= take a pass
regex, set LLVM's -pass-remarks options once per process and force codegen past
the DLL and object caches; under --jit-debug each remark carries the .das file
and line.

Fixes GaijinEntertainment/daScript#4470

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* jit: the typed-pointer aliasing check lands skipped

The emitter attaches no alias information to typed pointers, by the language's
own rule that a pointer is an address; the test documents the shape and stays
skipped behind a flag.

Refs GaijinEntertainment/daScript#4446

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* review: two checklists name their triggers and share one rule format

The tests checklist's AOT-run rule described what test_aot compiles in a way
that read as an example, and its opt-out rule named two of the four ways a
file leaves the AOT set; a test that starts a daslang child now has its cache
flags spelled out instead of sending the reader into the host's source. The
dasLLVM checklist's routing rules take the list form the rest of the file
uses, its fused tune rule splits into its two triggers, the child-artifact
rule accepts a working directory inside the test's own directory, and the
twinned-emitter definition is the closed list with a scalar criterion.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* review: the version-bump rule covers a pass that rejects source

A pass that newly rejects a shape leaves a cached module untouched, so
the src/ast checklist's getVersion rule names that trigger beside the
output change it already named. The AST-headers checklist cites its
architecture section by anchor, drops the version-comment rule the root
CLAUDE.md already carries, and narrows the binding-name rule to the names
ast_handle.h itself computes.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* ci: the tree-sitter parser test runs on ubuntu-24.04

ubuntu-latest moved to 26.04, where the swift toolchain the parser test
action installs fails to load libxml2.so.2 and swift-build exits 127.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

---------

Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
daScript-Commit: 21316206e9773ac0f07fa1fb060c4edab199c47c
This was referenced Nov 9, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants