Skip to content

fix(agent_sdk): make generated Express parser reproducible - #2371

Merged
jacobsimionato merged 3 commits into
a2ui-project:mainfrom
FelippeRoza:fix/antlr-reproducible-header
Sep 4, 2026
Merged

jacobsimionato merged 3 commits into
a2ui-project:mainfrom
FelippeRoza:fix/antlr-reproducible-header

Conversation

@FelippeRoza

Copy link
Copy Markdown
Contributor

Description

Fixes #2369.

A from-source build of the SDK left three generated files modified, for two independent reasons.

ANTLR writes the path it is given into the header of everything it generates, and the hook passed it an absolute path, so the committed output recorded whichever machine last regenerated it. This invokes ANTLR from the grammar's own directory with a bare filename, which makes the header Express.g4 regardless of checkout location.

                 "-o",
-                "generated",
-                g4_path,
+                generated_dir,
+                os.path.basename(g4_path),
             ]
...
             res = subprocess.run(
                 cmd,
-                cwd=express_dir,
+                cwd=os.path.dirname(g4_path),

-o is absolute because cwd is now the grammar directory. A relative -o would write next to the grammar rather than into the package. A relative input path would also get mirrored underneath -o, which a bare filename avoids.

The second reason is the visitor's fallback import. The rename step leaves no ExpressParser.py on disk, but the post-processing rewrote the non-package fallback to a relative import anyway, so a build always disagreed with the committed file:

                 content = content.replace(
-                    "from ExpressParser import", "from .express_parser import"
+                    "from ExpressParser import", "from express_parser import"
                 )

The else branch only runs when there is no package to be relative to, so the previous rewrite made it raise ImportError: attempted relative import with no known parent package. This keeps the bare form committed on main.

The three generated files are regenerated accordingly. Their diff is the header line, plus one blank line in the visitor that the generator does not emit. pyproject.toml:99 already excludes /generated/ from pyink, so the formatter does not fight the generator over it.

Also adds a line to the SDK README noting that the ANTLR step needs a Java runtime, which I could not find documented.

Verification

  • After this change, uv sync --reinstall-package a2ui-agent-sdk followed by git status --short is empty. Before it, three files were modified.
  • cd eval && uv run python -m pytest: 127 passed. Two TestArchiver failures remain. They are pre-existing on main and unrelated, filed as [BUG]: archive_run writes outside the repository when no repo root is detected #2370.
  • ./scripts/fix_format.sh --check: clean for prettier, pyink and ktfmt. Dart and swift-format are not installed locally and skip.

Note on #2251

That PR touches the same function about five lines away, making ANTLR failure non-fatal. I merged this branch against its head and git resolves both cleanly, so neither should block the other.

Pre-launch Checklist

One time:

For this PR:

  • I have updated the relevant CHANGELOG.md file.
  • I updated/added relevant documentation.
  • My code changes (if any) have tests. Asserting on the generated header would mean invoking ANTLR from the test suite, which needs a Java runtime. Verified by hand as above. Happy to add one if you want it.
  • If my branch is on a fork, I have verified that scripts/e2e_test.sh passes. Needs Flutter and a Gemini API key, and this change does not touch that path.

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request updates the Python SDK build process to invoke ANTLR from the grammar's directory, preventing absolute paths from being embedded in the generated files. It also fixes an import rewrite issue in the generated Express visitor and updates the README and CHANGELOG. A review comment suggests using os.path.abspath for the output directory in pack_specs_hook.py to ensure correct resolution when the working directory is changed.

"-o",
"generated",
g4_path,
generated_dir,

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

high

When cwd is changed to os.path.dirname(g4_path) in subprocess.run, any relative path passed to the -o option of ANTLR will be resolved relative to that new working directory. If project_root (and thus generated_dir) is a relative path, ANTLR will write the generated files to an incorrect directory relative to the grammar directory instead of the project root. Converting generated_dir to an absolute path using os.path.abspath ensures it is resolved correctly regardless of the current working directory.

Suggested change
generated_dir,
os.path.abspath(generated_dir),

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good catch, applied. generated_dir was absolute in practice, since it comes from hatchling's self.root, but moving cwd to the grammar directory is what made the code depend on that, so making it explicit is right.

Re-checked after the change: uv sync --reinstall-package a2ui-agent-sdk still leaves git status empty, the generated headers still read Express.g4, and pytest in eval/ is unchanged at 127 passed.

The build hook passed ANTLR an absolute path to Express.g4, and ANTLR writes
its input path into the header of every generated file. The committed output
therefore carried the absolute path of whichever machine last regenerated it,
so any from-source build left express_lexer.py, express_parser.py and
express_visitor.py modified with a header-only diff.

Invoke ANTLR from the grammar's own directory with a bare filename, which
keeps the output flat and makes the header independent of checkout location.
Regenerates the three committed files accordingly.

Also documents that a JDK is required to build from source, since the ANTLR
step needs one and that was not written down anywhere.
@FelippeRoza
FelippeRoza force-pushed the fix/antlr-reproducible-header branch from 47aa99d to 53c8a6f Compare August 24, 2026 15:50
@github-actions github-actions Bot added the status: needs-triage auto-managed: https://github.com/a2ui-project/a2ui/blob/main/scripts/triage.mjs label Aug 25, 2026

@jacobsimionato jacobsimionato left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you for this excellent change!

@jacobsimionato
jacobsimionato enabled auto-merge (squash) September 1, 2026 05:07
@github-actions github-actions Bot removed the status: needs-triage auto-managed: https://github.com/a2ui-project/a2ui/blob/main/scripts/triage.mjs label Sep 1, 2026
@jacobsimionato
jacobsimionato merged commit 0dc4a30 into a2ui-project:main Sep 4, 2026
34 checks passed
@github-project-automation github-project-automation Bot moved this from Todo to Done in A2UI Sep 4, 2026
@FelippeRoza
FelippeRoza deleted the fix/antlr-reproducible-header branch September 4, 2026 09:24
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[BUG]: from-source build cannot reproduce the committed Express parser

2 participants