fix(agent_sdk): make generated Express parser reproducible - #2371
Conversation
There was a problem hiding this comment.
Code Review
This pull request updates the Python SDK build process to invoke ANTLR from the grammar's directory, preventing absolute paths from being embedded in the generated files. It also fixes an import rewrite issue in the generated Express visitor and updates the README and CHANGELOG. A review comment suggests using os.path.abspath for the output directory in pack_specs_hook.py to ensure correct resolution when the working directory is changed.
| "-o", | ||
| "generated", | ||
| g4_path, | ||
| generated_dir, |
There was a problem hiding this comment.
When cwd is changed to os.path.dirname(g4_path) in subprocess.run, any relative path passed to the -o option of ANTLR will be resolved relative to that new working directory. If project_root (and thus generated_dir) is a relative path, ANTLR will write the generated files to an incorrect directory relative to the grammar directory instead of the project root. Converting generated_dir to an absolute path using os.path.abspath ensures it is resolved correctly regardless of the current working directory.
| generated_dir, | |
| os.path.abspath(generated_dir), |
There was a problem hiding this comment.
Good catch, applied. generated_dir was absolute in practice, since it comes from hatchling's self.root, but moving cwd to the grammar directory is what made the code depend on that, so making it explicit is right.
Re-checked after the change: uv sync --reinstall-package a2ui-agent-sdk still leaves git status empty, the generated headers still read Express.g4, and pytest in eval/ is unchanged at 127 passed.
The build hook passed ANTLR an absolute path to Express.g4, and ANTLR writes its input path into the header of every generated file. The committed output therefore carried the absolute path of whichever machine last regenerated it, so any from-source build left express_lexer.py, express_parser.py and express_visitor.py modified with a header-only diff. Invoke ANTLR from the grammar's own directory with a bare filename, which keeps the output flat and makes the header independent of checkout location. Regenerates the three committed files accordingly. Also documents that a JDK is required to build from source, since the ANTLR step needs one and that was not written down anywhere.
47aa99d to
53c8a6f
Compare
jacobsimionato
left a comment
There was a problem hiding this comment.
Thank you for this excellent change!
Description
Fixes #2369.
A from-source build of the SDK left three generated files modified, for two independent reasons.
ANTLR writes the path it is given into the header of everything it generates, and the hook passed it an absolute path, so the committed output recorded whichever machine last regenerated it. This invokes ANTLR from the grammar's own directory with a bare filename, which makes the header
Express.g4regardless of checkout location.-ois absolute becausecwdis now the grammar directory. A relative-owould write next to the grammar rather than into the package. A relative input path would also get mirrored underneath-o, which a bare filename avoids.The second reason is the visitor's fallback import. The rename step leaves no
ExpressParser.pyon disk, but the post-processing rewrote the non-package fallback to a relative import anyway, so a build always disagreed with the committed file:The
elsebranch only runs when there is no package to be relative to, so the previous rewrite made it raiseImportError: attempted relative import with no known parent package. This keeps the bare form committed onmain.The three generated files are regenerated accordingly. Their diff is the header line, plus one blank line in the visitor that the generator does not emit.
pyproject.toml:99already excludes/generated/from pyink, so the formatter does not fight the generator over it.Also adds a line to the SDK README noting that the ANTLR step needs a Java runtime, which I could not find documented.
Verification
uv sync --reinstall-package a2ui-agent-sdkfollowed bygit status --shortis empty. Before it, three files were modified.cd eval && uv run python -m pytest: 127 passed. TwoTestArchiverfailures remain. They are pre-existing onmainand unrelated, filed as [BUG]: archive_run writes outside the repository when no repo root is detected #2370../scripts/fix_format.sh --check: clean for prettier, pyink and ktfmt. Dart and swift-format are not installed locally and skip.Note on #2251
That PR touches the same function about five lines away, making ANTLR failure non-fatal. I merged this branch against its head and git resolves both cleanly, so neither should block the other.
Pre-launch Checklist
One time:
For this PR: