linux: make the read-deny glob walk about six times faster - #627
Merged
Merged
Conversation
A denyRead pattern with `**` is expanded by listing everything under its base, at every command and on the caller's thread. Under a large directory that holds the caller up for seconds, once per pattern. For every entry the walk spelled two paths and converted one before it asked whether the entry matched. A pattern that splits into path components is matched by name, so a plain file it does not match, which is nearly every entry of a large tree and which nothing further on records, now costs no path at all. The walk already kept each directory's listing for a second position to read. The patterns of one read configuration now share that map, so those with a base in common list each directory once between them. The map lives for one expansion: the next command lists afresh. Six patterns over 400,000 files in 20,000 directories: one wrap 6.2 s before, 0.97 s after; resident memory 469 MB before, 286 MB after; the same mounts.
ronleizrowice-ant
added a commit
that referenced
this pull request
Sep 30, 2026
Both it and the walk's budget (#607) hand every pattern of one read configuration something to share, so the listings go where the budget already is: made in readDenyGlobExpander(), and handed on in the options of expandReadDenyGlobLinux(). The walk's record of a directory gives up its own copy of the entries for the shared map. An entry is paid for before it is skipped: the budget bounds the entries looked at, and one the fast path passes over was looked at too.
ronleizrowice-ant
added a commit
that referenced
this pull request
Sep 30, 2026
The walk's budget (#607), its shared listings (#627) and its steps (#628) rewrite one code path, and the path entries of #620 to #622 sit on top of it. Everything above the walk is a generator now: the read-deny expansion, the function readDenyGlobExpander() returns, the allowRead expansion, the resolution of read path entries, and withOtherReadings(), which calls the expander once for each reading of an entry. The synchronous names finish those on the spot. In the walk the step comes first and the budget's clock is read after it, before the next listing. The deadline is wall time, so turns given to other work count against it: a wrap can be refused earlier for that, and a list is never cut short. The budget error is given its code around the delegated generator. An abort is raised by the driver and does not pass through there, so a wrap with a spent budget and an aborted signal rejects with the signal's reason. A wrap that starts over makes a new budget and new listings, and reads the new configuration's denyReadGlobBudget. In the scan's catch the abort is looked at first, then what ripgrep listed.
antdres
approved these changes
Sep 30, 2026
antdres
reviewed
Sep 30, 2026
antdres
left a comment
antdres
left a comment
Collaborator
There was a problem hiding this comment.
No blocking issues. I compared this branch with its base on a few thousand random trees, including symlinks and unreadable directories, and in a real bubblewrap run. The denied paths are identical and the walk is about 4x faster here.
Generated by Claude Code
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
On Linux a
denyReadpattern with**is expanded by listing everything under its base, at every command and on the caller's thread. Under a large directory (~/**/.env) that holds the caller up for seconds, once per pattern. This makes the walk about six times faster and denies exactly what it denied before.What changes
continue. Nothing further on records such an entry.getFsReadConfig()and the wrap now hand one map to all the patterns of a read configuration. The map lives for one expansion: the next command lists afresh.Measured
Six patterns over 400,000 files in 20,000 directories, one wrap:
continueonlyThe mounts are the same in all three.
Known limits
Testing
main: a sandboxedcatof every file in a tree with symlinks, an unreadable directory, directory-form and unsplittable patterns gives identical results. Files made between two wraps of one process are denied at the next.