variant: optimize variant object path access (79% lower latency)#10358
Open
friendlymatthew wants to merge 2 commits into
Open
variant: optimize variant object path access (79% lower latency)#10358friendlymatthew wants to merge 2 commits into
friendlymatthew wants to merge 2 commits into
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Rationale for this change
Based on benchmark #10357
This PR speeds up field extraction from unshredded variant objects by avoiding repeated field name decoding for every row
Variant object keys are stored as numeric field ids that map to a metadata dictionary.
variant_getcurrently stores a named path by repeatedly mapping those ids back to strings and comparing the strings for each row, even when a batch shares one metadata dictionary. This repeated per-row work dominates when dealing with wide dictionariesThis optimization adds metadata scoped field id lookups, caches parsed metadata, and resolves named path elements to ids once per metadata dictionary before scanning rows. It re-resolves paths when the metadata changes and retains name lookup for unsorted dictionaries