Repository navigation
Invalidations 3/6: drop the to_indices method for RangeIndex - #340
Draft
asinghvi17 wants to merge 1 commit into
Draft
asinghvi17 wants to merge 1 commit into
asinghvi17 wants to merge 1 commit into
Conversation
Delete to_indices(::Union{DenseArray,AbstractRange}, inds, ::Tuple{RangeIndex,Vararg}).
RangeIndex <: AbstractVector{Int}, so the method matched any Base array or
range indexed with an AbstractVector{Int} first, widened the inferred result
of those to_indices calls, and invalidated compiled callers such as
DataFrames' view(::Vector{Symbol}, ::AbstractVector{Int}).
Its job was to give DiskArrays' temp arrays a Vector{Int} to index with.
process_index now builds that Vector{Int} directly from the shifted ranges,
which also avoids the O(log n) per element of collecting a RangeIndex. Base
arrays indexed with a user RangeIndex take Base's generic AbstractVector path
and return the same values.
The test that asserted the Vector conversion now checks the user-facing
result of indexing Base arrays and ranges with a RangeIndex, and that the
temp-array indices are a Vector{Int}.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019Dfq6aDr3GqA8w61TsvMsc
asinghvi17
added this pull request to stack #342
October 8, 2026 14:03
to_indices method for RangeIndex on Base arrays
to_indices method for RangeIndex on Base arraysto_indices method for RangeIndex
to_indices method for RangeIndexto_indices method for RangeIndex
mkitti
reviewed
Oct 9, 2026
mkitti
left a comment
Member
There was a problem hiding this comment.
Looks good to me, with one performance trade-off worth stating in the description. I checked out the branch: the full suite passes on Julia 1.13, and the new shifted_indices and process_index assertions match what I see in the code.
Measurements (Julia 1.13, Chairmarks, #339 → #340)
| case | #339 | #340 |
|---|---|---|
v[r], Vector{Float64}, 100k ranges × 5 |
11.7 ms | 11.6 ms |
v[r], 2 long ranges (1M elements) |
2.72 ms | 2.15 ms |
m[r, :], in-memory 2000×1000 matrix, 100 ranges × 5 |
0.23 ms | 3.6 ms |
ChunkedDiskArray read with 1000 ranges (batched) |
0.18 ms | 0.071 ms |
- The disk path is about 2.5× faster here (the description reports 3–4× on its machine), so that claim holds.
- Vector indexing is unchanged.
- Matrix indexing with a
RangeIndexis about 15× slower.RangeIndex[k]is anO(log n)searchsortedfirst, and Base's multi-dimensionalgetindexloop now calls it once per element per column, where the removedto_indicesmethod collected to aVector{Int}first. That collect is exactly the case the deleted comment described ("In-memory arrays and ranges index fastest with a plainVector{Int}"). - It is probably an uncommon use of
RangeIndex, and it may be an acceptable price for removing the method. But it is a trade-off, so please say so in the description and the CHANGELOG (there is none in this PR), rather than only reporting the disk speedup.
Invalidations
- On 1.13,
uinvalidatedafterusing DiskArraysis unchanged by this PR in my runs (574 → 574 clean, 1003 → 1003 with DataFrames loaded). That matches the description's own table, where 1.13 changes only in the "many packages loaded" session. I could not reproduce a gain with DataFrames alone. - Maybe name the package that triggers the 80 invalidations in the many-packages session, so reviewers can verify.
@test !any(m -> m.module === DiskArrays, methods(Base.to_indices))would guard against the method coming back; the current tests check the values and do not catch that.- The
only(only(di.temparray_indices)) == ([1, 2, 6, 7, 8],)assertions depend on the internalDiskIndexlayout. That is fine for now, but they will break on any refactor of it.
Not blocking.
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Part 3/6, on #339.
Deletes
to_indices(::Union{DenseArray,AbstractRange}, _, ::Tuple{RangeIndex,Vararg}).RangeIndex <: AbstractVector{Int}, so the method matched every Base array indexed with anAbstractVector{Int}, such as DataFrames'view. DiskArrays now builds its temp-array indices as aVector{Int}directly.RangeIndexgives the same values.to_indicesnow returns theRangeIndexitself; the test checks values instead.RangeIndexare 3–4× faster (read 4.0 → 1.2 ms on 1.13).uinvalidatedafterusing DiskArrays(#339 → this PR):🤖 Generated with Claude Code
https://claude.ai/code/session_019Dfq6aDr3GqA8w61TsvMsc