Skip to content

Invalidations 3/6: drop the to_indices method for RangeIndex - #340

Draft
asinghvi17 wants to merge 1 commit into
inv/broadcast-unwrapfrom
inv/rangeindex
Draft

asinghvi17 wants to merge 1 commit into
inv/broadcast-unwrapfrom
inv/rangeindex

Conversation

@asinghvi17

@asinghvi17 asinghvi17 commented Oct 8, 2026 •

Copy link
Copy Markdown
Member

Part 3/6, on #339.

Deletes to_indices(::Union{DenseArray,AbstractRange}, _, ::Tuple{RangeIndex,Vararg}). RangeIndex <: AbstractVector{Int}, so the method matched every Base array indexed with an AbstractVector{Int}, such as DataFrames' view. DiskArrays now builds its temp-array indices as a Vector{Int} directly.

  • Indexing Base arrays with a RangeIndex gives the same values. to_indices now returns the RangeIndex itself; the test checks values instead.
  • Batched disk reads and writes with a RangeIndex are 3–4× faster (read 4.0 → 1.2 ms on 1.13).

uinvalidated after using DiskArrays (#339 → this PR):

session Julia 1.13 Julia 1.12
clean 153 → 153 503 → 503
CairoMakie session 2826 → 2826 3238 → 3238
many packages loaded 1785 → 1705 1845 → 1765
DataFrames session 807 → 807 1506 → 1506

🤖 Generated with Claude Code

https://claude.ai/code/session_019Dfq6aDr3GqA8w61TsvMsc

Delete to_indices(::Union{DenseArray,AbstractRange}, inds, ::Tuple{RangeIndex,Vararg}).
RangeIndex <: AbstractVector{Int}, so the method matched any Base array or
range indexed with an AbstractVector{Int} first, widened the inferred result
of those to_indices calls, and invalidated compiled callers such as
DataFrames' view(::Vector{Symbol}, ::AbstractVector{Int}).

Its job was to give DiskArrays' temp arrays a Vector{Int} to index with.
process_index now builds that Vector{Int} directly from the shifted ranges,
which also avoids the O(log n) per element of collecting a RangeIndex. Base
arrays indexed with a user RangeIndex take Base's generic AbstractVector path
and return the same values.

The test that asserted the Vector conversion now checks the user-facing
result of indexing Base arrays and ranges with a RangeIndex, and that the
temp-array indices are a Vector{Int}.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019Dfq6aDr3GqA8w61TsvMsc
@asinghvi17
asinghvi17 added this pull request to stack #342 October 8, 2026 14:03
@asinghvi17 asinghvi17 changed the title inv/rangeindex Invalidations 3/4: drop the to_indices method for RangeIndex on Base arrays Oct 8, 2026
@asinghvi17 asinghvi17 changed the title Invalidations 3/4: drop the to_indices method for RangeIndex on Base arrays Invalidations 3/4: drop the to_indices method for RangeIndex Oct 8, 2026
@asinghvi17 asinghvi17 changed the title Invalidations 3/4: drop the to_indices method for RangeIndex Invalidations 3/6: drop the to_indices method for RangeIndex Oct 8, 2026

@mkitti mkitti left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good to me, with one performance trade-off worth stating in the description. I checked out the branch: the full suite passes on Julia 1.13, and the new shifted_indices and process_index assertions match what I see in the code.

Measurements (Julia 1.13, Chairmarks, #339 → #340)

case #339 #340
v[r], Vector{Float64}, 100k ranges × 5 11.7 ms 11.6 ms
v[r], 2 long ranges (1M elements) 2.72 ms 2.15 ms
m[r, :], in-memory 2000×1000 matrix, 100 ranges × 5 0.23 ms 3.6 ms
ChunkedDiskArray read with 1000 ranges (batched) 0.18 ms 0.071 ms
  • The disk path is about 2.5× faster here (the description reports 3–4× on its machine), so that claim holds.
  • Vector indexing is unchanged.
  • Matrix indexing with a RangeIndex is about 15× slower. RangeIndex[k] is an O(log n) searchsortedfirst, and Base's multi-dimensional getindex loop now calls it once per element per column, where the removed to_indices method collected to a Vector{Int} first. That collect is exactly the case the deleted comment described ("In-memory arrays and ranges index fastest with a plain Vector{Int}").
  • It is probably an uncommon use of RangeIndex, and it may be an acceptable price for removing the method. But it is a trade-off, so please say so in the description and the CHANGELOG (there is none in this PR), rather than only reporting the disk speedup.

Invalidations

  • On 1.13, uinvalidated after using DiskArrays is unchanged by this PR in my runs (574 → 574 clean, 1003 → 1003 with DataFrames loaded). That matches the description's own table, where 1.13 changes only in the "many packages loaded" session. I could not reproduce a gain with DataFrames alone.
  • Maybe name the package that triggers the 80 invalidations in the many-packages session, so reviewers can verify.
  • @test !any(m -> m.module === DiskArrays, methods(Base.to_indices)) would guard against the method coming back; the current tests check the values and do not catch that.
  • The only(only(di.temparray_indices)) == ([1, 2, 6, 7, 8],) assertions depend on the internal DiskIndex layout. That is fine for now, but they will break on any refactor of it.

Not blocking.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants