Repository navigation
Invalidations 4/6: reach reduction hooks through Base internals - #341
Draft
asinghvi17 wants to merge 6 commits into
Draft
asinghvi17 wants to merge 6 commits into
asinghvi17 wants to merge 6 commits into
Conversation
A method on public any(f::Function, ::AbstractDiskArray) invalidates every compiled any(f, ::Any) call; on Julia 1.12 that is 553 MethodInstances in a clean session, through the poorly inferred any call in Compiler's compileable_specialization. On 1.11+ hook Base's internal _any/_all(f, itr, ::Colon) instead, which costs nothing there and also lets callable structs read chunk by chunk. On 1.10 that hook invalidates LinearAlgebra's _any(f∘transpose, ::Any, :) (~70), while the public methods cost 4, so 1.10 keeps the public methods. Also hook _any/_all(f, a, dims), so any(f, a; dims) and any(a; dims) keep reaching diskarrays_any_impl/diskarrays_all_impl with dims, as with the keyword methods before. On 1.10, dims excludes Colon to stay unambiguous with Base's _any(f, itr, ::Colon). Measured at no extra cost on 1.10, 1.12 and 1.13. The default dims implementation now calls Base's generic _any/_all(f, A, dims) directly (any(f, A; dims) would come back to the hook), which forwards to mapreduce as before. Supersedes #335 (thanks @Beforerr for the _any/_all hook). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019Dfq6aDr3GqA8w61TsvMsc
sum/prod/minimum/maximum/extrema(f, a; kwargs...) add a Core.kwcall method, which invalidates compiled keyword calls such as GeometryBasics' maximum(f, faces; init=0) (130-225 MethodInstances in every Makie session). Without the keyword methods, Base's methods take dims and init and forward to mapreduce(f, op, a; dims, init), which reaches the backend through diskarrays_mapreduce_impl. The calls without keywords still go through diskarrays_<f>_impl. This also fixes init, which threw a MethodError for sum, prod, minimum and maximum (and count(a; init)) because the _impl hooks only accept dims. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019Dfq6aDr3GqA8w61TsvMsc
…arly For sum, prod, maximum, minimum, extrema, count, any and all, with and without f, with dims=1, dims=(1, 2) and init, compare against Array and check with a counting backend which diskarrays_*_impl hook the call reaches: the reduction's own hook without keywords (and for any/all with dims), the mapreduce hook with dims or init. Check that any/all read chunk by chunk and stop at the first chunk that decides the result, also for callable structs on Julia 1.11+. Fold in the early-stopping test from #335 (@Beforerr). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019Dfq6aDr3GqA8w61TsvMsc
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019Dfq6aDr3GqA8w61TsvMsc
Every public sum/prod/minimum/maximum/extrema method with `dims` calls Base's internal `_sum(f, a, dims; kw...)` or its sibling, and count calls `_count(f, a, dims, init)`. Methods on those internals send `dims`, with or without `init`, to `diskarrays_<f>_impl` as the compute-backend branch did, and `count` now reaches `diskarrays_count_impl` with `dims` and `init` too. `dims` is typed to exclude `Colon`. A keyword method for the whole array would add a `Core.kwcall` method that invalidates GeometryBasics' `maximum(f, faces; init=0)` in every Makie session, so whole-array calls keep the public methods without keywords and `init` reaches `mapreduce`. The defaults call Base's generic `_sum(f, A, dims)`, since `sum(f, A; dims)` would come back to the hook. The test now expects each form's own hook. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019Dfq6aDr3GqA8w61TsvMsc
`sum`/`prod`/`maximum`/`minimum`/`extrema(f, a; init)` and `(a; init)` call `Base._sum(f, a, :; init)` and its siblings. A `Core.kwcall` method on `_sum(f, ::AbstractDiskArray, ::Colon)` sends them to `diskarrays_<f>_impl(f, a, backend; init)` instead of `diskarrays_mapreduce_impl`. Positional whole-array calls keep the public no-keyword methods. The keyword-call method invalidates compiled `maximum(f, v; init)` callers such as GeometryBasics': +13 MethodInstances in the CairoMakie session and +9 in the many-package session on Julia 1.13 (+6 and +6 on 1.12). Clean sessions and the DataFrames session are unchanged. The counting-backend test now asserts the own hook for whole-array `init` with and without `f`, for every reduction. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019Dfq6aDr3GqA8w61TsvMsc
asinghvi17
added this pull request to stack #342
October 8, 2026 14:03
any/all entry point; reductions reach their own hooks through Base's internal _sum/…
any/all entry point; reductions reach their own hooks through Base's internal _sum/…
mkitti
reviewed
Oct 9, 2026
mkitti
left a comment
Member
There was a problem hiding this comment.
Looks good to me. The approach is sound and I could not break it, but it leans on Base internals, so see comment 1.
What I ran
- The full test suite on Julia 1.10, 1.11, 1.12 and 1.13: all pass.
- A battery of 36 reduction forms against an in-memory array, on #340 and on this PR, on 1.13 and 1.10:
sum/prod/maximum/minimum/extrema/count/any/all, each with a function,dims,initand combinations, a callable struct, aview,mapreduceandreduce.- Values and read counts are identical to #340 everywhere except the 8
init=forms (sum,sum(f, …),sumwithdims+init,prod,maximum,maximum(f, …),minimumwithdims+init,count). Those throwMethodError: no method matching diskarrays_*_implon #340 and are correct with 16 reads here, so the claimed fix is real. - 1.10 and 1.13 give identical output for every case on this PR.
- Short-circuiting is preserved:
all(<(-90), a)reads 1 of 16 chunks,any(>(250), a)reads 13.
- Values and read counts are identical to #340 everywhere except the 8
uinvalidatedafterusing DiskArrayson 1.13, #340 → this PR: 574 → 572 in a clean session and 1003 → 982 with DataFrames loaded.
Comments
- It depends on Base internals. The PR adds methods to
Base._sum,_prod,_minimum,_maximum,_extrema,_any,_all,_count, and aCore.kwcallmethod for theinitwhole-array case. All exist on 1.10–1.13, but nothing guarantees they keep their signatures. The per-form hook tests would fail loudly if one changed, and the nightly job in the CI matrix is the early warning; maybe say that in a comment next to theCore.kwcallblock, since it is the least obvious one. - The
inithooks cost +4 invalidations in Makie-like sessions (GeometryBasics'maximum(f, faces; init=0)). The description andbackend.mdboth say so, which I appreciate. The trade is reasonable becauseinit=was throwing before, but if that count ever matters, that block is the one to gate. - FYI, not this PR:
Project.tomldeclaresjulia = "1.9", but on 1.9.4maximum(a; dims=1)andsum(view(a, …))already throwUndefVarError: rm_singleton_indicesonmainand here (CI starts at 1.10). Theinitforms work on 1.9 with this PR. - Docs: the table makes it clear that a callable struct
freachesdiskarrays_mapreduce_implrather than the per-reduction hook, and that on 1.10 non-Functioncallables iterate. That is a real inconsistency betweenFunctionand callablef, and it is documented. A one-line CHANGELOG entry (there is none) would help backend authors, since thedims=:, kwargs...hook signatures changed.
Not blocking.
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Part 4/6, on #340. Supersedes #335; thanks @Beforerr.
Moves the reductions off public Base methods. #316's backend hooks stay reachable without invalidating Base code:
any/all: on 1.11+, hookBase._any/_all(f, ::AbstractDiskArray, dims). On 1.10, keep the publicany(f::Function, a), because the_anyhook would invalidate LinearAlgebra code there. Per-chunk short-circuiting is unchanged.sum,prod,maximum,minimum,extremaandcount: drop; kwargs...from the public methods.dimsandinitreach each reduction's owndiskarrays_*_implthrough Base's internal_sumand friends. Tests assert this for every form.init=for these reductions, which threw aMethodErroron Add options to extend compute backends #316.uinvalidatedafterusing DiskArrays(#340 → this PR):The 1.12 clean drop comes from a compiler call site that 1.13 already fixes (JuliaLang/julia#62491). The CairoMakie and many-packages rows rise by 4 because the whole-array
inithooks collide with GeometryBasics'maximum(f, faces; init=0).🤖 Generated with Claude Code
https://claude.ai/code/session_019Dfq6aDr3GqA8w61TsvMsc