Skip to content

Invalidations 4/6: reach reduction hooks through Base internals - #341

Draft
asinghvi17 wants to merge 6 commits into
inv/rangeindexfrom
inv/any-all-hook
Draft

asinghvi17 wants to merge 6 commits into
inv/rangeindexfrom
inv/any-all-hook

Conversation

@asinghvi17

@asinghvi17 asinghvi17 commented Oct 8, 2026 •

Copy link
Copy Markdown
Member

Part 4/6, on #340. Supersedes #335; thanks @Beforerr.

Moves the reductions off public Base methods. #316's backend hooks stay reachable without invalidating Base code:

  • any/all: on 1.11+, hook Base._any/_all(f, ::AbstractDiskArray, dims). On 1.10, keep the public any(f::Function, a), because the _any hook would invalidate LinearAlgebra code there. Per-chunk short-circuiting is unchanged.
  • sum, prod, maximum, minimum, extrema and count: drop ; kwargs... from the public methods. dims and init reach each reduction's own diskarrays_*_impl through Base's internal _sum and friends. Tests assert this for every form.
  • Fixes init= for these reductions, which threw a MethodError on Add options to extend compute backends #316.

uinvalidated after using DiskArrays (#340 → this PR):

session Julia 1.13 Julia 1.12
clean 153 → 151 503 → 141
CairoMakie session 2826 → 2830 3238 → 3242
many packages loaded 1705 → 1709 1765 → 1769
DataFrames session 807 → 786 1506 → 1111

The 1.12 clean drop comes from a compiler call site that 1.13 already fixes (JuliaLang/julia#62491). The CairoMakie and many-packages rows rise by 4 because the whole-array init hooks collide with GeometryBasics' maximum(f, faces; init=0).

🤖 Generated with Claude Code

https://claude.ai/code/session_019Dfq6aDr3GqA8w61TsvMsc

asinghvi17 and others added 6 commits October 8, 2026 00:08
A method on public any(f::Function, ::AbstractDiskArray) invalidates every
compiled any(f, ::Any) call; on Julia 1.12 that is 553 MethodInstances in a
clean session, through the poorly inferred any call in Compiler's
compileable_specialization. On 1.11+ hook Base's internal
_any/_all(f, itr, ::Colon) instead, which costs nothing there and also lets
callable structs read chunk by chunk. On 1.10 that hook invalidates
LinearAlgebra's _any(f∘transpose, ::Any, :) (~70), while the public methods
cost 4, so 1.10 keeps the public methods.

Also hook _any/_all(f, a, dims), so any(f, a; dims) and any(a; dims) keep
reaching diskarrays_any_impl/diskarrays_all_impl with dims, as with the
keyword methods before. On 1.10, dims excludes Colon to stay unambiguous with
Base's _any(f, itr, ::Colon). Measured at no extra cost on 1.10, 1.12 and 1.13.

The default dims implementation now calls Base's generic _any/_all(f, A, dims)
directly (any(f, A; dims) would come back to the hook), which forwards to
mapreduce as before.

Supersedes #335 (thanks @Beforerr for the _any/_all hook).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019Dfq6aDr3GqA8w61TsvMsc
sum/prod/minimum/maximum/extrema(f, a; kwargs...) add a Core.kwcall method,
which invalidates compiled keyword calls such as GeometryBasics'
maximum(f, faces; init=0) (130-225 MethodInstances in every Makie session).
Without the keyword methods, Base's methods take dims and init and forward to
mapreduce(f, op, a; dims, init), which reaches the backend through
diskarrays_mapreduce_impl. The calls without keywords still go through
diskarrays_<f>_impl.

This also fixes init, which threw a MethodError for sum, prod, minimum and
maximum (and count(a; init)) because the _impl hooks only accept dims.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019Dfq6aDr3GqA8w61TsvMsc
…arly

For sum, prod, maximum, minimum, extrema, count, any and all, with and without
f, with dims=1, dims=(1, 2) and init, compare against Array and check with a
counting backend which diskarrays_*_impl hook the call reaches: the
reduction's own hook without keywords (and for any/all with dims), the
mapreduce hook with dims or init. Check that any/all read chunk by chunk and
stop at the first chunk that decides the result, also for callable structs on
Julia 1.11+. Fold in the early-stopping test from #335 (@Beforerr).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019Dfq6aDr3GqA8w61TsvMsc
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019Dfq6aDr3GqA8w61TsvMsc
Every public sum/prod/minimum/maximum/extrema method with `dims` calls Base's
internal `_sum(f, a, dims; kw...)` or its sibling, and count calls
`_count(f, a, dims, init)`. Methods on those internals send `dims`, with or
without `init`, to `diskarrays_<f>_impl` as the compute-backend branch did,
and `count` now reaches `diskarrays_count_impl` with `dims` and `init` too.

`dims` is typed to exclude `Colon`. A keyword method for the whole array would
add a `Core.kwcall` method that invalidates GeometryBasics'
`maximum(f, faces; init=0)` in every Makie session, so whole-array calls keep
the public methods without keywords and `init` reaches `mapreduce`.

The defaults call Base's generic `_sum(f, A, dims)`, since `sum(f, A; dims)`
would come back to the hook. The test now expects each form's own hook.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019Dfq6aDr3GqA8w61TsvMsc
`sum`/`prod`/`maximum`/`minimum`/`extrema(f, a; init)` and `(a; init)`
call `Base._sum(f, a, :; init)` and its siblings. A `Core.kwcall`
method on `_sum(f, ::AbstractDiskArray, ::Colon)` sends them to
`diskarrays_<f>_impl(f, a, backend; init)` instead of
`diskarrays_mapreduce_impl`. Positional whole-array calls keep the
public no-keyword methods.

The keyword-call method invalidates compiled `maximum(f, v; init)`
callers such as GeometryBasics': +13 MethodInstances in the CairoMakie
session and +9 in the many-package session on Julia 1.13 (+6 and +6
on 1.12). Clean sessions and the DataFrames session are unchanged.

The counting-backend test now asserts the own hook for whole-array
`init` with and without `f`, for every reduction.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019Dfq6aDr3GqA8w61TsvMsc
@asinghvi17
asinghvi17 added this pull request to stack #342 October 8, 2026 14:03
@asinghvi17 asinghvi17 changed the title inv/any all hook Invalidations 4/4: version-split any/all entry point; reductions reach their own hooks through Base's internal _sum/… Oct 8, 2026
@asinghvi17 asinghvi17 changed the title Invalidations 4/4: version-split any/all entry point; reductions reach their own hooks through Base's internal _sum/… Invalidations 4/4: reach reduction hooks through Base internals Oct 8, 2026
@asinghvi17 asinghvi17 changed the title Invalidations 4/4: reach reduction hooks through Base internals Invalidations 4/6: reach reduction hooks through Base internals Oct 8, 2026

@mkitti mkitti left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good to me. The approach is sound and I could not break it, but it leans on Base internals, so see comment 1.

What I ran

  • The full test suite on Julia 1.10, 1.11, 1.12 and 1.13: all pass.
  • A battery of 36 reduction forms against an in-memory array, on #340 and on this PR, on 1.13 and 1.10: sum/prod/maximum/minimum/extrema/count/any/all, each with a function, dims, init and combinations, a callable struct, a view, mapreduce and reduce.
    • Values and read counts are identical to #340 everywhere except the 8 init= forms (sum, sum(f, …), sum with dims+init, prod, maximum, maximum(f, …), minimum with dims+init, count). Those throw MethodError: no method matching diskarrays_*_impl on #340 and are correct with 16 reads here, so the claimed fix is real.
    • 1.10 and 1.13 give identical output for every case on this PR.
    • Short-circuiting is preserved: all(<(-90), a) reads 1 of 16 chunks, any(>(250), a) reads 13.
  • uinvalidated after using DiskArrays on 1.13, #340 → this PR: 574 → 572 in a clean session and 1003 → 982 with DataFrames loaded.

Comments

  1. It depends on Base internals. The PR adds methods to Base._sum, _prod, _minimum, _maximum, _extrema, _any, _all, _count, and a Core.kwcall method for the init whole-array case. All exist on 1.10–1.13, but nothing guarantees they keep their signatures. The per-form hook tests would fail loudly if one changed, and the nightly job in the CI matrix is the early warning; maybe say that in a comment next to the Core.kwcall block, since it is the least obvious one.
  2. The init hooks cost +4 invalidations in Makie-like sessions (GeometryBasics' maximum(f, faces; init=0)). The description and backend.md both say so, which I appreciate. The trade is reasonable because init= was throwing before, but if that count ever matters, that block is the one to gate.
  3. FYI, not this PR: Project.toml declares julia = "1.9", but on 1.9.4 maximum(a; dims=1) and sum(view(a, …)) already throw UndefVarError: rm_singleton_indices on main and here (CI starts at 1.10). The init forms work on 1.9 with this PR.
  4. Docs: the table makes it clear that a callable struct f reaches diskarrays_mapreduce_impl rather than the per-reduction hook, and that on 1.10 non-Function callables iterate. That is a real inconsistency between Function and callable f, and it is documented. A one-line CHANGELOG entry (there is none) would help backend authors, since the dims=:, kwargs... hook signatures changed.

Not blocking.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants