Density surfaces now wear the data's own colors (LOD doc §2)#219
Closed
masenf wants to merge 15 commits into
Closed
Density surfaces now wear the data's own colors (LOD doc §2)#219masenf wants to merge 15 commits into
masenf wants to merge 15 commits into
Conversation
The Tier-2 density map previously colormapped the binned COUNT, ignoring the points' own colors — so a colored scatter's aggregate view matched neither its marks nor its legend, and every density<->points zoom transition visibly recolored the chart. Now the surface wears the data's colors (LOD doc §2): each cell carries the alpha-weighted mean of its binned points' resolved colors — one law for continuous, categorical, and direct-RGBA channels — averaged in linear light through an integer-only pipeline (checked-in sRGB<->linear u16 tables, u64 sums; bitwise deterministic across thread counts and platforms). The log-tone-mapped count now drives only the alpha channel: more points, deeper color; fewer, lighter. Constant-color traces keep the compact count-only wire and a client tint (the mean of a constant is the constant). - Rust core: bin_2d_mean_color kernel (+ fused cell accumulator); pyramid mean-color planes (build_color/compose_color, count grids bit-identical to the count-only compose; colored pyramids refuse appends and rebuild lazily). C ABI v38. - Python: channels.resolve_bin_colors maps every channel mode onto the kernel color source; initial emit + density_view (exact and pyramid paths) ship the RGBA plane as density.rgba with color_agg: "mean"; color is no longer listed in dropped_channels; SVG/PNG exporters follow the same color law; continuous-channel density scatters render their colorbar again (the legend is truthful now). - Client: DENSITY_FS mean-color branch over a premultiplied RGBA8 texture (count tone curve baked at upload so bilinear filtering weights color by coverage); exposure easing re-encodes color grids; the standalone to_html re-bin worker aggregates the sample's colors with the same law; the drill handoff is intensity-only — points enter in native colors at their cell's count-alpha (density_colormap left the points wire). - Tests: NumPy oracle for the kernel, wire-shape coverage for payload/ exact/pyramid/export paths, Rust unit tests, ABI smoke checks, and a mean-color pixel probe in the headless render smoke. - Specs: LOD doc §2 rewritten around the shipped algebra (+§4/§5/§6 status), wire-protocol density_update/points messages, dossier F5 narrowed to the residual value-aggregation gap, renderer/rust-engine/ chart-kind-contract touch-ups, CHANGELOG entry.
… drills Field fixes from the first 100M-point live-drilldown run of the mean-color density surface: - The FastAPI demo hand-built count-only density replies on three paths (browser-local integral-image re-bin, server DensityOverview branch, home-grid re-serve), so mid-zoom windows flipped back to the old count-colormapped surface — a green wash unrelated to the points' colors, flashing old<->new across zoom levels. The browser-local re-bin now aggregates the shipped mean-color plane (same linear-light, countxalpha-weighted law as the kernel), the home re-serve carries the plane through, and the server branch is deleted outright: the engine's colored pyramid already serves wide windows in O(visible cells) with colors, drill bookkeeping, and recorded binning, which is everything the demo's pre-pyramid overview existed to fake. The density badge count is rounded (no more '48,636,815.177'). - T10 amended: the aggregate backdrop now RETIRES once a drill is settled inside its window — the marks are exact, so the mean-color wash under them read as data that isn't there. It stays painted through every transition (entry, hold, exit, revive; eased continuously, fast on restore) so zoom-outs never blank and interleaved replies never flash, and per-reply refreshes inside a settled drill keep it retired. New smoke probe (dretire) pins the contract; spec T10 rewritten. - The colored pyramid's fused base scan now fans out like bin_2d_mean_color (points-per-cell gated, <=4 workers, exact integer merge — bitwise-identical output), cutting the one-time 100M-row build roughly 4x. The parallel-oracle kernel test keeps a genuinely serial oracle.
The demo predated the kernel pyramid and carried two extra density implementations to stay interactive at 100M points: a 6144^2 integral-image overview server-side and ~350 lines of page JS (browser local re-bins, request parking/single-flight/timeout-retry, per-client staleness maps, pending-status heuristics). The engine has since absorbed every one of those jobs: Figure.density_view serves wide windows from the mean-color pyramid in O(visible cells), the render client debounces requests, seq-drops stale replies, and keeps the best cached texture drawn while a reply is in flight. The module is now the pattern it was always meant to demonstrate: build one figure (pyramid warmed at startup), serve payload + bundled client as HTML, and wire ChartView to a comm object that POSTs messages to one endpoint dispatching density_view/pick under a lock. Page JS drops from ~400 lines to ~60 (fetch + status badge); the HTML route also reuses the shared figure instead of regenerating 100M rows per GET. tests/test_drilldown_pan_alignment.py is removed with its subject: it guarded the browser-local re-bin's window-clamp bug, and engine-served grids always report exactly the window they binned, so the stretch failure mode is unreachable. The simplified page was verified live — uvicorn + headless Chromium at 3M points: mean-color rgba planes at home/mid zoom, drill-in to exact points with the backdrop retiring, clean zoom-out back to density, no page errors.
The integrated tree serves colored pyramids from adaptively finer bases (issue-153 work), so the LOD doc's residency note no longer calls the 2048^2 base fixed; the windowed-exact spatial tier's colorless gate is now recorded in the Phase-3 serving contract as well as the dossier; the changelog's ABI claim moves to the merged v40.
The #153 compose spreads a cluster-edge source cell across every output bin its extent overlaps, so a boundary bin can legitimately carry a fractional count sliver: lit in the color plane while the log-u8 count plane rounds it to 0 or 1. Assert the direction that is invariant -- nonzero count implies lit, unlit implies zero count -- instead of exact mask equality, which encoded the old whole-cell assignment.
Merging this PR will not alter performance
Comparing 
Footnotes
|
Greptile SummaryThis PR makes scatter density surfaces use the source data’s colors. The main changes are:
Confidence Score: 5/5Safe to merge after resolving the existing drill-state feedback. No additional blocking issues were found in the updated code. The remaining shared drill-state behavior is already covered by existing feedback.
 T-Rex LogsWhat T-Rex did
Important Files Changed
Reviews (5): Last reviewed commit: "Merge remote-tracking branch 'origin/cla..." | Re-trigger Greptile |
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
cargo clippy --all-targets -D warnings failed on two lints: - tiles.rs: fold the two negated partial-ord comparisons into a single !(count > 0 && weight > 0) guard (De Morgan-equivalent, same NaN-skipping behavior) to satisfy neg_cmp_op_on_partial_ord. - kernels.rs: iterate SRGB_TO_LINEAR_U16 with enumerate() instead of indexing by a range variable to satisfy needless_range_loop.
The fastapi example's live drilldown scatter needs ~400 lines of custom transport there (a Starlette callback endpoint, an HTTP comm bridge, a hand-built density-overview cache with stale-sequence bookkeeping). Serve the identical chart — same seed-11 chunked columns, same mark config, same XY_LIVE_POINTS override — from the reflex-xy adapter as one inline() token with zero transport code, so behavior differences between the two hosts isolate what that custom code adds versus what is fundamental to the engine (density tiers, pyramid binning, drill-in). - examples/reflex: §6 section + drilldown_chart()/drilldown_view(), README run notes (import-time build, memory, resize override) - tests/test_example_apps.py: source markers for §6; the import test pins XY_LIVE_POINTS=50000 (same override the fastapi test uses) and asserts the inline token mints - scripts/reflex_ws_smoke.py: count the seventh live source, wait for the drilldown payload, and pixel-check its density surface - spec: reflex-integration §7 inventory + production-readiness example focus bullet now name the deliberate cross-host overlap
The live drilldown endpoint base64-encoded every reply's binary buffers into a JSON envelope, so each zoom/pan paid a base64 encode server-side and a decode in the browser plus ~33% inflation on the density grids and point buffers that dominate a drill round-trip. Switch the round-trip transport to the XYBF binary frame (wire-protocol §7): the reply message is the frame's compact JSON metadata and each numeric buffer rides raw and 8-byte aligned, so the browser decodes one xy.decodeFrame and hands the kernel zero-copy views. First paint still embeds its blob as base64 (inline HTML has no binary channel, §6); malformed requests still return a JSON error the client rejects on res.ok before decoding. Update the example test to decode the frame and assert on its message and buffers rather than substring-matching binary bytes.
…-uhpq9z' into claude/density-mean-color-on-issue-153
A density-tier scatter surface now encodes count as alpha and wears the mean point color (not a count colormap), so the live view's gradient legend read as "color == density" and was misleading. Remove the density-tier special case in _buildLegend; a named density trace falls through to the plain marker swatch, matching the static SVG/raster exporters, and a count-only surface gets no legend entry. Update the chart-kind contract to match.
Once a points reply has shipped its window exactly (reduction "none" — the subset IS every point in the window), any view contained in that window is already answered by the marks on the GPU: the smaller window's points are a subset of the shipped ones. Drilling deeper therefore no longer round-trips the kernel — _scheduleViewRequest skips the trace's pending markers and its density_view message (lodDrillServesView, re-checked at debounced send time), while still bumping seq so an in-flight reply for an older, wider view dies stale instead of yanking exact marks out from under a view it cannot improve. Three conditions re-arm the request: leaving the window, a drill that is dying or not exact (only reduction "none" arms the elision, recorded as drill.exact), and depth — the shipped geometry is f32, offset-encoded on the window midpoint (dossier §16), so once the view span drops below 1/256 of the window span on either axis one request goes out purely to re-center the encoding. Data changes cannot serve stale marks through the elision: streaming append and full payload updates rebuild the GPU trace, which drops the drill. Recorded as invariant T12 in spec/design/lod-architecture.md §5, in the wire protocol's density_view entry, and in dossier §16/§28; covered by a headless-Chromium probe (tests/test_drill_zoomin_elides_request.py) asserting the elision and each re-arm condition.
…nd-1p3sfb' into claude/density-mean-color-on-issue-153
…zation-jfk57d' into claude/density-mean-color-on-issue-153
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Scatter density surfaces (Tier-2 aggregation) now display the alpha-weighted mean color of binned points instead of a uniform colormap tint. This makes zoomed-out views truthfully represent the data's own colors while count drives only the alpha channel.
Key Changes
Rust kernel (
src/kernels.rs,src/tiles.rs)bin_2d_mean_colorkernel: fused count+color accumulation over a grid, computing per-cell alpha-weighted mean point color in linear-light u16 (sRGB input/output, deterministic integer arithmetic)SRGB_TO_LINEAR_U16lookup table andlinear_u16_to_srgb_u8inverse for bitwise-deterministic color space conversion across all platformsPyramidstruct withcolor_levelsplanes (mean-color u16x4 per cell) alongside count levelsbuild_colorpyramid builder that fuses count and color accumulation in a single parallel scan, with 4→1 reduction that re-rounds color means once per level (≤0.5-lsb-per-level error, recorded in LOD doc §28)Python bindings (
python/xy/_native.py,python/xy/kernels.py)xy_bin_2d_mean_colorandxy_pyramid_build_color/xy_pyramid_compose_colorC ABI functionsBinColorSourceenum marshaling (indexed LUT or straight-alpha RGBA8)ABI_VERSIONto 40Interaction & payload (
python/xy/interaction.py,python/xy/_payload.py,python/xy/channels.py)colormap_lut_rgba8andpalette_rgba8helpers to export LUTs as straight-alpha RGBA8 for kernel binningrgbaplane (per-cell mean color) alongside count gridRender client (
js/src/45_lod.ts,js/src/50_chartview.ts,js/src/54_kernel.ts)lodWriteGridTexturenow handles mean-color planes: uploads RGBA texture with premultiplied RGB (sRGB space) and log-tone-mapped alphaStatic exporters (
python/xy/_svg.py,python/xy/_raster.py)Documentation & tests
test_density_mean_color.py: NumPy oracle validation of kernel against float reference, wire shape checks, and exporter color lawtest_drilldown_pan_alignment.py(browser-based pan test for old integral-image overview)Implementation Details
powf, so results are bitwise identical for any thread count and libm versionhttps://claude.ai/code/session_01GR6foqKYEMhuW78tn5whk2