[Fix branch via fast-forward push, NOT the merge button] Restore main as an ancestor of the HRRR shared-base branch#784
Merged
mrshll merged 24 commits intoJul 23, 2026
Conversation
…ndows (~15x) (#744) * HRRR-spatial ops latency: parallel commit flush + smaller manifest windows Operational update commits were p50 ~60s / max ~500s vs GEFS-spatial's ~5s. Profiled locally by replaying one cycle's file arrivals against a store seeded to prod manifest fill (26M refs, M=3.1k), with icechunk.testing.LatencyStorage simulating S3. Commit cost decomposes as, per changed array, one manifest read-modify-write: ~1 GET + ~1.2 PUTs serialized (max_concurrent_nodes=1) plus ~0.1s single-threaded CPU per MiB of active manifest window (decode + merge + re-serialize + zstd). An sfc file touches 142 root arrays x ~1.7 MiB windows = ~24s CPU + ~18s round trips. Neither lever alone fixes it: concurrency at 1.5 vCPU stays CPU-bound (measured 26s), smaller windows alone stay round-trip-bound. Changes (measured steady-state commits at 10ms simulated S3): - Pin icechunk fork (2.1.1 + Session.commit(max_concurrent_nodes) + O(1) manifest-info index) and pass max_concurrent_nodes=16 in commit_if_icechunk (env ICECHUNK_COMMIT_MAX_CONCURRENT_NODES). - Shrink HRRR-spatial manifest splits 2400/180/160 -> 240/45/40 along init_time (~190KiB/1.4MiB/1.6MiB full windows). Icechunk re-windows existing manifests on each array's next commit: one-time ~70s commit locally, longer on prod's full historical windows. M grows ~3.1k -> ~16.4k, covered by the fork's O(1) index. - Update pod cpu 1.5 -> 4: sfc/prs/nat commits 2.4/1.1/3.0s at 1.5 CPU vs 1.5/0.7/2.0s at 4 (vs prod today: 42/7.7/13.4s). - Rust toolchain in Dockerfile to build the git-pinned icechunk during uv sync. Not for main as-is: revert the pyproject pin + Dockerfile rust stage once the icechunk changes land upstream. GEFS-spatial needs no dataset change (its splits are already small); it gains the parallel flush automatically. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Moderate HRRR-spatial splits to 600/90/80 after manifest-count tradeoff review 240/45/40 minimized commit latency (1.6/0.7/2.1s) but at M ~16.4k manifests, ~10x whole-archive read amplification, and sub-200KiB manifest files. 600/90/80 keeps manifests at 0.55-3.2 MiB and M ~7.6k for sfc/prs/nat commits of 2.6/2.1/4.5s (c=16, 4 cpu, 10ms simulated S3; peak RSS 2.9G fits the pod), still ~10-15x faster than the current 42/7.7/13.4s. One-time re-window on the first commit per array: ~44s measured locally. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Pin icechunk fork tip: rewrite_manifests concurrency exposure Adds Repository.rewrite_manifests(max_concurrent_manifests=...) for the one-time manifest re-split (the Rust plumbing already existed; the binding hardcoded 1). Validated on the local harness: re-split of a prod-fill store in one atomic commit. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Trim fork pin to the minimum: max_concurrent_nodes exposure only Drop the O(1) manifest-info index from the deployed fork surface (~0.13s/commit at M~7.6k, near the noise floor at the 600/90/80 splits) to keep the prod test and the upstream PR to exactly the change that matters. The rewrite_manifests concurrency exposure is used only by the local one-time re-split wheel and is not deployed. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Hardcode commit manifest-write concurrency at 16 Drop the ICECHUNK_COMMIT_MAX_CONCURRENT_NODES env override; one less knob. Note in the pyproject pin comment to also remove the Dockerfile Rust stage when the pin reverts. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Operational update workers aren't checkpointed, so a mid-run Karpenter consolidation eviction restarts the whole job from scratch, doubling wall-clock. This caused noaa-hrrr-analysis update runs to finish after their validation cronjob started, firing spurious validation alerts. Add a pod_annotations hook on Job (empty by default) and set the karpenter.sh/do-not-disrupt annotation on ReformatCronJob so Karpenter won't voluntarily evict a running update. Backfill and validation pods are unaffected. Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* Quantize MRMS GRIB reads to the message's decimal precision GDAL's g2clib unpacks GRIB values in float32 arithmetic whose rounding error varies by architecture: on arm64, FMA contraction of refD + bdscale*X decodes packed zeros to ~4.5e-8 (mm) / -2.2e-5 (percent) instead of exact 0, which the archive then stores as 1.2391865e-11 kg m-2 s-1 and -2.2292137e-05 percent after deaccumulation and mantissa rounding. x86_64 rounds the product to float32 first and gets exact 0 from the same file. GRIB2 field values are exact multiples of 10^-D (decimal scale factor, section 5), so rounding the decoded float64 field to D decimals restores the encoded values on any architecture and makes materialized reads bit-identical to gribberish's float64 decode used by virtual datasets. Adds grib_decimal_scale_factors to parse D per field (asserting binary scale factor E is 0) and applies it in the MRMS read path. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01YVoAgQxtwjvu86nMfowU9a * Assert integer GRIB reference value; clarify quantization comment Values are exact multiples of 10^-D only when the binary scale factor E is 0 and the reference value R is an integer; previously only E was checked, so a fractional R would have silently passed and decimal rounding would corrupt values by up to half a quantum. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01YVoAgQxtwjvu86nMfowU9a * Split decimal scale lookup onto two lines for readability Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01YVoAgQxtwjvu86nMfowU9a * Document the GRIB2 byte layout behind the section 5 offsets Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01YVoAgQxtwjvu86nMfowU9a --------- Co-authored-by: Claude <noreply@anthropic.com>
…759) HRRR CPOFP encodes "no/undefined frozen precipitation" as a discrete -50 (verified: ~97-98% of the CONUS field where no precipitation is falling, with valid frozen-fraction values 0-100 elsewhere). The materialized HRRR datasets passed this through unmasked and undocumented, so the field mean reads ~-47 and CF-aware readers see the sentinel as data. Mirror the CF-compliant handling already used by the HRRR forecast virtual dataset: set missing_value=-50.0 with a matching -50.0 fill_value and a comment, so CF-aware readers mask it to NaN. Applies to both materialized datasets that share the common data var (forecast-48-hour and analysis); templates regenerated. Claude-Session: https://claude.ai/code/session_01Gd2qpwonTbByYkCzaNAyt9 Co-authored-by: Claude <noreply@anthropic.com>
* Validate recent init_times for NaNs, not just the newest check_forecast_recent_nans only ever inspected the single newest init_time (isel(init_time=[-1])), so an init_time was validated once, while it was the newest, and never again. A transient source read failure that leaves a whole forecast step NaN (e.g. all perturbed members at one lead_time, whose open-data enfo-ef file failed to download) is therefore invisible if it isn't caught in that one window. Add num_recent_init_times to check a window of recent init_times, each independently (a per-init threshold is not diluted), reporting which init_time failed. Default stays 1 (unchanged for existing datasets). Wire the ECMWF forecast datasets, which expect zero NaNs, to check the last 3. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Wvu6VqQjxWyRHZRMA4W28o * Drop explanatory comments on ECMWF recent-nans validators The rationale lives in check_forecast_recent_nans's docstring; the per-dataset repetition added no local information. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Wvu6VqQjxWyRHZRMA4W28o --------- Co-authored-by: Claude <noreply@anthropic.com>
…, manual GitHub action (#755) * Safer, simpler backfills: explicit overwrite API, auto append-dim end, manual GitHub action backfill-kubernetes now defaults --append-dim-end to "now" for a new store and to the store's current end for an existing one, and replaces --overwrite-existing with explicit --overwrite-chunks / --overwrite-metadata flags. Guards (assert_safe_overwrite, checked in the driver and again on worker 0) reject structural drift, never allow trimming an existing store, and only allow new arrays or extension when explicitly requested. --overwrite-metadata alone refreshes store metadata from the template in place via the shared refresh_store_metadata helper (extracted from VirtualRegionJob.refresh_metadata), creating newly added variables without launching workers. Overwrite backfills preserve job-written coordinate values (template-null coords like ingested_forecast_length are excluded from metadata copies), publish zarr3 metadata at finalize so a new variable appears only once its data is written, and suspend the dataset's update/validate cronjobs so an operational update can't move icechunk main and strand the backfill's finalize branch reset — which now raises loudly (with correct crash-retry detection) instead of silently skipping publication. New workflow_dispatch GitHub actions, generated alongside the existing manual workflows: "Manual: Backfill" exposing only the safe operations (create new store, overwrite chunks/metadata) using the deployed image, and "Manual: Suspend/Resume Dataset CronJobs" to resume updates after a backfill. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SAB2gwVGczEvJGx4d9DnMm * Run overwrite backfills concurrently with operational updates via icechunk rebase Replaces the cronjob-suspension approach: backfills can run for days and updates have a latency SLA, so neither side may block or lose to the other. Overwrite backfills (--overwrite-chunks / --overwrite-metadata) no longer use a temp branch. Worker 0 writes refreshed template metadata straight into the live stores (skip_unchanged, store-written coordinate values preserved), then each worker's chunk-only commit to icechunk main rebases over concurrent update commits, with BasicConflictSolver(UseTheirs) so the operational update wins any chunk both wrote. A new variable appears immediately and fills in progressively, matching the previous update-creates-then-backfill flow. The reverse race is handled in finalize: an update that finds main moved past its setup snapshot (a committed temp branch cannot be rebased) now replays itself onto the moved main — copying the branch's exact data-variable chunk bytes from repo.diff plus the final template metadata onto a fresh main session and committing with UseOurs so the update wins conflicts. The replay is bounded by one update's writes, atomic to readers, and idempotent across crash retries. Fresh-store backfill divergence (nothing else should write a store being created) still fails loudly. commit_if_icechunk gains a rebase solver parameter and skips sessions with no changes; refresh_store_metadata and the overwrite setup share write_template_metadata_to_stores. The suspend/resume workflow and cronjob suspension code are removed. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SAB2gwVGczEvJGx4d9DnMm * Mark replay commits via snapshot metadata, not commit message text The replay commit now carries the standard "Update at <time>" message public history consumers see, with the idempotence marker in icechunk snapshot metadata ({"replayed_branch": ...}) instead of machinery text. Also pass chunk buffers straight through to store.set so replay holds one shard's bytes at a time without an extra copy. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SAB2gwVGczEvJGx4d9DnMm * Manual: Backfill runs only from main with the image main's deploy built The workflow rejects any ref but main, then waits for the deploy of main's tip to complete (failing fast if Code Quality or the deploy failed) before submitting, and pins the kubernetes job to that deploy's image tag — no one-off docker build, and the driver's checked-out code and the workers' image are the same commit. If the tip already deployed, it submits immediately. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SAB2gwVGczEvJGx4d9DnMm * Limit the backfill workflow's deploy wait to 30 minutes Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SAB2gwVGczEvJGx4d9DnMm * Run backfill workflow setup steps before waiting on main's deploy Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SAB2gwVGczEvJGx4d9DnMm * Per-kind jobs-per-pod and max-parallelism guidance in backfill docs and action Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SAB2gwVGczEvJGx4d9DnMm * Split out concurrent backfill/update support; keep the temp-branch flow Per review: the direct-to-main commit, replay-onto-moved-main, and conflict solver machinery carried most of this branch's risk, so it moves to its own PR (branch claude/backfill-concurrent-updates-iwd69b). Overwrite backfills here use the existing temp-branch flow; an operational update publishing mid-backfill makes the backfill's finalize fail loudly (the update wins; re-run the backfill), and docs say to time backfills to avoid update publishes or manually suspend the update cronjob for the duration. Also applies review fixes: the default append-dim end is floored to whole seconds so the worker command parses as a CLI datetime; _open_existing_store treats only store-missing errors as "no store" so a transient error can't route a backfill past the overwrite guards; --overwrite-metadata alone rejects filter/--docker-image options it would silently ignore; workflow inputs pass to bash via env so free-form input text cannot inject shell; backfill workflow defaults to 2 jobs per pod and max parallelism 10; and the structural-drift docstring no longer claims backfills are exempt from structure checks. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SAB2gwVGczEvJGx4d9DnMm * Advise running overwrite backfills between update runs, not suspending crons Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SAB2gwVGczEvJGx4d9DnMm * Point backfill-kubernetes docstring at AGENTS.md Backfills section Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SAB2gwVGczEvJGx4d9DnMm --------- Co-authored-by: Claude <noreply@anthropic.com>
* Add 80m wind, temperature, and pressure to NOAA GFS Adds wind_u_80m, wind_v_80m, temperature_80m, and pressure_80m to the shared GFS data-variable config, so both noaa-gfs-forecast and noaa-gfs-analysis gain the 80m level. Template metadata only; backfill is a separate follow-up. Also exempts 80t/80sp and their long names from the ECMWF parameter compliance check (ECMWF has no 80m level), matching the existing 80u/80v HRRR exemption. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015ZY5VagHhMYq9D6WXVXxvq * test: update GFS var count and list for 80m variables Adding the four 80m variables changes the analysis operational job split (25 vars into 3 jobs) and the expected forecast data var list. Claude-Session: https://claude.ai/code/session_01HX6W8Ru3TvKaUBgQFCUgAZ --------- Co-authored-by: Claude <noreply@anthropic.com>
…ups guard (#767) Holistic review follow-ups for the virtual datasets offering: - Tests closing coverage gaps in the common virtual machinery (chunk-key asserts, unregistered-container assert, refresh on replicas/empty store, multi-prefix + MemoryStore source listing, finalize multi-repo retry and divergence paths, virtual update drift rejection, virtual-with-replica validation, single-writer asserts, tick pacing bounds, skipped-file drop semantics) and in the HRRR 48h virtual dataset (idx element alternatives, stale-index guard, discovery wiring, manifest split sizing, validator thresholds). - Unequal committed group sizes now fail loudly: the write loop asserts all groups agree (they grow in one atomic commit) instead of silently growing from the min, and refresh_store_metadata trims to the single committed extent. - Comment/docstring cleanups per AGENTS.md conventions (dedupe, drop change-narration and benchmark-session details, typos) and removal of the GEFS download_concurrency override that now equals the base default. - Doc corrections: discover_available_by_obstore_listing signature and download_concurrency default in virtual_datasets.md, finalize retry description and structure-guard attribution in parallel_processing.md, registration snippet/anchor/heading fixes in dataset_integration_guide.md, --kind flag in AGENTS.md; validation.md tightened (remote-session and batch-sizing guidance compressed, outage-verification rule stated once in 3d and cross-referenced, typos). Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
…ile (#768) * Tolerate GRIB message padding; read at float32 to restore memory profile The first MRMS overwrite backfill with decimal quantization surfaced two issues. Iowa Mesonet MRMS files from 2014-2015 append zero padding after the GRIB message, which grib_decimal_scale_factors rejected (nulling those hours as failed reads) while GDAL scans past it; locate messages by their magic bytes instead. And reading at GDAL's native float64 tripled per-read transient memory (read_parallelism scales with node CPUs), memory-killing backfill workers; read at float32 and round in place — bit-identical to the float64 path and to gribberish's decode because the float32 cast error is far below half a decimal quantum and scaled values stay under 2^24 (verified on 2014-2020 source files). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01YVoAgQxtwjvu86nMfowU9a * Assert decimal scale is small enough for exact float32 rounding Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01YVoAgQxtwjvu86nMfowU9a --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
The dataset catalog on dynamical.org builds from the STAC catalog (dynamical-org/dynamical-stac), which is the source of truth — the prior text wrongly described editing the website directly. Since the STAC generator derives variables from each dataset's Icechunk store, a new variable needs no catalog edit, just a regenerate-and-commit after backfill. Claude-Session: https://claude.ai/code/session_01Jxv5Wc8bwWAaGCtm9KdGBC
On an analysis dataset the point series has no non-time dims, so the null-fraction mean over an empty dim list is an identity op that keeps the mask's bool dtype. The two-point fraction combine then does bool addition (logical or), rendering fully-missing positions at fraction 0.5 — the brown midpoint of the availability colormap — instead of 0.0 light red, and hiding the distinction from half-missing positions. Forecast datasets were unaffected (their mean over lead_time produces floats), which is why existing tests passed. Claude-Session: https://claude.ai/code/session_01YVoAgQxtwjvu86nMfowU9a Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
* Add 80m wind, temperature, and pressure to NOAA GEFS Adds wind_u_80m, wind_v_80m, temperature_80m, and pressure_80m to the shared GEFS data-variable config (from the 0.5 degree b file, like the existing 100m wind), so noaa-gefs-forecast-35-day and noaa-gefs-analysis gain the 80m level. The 0.25 degree 10-day-spatial dataset excludes b-file variables by design and is unaffected. Template metadata only; backfill is a separate follow-up. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015ZY5VagHhMYq9D6WXVXxvq * test: mark GEFS 80m vars missing from v12 reforecast The 80 m above ground level is not present in the GEFS v12 reforecast archive (2000-2019), so the reforecast integration test must exclude the new 80m variables. Claude-Session: https://claude.ai/code/session_01HX6W8Ru3TvKaUBgQFCUgAZ --------- Co-authored-by: Claude <noreply@anthropic.com>
* Add dataset development coordinator guide Add docs/dataset_development_guide.md, an agent-first doc for a coordinator agent that runs dataset development end to end (explore → implement → backfill → validate → publish) by spawning a sub-agent per stage, with two human checkpoints. Supports both new-dataset and add-a-variable modes. Rename dataset_integration_guide.md → implementation_guide.md and trim its validate/publish tail (now coordinator stages), and update references in AGENTS.md, README.md, and add_new_variable.md. Implements dynamical-org/meta#137. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * Refine coordinator guide per review Cut contemporaneous validation.md cross-reference; reframe multiple-products as multiple variants in one run sharing a single Implement agent; make Checkpoint A about aligning on provider/model/variant(s) plus the non-obvious questions from mapping source to datacube (not a prescribed checklist); and never suspend an active operational update cron for a backfill. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * Consolidate STAC publish into the development guide Move the STAC-refresh detail solely into the development guide's Publish stage; add_new_variable.md now points there instead of duplicating it. Name the real external catalogs (Source Coop, Earthmover Marketplace, AWS Open Data). Reword the AGENTS.md pointer. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * Retitle implementation guide header Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * Extract backfill into its own doc Add docs/backfill.md consolidating the backfill process from the two implementation docs and AGENTS.md (new-store, new-variable, and re-backfill operations; where-to-run; parallelism tuning; update-cron concurrency). Trim implementation_guide.md and add_new_variable.md to only their implementation-specific content — both now end at a merged PR and hand off to backfill.md and the development guide's validate/publish stages. Give the coordinator's Publish stage the concrete STAC-regenerate commands so it is self-contained. Wire backfill.md into AGENTS.md. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * Tighten doc entry points and draft-vs-publish semantics Drop the invoke-phrase line from the development guide (only read once the doc is already open). In AGENTS.md, add a top-level "to add a dataset or variable, follow the development guide" pointer and slim the now-redundant references. In validation.md, clarify that uploading a draft does not make a report externally viewable and should be redone after every summary update, while --publish overwrites the website-linked report and needs explicit user direction. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * Publish the report at approval, deploy via the STAC change Move the report publish to immediately after Checkpoint B approval; the STAC change's site deploy is what surfaces it on dynamical.org. Reword the generate comment to "each Zarr store". Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * STAC update lands via a PR to main Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * Allow batching multiple datasets into one STAC PR Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * Add --filter-contains to Manual: Backfill; refresh parallelism guidance Expose filter_contains on the Manual: Backfill action (repeatable per append-dim timestamp, expanded from a comma-separated input) — the most efficient re-backfill of specific flagged positions, running only the region jobs those timestamps touch. Update jobs_per_pod (materialized: 2-4 non-ensemble, 1 ensemble) and max_parallelism (materialized: 50-200, mind cluster quotas so operational updates can schedule) guidance in the action inputs and docs/backfill.md. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * Require seconds-precision ISO for filter timestamps Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * Document filter endpoint inclusivity Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * Align on variable set at Checkpoint A; drop Sentry-secret prereq Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * Scaffold cron jobs suspended until backfill completes Set suspend=True (via the suspend variable) on the commented ReformatCronJob and ValidationCronJob stubs in both example teaching templates, and add the virtual example the same suspend variable the materialized one already had. Document that a follow-up PR removes it once the backfill is complete so operational updates begin keeping the store current. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * Simplify the suspend scaffold comment Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * Remove the Backfills section from AGENTS.md backfill.md is now its home; repoint the backfill_kubernetes docstring to it. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * Address code-review findings - Trim whitespace and skip empties when splitting the action's comma-separated filter_contains / filter_variable_names inputs, so "a, b" no longer yields a leading-space value the CLI datetime/name parse rejects. - Drop the repeated --filter-contains retry strings from availability.py's unavailable_timestamps.txt; it keeps the newline-separated timestamps, and backfill.md now points readers to that file. - Restructure backfill.md Operations so metadata-only (overwrite-metadata) and chunk rewrite/re-backfill (overwrite-chunks) are distinct from new-variable (overwrite-chunks-and-metadata), instead of conflating refresh-metadata. - Remove the local-backfill path from backfill.md. - Fix stale add_new_variable.md section refs (heading is no longer numbered). - Note that removing suspend=True at backfill makes Validate re-backfills follow the between-update-fires rule. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * Warn that overwrite-metadata is for attribute changes, not encoding Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
) * GEFS: skip 80m vars in reforecast era via available_from The four 80m fields (wind_u/v, temperature, pressure) added in #749 do not exist in the GEFS v12 reforecast archive (2000 to 2020-01-01), so the analysis backfill's get_url raised KeyError('80 m above ground') on every reforecast-era region. They are present from the pre-v12 period (2020-01-01) onward. Add an available_from field to GEFSInternalAttrs and set it to GEFS_REFORECAST_END on the four 80m vars; the analysis region job now skips source coords for times before available_from, leaving those chunks as fill value. Mirrors the MRMS available_from mechanism. available_from is internal, so templates are unchanged. * GEFS: address review comments on 80m available_from - Drop the explanatory comments on GEFSInternalAttrs.available_from and on the reforecast-era skip in GefsAnalysisRegionJob.generate_source_file_coords. - Rework the available_from test into a single job whose times straddle the reforecast end, asserting an explicit count of returned coords. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BvCufXN3oucnRBzV9Udvcr --------- Co-authored-by: Claude <noreply@anthropic.com>
…l Pages auto-deploy (#773) * Fix validation --lead-time parsing; restore remote-session monitoring guidance - compare_spatial: convert the --lead-time hours argument to a Timedelta before .sel(); the raw string raised KeyError on lead_time when pinning a forecast spatial snapshot. - plots.py: note that --lead-time is in hours. - docs/validation.md: restore the explanation that a monitor's periodic emitted events are what keep a remote session warm (a monitor that emits only at completion lets the session get reclaimed mid-run), plus the setsid / watch-the-worker-PID / avoid-loose-pgrep / re-arm-on-timeout gotchas and the measured RAM budget. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01A6bz5xuwNwPEmJLGDqJZ9P * Auto-trigger dynamical.org Pages deploy on validation report publish Add an optional PAGES_DEPLOY_HOOK_URL: a `--publish` upload POSTs the Cloudflare Pages deploy hook after the files land, so dynamical.org rebuilds with the new report without a manual redeploy. Draft uploads never trigger a deploy, and it is a no-op when the env var is unset. Documented in validation.md (§5d + Configuration); test covers the set and unset branches. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01A6bz5xuwNwPEmJLGDqJZ9P --------- Co-authored-by: Claude <noreply@anthropic.com>
…ariable (#779) Adding a variable to a dataset that already has a published validation report should run the full run-all and carry the approved ## Summary forward, updating only the new variable's pieces — not regenerate the whole summary (which would drop other variables from the website-linked report and re-litigate vetted commentary). Document this in validation.md §5 and the dev guide's Validate stage.
Prompt exploration to locate the source data license and record it in the report, flagging any non-open terms so redistribution constraints surface before integration. Claude-Session: https://claude.ai/code/session_015TrAjqYYp2T4v1eeB2Qw4B Co-authored-by: Claude <noreply@anthropic.com>
Re-establish main as a first-class ancestor of this branch. PR #783 was squash-merged, which flattened the earlier merge into a single commit and left main un-merged in history, so this branch's diff against main showed all of main's recent commits as if they belonged to it. The tree is unchanged from the already-resolved, tested state.
mrshll
marked this pull request as ready for review
July 23, 2026 16:00
mrshll
changed the title
mrshll
marked this pull request as draft
July 23, 2026 16:03
mrshll
merged commit Jul 23, 2026
f197b4d
into
claude/hrrr-virtual-shared-base-a8s89q
2 checks passed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This repo disallows merge commits (squash-only), and a squash is exactly what broke things: PR #783 was squash-merged, which flattened the resolution into a single commit (
28f07bf) and discarded the parent pointing atmain. Somainis no longer an ancestor ofclaude/hrrr-virtual-shared-base-a8s89q, #743's merge base reverted to the old6c04b47, and its diff ballooned to 99 files / +8205 (showing all of main's recent commits) and wentdirty.Squash-merging or rebase-merging this PR would just repeat the problem. The branch needs a real fast-forward instead.
The fix (one fast-forward push, no
--force)This PR's head branch
claude/pr-743-merge-conflicts-jzlze0points at commitf197b4d, which is:28f07bf(its first parent), so no force is needed;main(dd085ad), so it re-establishesmainas an ancestor;git diffempty);8abb4c4stays a first-parent ancestor, leaving PR Add noaa-hrrr-forecast-18-hour-virtual and noaa-hrrr-forecast-48-hour-virtual-fast #740's stacking undisturbed.Advance the shared-base branch to it:
(Plain fast-forward. Equivalent: check out
claude/hrrr-virtual-shared-base-a8s89q,git merge --ff-only origin/claude/pr-743-merge-conflicts-jzlze0, push.)Result
mainbecomes an ancestor of the shared-base branch again.8abb4c4) is unaffected.Verification (from #783, re-confirmed on this tree)
uv run main noaa-hrrr-forecast-48-hour-virtual update-template→ zero diff.ruff format --check,ruff check,ty checkall pass.🤖 Generated with Claude Code
https://claude.ai/code/session_01PABJSjFNGTDsrqg4nASFiU