feat: chunk fan-out fault-domain seeding (F + B1 + B2 + dashboard)#812
Open
vilenarios wants to merge 11 commits into
Open
feat: chunk fan-out fault-domain seeding (F + B1 + B2 + dashboard)#812vilenarios wants to merge 11 commits into
vilenarios wants to merge 11 commits into
Conversation
getSortedChunkPostPeers memoized its weighted peer ordering with a normalizer of `eligiblePeers.length`, so two *different* eligible-peer sets of equal size shared one cached ordering for the full CHUNK_POST_SORTED_PEERS_CACHE_DURATION_MS window (10s default). That quietly narrowed chunk-POST seeding diversity: a churning-but-same-size peer set kept reusing the first set's ordering. Key by the set of peer ids instead (order-independent sort+join). Still cheap — a sort+join over the bounded postChunk peer set, far less than the weighted selection it guards — and the 10s maxAge still absorbs gradual weight drift. Extracted as the pure, exported `chunkPostPeersCacheKey` so the keying is unit-tested directly (distinct same-length sets ⇒ distinct keys; order- independent; no input mutation; empty set). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XQPK4TXcVoXoyFp6sNW2Lr
Adds a pure helper that reduces a peer host to a stable "fault domain" key — the network block it belongs to — for chunk-seeding diversity accounting: - IPv4 (incl. IPv4-mapped IPv6): masked to /24 by default -> "a.b.c.0/24" - IPv6: expanded + truncated to /48 by default -> "2001:db8:abcd::/48" - non-IP (unresolved hostname): returned verbatim (its own domain) Also exports `expandIpv6` (8 zero-padded groups, handles `::`), the minimum IPv6 parsing needed for prefix bucketing — ip-utils' existing `isIpInCidr` is IPv4-only. No behavior change: pure additions, not yet wired. This is PR B1 of the chunk-fanout SPOF fix (see architect-handoff/chunk-seeding-durability-plan.md §17): the five tip-*.arweave.xyz preferred nodes all collapse into one 38.29.227.0/24, which the upcoming soft-preferred quorum (B2) uses to detect and route around a single-fault-domain tips outage. Tested: the five tips collapse to one /24; distinct /24s stay distinct; IPv6 /48 bucketing; IPv4-mapped normalization; custom widths; hostname fallback. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XQPK4TXcVoXoyFp6sNW2Lr
Makes chunk fanout fault-domain-aware and removes the single-fault-domain
tips SPOF (all five tip-*.arweave.xyz resolve into one 38.29.227.0/24, so a
tips outage hard-fails every POST today). Both behaviors are opt-in and
default to the legacy quorum exactly.
- Distinct-domain counting: buckets each successful peer by IP /24 (v4) /48
(v6) via ipFaultDomain; CHUNK_POST_MIN_DISTINCT_DOMAINS (default 0=off) adds
a soft target among successes that degrades gracefully when the eligible
peer set can't supply N (never hard-fails on scarcity).
- Soft-preferred fallback: CHUNK_POST_PREFERRED_SOFT_FALLBACK (default false)
lets a strong distinct-domain discovered quorum satisfy a POST when NO tip
was eligible (tips down/over-queue) — but never when tips merely failed,
so tips stay the hard primary on-ranp on a real outage only.
- Centralized verdict: evaluateChunkBroadcastVerdict is the single source of
truth for "seeded sufficiently?"; the handler reads result.succeeded instead
of recomputing the thresholds (removes latent drift). With both flags at
defaults it reduces EXACTLY to the legacy rule (regression test).
- Placement-evidence contract: BroadcastChunkResult now carries
distinctDomainCount/preferredEligibleCount/succeeded, and POST /chunk emits
X-AR-IO-Chunk-Placement-Domains — so the bundler can consume where a chunk
landed instead of collapsing the result to pass/fail (see plan §18).
- Metrics: chunk_post_distinct_domains (histogram),
chunk_post_preferred_shortfall_total{reason}, chunk_post_domain_shortfall_total{reason}.
Preferred shortfall is emitted even when the fallback makes the POST succeed,
so a tips outage is never silently masked.
Depends on #805 (ipFaultDomain). Default-off; soak before any default flip.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XQPK4TXcVoXoyFp6sNW2Lr
…oard Adds a "Chunk Fan-out Seeding (placement)" row to the provisioned Grafana dashboard visualizing the metrics introduced with the fault-domain-aware quorum (#806): - Distinct fault domains per POST (avg / p50 / p10) — p10 is the seeding-risk tail; low p10 = chunks landing on few independent operators. - Preferred (tip) shortfall rate by reason — makes a tips-/24 outage (tips_unavailable) visible even when the soft-preferred fallback still succeeds. - Distinct-domain target shortfall rate by reason (unmet / degraded). - Chunk broadcast success rate — quorum context. Panels are inert ("No data") until #806's metrics flow; they reference chunk_post_distinct_domains, chunk_post_{preferred,domain}_shortfall_total, and the existing arweave_chunk_broadcast_total. Additive only — existing panels unchanged. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XQPK4TXcVoXoyFp6sNW2Lr
# Conflicts: # src/arweave/composite-client.test.ts # src/arweave/composite-client.ts
This was referenced Jul 1, 2026
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## develop #812 +/- ##
===========================================
+ Coverage 78.65% 78.72% +0.06%
===========================================
Files 132 132
Lines 49980 50468 +488
Branches 3763 3812 +49
===========================================
+ Hits 39312 39729 +417
- Misses 10620 10690 +70
- Partials 48 49 +1 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
Contributor
📝 WalkthroughWalkthroughThis PR adds fault-domain-aware chunk POST quorum handling. It introduces IP fault-domain bucketing, new quorum configuration and metrics, centralized verdict evaluation, placement-evidence response wiring, and Grafana panels for the new placement signals. ChangesFault-domain quorum feature
Estimated code review effort: 4 (Complex) | ~60 minutes Possibly related PRs
Suggested reviewers: 🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches📝 Generate docstrings
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Actionable comments posted: 5
🧹 Nitpick comments (3)
src/config.ts (1)
807-823: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick winConvert these exported config docs to TSDoc.
These are new exported
src/**/*.tssymbols, but the added docs are line comments rather than TSDoc blocks. As per coding guidelines, "Add or improve TSDoc comments on code you touch".🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@src/config.ts` around lines 807 - 823, The new exported config constants in config.ts need TSDoc instead of line comments. Convert the comments above CHUNK_POST_MIN_DISTINCT_DOMAINS and CHUNK_POST_PREFERRED_SOFT_FALLBACK into proper TSDoc block comments so the exported symbols are documented per guideline, keeping the same meaning and placement directly above each export.Source: Coding guidelines
src/metrics.ts (1)
559-586: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick winConvert the new metric docs to TSDoc.
These are exported metrics in
src/**/*.ts, so the repo rule expects TSDoc rather than line comments. As per coding guidelines, "Add or improve TSDoc comments on code you touch".🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@src/metrics.ts` around lines 559 - 586, The exported metric docs for chunkPostDistinctDomainsHistogram, chunkPostPreferredShortfallCounter, and chunkPostDomainShortfallCounter are currently written as line comments and should be converted to TSDoc. Update the documentation immediately above each promClient.Histogram and promClient.Counter export in src/metrics.ts to use proper TSDoc block comments so the exported metrics comply with the repo’s documentation rule.Source: Coding guidelines
src/types.d.ts (1)
804-812: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick winUse TSDoc for these public response fields.
These properties are part of the
/chunkcontract, so/** ... */comments are a better fit than//comments in the declaration file and will surface more reliably in TS tooling. As per coding guidelines, "Add or improve TSDoc comments on code you touch".🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@src/types.d.ts` around lines 804 - 812, The public `/chunk` response fields in the `types.d.ts` declaration use line comments instead of TSDoc. Update the `distinctDomainCount`, `preferredEligibleCount`, and `succeeded` declarations in `src/types.d.ts` to use `/** ... */` comments so the contract is surfaced correctly in TypeScript tooling and matches the project’s TSDoc guideline.Source: Coding guidelines
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@monitoring/grafana/dashboards/default.json`:
- Around line 1377-1384: The Grafana success-ratio query in the panel expression
is turning idle periods into false 0% success by using clamp_min on the total
rate. Update the query for the success ratio panel to avoid forcing a minimum
denominator for zero-traffic windows, so it returns no data/blank instead of 0
when arweave_chunk_broadcast_total has no samples. Keep the change localized to
the affected expression in default.json and preserve the existing success-rate
numerator and legendFormat.
In `@src/arweave/composite-client.ts`:
- Around line 2053-2060: The early-exit check in composite-client.ts should use
the same verdict logic as evaluateChunkBroadcastVerdict() instead of
hard-requiring raw preferred quorum and raw CHUNK_POST_MIN_DISTINCT_DOMAINS.
Update the branch in the chunk broadcast loop to honor the degraded-domain and
soft-preferred-fallback success paths, so it can stop posting once the broadcast
is already effectively succeeded. Keep the condition aligned with the same
success-count, preferred-count, and domain-state rules used by
evaluateChunkBroadcastVerdict() and the surrounding chunk post flow.
In `@src/config.ts`:
- Around line 811-814: CHUNK_POST_MIN_DISTINCT_DOMAINS currently accepts any
numeric coercion from env.varOrDefault, so NaN, negative, and fractional values
can leak into evaluateChunkBroadcastVerdict and break chunk POST behavior.
Update the CHUNK_POST_MIN_DISTINCT_DOMAINS parsing in config.ts to validate that
the env value is a non-negative integer before exporting it, and fall back to
the default or reject the value when it is invalid; keep the check close to the
existing env.varOrDefault usage so the exported constant can only be a safe
threshold.
In `@src/constants.ts`:
- Around line 49-51: Add X-AR-IO-Chunk-Placement-Domains to the signed response
header allowlist in src/lib/httpsig.ts so POST /chunk responses include this
placement evidence in the HTTP signature. Update the CO_SIGNABLE_HEADERS
definition used by the response signing path, ensuring the new constant from
constants.ts (chunkPlacementDomains) is included alongside the other signable
headers. Keep the change limited to the header registration so intermediaries
can’t rewrite the emitted placement evidence.
In `@src/routes/chunk/handlers.ts`:
- Around line 888-891: The `/chunk` response currently sets the chunk placement
domains header without protecting it in the signed response surface. Update the
chunk handler that writes `headerNames.chunkPlacementDomains` so it is covered
by the same signing/trigger path used for response integrity, and add
`headerNames.chunkPlacementDomains` to `CO_SIGNABLE_HEADERS` in
`src/lib/httpsig.ts` so intermediaries cannot alter the reported domain count.
---
Nitpick comments:
In `@src/config.ts`:
- Around line 807-823: The new exported config constants in config.ts need TSDoc
instead of line comments. Convert the comments above
CHUNK_POST_MIN_DISTINCT_DOMAINS and CHUNK_POST_PREFERRED_SOFT_FALLBACK into
proper TSDoc block comments so the exported symbols are documented per
guideline, keeping the same meaning and placement directly above each export.
In `@src/metrics.ts`:
- Around line 559-586: The exported metric docs for
chunkPostDistinctDomainsHistogram, chunkPostPreferredShortfallCounter, and
chunkPostDomainShortfallCounter are currently written as line comments and
should be converted to TSDoc. Update the documentation immediately above each
promClient.Histogram and promClient.Counter export in src/metrics.ts to use
proper TSDoc block comments so the exported metrics comply with the repo’s
documentation rule.
In `@src/types.d.ts`:
- Around line 804-812: The public `/chunk` response fields in the `types.d.ts`
declaration use line comments instead of TSDoc. Update the
`distinctDomainCount`, `preferredEligibleCount`, and `succeeded` declarations in
`src/types.d.ts` to use `/** ... */` comments so the contract is surfaced
correctly in TypeScript tooling and matches the project’s TSDoc guideline.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Repository UI
Review profile: CHILL
Plan: Pro
Run ID: e77f1571-aa7a-412e-b8b9-64b630b75c85
📒 Files selected for processing (13)
docker-compose.yamldocs/envs.mdmonitoring/grafana/dashboards/default.jsonsrc/arweave/composite-client.test.tssrc/arweave/composite-client.tssrc/config.tssrc/constants.tssrc/data/rebroadcasting-chunk-source.test.tssrc/lib/ip-utils.test.tssrc/lib/ip-utils.tssrc/metrics.tssrc/routes/chunk/handlers.tssrc/types.d.ts
Comment on lines
+49
to
+51
| // Placement-evidence contract: distinct fault domains (IP /24·/48) a POSTed | ||
| // chunk landed on, surfaced on the POST /chunk response for the poster. | ||
| chunkPlacementDomains: 'X-AR-IO-Chunk-Placement-Domains', |
Contributor
There was a problem hiding this comment.
🔒 Security & Privacy | 🟡 Minor | ⚡ Quick win
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
rg -n "CO_SIGNABLE_HEADERS|chunkPlacementDomains|X-AR-IO-Chunk-Placement-Domains" src/constants.ts src/routes src/system.tsRepository: ar-io/ar-io-node
Length of output: 534
🏁 Script executed:
#!/bin/bash
set -euo pipefail
printf '\n== constants ==\n'
sed -n '1,140p' src/constants.ts
printf '\n== signing/header references ==\n'
rg -n "CO_SIGNABLE_HEADERS|signable|signable headers|Content-Digest|chunkPlacementDomains|X-AR-IO-Chunk-Placement-Domains" src -g '!**/*.map'
printf '\n== chunk handler around the response header ==\n'
sed -n '860,910p' src/routes/chunk/handlers.tsRepository: ar-io/ar-io-node
Length of output: 12885
🏁 Script executed:
#!/bin/bash
set -euo pipefail
# Locate the exact definition of CO_SIGNABLE_HEADERS and any header list wiring.
rg -n -C 4 "CO_SIGNABLE_HEADERS|Content-Digest|Signature-Input|Signature" src
# Show the surrounding code for src/routes/chunk/handlers.ts where the header is set.
sed -n '880,900p' src/routes/chunk/handlers.tsRepository: ar-io/ar-io-node
Length of output: 50373
🏁 Script executed:
#!/bin/bash
set -euo pipefail
# Find the actual signable-header list and its consumers across the repo.
rg -n -C 3 "CO_SIGNABLE_HEADERS|Content-Digest|Signature-Input|Signature" .
# Narrow view around any declaration of the list if found.
python3 - <<'PY'
from pathlib import Path
import re
for path in Path('.').rglob('*.ts'):
text = path.read_text(errors='ignore')
if 'CO_SIGNABLE_HEADERS' in text:
print(f'\n## {path}')
for m in re.finditer(r'.{0,80}CO_SIGNABLE_HEADERS.{0,120}', text, re.S):
s = m.group(0).replace('\n',' ')
print(s[:220])
PYRepository: ar-io/ar-io-node
Length of output: 50373
🏁 Script executed:
#!/bin/bash
set -euo pipefail
python3 - <<'PY'
from pathlib import Path
import re
text = Path('src/lib/httpsig.ts').read_text()
m = re.search(r'export const CO_SIGNABLE_HEADERS = new Set\(\[(.*?)\]\);', text, re.S)
if not m:
raise SystemExit('CO_SIGNABLE_HEADERS not found')
items = re.findall(r"'([^']+)'", m.group(1))
target = 'x-ar-io-chunk-placement-domains'
print('contains_target=', target in items)
print('index=', items.index(target) if target in items else None)
print('count=', len(items))
print('headers=', ','.join(items))
PYRepository: ar-io/ar-io-node
Length of output: 356
Add X-AR-IO-Chunk-Placement-Domains to CO_SIGNABLE_HEADERS in src/lib/httpsig.ts. It’s emitted on POST /chunk, but it isn’t currently covered by the response signature, so an intermediary can rewrite the placement evidence.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@src/constants.ts` around lines 49 - 51, Add X-AR-IO-Chunk-Placement-Domains
to the signed response header allowlist in src/lib/httpsig.ts so POST /chunk
responses include this placement evidence in the HTTP signature. Update the
CO_SIGNABLE_HEADERS definition used by the response signing path, ensuring the
new constant from constants.ts (chunkPlacementDomains) is included alongside the
other signable headers. Keep the change limited to the header registration so
intermediaries can’t rewrite the emitted placement evidence.
Source: Coding guidelines
Comment on lines
+888
to
+891
| res.setHeader( | ||
| headerNames.chunkPlacementDomains, | ||
| String(result.distinctDomainCount), | ||
| ); |
Contributor
There was a problem hiding this comment.
🔒 Security & Privacy | 🟠 Major | ⚡ Quick win
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
set -euo pipefail
echo '== signing-related definitions =='
rg -n -C2 'CO_SIGNABLE_HEADERS|Content-Digest|HTTPSIG|chunkPlacementDomains|X-AR-IO-Chunk-Placement-Domains' src
echo
echo '== chunk POST handler coverage =='
fd -i 'handlers.test.ts' src
rg -n -C2 'chunkPlacementDomains|X-AR-IO-Chunk-Placement-Domains|result\.succeeded|broadcast\.success' src/routesRepository: ar-io/ar-io-node
Length of output: 30527
🏁 Script executed:
#!/bin/bash
set -euo pipefail
echo '== CO_SIGNABLE_HEADERS / HTTPSIG signing set =='
rg -n -C3 'CO_SIGNABLE_HEADERS|signable headers|Content-Digest|chunkPlacementDomains|X-AR-IO-Chunk-Placement-Domains' src
echo
echo '== chunk route response path =='
ast-grep outline src/routes/chunk/handlers.ts --view expandedRepository: ar-io/ar-io-node
Length of output: 26505
🏁 Script executed:
#!/bin/bash
set -euo pipefail
echo '== src/lib/httpsig.ts =='
sed -n '1,140p' src/lib/httpsig.ts
echo
echo '== src/routes/chunk/handlers.ts (createChunkPostHandler) =='
sed -n '824,930p' src/routes/chunk/handlers.tsRepository: ar-io/ar-io-node
Length of output: 9057
🏁 Script executed:
#!/bin/bash
set -euo pipefail
echo '== trigger headers in chunk POST handler =='
rg -n -C2 'x-ar-io-trusted|x-ar-io-verified|x-arweave-chunk-data-root|x-arweave-chunk-tx-id|x-ar-io-chunk-source-type|setHeader\(' src/routes/chunk/handlers.ts
echo
echo '== nearby response header logic =='
sed -n '824,940p' src/routes/chunk/handlers.tsRepository: ar-io/ar-io-node
Length of output: 7557
Protect the placement count header in /chunk responses src/routes/chunk/handlers.ts:888-891 emits X-AR-IO-Chunk-Placement-Domains, but this response doesn’t set any trigger header and src/lib/httpsig.ts doesn’t include it in CO_SIGNABLE_HEADERS. Add it to the signed surface so intermediaries can’t rewrite the reported domain count.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@src/routes/chunk/handlers.ts` around lines 888 - 891, The `/chunk` response
currently sets the chunk placement domains header without protecting it in the
signed response surface. Update the chunk handler that writes
`headerNames.chunkPlacementDomains` so it is covered by the same signing/trigger
path used for response integrity, and add `headerNames.chunkPlacementDomains` to
`CO_SIGNABLE_HEADERS` in `src/lib/httpsig.ts` so intermediaries cannot alter the
reported domain count.
Source: Coding guidelines
…rget gate Live soak on #812 (real bundler traffic, tip /24 blocked) surfaced a case where the SPOF fix didn't fire when it mattered: a chunk seeded across 14 independent fault domains but the POST was still marked FAILED. Two coupled bugs + a latency gap, all in the quorum path: #2 Soft-preferred fallback keyed on the wrong condition. meetsFallback required preferredEligibleCount === 0, which only holds under tip queue-saturation (or no tips configured). A real tip partition leaves tips eligible-but-failing, and one surviving preferred node in another domain keeps eligibleCount >= 1 forever — so the fallback never fired for the exact outage it was built for. Trigger is now `!meetsPreferred` (fires whenever the preferred quorum falls short, unavailable OR failing). Safe because tips are dispatched first, so a healthy tip set meets the normal path before the fallback is consulted. preferredEligibleCount now only labels the shortfall reason (tips_unavailable vs tips_failed). #4 Distinct-domain target hard-gated success off the ELIGIBLE set. domainsAvailable counted eligible peers (~14), but on fresh chunks only the preferred nodes 200 (discovered peers reject an unknown data_root until the tx propagates). So the soft-cap never engaged and MIN_DISTINCT_DOMAINS>=2 turned legitimate first-posts into failures. The target is now best-effort: it drives fan-out breadth (early termination) + a chunk_post_domain_shortfall_total {reason=below_target} metric, but never fails a POST. Broader diversity accrues over the seeding lifecycle, not a single POST. Removed domainsAvailable. #5 (latency) Fallback-aware early termination. With tips down the preferred quorum is unreachable, so early-termination never fired and every POST walked the whole peer list (~12s in the soak). Added a fallback early-term branch, gated on preferredAttempted >= preferred.length so a healthy tip set is never skipped. (Per-peer POSTs were already bounded at abort=2s/response=5s; the 12s was the aggregate, not per-peer.) Default-off behavior unchanged (both branches gated on the flag; regression test holds). Adds a regression test for the exact soak trace (14 domains, 1 preferred, tips_failed -> succeeds). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XQPK4TXcVoXoyFp6sNW2Lr
…n steady state Per live-soak review of #812: with CHUNK_POST_PREFERRED_SOFT_FALLBACK on, the relaxed trigger (!meetsPreferred) also applies in steady state, not just during a full tip outage. The effective quorum becomes "minPreferred tip successes OR minSuccess total successes across >= max(minPreferred, minDistinctDomains) distinct domains" — a normal POST where only one tip acks but several independent discovered peers land will pass. Intended availability win, but operators must not assume the unconditional 2-tip guarantee still holds with the flag on. Documented in envs.md + a code note; no behavior change. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XQPK4TXcVoXoyFp6sNW2Lr
- config: parse CHUNK_POST_MIN_DISTINCT_DOMAINS via nonNegativeIntOrDefault so a bad env value (NaN/negative/fraction) can't leak into the verdict and skew the threshold (CodeRabbit: Reject invalid values). - dashboard: drop clamp_min from the broadcast success-ratio panel — it forced zero-traffic windows to render as 0% (false outage) instead of no-data (CodeRabbit: Don't turn idle periods into 0% success). - composite-client: document why the early-exit path intentionally does NOT reuse evaluateChunkBroadcastVerdict — both branches are provably stricter than the verdict (each implies verdict.succeeded), so we never terminate before the POST would succeed; mirroring it would remove the diversity-seeking breadth. (CodeRabbit: Mirror verdict in early-exit — declined, by design.) Not changed: adding X-AR-IO-Chunk-Placement-Domains to CO_SIGNABLE_HEADERS — POST /chunk responses aren't HTTPSIG-signed (no trigger header), the header is advisory operational metadata (not data-provenance), and the gateway<->bundler channel is TLS-protected, so adding it to the signable set signs nothing. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XQPK4TXcVoXoyFp6sNW2Lr
Ops-facing guide (docs/chunk-fanout-seeding.md) so operations agents can
understand the effects and set up/manage the feature safely. Leads with the
data-safety model — why this cannot lose data:
1. default-off is byte-identical to the previous release (regression-tested +
soak-verified);
2. the gateway relays chunks, never holds the only copy, and doesn't touch the
local ingest cache;
3. the uploader (Turbo) owns durability and independently re-verifies
permanence + re-seeds, so a gateway false-success can't lose data.
Includes the operational precondition for SOFT_FALLBACK (safe when the uploader
re-verifies — Turbo does; leave off for naive uploaders), staged rollout,
steady-state quorum semantics, monitoring, troubleshooting, and rollback.
Also: linked from docs/INDEX.md and added glossary terms (fault domain, tip
node, chunk fan-out). All metric/env/header names cross-checked against the code.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XQPK4TXcVoXoyFp6sNW2Lr
Contributor
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@docs/chunk-fanout-seeding.md`:
- Around line 168-171: The fenced environment snippet in the docs is unlabeled,
which triggers markdownlint and reduces clarity; update the markdown fence
around the CHUNK_POST_MIN_DISTINCT_DOMAINS and
CHUNK_POST_PREFERRED_SOFT_FALLBACK example to use the appropriate dotenv label
so the snippet is clearly identified as environment data.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Repository UI
Review profile: CHILL
Plan: Pro
Run ID: f780ffeb-dba5-4c18-b2a9-850f70864050
📒 Files selected for processing (6)
docs/INDEX.mddocs/chunk-fanout-seeding.mddocs/glossary.mdmonitoring/grafana/dashboards/default.jsonsrc/arweave/composite-client.tssrc/config.ts
✅ Files skipped from review due to trivial changes (2)
- docs/glossary.md
- docs/INDEX.md
🚧 Files skipped from review as they are similar to previous changes (3)
- src/config.ts
- monitoring/grafana/dashboards/default.json
- src/arweave/composite-client.ts
Comment on lines
+168
to
+171
| ``` | ||
| CHUNK_POST_MIN_DISTINCT_DOMAINS=0 | ||
| CHUNK_POST_PREFERRED_SOFT_FALLBACK=false | ||
| ``` |
Contributor
There was a problem hiding this comment.
📐 Maintainability & Code Quality | 🟡 Minor | ⚡ Quick win
Label the fenced env snippet.
This block is plain environment data, so leaving it unlabeled trips markdownlint and makes the example less clear.
Suggested fix
-```
+```dotenv
CHUNK_POST_MIN_DISTINCT_DOMAINS=0
CHUNK_POST_PREFERRED_SOFT_FALLBACK=false
</details>
<!-- suggestion_start -->
<details>
<summary>📝 Committable suggestion</summary>
> ‼️ **IMPORTANT**
> Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
```suggestion
🧰 Tools
🪛 markdownlint-cli2 (0.22.1)
[warning] 168-168: Fenced code blocks should have a language specified
(MD040, fenced-code-language)
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@docs/chunk-fanout-seeding.md` around lines 168 - 171, The fenced environment
snippet in the docs is unlabeled, which triggers markdownlint and reduces
clarity; update the markdown fence around the CHUNK_POST_MIN_DISTINCT_DOMAINS
and CHUNK_POST_PREFERRED_SOFT_FALLBACK example to use the appropriate dotenv
label so the snippet is clearly identified as environment data.
Source: Linters/SAST tools
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Integration branch consolidating the chunk fan-out seeding work into one coherent, atomically-testable feature. Supersedes #804, #805, #806, #811 (closed in favor of this). The typecheck CI gate (#809) stays a separate PR — different concern.
The problem
Chunk fan-out treated "N peers returned 200" as "seeded" — fault-domain-blind. All five
tip-*.arweave.xyzpreferred nodes resolve into one38.29.227.0/24, so the hardpreferred ≥ 2quorum was a single-fault-domain SPOF: a tips outage hard-failed every chunk POST even when independent discovered peers accepted it.What's in here
chunkPostPeersCacheKey): the sorted-peer cache was keyed by list length, so distinct same-length peer sets shared one cached ordering for 10s, narrowing seeding diversity. Now keyed by peer set. Pure fix, always-on.ipFaultDomain(src/lib/ip-utils.ts): pure helper bucketing peers by IP/24(v4) //48(v6); the tips collapse to one domain. No hot-path DNS (peer list is IP-literal after resolution).CHUNK_POST_MIN_DISTINCT_DOMAINS, default0), soft-preferred fallback (CHUNK_POST_PREFERRED_SOFT_FALLBACK, defaultfalse) that rescues a tips-down POST via a strong distinct-domain discovered quorum, and a centralizedevaluateChunkBroadcastVerdict. Placement-evidence contract:X-AR-IO-Chunk-Placement-Domainsheader + result fields. New metrics:chunk_post_distinct_domains,chunk_post_{preferred,domain}_shortfall_total.Safety
Verification (this branch)
develop, passes in CI).tsc --noEmit -p tsconfig.json: 0 errors (test-inclusive).eslint: clean.composite-client) resolved keeping both;composite-client.test.ts= 28 pass (both suites).Rollout
Merge → deploy to rig → observe
chunk_post_distinct_domains(flags off) → enableCHUNK_POST_PREFERRED_SOFT_FALLBACK, simulate a tips-/24outage, confirm POSTs succeed via fallback +chunk_post_preferred_shortfall_total{reason=tips_unavailable}ticks → then consider default flips.🤖 Generated with Claude Code