RS-12 opening session — the bulk floor is a silent L072 URC drop, not RF - #108
Merged
Conversation
Campaign anchor for issue #107. Records the opening desk finding: at the clean 902.5 MHz channel, penultimate-fragment losses are NEVER-DEMODULATED (corrupt captures with readable headers sit ~uniform across indices, idx-11 = 9% ~ 1/13, while attributed losses concentrate there at 35%). Target: a base-board receiver-deafness mechanism phase-locked to the second-to-last fragment, invisible to crc_dump. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Opening session of the bulk-floor campaign (issue #107): four legs at the clean 902.5 MHz channel, closing with an L072-counter-bracketed leg that localizes the mechanism. THE FINDING. The clustered losses are not an RF phenomenon: radiated 2414 | radio handled (rx_ok+crc_err) 2429 -> nothing undemodulated crc_err 126 == crc_dumps 126 (exact instrument closure) rx_ok 2303 vs host URCs 2215 -> 88 frames demodulated then LOST host_dropped / queue_full / ring_ovf / parse_err: ALL ZERO The SX1276 demodulates the "lost" fragments successfully; they vanish inside the L072's URC path without incrementing any counter, and nothing truncated reaches the host. Combined with the corrupt-dump index readout (corruptions uniform across indices; losses locked to the penultimate at 28-35% on 13-frag and 53% on 7-frag trains -- the lock FOLLOWS train structure on the clean channel), the target is now: an L072 firmware race in the RX->URC path, triggered once per train near the transmitter's end-of-train pattern, uncounted by every existing counter. This also dissolves the RS-11.5 role-swap puzzle: the trigger is the TX-side end-of-train pattern, so identical firmware shows the lock only in one direction. Variance: loss swings 3.3-8.2% across same-config legs while the boundary-event RATE stays fixed (~86 timeouts, ~100 published per leg) -- bad runs deepen doomed trains rather than dooming more. Correction to the RS-11.6 leg-3 record included: "escape halves the loss" compared n=1 vs n=1 under ~2x spread; the escape's confirmed effect is the removal of the periodic interference population, and the honest total-loss statement is "bulk dominates and varies 3-8%". Next: firmware instrumentation (rx_urc_lost counter -- needs flash, bench presence) and TX-pattern discriminators (TrainGapMs/TxPipelineDepth, no flash). Activity-modulation legs deprioritized: the drop is inside the L072, not board-level EMI. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Contributor
There was a problem hiding this comment.
Pull request overview
This PR opens the RS-12 “clustered bulk loss floor” campaign by adding a new TODO entry and archiving a bench-evidence bundle (results narrative + run artifacts) for the 2026-08-16 opening session at 902.5 MHz, intended to support issue #107’s investigation.
Changes:
- Add an RS-12 campaign entry to
TODO.mdwith the opening finding and planned next steps. - Add a new RS-12 evidence folder (
RS_12_bulk_floor_2026-08-16) including counter snapshots and a written session summary. - Archive three radio-monitor runs (logs + params) used as evidence in the writeup.
Reviewed changes
Copilot reviewed 7 out of 16 changed files in this pull request and generated 2 comments.
Show a summary per file
| File | Description |
|---|---|
| LifeTrac-v25/DESIGN-CONTROLLER/TODO.md | Adds an RS-12 campaign TODO entry referencing issue #107 and summarizing findings/plan. |
| LifeTrac-v25/DESIGN-CONTROLLER/bench-evidence/RS_12_bulk_floor_2026-08-16/RESULTS.md | New RS-12 session writeup summarizing legs A–D and the URC-drop conclusion. |
| LifeTrac-v25/DESIGN-CONTROLLER/bench-evidence/RS_12_bulk_floor_2026-08-16/stats_before.txt | Pre-leg stats snapshot used for counter bracketing. |
| LifeTrac-v25/DESIGN-CONTROLLER/bench-evidence/RS_12_bulk_floor_2026-08-16/stats_after.txt | Post-leg stats snapshot used for counter bracketing. |
| LifeTrac-v25/DESIGN-CONTROLLER/bench-evidence/radio_monitor_20260816_204928_29c8dea8/tx_daemon.log | Archived TX daemon log for leg B. |
| LifeTrac-v25/DESIGN-CONTROLLER/bench-evidence/radio_monitor_20260816_204928_29c8dea8/synth_pub.log | Archived synthetic publisher log for leg B. |
| LifeTrac-v25/DESIGN-CONTROLLER/bench-evidence/radio_monitor_20260816_204928_29c8dea8/params.txt | Archived run parameters for leg B. |
| LifeTrac-v25/DESIGN-CONTROLLER/bench-evidence/radio_monitor_20260816_205609_29c8dea8/tx_daemon.log | Archived TX daemon log for leg C. |
| LifeTrac-v25/DESIGN-CONTROLLER/bench-evidence/radio_monitor_20260816_205609_29c8dea8/synth_pub.log | Archived synthetic publisher log for leg C. |
| LifeTrac-v25/DESIGN-CONTROLLER/bench-evidence/radio_monitor_20260816_205609_29c8dea8/params.txt | Archived run parameters for leg C. |
| LifeTrac-v25/DESIGN-CONTROLLER/bench-evidence/radio_monitor_20260816_210320_29c8dea8/tx_daemon.log | Archived TX daemon log for leg D (stats-bracketed leg). |
| LifeTrac-v25/DESIGN-CONTROLLER/bench-evidence/radio_monitor_20260816_210320_29c8dea8/synth_pub.log | Archived synthetic publisher log for leg D. |
| LifeTrac-v25/DESIGN-CONTROLLER/bench-evidence/radio_monitor_20260816_210320_29c8dea8/params.txt | Archived run parameters for leg D. |
Suppressed comments (1)
LifeTrac-v25/DESIGN-CONTROLLER/TODO.md:2270
- The plan list here still emphasizes software-only activity-modulation legs, but RESULTS.md (section 4) says those are deprioritized now that the drop is inside the L072 URC path, and instead calls out firmware instrumentation (rx_urc_lost) and TX-pattern discriminators as next steps. Align the TODO plan with the updated conclusion in the evidence doc.
second-to-last fragment, invisible to the crc_dump instrument. Plan in
#107: n=2 @ 902.5 → train-length sweep @ 902.5 → stats-probe deltas
(incl. the never-read tx_done_early counter) → software-only
activity-modulation legs (ethernet flood on the gigabit-marginal base
cable, USB, CPU) → hands only after the coupling path is named.
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
Comment on lines
+91
to
+93
| - The 88-frame firmware-drop figure is one bracketed leg; the base's dead | ||
| NRST makes counters cumulative, so brackets are cheap to repeat — do | ||
| n=2 next session. |
Comment on lines
+2259
to
+2261
| ~3.3 % is the RS-11.4 clustered process. **Opening desk finding: the | ||
| penultimate losses are NEVER-DEMODULATED, not corrupted** — at 902.5, | ||
| corrupt captures with readable frag headers are ~uniform across indices |
…ajor trigger component Legs E-F, counter-bracketed, plus tools/rs12_leg_report.py (one-command per-leg analysis: loss, idx histograms from both instruments, radio deltas, firmware-drop count). leg TX pattern fw drops penultimate lock loss D gap 40, depth 2 (base) 88 28% 8.2% E gap 120, depth 2 37 26% 5.0% F gap 40, depth 1 26 15% 3.9% Tripling the boundary gap left the lock intact (boundary-relative timing is NOT the trigger; gap width appears to modulate event DEPTH -- leg E's 37 drops were purely single-penultimate events). Removing the depth-2 drain halved the lock to 15%, below every prior 3000 B leg -- the two-stage mailbox drain that fingerprints each train's ending is a major component of the trigger, though a ~2x-uniform residual persists (at depth 1 the mailbox still drains 1->0). Instrument closure held in all three brackets (dcrc_err vs crc_dumps: 126/126, 126/124, 94/94) -- corruption and the silent drop remain fully separate channels. Sharpened target for the firmware pass: the L072 RX-URC path loses a pending frame when the transmitter's end-of-train drain cadence coincides with it; drain depth scales probability, gap width scales event depth. rx_urc_lost counter (flash, bench presence) names the line; depth-1 legs are the low-rate control. Also noted: depth 1 cost -6.5% offered throughput but delivered the series' lowest loss -- worth an n=3 A/B as an interim mitigation if the firmware fix stalls. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…rishable Overnight mains outage (neither carrier rebooted; uptimes 21:45/22:34 and counters continuous, so this is a pure RF-environment change). Re-swept 902-928 identically to yesterday: 11 of 53 channels (21%) flipped state overnight FOUR of the five channels clean yesterday are HOT today, including 902.5 -- the pinned escape -- which went -94 -> -44 dBm 909.0 went the other way: -43 -> -94 Two corrections to the RS-11.6 record, in-document: 1. "Standing recommendation: -ForceFrfHz 902500000" is WITHDRAWN. -ForceFrfHz is a per-session parameter chosen from a same-day survey, never a config constant. 2. "Device A is wideband or front-end leakage, carrier unlocalised" took the wrong branch: the device is a frequency HOPPER. Yesterday's "-43 on essentially every channel" was one emitter seen across a 24-minute sweep; the outage restarted it into a different sequence. This restores the AMI/utility-meter hypothesis leg 2 had discarded. Also recorded: an intermediate "the emitter is gone" claim made during this session off a single 915 MHz sniff was WRONG -- targeted sweeps found it immediately at -42..-46 on 922-925. One channel is never evidence about a hopper. RS-12 IS UNAFFECTED. Re-baseline at the freshly-validated 909.0 (leg G, counter-bracketed) reproduces the mechanism on a different clean channel: penultimate lock 35% (vs 28%), firmware drop 78 frames (vs 88), crc closure and identity residue both intact. The race is channel-independent and the day-1 conclusions stand. New tool: tools/survey_compare.py -- ranks zero-hot legal centers (500 kHz occupied BW inside 902-928) and diffs two surveys. Validated by re-deriving yesterday's 902.5 pick from yesterday's data; it also correctly rejects 902.0, which is not a legal center. Operational consequence: survey before any measurement campaign -- had we trusted yesterday's pin, leg G would have run on a -44 dBm channel and shown a "regression" that would have been hunted in firmware. RS-11.7's v0 button is promoted from nice-to-have to practical prerequisite; survey_compare.py already implements its recommendation logic. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…t cause them Six interleaved 300 s legs at 909.0 (2,1,2,1,2,1), every leg counter-bracketed. New driver run_rs12_depth_ab.ps1 + aggregator tools/rs12_ab_summary.py (exact permutation test; validated to return 0.50 on overlapping arms and 0.10 -- its floor at n=3 -- at maximum separation). metric depth2 depth1 perm p penultimate lock 37.1% +/-2.5 6.6% +/-2.0 0.10 raw loss 3.8% +/-0.4 2.1% +/-0.1 0.10 goodput 2010 B/s +/-14 1858 +/-8 0.10 FIRMWARE DROPS 25.7 +/-11.0 24.0 +/-8.5 0.90 <- NO DIFFERENCE THE CORRECTION. Dropping mailbox depth removes the penultimate lock but does NOT reduce the number of silently-dropped frames. The depth-2 drain is a PHASE SELECTOR, not a trigger: the same ~25 demodulated frames per leg vanish inside the L072 either way, landing on the penultimate at depth 2 and scattering at depth 1. The 2026-08-16 addendum called the drain "a major component of the trigger" from a single leg that showed both a halved lock and the series' lowest drop count; at n=3 the drop counts are indistinguishable, so that leg's low count was spread and the trigger claim is withdrawn (superseded in place in the day-1 document). Loss still improves because ~25 aimed losses kill trains outright while ~25 scattered ones damage many trains shallowly. Also corrected: depth-1 losses are NOT uniform, only no longer penultimate-locked -- chi2(12df) 155.9 -> 25.6, still above the 21.0 critical value, with an unexplained mild excess at idx 2. An earlier "essentially uniform" phrasing overstated it. RECOMMENDATION: do NOT adopt depth 1 as a mitigation. It buys -1.7 pp loss for -7.6% goodput while leaving the defect untouched, and it erases the phase-lock -- currently our most legible signature. It IS the right substrate for the rx_urc_lost instrumentation pass, where scattered drops are easier to localize than phase-locked ones. Caveat recorded: pre/post channel checks caught the hopper wandering onto 909.0 mid-series (-41 dBm vs -94 an hour earlier). Interleaving absorbed it -- the arm comparison holds -- but these absolute loss figures sit on a mildly contaminated channel and do not compare with day-1 legs. Tooling: the driver first wrote EMPTY archive paths (the harness reports via Write-Host, which bypasses pipeline capture); mapping was rebuilt from timestamps and verified leg-by-leg against each params.txt, and the driver now detects new archive directories instead. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…3 clean Possible second mains blink; both carriers again rode through on continuous uptime (>1 day each), so this is a pure RF re-check. The band reshuffled AGAIN in ~4.5 h: 10/53 channels flipped, the morning's 909.0 pick went -94 -> -41 (confirming the mid-depth-A/B contamination the pre/post checks caught), and 924.0 lit up at -30 dBm -- device B's signature level. Flip rate is ~20% per interval at both ~14 h and ~4.5 h spacings: a channel pick is valid for HOURS, not days. survey_compare.py gains a --history stability mode (clean-in-N-of-M ranking across accumulated surveys). Across all three sweeps exactly one channel has a perfect record: 927.5 MHz clean 3/3 (every other channel: at most 1/3) and it is also today's absolute quietest (-96 dBm). STANDING PICK: -ForceFrfHz 927500000, still subject to the same-day check before any measurement leg. Plausible physics recorded as hypothesis: 927.5's occupied BW grazes the 928 band edge, which hopping devices back away from -- the hopper may never park there. RS-11.7 v1 consequence: rank by stability across accumulated surveys, not per-day quietness -- the --history mode is that logic. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
… ride; host-side fix validated The campaign closes its search phase. Dual-clock instrumentation (RX frag_arrival with firmware us demod timestamps; TX txdone_arrival with toa_us) across legs H/I/J at the day's stability channel found the mechanism and validated a no-flash fix. MECHANISM. The last fragment of every train is SHORT (the ~36 B frame remainder, ~20 ms ToA) and rides the firmware's fire-on-TX_DONE ~42 ms behind the penultimate -- by design. The penultimate is therefore the only fragment whose 255 B URC must clear the L072's single pending-URC path in 42 ms instead of 117 ms; when emission slips, the successor silently overwrites the pending URC. This explains the penultimate lock, its train-length tracking, depth-1 collapsing it, gap/channel/power irrelevance, the role-swap asymmetry, and corrupt-capture uniformity. Confirmed at both ends: TX_DONE pair spacing med 42 ms with full 99904 us penultimate ToA (192 trains); RX demod pair gap med 41.7 ms vs 117.0 mid-train control. TWO WRONG TURNS RECORDED IN-DOCUMENT: (1) leg H's "TX compression" reading -- the 1.16-1.36-slot drop gaps are NORMAL once the short last fragment is accounted (117+42=159 ms); the len= field that settles it was in the instrument's own output, unread for two legs. (2) The "firmware drop" metric (drx_ok - host URCs) is ack-contaminated (~32/leg): host counts exclude command/ack frames, radio counts include them -- absolute drop figures in the earlier sections and the depth A/B carry that inflation; the localisation survives, the counts do not separate. FIX (host-side, no flash): LIFETRAC_NO_PARK_LAST=1 / -NoParkLast 1 -- the TX daemon holds the final fragment until the pipeline drains, so the last pair is host-paced. Leg J, n=1, bracketed: last-pair gap 41.7 -> 114.7 ms; penultimate lock 42% -> 7% (= uniform); loss 3.3% -> 1.7% (campaign best); timeouts 69 -> 37; published 117 -> 144; offered -2.4% (a third of depth-1's cost). Pending n=3. Flash-session brief shrinks from search to confirmation: rx_urc_lost at the overwrite site, then double-buffer the URC path or enforce minimum inter-fire spacing on parked fragments in firmware. Also: rs12_txrx_join.py (wrap-safe TX/RX train join), arrival analyzers, depth-AB driver capture fix, and the full instrumented-leg evidence. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…m-verified per leg Six legs at 927.5 (channel clean at both ends of the series), interleaved ctrl/fix, every leg bracketed and arrival-instrumented: per-leg mechanism check: ctrl 41.7 ms last-pair gap / 87% riding in all three; fix 114.7 ms / ~26% in all three -- no crossovers loss ctrl 3.3 +/-0.1 fix 1.8 +/-0.2 p=0.10 (floor) penultimate lock ctrl 42.6 +/-1.7 fix 11.6 +/-1.3 p=0.10 (floor) goodput cost -2.9% The arms are the tightest of the campaign (loss sd 0.1-0.2 pp vs the 2x swings of the search phase) -- the mechanism was the dominant variance source and is now controlled. Residuals stated in-document: ~26% of last pairs still ride (hold releases on TX_DONE receipt; stricter hold or the firmware double-buffer closes it), penultimate 11.6% vs 7.7% uniform tracks it, fw-drop metric remains ack-contaminated and non-load-bearing. DECISION: -NoParkLast 1 is the standing bench operating point. Default flip = maintainer call, framed both ways in the PR. Tooling: run_rs12_noparklast_ab.ps1 + rs12_ab_summary.py generalized to labeled arms (back-compatible with the depth series). Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…ditional The plain hold releases on TX_DONE receipt and raced the pacer (~26% of last pairs still rode <80 ms). LIFETRAC_NO_PARK_LAST_GAP_MS (default 80) now enforces a minimum delay between the penultimate's TX_DONE and the final submission; the hold branch keeps the frame alive with the standard 50 ms poll instead of abandoning it when nothing is inflight. Leg K (bracketed, channel clean): last pairs <80 ms = 0/174 (min 130.9 ms); penultimate 6% (at/below uniform); loss 1.5% -- campaign best; timeouts 32. Cost: offered -6.2% vs control (gap + poll quantization; median realized gap 161 ms vs the 117 target) -- tuning headroom noted in-document, correctness-first default kept while the firmware fix is pending. Flash-session prediction is now maximally crisp: rx_urc_lost == 0 on a strict-hold leg, ~= timeouts on a control leg. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Contributor
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 69 out of 156 changed files in this pull request and generated 1 comment.
Suppressed comments (2)
LifeTrac-v25/DESIGN-CONTROLLER/TODO.md:2263
- This TODO records the opposite of the later evidence in this PR: the final analysis says the penultimate is demodulated and then lost in the RX→URC path, and the depth/NoParkLast experiments have already replaced the listed search plan. Leaving the campaign tracker at the opening hypothesis will direct future work toward a receiver-deafness mechanism that this PR supersedes.
~3.3 % is the RS-11.4 clustered process. **Opening desk finding: the
penultimate losses are NEVER-DEMODULATED, not corrupted** — at 902.5,
corrupt captures with readable frag headers are ~uniform across indices
(idx-11 = 9 % ≈ 1/13) while attributed losses concentrate at idx-11
(35 %). Combined with the RS-11.5 exonerations (TX FIFO readback clean,
LifeTrac-v25/DESIGN-CONTROLLER/bench-evidence/RS_12_bulk_floor_2026-08-16/RESULTS.md:169
- This correction contradicts the PR description and §1's exact claim that all 88 excess
rx_okframes were lost URCs. Because roughly 32 are expected command/ack frames andrx_okcannot identify which radiated data fragments it counted, the counter delta does not prove “the radio demodulates everything” or an exact 88-frame URC loss. Update the PR description and the earlier headline/count claims to retain only the supported localization evidence.
**Wrong turn 2 (this document's §1): the "firmware drop" metric is
ack-contaminated.** Host `rx_frames` excludes command/ack frames (they are
dispatched before the counter increments) while radio `rx_ok` includes
them, so every Δrx_ok−URCs figure in §1/§6 and the depth A/B carries
roughly the leg's ack traffic (~32 frames: 16 base commands × 2 tractor
| # take the same 50 ms poll the wait-loop uses, then | ||
| # re-check the gate. Without this the plain | ||
| # `if not inflight: break` would abandon the frame. | ||
| link.read_frames(0.05) |
…g + clock caveats Three review findings (Copilot), all verified before acting: 1. FUNCTIONAL: the strict hold's discard-poll bypassed the event body that dispatches inbound RX_FRAME_URCs, opening an 80-130 ms command-deaf window per train -- invisible on the bench (commands fly only at session start) but wrong for production. The hold now FALLS THROUGH to the normal event body (same 50 ms poll, full dispatch). Verification leg L (dde2c8a): guarantee intact (0/175 pairs <80 ms, min 131.0), penultimate 8% = uniform, loss 0.9% / timeouts 22 / published 154 -- all campaign bests. 2. TODO's "NEVER-DEMODULATED" phrasing was the campaign's OPENING finding, overturned by the final mechanism (frame demodulates fully; its URC is overwritten). Dated supersession note added pointing at the final evidence. 3. Per-board log clocks are unsynchronized (~9 days apart) -- now stated as a caveat in the evidence bundle itself: per-board deltas only, never cross-log wall-time correlation. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Contributor
Author
|
All three review findings verified and addressed (commit on branch):
The command-plane catch also sharpens the default-flip framing: the fix's loss numbers are solid, but its command-plane interaction has had one review pass and zero bench exercise with live mid-train traffic — another reason |
dorkmo
added a commit
that referenced
this pull request
Aug 22, 2026
…te red since #108 The strict-hold review fix (5cc4dd9, PR #108) introduced a bare int(os.environ.get(...)) parse that the EnvContractTests gate rejects; main CI has been red since that merge (2026-08-18) and PR #111 inherited it. Now uses the sanctioned _env_int helper with lo=0 (negative gap is meaningless) and hi=5000 ms (a gap beyond that starves the pacer). Verified locally: EnvContractTests 5/5 pass. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Opens the bulk-floor campaign (#107). Four legs at 902.5 MHz, closing with an L072-counter-bracketed leg.
The clustered bulk loss is not RF. The radio demodulates everything (
Δrx_ok+Δcrc_err≥ radiated;Δcrc_err == crc_dumpsexactly), then 88 demodulated frames vanish inside the L072's URC path with every drop counter at zero and zero host parse errors. The loss locks to the second-to-last fragment and follows train structure (idx 11 → idx 5 when trains shrink to 7 fragments; 53 % there, matching RS-11.4's 50 %). Since the L072 doesn't parse fragment headers, the phase-lock must be induced by the transmitter's end-of-train pattern — which also finally explains the RS-11.5 role-swap asymmetry.Also in the evidence doc: run-to-run variance is event-depth not event-rate (loss 3.3–8.2 % while timeouts/publishes stay fixed), and an on-the-record correction to RS-11.6 leg 3's "escape halves the loss" (n=1 vs n=1 under ~2× spread; the confirmed effect is removal of the periodic interference population).
Next steps per #107: a firmware
rx_urc_lostcounter at the suspected race (needs a flash — bench presence), and no-flash TX-pattern discriminators (TrainGapMs,TxPipelineDepth).Not merging until you say so.
🤖 Generated with Claude Code