Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
21 changes: 21 additions & 0 deletions LifeTrac-v25/DESIGN-CONTROLLER/TODO.md
Original file line number Diff line number Diff line change
Expand Up @@ -2254,6 +2254,27 @@ Two consequences for what is worth doing next:
ARTIFACT (a fold scan peaks at ~the mean inter-arrival with no clock
present) and is retracted — the shuffled-interval null that catches it
is now part of the tool.
- [ ] **RS-12 — the clustered bulk loss floor (campaign opened 2026-08-16,
issue #107). ⚠️ WORDING SUPERSEDED (review catch, PR #108): the
"NEVER-DEMODULATED" phrasing below reflects the opening desk finding
and was overturned by the final mechanism — the radio DOES demodulate
the penultimate (full ToA at TX, rx_ok counts it); its URC is then
silently overwritten when the short final fragment rides fire-on-
TX_DONE 42 ms behind it. Mechanism + validated fix (NoParkLast):
bench-evidence/RS_12_bulk_floor_2026-08-16 §7 and
RS_12_noparklast_ab_2026-08-17. Loss floor now 1.5%.** With the interferers escaped (902.5 MHz), the remaining
~3.3 % is the RS-11.4 clustered process. **Opening desk finding: the
penultimate losses are NEVER-DEMODULATED, not corrupted** — at 902.5,
corrupt captures with readable frag headers are ~uniform across indices
Comment on lines +2266 to +2268
(idx-11 = 9 % ≈ 1/13) while attributed losses concentrate at idx-11
(35 %). Combined with the RS-11.5 exonerations (TX FIFO readback clean,
receiver-side by role swap, power-independent) the target is: a
base-board receiver-deafness mechanism, phase-locked to the
second-to-last fragment, invisible to the crc_dump instrument. Plan in
#107: n=2 @ 902.5 → train-length sweep @ 902.5 → stats-probe deltas
(incl. the never-read tx_done_early counter) → software-only
activity-modulation legs (ethernet flood on the gigabit-marginal base
cable, USB, CPU) → hands only after the coupling path is named.
- [ ] **RS-11.7 Operator spectrum survey in web_ui (proposed 2026-08-16).**
Turn the channel-survey diagnostic into a field feature: a
maintenance-mode button that pauses the RX daemon, sweeps 902–928 MHz
Expand Down
15 changes: 15 additions & 0 deletions LifeTrac-v25/DESIGN-CONTROLLER/base_station/image_rx_daemon.py
Original file line number Diff line number Diff line change
Expand Up @@ -220,6 +220,16 @@ def _env_float(name: str, default: float, lo: "float | None" = None,
# convergence soak actually measured, kept as a bench A/B control.
ALIGNED_PUMP_ENABLE = os.environ.get("LIFETRAC_ALIGNED_PUMP", "1") != "0"

# RS-12 (2026-08-17): per-fragment arrival logging. One INFO line per data
# fragment carrying the FIRMWARE microsecond RX timestamp, so the silent
# L072 URC drop's timing shape can be read without a firmware flash: in a
# train missing idx N-2, the fw_us gap between idx N-3 and idx N-1 says
# whether the lost fragment's slot passed on schedule (pure emission loss),
# was compressed (TX timing anomaly), or was followed by a stall (main-loop
# blockage). ~2300 lines per 300 s leg at the bench rate — diagnostic only.
LOG_FRAG_ARRIVALS = os.environ.get(
"LIFETRAC_LOG_FRAG_ARRIVALS", "0") == "1"


class KeyframeRequester:
"""Rate-limited req_keyframe on reassembly failure. Publishes to the
Expand Down Expand Up @@ -1223,6 +1233,11 @@ def _rx_worker(self) -> None:
self._gap_samples.append(
(d, len(data), self._last_gap_class))

if LOG_FRAG_ARRIVALS and frag_idx is not None:
LOG.info("frag_arrival: seq=%d idx=%d total=%d fw_us=%s "
"len=%d", frag_seq, frag_idx, frag_total,
parsed.get("timestamp_us"), len(data))

saw_rx = True
with self._lock:
self.stats.rx_frames_seen += 1
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,100 @@
# Day-2 survey (post-outage) — the interferer HOPS; a pinned channel is perishable

SHA `6d248b63`. Trigger: a thunderstorm/mains outage overnight
2026-08-16→17. Neither carrier rebooted (uptime 21:45 / 22:34, counters
continuous), so this is a pure RF-environment change.

Transcript: [`survey_base_day2.jsonl`](survey_base_day2.jsonl).
Tools: `channel_survey_sniff.py`, `tools/survey_compare.py` (new).

## 1. What the outage did to the band

Identical sweep to 2026-08-16 (902–928 MHz, 500 kHz steps, 30 s dwell):

| | 2026-08-16 | 2026-08-17 |
|---|---:|---:|
| hot channels | 46 / 53 | 43 / 53 |
| zero-hot legal centers | 5 | 8 |

**11 of 53 channels (21 %) flipped state overnight**, and the flip was
catastrophic for the pin we had chosen:

| channel | 2026-08-16 | 2026-08-17 |
|---|---|---|
| **902.5 — the pinned escape** | clean, −94 | **HOT, −44** |
| 905.5 | clean, −86 | **HOT, −43** |
| 907.0 | clean, −77 | **HOT, −43** |
| 925.5 | clean, −94 | **HOT, −44** |
| 927.5 | clean, −75 | clean, −94 |
| 909.0 | HOT, −43 | clean, −94 |
| 918.0 / 918.5 / 921.0 / 924.0 | HOT, −42…−53 | clean, −91…−94 |

**Four of the five channels that were clean yesterday are hot today.**

### Correction to the RS-11.6 record

Two earlier readings are superseded:

1. **"Standing recommendation: bench legs run `-ForceFrfHz 902500000`"**
(leg-3 RESULTS §3.2) — withdrawn. That channel is now among the worst.
`-ForceFrfHz` is a **per-session parameter chosen from a same-day
survey**, never a config constant.
2. **"Device A is wideband or front-end leakage; carrier unlocalised"**
(leg-3 RESULTS §1) — wrong branch. The device is a **frequency
hopper**: yesterday's "−43 dBm on essentially every channel" was one
emitter observed across a 24-minute sweep, and the outage restarted it
into a different channel sequence. This restores the original
utility-meter/AMI hypothesis that leg 2 had talked itself out of.

Also worth recording: an intermediate claim made during this session —
"the emitter is gone" — was made off a single 120 s sniff at 915 MHz and
was **wrong**; targeted sweeps immediately found it alive at −42…−46 dBm
on 922–925 with clean ~7.0 s intervals. One channel is never evidence
about a hopper.

## 2. Today's pick, and a stability alternative

`survey_compare.py` ranks zero-hot legal centers by quietest peak:

```
RECOMMENDED: -ForceFrfHz 909000000 (909.0 MHz, hot=0, max=-94 dBm)
runners-up: 918.0, 924.0, 927.5 (all hot=0, max -94)
```

**927.5 MHz was the only channel clean on BOTH days** (−75 → −94). If
run-to-run comparability matters more than today's absolute quietest, it
is the better long-run pin — two days is a thin basis, but it is the only
stability signal available and worth accumulating.

## 3. Re-baseline at 909.0 (leg G) — and RS-12 is untouched by any of this

Standard 300 s leg, counter-bracketed
(`../RS_12_bulk_floor_2026-08-16/stats_pre_G.txt` / `stats_post_G.txt`):

| metric | leg G @ 909.0 (day 2) | leg D @ 902.5 (day 1) |
|---|---:|---:|
| raw loss | 6.8 % | 8.2 % |
| firmware drop (Δrx_ok − URCs) | **78** | 88 |
| penultimate lock | **35 %** | 28 % |
| crc closure | 94 vs 91 | 126 vs 126 |
| identity residue | 8 | 9 |

The RS-12 mechanism reproduces on a completely different, freshly-clean
channel: the penultimate lock is intact at 35 % and ~78 demodulated
frames per leg still vanish inside the L072. **The firmware race is
channel-independent**, as the counter separation always implied — the
day-1 conclusions stand without re-work.

## 4. Operational consequences

1. **Survey before any measurement campaign.** Not hygiene — correctness.
Had we trusted yesterday's pin, leg G would have run on a −44 dBm
channel and shown a large "regression" that would have been hunted in
firmware. 28 minutes unattended prevents that.
2. **RS-11.7 (operator survey in web_ui) is promoted** from
nice-to-have to the practical prerequisite for field measurement. Its
v0 (survey + recommend, manual apply) is exactly the workflow used
here, and `survey_compare.py` already implements the recommendation
logic the button needs.
3. Record `force_frf_hz` with every citation of a run — it is now a
varying parameter (already in `params.txt`).
Original file line number Diff line number Diff line change
@@ -0,0 +1,132 @@
SURVEY_META {"start_hz": 902000000, "stop_hz": 928000000, "step_hz": 500000, "dwell_s": 30.0, "ts_start": 1786979701.1183116}

Initializing regulatory profile 2 on co-processor...

FHSS mask: single channel 0 (0100000000000000)

CFG_SET_REQ(FHSS_CHANNEL_MASK) OK: 07000800

CFG_SET_REQ(ANTENNA_GAIN_DBI) OK: 15000100

CFG_SET_REQ(HW_CEILING_DBM) OK: 16000100

FHSS seed: (0,0) legacy shared permutation [set LIFETRAC_FHSS_FARM_ID / LIFETRAC_FHSS_LINK_ID to separate pairs]

CFG_SET_REQ(FHSS_FARM_ID) OK: 17000800

CFG_SET_REQ(FHSS_NODE_ID) OK: 18000800

CFG_SET_REQ(REG_PROFILE=2) OK: 14000100

forcing FRF -> 915.000 MHz (raw=0xE4C000 -> Msb=0xE4 Mid=0xC0 Lsb=0x00)

FRF readback: 0xE4C000 = 915.000 MHz (OK)

SURVEY_CH {"freq_hz": 902000000, "n": 457, "hot": 0, "max_dbm": -94, "hot_ts": []}

SURVEY_CH {"freq_hz": 902500000, "n": 456, "hot": 4, "max_dbm": -44, "hot_ts": [1786979739.099, 1786979745.667, 1786979751.279, 1786979762.275]}

SURVEY_CH {"freq_hz": 903000000, "n": 457, "hot": 1, "max_dbm": -43, "hot_ts": [1786979778.282]}

SURVEY_CH {"freq_hz": 903500000, "n": 456, "hot": 2, "max_dbm": -43, "hot_ts": [1786979800.667, 1786979812.198]}

SURVEY_CH {"freq_hz": 904000000, "n": 456, "hot": 1, "max_dbm": -43, "hot_ts": [1786979843.182]}

SURVEY_CH {"freq_hz": 904500000, "n": 456, "hot": 1, "max_dbm": -43, "hot_ts": [1786979875.092]}

SURVEY_CH {"freq_hz": 905000000, "n": 456, "hot": 3, "max_dbm": -45, "hot_ts": [1786979884.148, 1786979892.704, 1786979905.211]}

SURVEY_CH {"freq_hz": 905500000, "n": 456, "hot": 1, "max_dbm": -43, "hot_ts": [1786979916.164]}

SURVEY_CH {"freq_hz": 906000000, "n": 456, "hot": 2, "max_dbm": -45, "hot_ts": [1786979948.452, 1786979966.195]}

SURVEY_CH {"freq_hz": 906500000, "n": 455, "hot": 1, "max_dbm": -42, "hot_ts": [1786979999.179]}

SURVEY_CH {"freq_hz": 907000000, "n": 456, "hot": 1, "max_dbm": -43, "hot_ts": [1786980015.629]}

SURVEY_CH {"freq_hz": 907500000, "n": 455, "hot": 2, "max_dbm": -33, "hot_ts": [1786980040.172, 1786980055.148]}

SURVEY_CH {"freq_hz": 908000000, "n": 456, "hot": 3, "max_dbm": -42, "hot_ts": [1786980068.239, 1786980075.227, 1786980082.209]}

SURVEY_CH {"freq_hz": 908500000, "n": 455, "hot": 0, "max_dbm": -82, "hot_ts": []}

SURVEY_CH {"freq_hz": 909000000, "n": 456, "hot": 0, "max_dbm": -94, "hot_ts": []}

SURVEY_CH {"freq_hz": 909500000, "n": 455, "hot": 3, "max_dbm": -42, "hot_ts": [1786980157.138, 1786980171.123, 1786980178.176]}

SURVEY_CH {"freq_hz": 910000000, "n": 455, "hot": 3, "max_dbm": -40, "hot_ts": [1786980192.056, 1786980199.114, 1786980213.051]}

SURVEY_CH {"freq_hz": 910500000, "n": 455, "hot": 3, "max_dbm": -41, "hot_ts": [1786980219.163, 1786980233.232, 1786980240.24]}

SURVEY_CH {"freq_hz": 911000000, "n": 456, "hot": 4, "max_dbm": -41, "hot_ts": [1786980247.135, 1786980254.184, 1786980261.111, 1786980275.416]}

SURVEY_CH {"freq_hz": 911500000, "n": 455, "hot": 2, "max_dbm": -41, "hot_ts": [1786980280.118, 1786980287.128]}

SURVEY_CH {"freq_hz": 912000000, "n": 456, "hot": 2, "max_dbm": -42, "hot_ts": [1786980312.179, 1786980333.268]}

SURVEY_CH {"freq_hz": 912500000, "n": 455, "hot": 1, "max_dbm": -43, "hot_ts": [1786980351.045]}

SURVEY_CH {"freq_hz": 913000000, "n": 455, "hot": 4, "max_dbm": -40, "hot_ts": [1786980372.25, 1786980380.108, 1786980386.438, 1786980394.095]}

SURVEY_CH {"freq_hz": 913500000, "n": 455, "hot": 1, "max_dbm": -43, "hot_ts": [1786980401.19]}

SURVEY_CH {"freq_hz": 914000000, "n": 455, "hot": 2, "max_dbm": -40, "hot_ts": [1786980440.399, 1786980455.052]}

SURVEY_CH {"freq_hz": 914500000, "n": 455, "hot": 2, "max_dbm": -40, "hot_ts": [1786980471.082, 1786980485.268]}

SURVEY_CH {"freq_hz": 915000000, "n": 455, "hot": 2, "max_dbm": -40, "hot_ts": [1786980499.1, 1786980513.108]}

SURVEY_CH {"freq_hz": 915500000, "n": 454, "hot": 2, "max_dbm": -40, "hot_ts": [1786980522.187, 1786980536.218]}

SURVEY_CH {"freq_hz": 916000000, "n": 455, "hot": 1, "max_dbm": -41, "hot_ts": [1786980551.402]}

SURVEY_CH {"freq_hz": 916500000, "n": 454, "hot": 1, "max_dbm": -40, "hot_ts": [1786980605.236]}

SURVEY_CH {"freq_hz": 917000000, "n": 454, "hot": 1, "max_dbm": -40, "hot_ts": [1786980632.41]}

SURVEY_CH {"freq_hz": 917500000, "n": 454, "hot": 2, "max_dbm": -40, "hot_ts": [1786980647.159, 1786980654.158]}

SURVEY_CH {"freq_hz": 918000000, "n": 455, "hot": 0, "max_dbm": -94, "hot_ts": []}

SURVEY_CH {"freq_hz": 918500000, "n": 454, "hot": 0, "max_dbm": -93, "hot_ts": []}

SURVEY_CH {"freq_hz": 919000000, "n": 454, "hot": 3, "max_dbm": -40, "hot_ts": [1786980731.23, 1786980738.242, 1786980760.123]}

SURVEY_CH {"freq_hz": 919500000, "n": 454, "hot": 1, "max_dbm": -40, "hot_ts": [1786980780.166]}

SURVEY_CH {"freq_hz": 920000000, "n": 454, "hot": 2, "max_dbm": -41, "hot_ts": [1786980812.214, 1786980818.428]}

SURVEY_CH {"freq_hz": 920500000, "n": 454, "hot": 3, "max_dbm": -41, "hot_ts": [1786980833.423, 1786980840.169, 1786980847.173]}

SURVEY_CH {"freq_hz": 921000000, "n": 454, "hot": 0, "max_dbm": -91, "hot_ts": []}

SURVEY_CH {"freq_hz": 921500000, "n": 454, "hot": 2, "max_dbm": -42, "hot_ts": [1786980889.258, 1786980910.677]}

SURVEY_CH {"freq_hz": 922000000, "n": 454, "hot": 0, "max_dbm": -79, "hot_ts": []}

SURVEY_CH {"freq_hz": 922500000, "n": 453, "hot": 4, "max_dbm": -35, "hot_ts": [1786980950.038, 1786980956.393, 1786980964.049, 1786980971.06]}

SURVEY_CH {"freq_hz": 923000000, "n": 454, "hot": 3, "max_dbm": -42, "hot_ts": [1786980988.092, 1786980995.174, 1786981002.125]}

SURVEY_CH {"freq_hz": 923500000, "n": 454, "hot": 3, "max_dbm": -43, "hot_ts": [1786981008.13, 1786981022.172, 1786981029.063]}

SURVEY_CH {"freq_hz": 924000000, "n": 454, "hot": 0, "max_dbm": -94, "hot_ts": []}

SURVEY_CH {"freq_hz": 924500000, "n": 455, "hot": 3, "max_dbm": -43, "hot_ts": [1786981065.064, 1786981072.048, 1786981086.05]}

SURVEY_CH {"freq_hz": 925000000, "n": 454, "hot": 1, "max_dbm": -44, "hot_ts": [1786981108.177]}

SURVEY_CH {"freq_hz": 925500000, "n": 455, "hot": 1, "max_dbm": -44, "hot_ts": [1786981136.042]}

SURVEY_CH {"freq_hz": 926000000, "n": 456, "hot": 3, "max_dbm": -45, "hot_ts": [1786981158.06, 1786981176.039, 1786981182.11]}

SURVEY_CH {"freq_hz": 926500000, "n": 453, "hot": 3, "max_dbm": -44, "hot_ts": [1786981187.198, 1786981193.417, 1786981205.2]}

SURVEY_CH {"freq_hz": 927000000, "n": 454, "hot": 2, "max_dbm": -46, "hot_ts": [1786981224.041, 1786981238.603]}

SURVEY_CH {"freq_hz": 927500000, "n": 453, "hot": 0, "max_dbm": -94, "hot_ts": []}

SURVEY_CH {"freq_hz": 928000000, "n": 455, "hot": 0, "max_dbm": -100, "hot_ts": []}

SURVEY_SUMMARY {"hot_channels": [{"freq_hz": 902500000, "hot": 4, "max_dbm": -44}, {"freq_hz": 903000000, "hot": 1, "max_dbm": -43}, {"freq_hz": 903500000, "hot": 2, "max_dbm": -43}, {"freq_hz": 904000000, "hot": 1, "max_dbm": -43}, {"freq_hz": 904500000, "hot": 1, "max_dbm": -43}, {"freq_hz": 905000000, "hot": 3, "max_dbm": -45}, {"freq_hz": 905500000, "hot": 1, "max_dbm": -43}, {"freq_hz": 906000000, "hot": 2, "max_dbm": -45}, {"freq_hz": 906500000, "hot": 1, "max_dbm": -42}, {"freq_hz": 907000000, "hot": 1, "max_dbm": -43}, {"freq_hz": 907500000, "hot": 2, "max_dbm": -33}, {"freq_hz": 908000000, "hot": 3, "max_dbm": -42}, {"freq_hz": 909500000, "hot": 3, "max_dbm": -42}, {"freq_hz": 910000000, "hot": 3, "max_dbm": -40}, {"freq_hz": 910500000, "hot": 3, "max_dbm": -41}, {"freq_hz": 911000000, "hot": 4, "max_dbm": -41}, {"freq_hz": 911500000, "hot": 2, "max_dbm": -41}, {"freq_hz": 912000000, "hot": 2, "max_dbm": -42}, {"freq_hz": 912500000, "hot": 1, "max_dbm": -43}, {"freq_hz": 913000000, "hot": 4, "max_dbm": -40}, {"freq_hz": 913500000, "hot": 1, "max_dbm": -43}, {"freq_hz": 914000000, "hot": 2, "max_dbm": -40}, {"freq_hz": 914500000, "hot": 2, "max_dbm": -40}, {"freq_hz": 915000000, "hot": 2, "max_dbm": -40}, {"freq_hz": 915500000, "hot": 2, "max_dbm": -40}, {"freq_hz": 916000000, "hot": 1, "max_dbm": -41}, {"freq_hz": 916500000, "hot": 1, "max_dbm": -40}, {"freq_hz": 917000000, "hot": 1, "max_dbm": -40}, {"freq_hz": 917500000, "hot": 2, "max_dbm": -40}, {"freq_hz": 919000000, "hot": 3, "max_dbm": -40}, {"freq_hz": 919500000, "hot": 1, "max_dbm": -40}, {"freq_hz": 920000000, "hot": 2, "max_dbm": -41}, {"freq_hz": 920500000, "hot": 3, "max_dbm": -41}, {"freq_hz": 921500000, "hot": 2, "max_dbm": -42}, {"freq_hz": 922500000, "hot": 4, "max_dbm": -35}, {"freq_hz": 923000000, "hot": 3, "max_dbm": -42}, {"freq_hz": 923500000, "hot": 3, "max_dbm": -43}, {"freq_hz": 924500000, "hot": 3, "max_dbm": -43}, {"freq_hz": 925000000, "hot": 1, "max_dbm": -44}, {"freq_hz": 925500000, "hot": 1, "max_dbm": -44}, {"freq_hz": 926000000, "hot": 3, "max_dbm": -45}, {"freq_hz": 926500000, "hot": 3, "max_dbm": -44}, {"freq_hz": 927000000, "hot": 2, "max_dbm": -46}]}

Original file line number Diff line number Diff line change
@@ -0,0 +1,52 @@
# Third band survey — 927.5 MHz is the stability pick (3/3 clean)

SHA at sweep: `8ef0aa5d`. Trigger: possible second mains blink (both
carriers again survived on continuous uptime — this is a pure RF check).
Transcript: [`survey_base_day2b.jsonl`](survey_base_day2b.jsonl).
Analysis: `tools/survey_compare.py` (now with `--history` stability mode).

## 1. The band reshuffled again — in ~4.5 hours this time

10 of 53 channels flipped vs the morning survey (19 %), including:

- **909.0 — the morning's pick and the depth-A/B channel — went −94 → −41**
(confirming the mid-A/B contamination the pre/post checks caught)
- 918.0 went −94 → −40; 921.0 went −91 → −41
- **924.0 lit up at −30 dBm — device B's signature level**, its first
reappearance at that strength since the day-1 survey localized it

Flip rates are now ~20 % per survey interval at both ~14 h and ~4.5 h
spacings, so the reshuffle cadence is faster than daily and may not need
a power event at all. Either way the operational rule is unchanged and
now triply confirmed: **a channel pick is valid for hours, not days.**

## 2. The stability ranking — one channel stands alone

Across all three surveys (`--history` mode):

```
927.5 MHz clean 3/3 worst max −75 dBm <== clean in every survey
(every other channel: clean in at most 1 of 3)
```

927.5 is also today's absolute quietest (−96 dBm). **Standing bench
recommendation: `-ForceFrfHz 927500000`**, held not because any pin is
durable but because it is the only channel with a perfect record — and
still subject to the same-day check before any measurement leg
(`hunt_sniff.ps1`-style spot check or the sweep).

Plausible physics for why the top edge stays clean: 927.5+250 kHz grazes
the 928 MHz band edge, which hopping devices typically back away from to
keep their occupied bandwidth legal — the same reason our own 902.0 was
rejected as a center. The hopper may simply never park there. Two more
clean surveys would make that a confident claim; it is a hypothesis today.

## 3. Consequences

1. RS-12's flash-session leg should run at **927.5** with the usual
same-day verification.
2. RS-11.7 v1 should rank by **stability across accumulated surveys**,
not per-day quietness — `survey_compare.py --history` is that logic,
already written and validated.
3. The depth-A/B absolute loss numbers (909.0, mid-drift) keep their
contamination caveat; the arm comparison stands.
Loading
Loading