Skip to content

FilterReview: show how much raw IMU data is usable in the analysis time - #376

Open
j-w9 wants to merge 6 commits into
ArduPilot:mainfrom
j-w9:filter-review-missing-data
Open

j-w9 wants to merge 6 commits into
ArduPilot:mainfrom
j-w9:filter-review-missing-data

Conversation

@j-w9

@j-w9 j-w9 commented Oct 24, 2026 •

Copy link
Copy Markdown
Contributor

"Warn if lots of missing data" from #10.

For raw IMU logs, each gyro's info now shows "Usable data: N% of analysis time", the share of the analysis time covered by the FFT windows that are used. Data is lost where it is missing from the log (the raw loader splits the data into sections at gaps), and at the end of each section where there isn't room for another whole window, so the number changes with the window size. It is clipped to the time covered by all IMUs, so an IMU that starts late or stops early counts as lost.

The line is always there so the gyro boxes stay the same height, and turns red below 90% (usable_data_warn_fraction, happy to change it).

Batch logs are not shown. The batch sampler logs short bursts by design and waits rather than dropping data when the log buffer is full (BatchSampler.cpp, "maybe later"), so the coverage there isn't lost data.

Checked with continuous synthetic raw data over a 47 s analysis time at 1 kHz, with the expected window coverage:

256 1024 8192
continuous 100% 99% 96%
one 25% gap 75% 74% 70%
four 1% gaps 95% 94% 87%

@tridge tridge added the AIReview label Nov 9, 2026
@AP-Review

AP-Review commented Nov 9, 2026 •

Copy link
Copy Markdown

Deprecated — see below for the updated review.

Previous review (2026-11-09)

Automated review note — AI-generated (Claude+Codex), validated against the live diff. Please sanity-check before acting.
Verdict: COMMENT

Reviewed at head 2a4ff15887.
Full report: https://firmware.ardupilot.org/Tools/APReview/DevCallReviews/PRReviews/ardupilot/webtools/376/1.html#prWebTools-376

This adds a useful missing-data warning, and interior gaps are reported correctly, but gaps at the edges of the analysis window are never counted.

Should fix

  • FilterReview/FilterReview.js:300: get_missing_fraction clips the window to this IMU's own first/last batch. An IMU that stops or starts partway through the window, or has no data in it at all, shows 0% and the warning stays hidden (multi-IMU or pre+post raw logs). Clip to Gyro_batch.start_time / Gyro_batch.end_time instead (the union over all instances, already set by load_from_raw_log) and drop first/last. In testing this gives 50%/90%/100% for those cases and still 0% for continuous data.

Short batches that the FFT skips are counted as covered, but that is existing FFT behaviour, not missing data, so it is not raised as an issue.

Clip to the time covered by all IMUs rather than this IMU's own data, so
an IMU that starts late, stops early or has no data in the window is
counted as missing.
@AP-Review

AP-Review commented Nov 9, 2026 •

Copy link
Copy Markdown

Deprecated — see below for the updated review.

Previous review (2026-11-09)

Automated review note — AI-generated (Claude+Codex), validated against the live diff. Please sanity-check before acting.
Verdict: ACCEPT

Reviewed at head 34bde32cf1.
Full report: https://firmware.ardupilot.org/Tools/APReview/DevCallReviews/PRReviews/ardupilot/webtools/376/2.html#prWebTools-376

This PR now looks good to merge: the missing-data warning is correct for interior gaps and for gaps at the edges of the analysis window. Since the last review, 34bde32 clips to the shared Gyro_batch.start_time/end_time, which fixes the edge-of-window false negative (FilterReview/FilterReview.js:299). An IMU that starts late, stops early or has no data in the window now reports 50%/100% instead of 0%, and continuous data still shows 0%.

The earlier note about short batches the FFT skips still stands as a non-issue.

Not fully reviewed: tested only with synthetic gyro data, not with a real raw-IMU .bin log loaded in a browser.

@IamPete1 IamPete1 left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I like the idea, but I think this needs to recalculated on the FFT object because it can change with window size. Smaller batches are less likely to run into missing bits. Larger batches can waste a lot of data if there is not quite enough room for another whole window.

So the end result would be a percentage of usable data which would be some percentage from the "batching" and then a calculation on the FFT coverage of the batches based on the window size.

I think we could always show the number, but only make it red if there is lots of dropped data. This also means the gyro boxes all stay the same height.

The share of the analysis time used is now taken from the FFT windows rather
than the raw data sections, so it includes the data lost at the end of each
section that is too short for another whole window, and changes with the
window size. It is always shown for raw logs and turns red below 90%, the
gyro boxes keep the same height.
@j-w9 j-w9 changed the title FilterReview: warn if raw IMU data is missing for much of the analysis time FilterReview: show how much raw IMU data is usable in the analysis time Nov 19, 2026
@j-w9

j-w9 commented Nov 19, 2026

Copy link
Copy Markdown
Contributor Author

Reworked: it's now "Usable data", the share of the analysis time covered by the FFT windows used, so it includes what's lost at the end of each section and changes with window size. Always shown for raw logs, red below 90%, so the boxes keep the same height.

@AP-Review

AP-Review commented Nov 20, 2026 •

Copy link
Copy Markdown

Deprecated — see below for the updated review.

Previous review (2026-11-20)

Automated review note — AI-generated (Claude+Codex), validated against the live diff. Please sanity-check before acting.
Verdict: COMMENT

Reviewed at head f6d04e8f5c.
Full report: https://firmware.ardupilot.org/Tools/APReview/DevCallReviews/PRReviews/ardupilot/webtools/376/3.html#prWebTools-376

This rework looks right, and it is worth merging once one small performance fix is in. Since the last review, f6d04e8 replaced the missing-data figure with "Usable data" from the FFT windows, as IamPete1 asked. The figure now follows window size and includes the end-of-section loss, and the earlier edge-of-window fix still holds.

Should fix

  • FilterReview/FilterReview.js:319: find_end_index(FFT.time) is re-run in the loop condition, and it scans the whole array each time, so the label update is quadratic in the window count. That is about 35 ms per IMU stream at the default window, but about 1 s per stream for a long log at window 64. Store it in a const before the loop, as redraw does.
Minor (2)

The earlier short-batch note no longer applies: short sections now count as unusable by design.

Not fully reviewed: no real raw-IMU .bin log was loaded through the parser (all testing used synthetic gyro data), and the eslint CI check was still pending for this head.

@AP-Review

AP-Review commented Nov 20, 2026 •

Copy link
Copy Markdown

Deprecated — see below for the updated review.

Previous review (2026-11-20)

Automated review note — AI-generated (Claude+Codex), validated against the live diff. Please sanity-check before acting.
Verdict: ACCEPT

Reviewed at head cd2dde5054.
Full report: https://firmware.ardupilot.org/Tools/APReview/DevCallReviews/PRReviews/ardupilot/webtools/376/4.html#prWebTools-376

This looks good to merge: the usable-data figure is correct and nothing is left open. Since the last review, cd2dde5 fixed all three earlier points. The window range is now found once before the loop (FilterReview.js:321), removing the quadratic scan. The red colour now follows the rounded percent that is shown (FilterReview.js:296). The stray <br><br> lines in index.html are tidied (index.html:233).

Not fully reviewed: no real raw-IMU .bin log was loaded through the parser or in a browser at this head (all testing used synthetic gyro data), and the eslint CI check is still pending (a local ESLint run passes).

Comment thread FilterReview/index.html Outdated
Logging rate: <label id="Gyro0_FFT_infoA"></label> Hz<br><br>
Frequency resolution: <label id="Gyro0_FFT_infoB"></label> Hz
Frequency resolution: <label id="Gyro0_FFT_infoB"></label> Hz<br><br>
<label id="Gyro0_usable" style="visibility:hidden">Usable data:</label>

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We get a blank space, I think it should be "display: none" until its filled in.

image

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done, it's display none now until there's a value

Hidden rather than invisible, so batch logs don't get a blank line.
@AP-Review

Copy link
Copy Markdown

Automated review note — AI-generated (Claude+Codex), validated against the live diff. Please sanity-check before acting.
Verdict: ACCEPT

Reviewed at head f32f794668.
Full report: https://firmware.ardupilot.org/Tools/APReview/DevCallReviews/PRReviews/ardupilot/webtools/376/5.html#prWebTools-376

This looks good to merge, and nothing is left open. Since the last review, f32f794 hides the Usable data line until it has a value, as asked. The wrapper span starts as display:none (index.html:234) and is shown only when there is a figure (FilterReview.js:293), so the blank space is gone and batch logs look as they did before.

Not fully reviewed: no real raw-IMU .bin log was loaded through the parser in a browser (testing used synthetic gyro data), and the CI result is taken from job metadata (a local ESLint run passes).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants