Skip to content

viewer: open GStreamer Matroska and SkyViper MP4 recordings - #75

Merged
tridge merged 11 commits into
ArduPilot:masterfrom
peterbarker:viewer-matroska-nul-strings
Nov 18, 2026
Merged

tridge merged 11 commits into
ArduPilot:masterfrom
peterbarker:viewer-matroska-nul-strings

Conversation

@peterbarker

Copy link
Copy Markdown
Contributor

The viewer rejected two kinds of recording that ffmpeg reads without
complaint.

Matroska from GStreamer: "No supported Matroska video track"

GStreamer's matroskamux ends CodecID with a NUL (V_MPEG4/ISO/AVC\0).
EBML allows string elements to be NUL-padded (RFC 8794 section 7.4), but
the viewer kept the NUL, so the codec ID didn't match any it supports and
the video track was dropped. This affects any MKV recorded with
rtspsrc ! rtph264depay ! h264parse ! matroskamux.

The codec ID is now read up to its first NUL. It is the only Matroska
string the viewer reads.

SkyViper SD-card MP4: "MP4 sample outside file"

The SkyViper's recorder keeps a constant frame rate by writing an empty
sample for each frame its encoder misses. One recording whose encoder
stalled for about 0.4 s just after recording started had 12 of them.
ISO BMFF allows empty samples, and ffmpeg skips them, but the viewer took
an empty sample for one outside the file and rejected the whole recording.

Empty samples are now skipped, and their time is kept, so the frames
after them stay at the right time. This applies to fragmented MP4 as well
as classic MP4. The track's default frame duration now comes from the
first sample kept, so an empty first sample with no duration doesn't
leave the frame rate unknown.

Testing

  • New tests in tests/media-containers.test.js:

    • an AVC Matroska track with a NUL-padded codec ID;
    • a classic MP4 with two empty samples between two real ones, checking
      the second frame's PTS and packet;
    • a classic MP4 and a fragmented MP4 whose first sample is empty with
      no duration, checking the default frame duration;
    • a fragmented MP4 with an empty sample, checking the next frame's
      timestamps, keyframe flag and packet.
  • npm test on Node 24: everything passes except
    terrain-elevation.test.js, which needs cesium and wasn't installed.
    That test doesn't touch this code.

  • prettier --check src tests passes.

  • I ran the parser on real recordings, all H.264 with APCG telemetry SEI
    on every frame:

    Recording Before After
    GStreamer MKV of the SkyViper's RTSP stream No supported Matroska video track 503 frames, telemetry on all
    SkyViper MP4 with 12 empty samples MP4 sample outside file 438 frames, telemetry on all
    SkyViper MP4 with no empty samples not affected 705 frames, telemetry on all

@peterbarker

Copy link
Copy Markdown
Contributor Author
Screenshot from 2026-11-12 13-03-15

@AP-Review

AP-Review commented Nov 12, 2026 •

Copy link
Copy Markdown

Deprecated — see below for the updated review.

Previous review (2026-11-12)

Automated review note — AI-generated (Claude+Codex), validated against the live diff. Please sanity-check before acting.
Verdict: COMMENT

Reviewed at head 8fb23fb4d6.
Full report: https://firmware.ardupilot.org/Tools/APReview/DevCallReviews/PRReviews/ardupilot/ap_cameragimbal/75/1.html#prAP_CameraGimbal-75

Looks good overall: the NUL-padded Matroska codec IDs and empty MP4 sample skipping work, with correct timestamps and the new tests passing, but two nonblocking issues are worth addressing.

Should fix

  • containers.js:304 / containers.js:419: skipped empty samples keep their time in the next pts, but no frame's duration covers the gap, so play() (paced by frameDuration()) runs fast across mid-recording gaps, and trailing empties shorten extent(). Pace playback by the pts gap, or carry skipped time into the previous frame's duration across index batches.
Minor (1)
  • containers.js:410: with !size gone, a trun with no per-sample fields and a trex default size of 0 loops through up to 10M empty samples per 16-byte trun, so a tiny crafted file can keep the index worker busy. Accept size 0 only from a per-sample size (f & 0x200), or bound the work.

@AP-Review

AP-Review commented Nov 12, 2026 •

Copy link
Copy Markdown

Deprecated — see below for the updated review.

Previous review (2026-11-12)

Automated review note — AI-generated (Claude+Codex), validated against the live diff. Please sanity-check before acting.
Verdict: COMMENT

Reviewed at head 4731945b14.
Full report: https://firmware.ardupilot.org/Tools/APReview/DevCallReviews/PRReviews/ardupilot/ap_cameragimbal/75/2.html#prAP_CameraGimbal-75

Not ready to approve yet: the pts-gap pacing introduced one small regression, and one rare pacing case remains. Since the last review, 5bb7d97 fixed the default-size-0 fragmented MP4 issue, and 4731945 now holds frames for the full pts gap across skipped samples.

Should fix

  • main.js:1302: state.frame.pts is read without a null guard. Pressing Space before any frame is decoded (e.g. while every indexed frame is still before the first keyframe) throws an unhandled TypeError and leaves the button showing Pause. Use state.frame?.pts, which falls back to frameDuration(null) as before.
Minor (1)
  • main.js:1304: pacing at the index frontier (still open, narrowed). When the current frame is the last one indexed, the fallback frameDuration() hold isn't revisited once the next batch arrives, so a gap at a 64-frame batch boundary plays in one frame time. Only hit while playback has caught up with indexing.

Withdrawn: trailing empty samples shortening extent() only affect the seek maximum and duration label, so that is fine as is.

Not fully reviewed: real SkyViper/GStreamer recordings, browser playback, Node 24 and the production build were not exercised.

GStreamer's matroskamux terminates CodecID with a NUL, which EBML
allows (RFC 8794 section 7.4), but the viewer kept it, so the codec
wasn't recognised and recordings made with matroskamux, e.g. from an
RTSP stream, failed with "No supported Matroska video track".  Read
the codec ID up to its first NUL.
The SkyViper's recorder keeps a constant frame rate by writing an empty
sample for each frame its encoder missed: a recording whose encoder
stalled for 0.4s as it started had 12.  ISO BMFF allows empty samples,
and ffmpeg skips them, but the viewer took one for a sample outside the
file and rejected the whole recording.  Skip them, keeping their time,
so the frames after them stay at the right time.
…eal samples

Fragmented MP4 still rejected an empty sample, unlike classic MP4 since
the last commit: skip them the same way, keeping their time.

Both paths took the track's default frame duration from the first
sample even if it was empty, so an empty first sample with no duration
left the frame rate unknown.  Take it from the first sample kept.
A trun with no per-sample sizes takes the trex default, and if that is
0 a 16-byte trun could describe millions of empty samples, all
indexed without yielding a frame.  Reject a size of 0 unless the
sample gives it itself, as all files did before empty samples were
skipped.
Playback showed each frame for the previous frame's duration.  A
recorder that skips frames (writing empty MP4 samples, now skipped)
leaves gaps in the timestamps that no frame's duration covers, so
playback ran fast across them.  Hold each frame until the next one's
presentation time, falling back to the frame duration for the last
frame or one whose successor isn't indexed yet.
…index's end

Pressing play before any frame had been decoded read state.frame.pts
from null, threw, and left the button showing Pause.  And when the
frame shown was the newest one indexed, its hold fell back to the
frame duration and wasn't revisited, so a gap at the end of an index
batch played in one frame time.  Guard the frame, and while the next
frame isn't indexed keep looking for it every 50ms, holding for the
real gap once it arrives.
@peterbarker
peterbarker force-pushed the viewer-matroska-nul-strings branch from 4731945 to 9ad67fb Compare November 12, 2026 23:50
@AP-Review

AP-Review commented Nov 13, 2026 •

Copy link
Copy Markdown

Deprecated — see below for the updated review.

Previous review (2026-11-13)

Automated review note — AI-generated (Claude+Codex), validated against the live diff. Please sanity-check before acting.
Verdict: COMMENT

Reviewed at head 9ad67fb202.
Full report: https://firmware.ardupilot.org/Tools/APReview/DevCallReviews/PRReviews/ardupilot/ap_cameragimbal/75/3.html#prAP_CameraGimbal-75

Close: the null-frame crash is fixed, and only two minor pacing cases are left, both seen only while playback has caught up with indexing. Since the last review, 9ad67fb fixed the state.frame.pts TypeError when playing before a frame is shown, and it now holds a gap frame that is indexed during the fallback hold.

Minor (2)
  • main.js:1390: frontier pacing (still open, narrowed). The wait <= 0 exit ends the poll once the fallback hold runs out, so a gap frame indexed after that plays early (0.14 s vs due 0.46 s in a harness).
  • main.js:1391: with a successor already indexed, the whole gap is slept without re-checking state.frames. B frames a later batch sorts in ahead of it are shown about 80 ms late, which base got right.

A fix for both: keep polling while !state.indexed, and leave only when indexing is done, there is no frame, or a next frame exists and is due. For example if (state.indexed || !frame || (next && wait <= 0)), with a 50 ms poll when wait <= 0. In the harness this holds the late gap for its full time and shows the inserted B frame at 50 ms.

Still withdrawn: trailing empty samples shortening extent(). It also feeds the splat panel, but no frame or endpoint is lost.

Not fully reviewed: real SkyViper/GStreamer recordings, browser playback, Node 24, the production build and the terrain-elevation test were not exercised.

While indexing was still going, the hold loop stopped looking once
the next frame was known, or once the fallback hold ran out.  So a
gap frame indexed after the fallback hold played early, and an
earlier B frame a later batch sorted in ahead of the known next frame
was shown late.  Keep checking every 50ms until indexing is finished,
there is no frame, or a next frame exists and is due.
@AP-Review

AP-Review commented Nov 13, 2026 •

Copy link
Copy Markdown

Deprecated — see below for the updated review.

Previous review (2026-11-13)

Automated review note — AI-generated (Claude+Codex), validated against the live diff. Please sanity-check before acting.
Verdict: COMMENT

Reviewed at head 97ea617832.
Full report: https://firmware.ardupilot.org/Tools/APReview/DevCallReviews/PRReviews/ardupilot/ap_cameragimbal/75/4.html#prAP_CameraGimbal-75

Nearly there: the remaining pacing issues are fixed, and one optional minor point is left. Since the last review, 97ea617 keeps polling while indexing. That fixes both the late-frontier gap frame playing early and the inserted B frame showing late.

Minor (1)
  • main.js:1386: the hold loop starts only after Play has already shown current + 1. A gap right after the frame on screen when Play is pressed (after a seek or file open) is therefore skipped at once, while the same gap during playback is held. This is the same as base. If you want it covered, wait out the displayed frame's hold (next.pts - state.frame.pts) before the first advance.

Still withdrawn: trailing empty samples shortening extent().

Not fully reviewed: real SkyViper/GStreamer recordings, browser playback, Node 24, the production build and the terrain-elevation test were not exercised.

Play showed the next frame at once, so a gap after the frame on screen
(frames the recorder skipped) was jumped over rather than held.  Hold
that frame until the next one is due, timed from the press, before
moving on, as for every other frame.
@AP-Review

AP-Review commented Nov 15, 2026 •

Copy link
Copy Markdown

Deprecated — see below for the updated review.

Previous review (2026-11-15)

Automated review note — AI-generated (Claude+Codex), validated against the live diff. Please sanity-check before acting.
Verdict: COMMENT

Reviewed at head 6cd97cb7d7.
Full report: https://firmware.ardupilot.org/Tools/APReview/DevCallReviews/PRReviews/ardupilot/ap_cameragimbal/75/5.html#prAP_CameraGimbal-75

Nearly there: one optional minor point is left. Since the last review, 6cd97cb holds the frame on screen when Play is pressed, so a gap right after it is no longer skipped.

Minor (1)
  • main.js:1383: once indexing is done, a gap is waited out in one setTimeout and Speed is not reread, so changing Speed during a long skipped-sample gap does nothing until the gap ends (a 10 s gap at 1x still takes 10 s after picking 4x). Keeping the 50 ms polling after indexing too, leaving once wait <= 0, would fix it.

Still withdrawn: trailing empty samples shortening extent().

Not fully reviewed: real SkyViper/GStreamer recordings, browser playback and Node 24 were not exercised.

Once a recording was fully indexed, the wait for the next frame was
one setTimeout, so changing Speed in a long gap had no effect until
the gap ended.  Wait in steps of at most 50 ms, working out again
each time, as while indexing.
@AP-Review

AP-Review commented Nov 15, 2026 •

Copy link
Copy Markdown

Deprecated — see below for the updated review.

Previous review (2026-11-15)

Automated review note — AI-generated (Claude+Codex), validated against the live diff. Please sanity-check before acting.
Verdict: COMMENT

Reviewed at head a13a3edce7.
Full report: https://firmware.ardupilot.org/Tools/APReview/DevCallReviews/PRReviews/ardupilot/ap_cameragimbal/75/6.html#prAP_CameraGimbal-75

Nearly there: one optional minor point is left. Since the last review, a13a3ed rereads Speed during a held gap, so a Speed change now takes effect straight away.

Minor (1)
  • main.js:1382: a Speed change mid-gap rescales the whole gap, time already played included, because wait divides hold by the new speed but subtracts unscaled wall time. At 4x, switching to 1x 2 s into a 10 s gap shows the next frame at 10 s rather than 4 s; speeding up late skips straight on. Keeping a media-time position (add (now - last) * speed each poll and compare with hold) would fix it.

Still withdrawn: trailing empty samples shortening extent().

Not fully reviewed: real SkyViper/GStreamer recordings, browser playback and Node 24 were not exercised.

A Speed change during a held gap rescaled the time already played,
since the wait divided the whole hold by the new speed.  Add up the
media time shown instead, each poll's wall time at the speed in force
during it, so a change applies only from when it is made.
@AP-Review

AP-Review commented Nov 15, 2026 •

Copy link
Copy Markdown

Deprecated — see below for the updated review.

Previous review (2026-11-15)

Automated review note — AI-generated (Claude+Codex), validated against the live diff. Please sanity-check before acting.
Verdict: COMMENT

Reviewed at head a962d37ba8.
Full report: https://firmware.ardupilot.org/Tools/APReview/DevCallReviews/PRReviews/ardupilot/ap_cameragimbal/75/7.html#prAP_CameraGimbal-75

One should-fix playback point is left. Since the last review, a962d37 fixed the Speed-change rescaling: a held gap now counts its time at the speed in force.

Should fix

  • main.js:1388: pressing Play while a forward seek is still decoding stalls playback for about the length of the seek. The loop keeps the old frame but reads next from the new state.number, so hold spans the whole jump. In a harness, a 100 s seek followed at once by Play waited 100 s, and base advances at once. Restarting the hold when state.frame changes (frame = state.frame; played = 0) would fix it.

Still withdrawn: trailing empty samples shortening extent(). The failing CI check is the camera firmware exposure job timing out, which this viewer-only change cannot affect.

Not fully reviewed: real SkyViper/GStreamer recordings and browser playback were not exercised.

Pressing Play while a forward seek was still decoding held the old
frame against the next frame of the new position, so playback waited
about as long as the seek jumped.  Start the hold again when the frame
shown changes.
@AP-Review

Copy link
Copy Markdown

Automated review note — AI-generated (Claude+Codex), validated against the live diff. Please sanity-check before acting.
Verdict: ACCEPT

Reviewed at head df681ee0dc.
Full report: https://firmware.ardupilot.org/Tools/APReview/DevCallReviews/PRReviews/ardupilot/ap_cameragimbal/75/8.html#prAP_CameraGimbal-75

No open findings remain at df681ee. Since the last review, df681ee fixed the stall when Play is pressed while a forward seek is still decoding: the hold now restarts from the seek's frame (main.js:1388).

Still withdrawn: trailing empty samples shortening extent().

Not fully reviewed: real SkyViper/GStreamer recordings and browser playback were not exercised.

@peterbarker
peterbarker marked this pull request as ready for review November 15, 2026 23:57
@tridge
tridge merged commit c9001a0 into ArduPilot:master Nov 18, 2026
8 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants