[RNE Rewrite] feat: add voice activity detection pipeline by msluszniak · Pull Request #1298 · software-mansion/react-native-executorch

msluszniak · 2026-07-02T15:52:14Z

Description

Adds a Voice Activity Detection (VAD) task pipeline and a corresponding speech example app. Chunked inference, segment postprocessing and streaming run in TypeScript on top of the core model.execute primitive. The per-frame feature extraction (framing, mean-removal, pre-emphasis, Hann window) is a native speech.frameWaveform C++ op: on device it dominated a detect() call (~86%, ~40 ms of Hermes vs ~6 ms for the model forward pass), so per the extension guidelines it lives in C++. It writes straight into the pre-allocated model-input tensor, fusing mean-removal + pre-emphasis + Hann into one dependency-free (vectorizable) pass; framing drops to ~3.4 ms (~12×), below the model's own inference cost.

Introduces a breaking change?

Yes
No

Type of change

Bug fix (change which fixes an issue)
New feature (change which adds functionality)
Documentation update (improves or adds clarity to existing documentation)
Other (chores, tests, code style improvements etc.)

Tested on

iOS
Android

Testing instructions

Build the speech app on iOS and Android
Test the Voice Activity Detection screen on a physical device (mic + xnnpack; simulator can't record)
Verify speech toggles SPEAKING/SILENT and logs begin/end events
Check the HF repo: https://huggingface.co/software-mansion/react-native-executorch-fsmn-vad

Screenshots

Related issues

Closes #1249

Checklist

I have performed a self-review of my code
I have commented my code, particularly in hard-to-understand areas
I have updated the documentation accordingly
My changes generate no new warnings

Additional notes

Depends on the per-method get_dynamic_dims_forward input validation from the embeddings PR ([RNE Rewrite] Add text and image embeddings pipelines #1292): VAD feeds a variable-length [frames, 512] input tensor per chunk. Outputs are still validated exactly, so the output tensor is pre-allocated at the model-declared shape. Requires [RNE Rewrite] Add text and image embeddings pipelines #1292 to land. The fsmn-vad model is re-exported (tag v0.10.0) with a get_dynamic_dims_forward method returning int32 [rank, 3] bounds per input.
Segments are returned in seconds (the old native path returned raw sample indices).
The FSMN output contract is assumed to be [1, frames, classes] with class 0 = non-speech (speech = 1 - p0), matching the current native implementation.

Port the VAD feature to the rewrite as a pure-TypeScript pipeline on top of the core model.execute primitive (no new C++): - src/extensions/speech/tasks/vad.ts: createVAD runner replicating the native FSMN-VAD algorithm (framing + Hann window + pre-emphasis, chunked inference, thresholding / min-duration / padding / merge). Segments are returned in seconds. Relies on the get_dynamic_dims relaxed input validation for the dynamic frame dimension; the fsmn-vad model is re-exported with it. - src/extensions/speech/vadStreamer.ts: pure streaming state machine driving onSpeechBegin / onSpeechEnd over an accumulating buffer. - src/hooks/useVAD.ts: hook wrapping createVAD + streamer lifecycle. - Register models.vad.FSMN_VAD and export the speech extension. - apps/speech: expo-router demo (mirrors apps/nlp) with a real-time mic VAD screen via react-native-audio-api.

…s re-export)

…arams into model config Frame geometry (sample rate, window/hop, FFT size, pre-emphasis, min frames) is FSMN-specific and now lives on VADModel.featureConfig (supplied by the models registry) instead of hardcoded constants in the task. The pipeline and streamer are parameterized by it; detection thresholds remain generic VADOptions.

Framing (windowing, mean-removal, pre-emphasis, Hann) was ~86% of a detect() call on device — ~40ms of Hermes vs ~6ms for the model forward pass. Port it to a native `speech.frameWaveform` op that writes directly into the pre-allocated model-input tensor, fusing mean-removal + pre-emphasis + Hann into a single dependency-free (vectorizable) pass. Framing drops to ~3.4ms on device (~12x), leaving it below the model's own inference cost.

The streamer re-ran the model over the whole growing buffer every tick (up to ~8ms on a 10s buffer on device), even though only ~100ms is new and the streaming decision only needs recent context. Cap the sliding window to 2.5s so per-tick detect stays flat and low (~3x cheaper, ~2.7ms) — still well above FSMN's receptive field and the 250ms min-speech-duration, so detection is unaffected. On-device benchmark showed the alternative 'O(1) insert' buffer only saved ~6us/tick (0.2%), so it was dropped.

Fold the standalone vadStreamer into the task (createFsmnVad) as a pull-based async generator woken by streamInsert, mirroring createWhisperSpeechToText: stream()/streamInsert()/streamStop() replace the callback-based streamer. Rename createVAD -> createFsmnVad and useVAD -> useFsmnVad for model-specific naming, matching the Whisper STT pipeline so the two speech tasks share one streaming structure.

…mments Address review: the model config type is model-specific, so name it FsmnVadModel (matching createFsmnVad/useFsmnVad). Also drop redundant comments in fsmnVad.ts.

Address review: wrap the waveform/hann/dst buffers in std::span and take per-frame subspans instead of raw float pointers; zero with std::ranges::fill and sum the frame with a range-for. Trim the redundant Pass 1/Pass 2 comments.

- frameWaveform: validate tensor shapes via symbolic shapes in fromJs instead of manual rank checks; take startSample/numFrames/hopLength as uint64_t for automatic non-negativity checks; use asType<float> for preemphasis; add the missing waveform/hann pairwise aliasing check. - speech extension barrel now exports ops (like cv), not the task. - fsmnVad: drop the SampleSegment duplicate of Segment, inline Required<VADOptions>, and narrow the tInput try/finally to its actual use.

…ng, op API - Replace the streaming generator with push()/resetStream(): a rolling window fed straight from a recorder callback, returning speechStart/speechEnd. The window is required — one 100ms chunk yields ~7 frames, far below the 25 hops minSpeechDurationMs needs, so detect() on a bare chunk never fires. - Rename to createFsmnVoiceActivityDetector / useVoiceActivityDetection, mirroring createWhisperSpeechToText / useSpeechToText; frameWaveform -> extractFrames. - Case acronyms per repo convention (cf. NmsOptions): VADOptions -> VadOptions etc. - Drop VadFeatureConfig: model geometry is now file consts (fftLength read from model metadata) and the sample rate is exported as FSMN_VAD_SAMPLE_RATE_HZ, mirroring WHISPER_SAMPLE_RATE_HZ. - extractFrames takes an options object and no startSample; the caller passes the per-chunk waveform slice, which removes the nested try. - Fold scoresToSegments/mergeSegments into a single postprocess().

The FSMN model's output is dynamic, not just its input: N input frames produce [1, N, classes], so execute() validates outputTensors against the shape produced for the current chunk rather than the model-declared maximum. The output tensor was pre-allocated once at that maximum, so every execute threw 'outputTensors[0] must have shape [1, 247, 248] ... got 1000'. Inside the recorder callback nothing surfaced the throw, so live detection silently never fired. Allocate the output per chunk with its frame dimension matched to the chunk and dispose it alongside the other per-chunk tensors. Verified on a physical Android device: live mic VAD toggles SPEAKING/SILENT again.

barhanc

Tested example app on Android and it works great. I don't have an iOS device to test unfortunately.

barhanc · 2026-07-15T14:17:23Z

Should be named ...Detector to be consistent with other hooks' naming.

barhanc · 2026-07-15T14:18:20Z

      },
    },
  },
+  vad: {


Here we should also use the full name voiceActivityDetection.

barhanc · 2026-07-15T14:21:54Z

+   * @param options Optional per-call overrides of the detection thresholds.
+   * @returns A promise resolving to the detected speech segments, in seconds.
+   */
+  detect: (waveform: Float32Array, options?: VadOptions) => Promise<Segment[]>;


Maybe detectVoice would be more descriptive. In general the pattern in other tasks e.g. in CV is that the inference method name is rather specific for a given task.

barhanc · 2026-07-15T14:22:31Z

+import { wrapAsync } from '../../../core/runtime';
+import { extractFrames } from '../ops';
+
+/** Sample rate (Hz) the FSMN-VAD model expects its input waveform to be at. */


This should be a proper jsdoc similar to the ones in constants.ts with category @constants.

barhanc · 2026-07-15T14:25:21Z

+
+  const detectWorklet = (waveform: Float32Array, options?: VadOptions): Segment[] => {
+    'worklet';
+    if (waveform.length < FRAME_LENGTH) return [];


Isn't this redundant because of the check if (numFrames <= 0) return []; below?

barhanc · 2026-07-15T15:05:17Z

+      // The model needs at least MIN_FRAMES rows, so a short final chunk is
+      // zero-padded up to it and its padding scores are discarded below.


I'm not sure if zero padding is the best strategy in here. In Whisper I just discard the trailing chunk if it has less then required min number of samples and it is what we also did in the legacy implementation.

barhanc · 2026-07-15T15:07:55Z

+
+  const detect = wrapAsync(detectWorklet, runtime);
+
+  const windowSamples = DETECTION_WINDOW_SECONDS * FSMN_VAD_SAMPLE_RATE_HZ;


Maybe this could be moved as a constant to const WINDOW_SAMPLES?

barhanc · 2026-07-15T15:11:49Z

+
+  const windowSamples = DETECTION_WINDOW_SECONDS * FSMN_VAD_SAMPLE_RATE_HZ;
+  let window = new Float32Array(0);
+  let isSpeaking = false;


The naming with isSpeaking and speaking variables is slightly confusing. Maybe rename speaking -> isSpeaking and isSpeaking -> wasSpeaking?

barhanc · 2026-07-15T15:21:10Z

+ * changes. The rolling-window state driven by `push` lives inside the task (see
+ * {@link createFsmnVoiceActivityDetector}), so it is exposed here directly.


Suggested change

* changes. The rolling-window state driven by `push` lives inside the task (see

* {@link createFsmnVoiceActivityDetector}), so it is exposed here directly.

* changes.

barhanc · 2026-07-15T15:30:15Z

Perhaps we could name it vadUtils.ts and place it under utils/. In TTS PR I will also be adding some utils files (like supertonicUtils.ts, textPartitioner.ts, etc.) so I thought we could group them.

msluszniak self-assigned this Jul 2, 2026

msluszniak added refactoring feature PRs that implement a new feature labels Jul 2, 2026

msluszniak linked an issue Jul 2, 2026 that may be closed by this pull request

[RNE Rewrite] Speech - add VAD pipeline implementation #1249

Open

msluszniak force-pushed the @ms/rewrite-vad branch 2 times, most recently from f76a5c0 to ae3f0da Compare July 9, 2026 11:56

msluszniak added 5 commits July 13, 2026 16:16

[RNE Rewrite] chore: point fsmn-vad model at v0.10.0 (get_dynamic_dim…

ac7aebf

…s re-export)

msluszniak force-pushed the @ms/rewrite-vad branch from 2b52139 to e934ace Compare July 13, 2026 14:18

msluszniak marked this pull request as ready for review July 13, 2026 14:40

barhanc self-requested a review July 13, 2026 20:55

msluszniak commented Jul 14, 2026

View reviewed changes

Comment thread packages/react-native-executorch/src/models.ts Outdated

[RNE Rewrite] refactor(vad): rename VADModel to FsmnVadModel, trim co…

0ee0fc1

…mments Address review: the model config type is model-specific, so name it FsmnVadModel (matching createFsmnVad/useFsmnVad). Also drop redundant comments in fsmnVad.ts.

msluszniak commented Jul 14, 2026

View reviewed changes

Comment thread packages/react-native-executorch/cpp/extensions/speech/operations.cpp Outdated

msluszniak mentioned this pull request Jul 14, 2026

[RNE Rewrite] Use std::span instead of raw pointers in native C++ ops #1308

Open

barhanc reviewed Jul 15, 2026

View reviewed changes

msluszniak added 3 commits July 15, 2026 12:42

msluszniak requested a review from barhanc July 15, 2026 12:09

barhanc reviewed Jul 15, 2026

View reviewed changes

		// The model needs at least MIN_FRAMES rows, so a short final chunk is
		// zero-padded up to it and its padding scores are discarded below.


		const detect = wrapAsync(detectWorklet, runtime);

		const windowSamples = DETECTION_WINDOW_SECONDS * FSMN_VAD_SAMPLE_RATE_HZ;

		* changes. The rolling-window state driven by `push` lives inside the task (see
		* {@link createFsmnVoiceActivityDetector}), so it is exposed here directly.

	* changes. The rolling-window state driven by `push` lives inside the task (see
	* {@link createFsmnVoiceActivityDetector}), so it is exposed here directly.
	* changes.

Uh oh!

Conversation

msluszniak commented Jul 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Introduces a breaking change?

Type of change

Tested on

Testing instructions

Screenshots

Related issues

Checklist

Additional notes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

barhanc left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

barhanc Jul 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

msluszniak commented Jul 2, 2026 •

edited

Loading

barhanc Jul 15, 2026 •

edited

Loading