Local speech output (Kokoro TTS) and voice input (Whisper STT)

[suinswofi]

Hi Andrew — this is a draft proposal for optional, fully local speech features. It works end to end on Linux with a real microphone and speakers; it's marked draft because it hasn't been tried on Mac/Windows builds yet, and because it adds two npm dependencies and ~70 MB to each package, which is your call to make. Happy to adjust, split up, or shelve it as you see fit.

What it does

Two features, each independently toggleable (Preferences → new Audio tab, or the View menu), both off by default:

• Speech output — reads each turn's new story text aloud with the Kokoro-82M neural voice (kokoro-js), streamed sentence by sentence. Choice of voices and speaking rate; reading is interrupted by new input; Stop Speaking (Cmd/Ctrl-.) and Repeat Last Turn (Cmd/Ctrl-Shift-R). Optional OS-voice fallback while the model downloads.
• Voice input — push-to-talk speech recognition with Whisper (@huggingface/transformers; base/small/large-v3-turbo selectable). The transcript is typed into the game's input line and (optionally) sent; single-key prompts get the first letter or spoken digit. Talk key: presets including bare Ctrl/Alt, or any custom key/combination; a Listen (Toggle) command for hands-free use.

Models are downloaded on first enable into <userData>/models with progress shown, and nothing is ever sent anywhere.

Screenshots

The Audio tab in Preferences (both features on, models loaded):

The bundled Advent at its opening, being read aloud (this is the same turn as the audio sample below); the small bar at bottom right shows the mic button and the current speech state:

Audio sample

What the speech output sounds like: the opening of the bundled Advent (banner and the first room, "At End Of Road"), read by the default Kokoro voice (af_heart) at speed 1.0. This is the raw audio the engine hands to the game window, captured from a fresh start of the game and saved as-is (48 s, 24 kHz mono WAV, 2.3 MB):

▶ advent-opening-kokoro.wav

Implementation

• audioengine.js — one shared Electron utilityProcess owning the models (ONNX Runtime, CPU, off the UI thread); shuts down after 30 s idle.
• speech.js — game-window side: text extraction from the GlkOte recording handler (buffer windows only, echoed commands skipped), Web Audio playback queue, mic capture (AudioWorklet) + resample to 16 kHz, injection into GlkOte's line/char input via a synthetic Enter keypress, small status bar.
• main.js — engine lifecycle, IPC relay, prefs, menu items; audioconfig.js — shared constants; prefs.html/js — Audio tab.
• makedist.py — copies just the needed module files per platform (--no-audio and lectroteAudioFeatures:false for bound games opt out; win32-ia32 never includes them, since ONNX Runtime has no 32-bit build). Mac plist/entitlements gain microphone permission.
• Deps: kokoro-js@1.2.1, @huggingface/transformers@3.8.1 (one deduped copy of onnxruntime-node, N-API, so no rebuild). No changes to Quixe/GlkOte. License notes added to the README (phonemizer embeds espeak-ng, GPL-3.0).

Testing so far

• Linux, hands-on with real speakers and microphone: story text is read aloud in the Kokoro voice, interrupted by new input; push-to-talk commands are recognized and submitted; the game's response is spoken back.
• Linux, dev and packaged builds, also driven over the DevTools protocol with Chromium's fake microphone: TTS streaming/interrupt/repeat, prefs↔menu sync, model status/progress, idle engine shutdown, custom/bare-modifier talk keys, --no-audio build.
• Not yet: macOS (mic permission prompt, notarizing the ONNX dylib), Windows.

🤖 Generated with Claude Code

[suinswofi]

In case it's useful for anyone wanting to try the speech features without building from source: I've published unofficial prebuilt test builds of this branch on my fork's releases page — https://github.com/suinswofi/lectrote/releases/tag/v1.5.6-speech.1 (Linux x64/arm64 and Windows x64/arm64/ia32 zips, plus a Windows installer). They're unsigned, and the win32-ia32 build omits the speech features since there's no 32-bit ONNX runtime. If the feature lands here I'll retire them in favor of the official releases, of course.

[suinswofi]

A side note unrelated to the speech changes: while building and testing the Windows installer for the fork release (silent install/uninstall cycles under Wine), I noticed a few small quirks in resources/wininstaller.nsi that also apply to the official installers:

• The installer doesn't write an entry under HKCU\...\CurrentVersion\Uninstall, so Lectrote never appears in Windows' Add/Remove Programs list — uninstalling only works via the Start Menu shortcut or Uninstall Lectrote.exe directly.
• The uninstaller's DeleteRegKey /ifempty HKCU "Software\Lectrote" never fires in practice, because the key always still holds the install-path default value and the Start Menu Folder value at that point, so the key survives uninstall.
• Uninstalling while Lectrote is running silently leaves the locked Lectrote.exe behind (the Delete just skips it). A running-instance check, or at least a note, might help.

Also, for anyone else building the installer: wininstaller.nsi expects a resources/wininstall-link symlink/junction pointing at dist/Lectrote-win32-x64, which isn't in the repo or mentioned in the README — might be worth a line in the packaging docs.

Happy to file these as a separate issue (or a small follow-up PR) if that's preferred — just didn't want the observations to get lost.