Portable Rainbow LEDs using Kalman Filters and Voice Commands
Watch with sound (MP4, 15 s): an invisible rainbow fog hangs in the air, and the Dong lights up only the slice it passes through, like a laser pointer in a smoky room. The fog and the Pixels' colors come from the firmware's own Cloud Field.
A fur-covered LED staff that uses its motion sensor to make you feel like you are slicing through slowly drifting rainbow clouds hanging in the air around you. The vocabulary (Dong, Blade, Line, Cloud Field, Girth, Push, ...) is defined in CONTEXT.md; key decisions are in docs/adr/.
- Rainbow clouds that stay put in the air. The Blade shows only the slice of a slowly drifting 3D Cloud Field it passes through, anchored by the Dong's orientation; twisting it rolls color around the Blade (Girth).
- Motion tracking at about 817 Hz with a Mahony filter (Kalman to compare): a lightsaber-style Ignition while it learns which way is down, Re-level, Travel and Push for moves without turning, and drift re-learned whenever it lies still.
- Sound drives brightness from the onboard mic: the Blade glows at a set quiet glow in silence and brightens with beats and shouts.
- Voice commands (prototype): hold USR3, or say the wake word, and say two words, like next palette or faster clouds.
- Three buttons with quick, long and held presses, and an optional 8-Encoder knob unit that gives every setting its own knob.
- Sleeps when set down: after a minute lying still the Blade dims and the screen turns off; picking it up wakes it, a bump of the table doesn't.
- Bench mode: the
xiao_screenbuild draws the Blade on the board's 1.14" display, with a live position and rotation readout. - Power Budget: brightness is scaled to stay within what the battery pack can deliver.
- A serial console with one-key commands and a
?help page; settings survive power cycles. - Tested on the PC: all logic is plain C++ with host tests, including replays of real IMU recordings.
LED build guide (docs/led-build.md): wiring, power, buttons and first light, starting with the 1 m prototype.
Prototype, in the xiao_screen build only.
Hold USR3 and say a two-word command; let go when you're done. The screen shows Listening... while you hold it. A command heard replaces the button's own press; if nothing is heard, the press counts as usual (a quick press of A on this build still steps the Palette). A tap under 0.4 s skips the recognizer and acts at once. After a longer press the recognizer needs about half a second of silence to close the last word, so results come 0.6-0.9 s after letting go; pausing a beat before letting go lets it finish while you still hold the button.
| Say | Does |
|---|---|
| next palette, next colors / previous palette, previous colors | step the Palette |
| lights up, more light / lights down, less light | step brightness |
| faster clouds / slower clouds | step cloud motion |
| freeze clouds | freeze the clouds |
| bigger clouds / smaller clouds | step cloud size |
| more clouds / fewer clouds, more sky | step Clear Sky |
| sound on / sound off | sound drives brightness, or not |
| reset heading | Re-level |
| show status | open or close the Status page |
It runs Espressif's MultiNet7 English recognizer (ESP-SR, part of the Arduino core) on core 0, below the IMU and mic tasks, so the IMU keeps its rate and the LED core its frames. Listening takes about 27 ms of every 32 ms of audio, so the Status display redraws slowly while you hold the button. The model (3.3 MB) lives in its own flash partition; write it once per board:
pio run -e xiao_screen -t upload_voice_modelWake word (prototype): say Hi ESP, then a command within 4 s. It
uses Espressif's free WakeNet9 "Hi ESP" model, already in the model
partition, until a "Yo Dong" model exists
(docs/research/wake-word.md). Listening for
it costs 14% of core 0, all of it taken from the Status display; the IMU
keeps 818 Hz and the LEDs 240 fps. It is on at boot; serial W turns it
off and on.
Serial t listens for 3 s as if USR3 were held, g steps the gain the mic
gets before the recognizer, u steps that closing silence (600 ms; 450,
300 and 200 to experiment), V lists every phrase, and i shows the
recognizer's timing. Each attempt logs what was heard, how much silence it
took and how loud you were.
The firmware talks over the board's USB-C port. Open a console on it with PlatformIO's monitor, naming the env you flashed (it sets the 921600 baud):
pio device monitor -e xiao_screen --port /dev/ttyACM0--port is only needed when more than one serial device is plugged in; the
board shows up as /dev/ttyACM0 on Linux (/dev/cu.usbmodem* on a Mac,
COMn on Windows; pio device list shows them).
- Type
?first. It prints the firmware version, which USR pad each button is on for this build, and every command. - One key per command, no Enter. The monitor sends each key as you type it and doesn't echo it.
- Ctrl+C quits.
- Quit before flashing or running a tool.
pio run -t upload,tools/capture_imu.pyandtools/bringup.pyneed the port to themselves: with the monitor open, flashing fails or the monitor swallows what the tool was waiting for. - Start it again after a reset or replug. The board's USB port goes away while it restarts, and the monitor doesn't reconnect by itself.
An unknown key says to type ?. The board prints the same list, plus its
version and which pads A, B and C are on, when you type ? (or h):
| Key | Does |
|---|---|
| Status | |
? |
this help, starting with the buttons and their pads (also h) |
i |
status: profile, IMU, filter, rates, per-stage frame and display timings, mic, knobs |
p |
raw button levels (0 = pressed) and how GPIO43 is wired |
| Controls | |
w |
as a quick press of C: open or close the Status page |
v |
Blade view on or off |
c |
start Mounting Calibration (as holding C for 5 s) |
k |
log the 8-Encoder unit's counts, pushes and switch as they change (again: stop) |
| Sound | |
m |
sound drives brightness, or not (as the 8-Encoder unit's switch) |
l |
log the mic's level and Loudness 10 times a second (again: stop) |
Voice (xiao_screen) |
|
t |
listen for 3 s, as if USR3 were held |
g |
next voice gain (1x to 32x) |
u |
next silence wait after letting go (600, 450, 300, 200 ms) |
V |
list every spoken command |
W |
wake word on or off |
| Motion | |
f |
next orientation filter (Mahony, Kalman) |
a |
axis check, 5 times a second: which sensor axis sees gravity, where the Blade points (again: stop) |
R / S |
start / stop streaming raw IMU samples as CSV (tools/capture_imu.py) |
| Performance | |
b |
Cloud Field micro-benchmark |
z |
log a line every frame for 3 s; then i shows the worst loop, which should stay a few ms even with the port closed |
Each env in platformio.ini is one Dong Profile plus a few
flags in its build_flags:
| env | profile | flags |
|---|---|---|
xiao_screen |
bare board on the bench; Blade drawn on the display | -DDONG_PROFILE_SCREEN -DKD_SCREEN_BLADE -DKD_VOICE |
xiao |
1 m, 4×60 Pixels, one folded Strand, 3 A budget | -DDONG_PROFILE_PROTO |
xiao_2m |
2 m, 4×120 Pixels, one folded Strand, 6 A, pack sensing | -DDONG_PROFILE_2M |
xiao_lightswarm |
LightSwarm Dong: 2 Strands × 150, software SPI, 1.6 A | -DDONG_PROFILE_LIGHTSWARM |
| Flag | Does |
|---|---|
-DDONG_PROFILE_PROTO, _SCREEN, _2M, _LIGHTSWARM |
picks the Dong Profile in src/profiles.h: Blade length, Pixels, pins, IMU axes, Power Budget, battery. Exactly one is required. |
-DKD_SCREEN_BLADE |
boots into the Blade view on the display instead of the Status page |
-DKD_VOICE |
voice commands on USR3. Needs board_build.partitions = partitions/voice_16MB.csv and the model written once with pio run -e <env> -t upload_voice_model (both set up for xiao_screen). |
-DKD_FILTER=Kalman |
starts with the Kalman orientation filter instead of Mahony (Orientation filters) |
KD_VERSION and KD_COMMIT are added by
tools/pio_version.py (Versioning).
To try a flag without editing the file, set it for one build:
PLATFORMIO_BUILD_FLAGS="-DKD_FILTER=Kalman" pio run -e xiao -t upload- Seeed XIAO ESP32-S3 Plus with the 1.14" display carrier: onboard LSM6DS3-family IMU (I2C, INT1 on GPIO41), ST7789 display, buttons A/B/C and a PDM microphone (clock GPIO1, data GPIO2).
- SK9822/APA102 4-pin Pixels in 4 Lines along a 1-2 m Blade. Data and clock go on the pads D11/D12 (GPIO38/39; a second Strand uses D13/D15). The 1 m prototype drives them directly at 4 MHz; longer builds add a 74AHCT125 level shifter.
- The prototype runs from a two-port USB power bank: one port to the board's USB-C, one to the Pixels' 5 V at the Hilt. Three external buttons wire to the U1-U3 pads.
- Optional: an M5Stack 8-Encoder unit (U153) on the Grove port gives every setting its own knob (see Knobs).
Wiring, power, buttons and first light: docs/led-build.md, with the exact pads on the carrier.
Each physical build is a Dong Profile in src/profiles.h: Blade length, Pixels per Line, how Strands fold into Lines, pins, IMU axis mapping, Power Budget, battery. The IMU must be mapped so body +X points from the Hilt to the tip. Each has its own PlatformIO env; see Compile flags.
Every button has three presses: Quick (let go before 2 s), Long (let go between 2 and 5 s) and Held (reaches 5 s; acts then). While a button is down, the display's header shows a bar of half-second segments (teal for Quick, amber for Long) and what letting go will do.
| Button | Quick | Long | Held | Bare board | Other builds |
|---|---|---|---|---|---|
| A | next Palette | cloud size: 0.3, 0.6, 1.2, 2.4 m | step Clear Sky | USR3 | USR1 |
| B | cloud motion: frozen, 1x, 2x | step brightness | step Girth | USR2 | USR2 |
| C | Status page (again to close; closes itself after 30 s) | Re-level | Mounting Calibration | USR1 | USR3 |
A, B and C are wired to the carrier's USR1-USR3 buttons, and which is which
depends on the build. On the LED builds (xiao, xiao_2m,
xiao_lightswarm) C is USR3. On the bare board (xiao_screen) A and C swap
so the Palette lands on USR3, the easiest button to press, and C is USR1.
Each Dong Profile picks its button pins (BUTTON_*_PIN in
src/profiles.h). Type ? on the serial console to see the
pads of the build that's running.
With an M5Stack 8-Encoder unit plugged into the Grove port (before power-on), each setting has its own knob, counted 1-8 as printed on the unit. One click of a knob moves its setting one step; settings stop at their ends, and the Palette wraps around. The buttons keep working alongside.
| Knob | Turn | Push: Quick | Long (2 s) | Held (5 s) |
|---|---|---|---|---|
| 1 | Palette | Status page | back to default | |
| 2 | brightness | Status page | back to default | |
| 3 | Clear Sky | Status page | back to default | |
| 4 | cloud size | Status page | back to default | |
| 5 | cloud motion | Status page | back to default | |
| 6 | Girth | Status page | back to default | |
| 7 | quiet glow: 5, 15, 30, 50, 75% | Status page | back to default | |
| 8 | spare | Status page | Re-level | Mounting Calibration |
Knob 8's push works like button C. A quick push never changes a setting, so
bumping a knob while turning it is harmless. Quiet glow (knob 7) is how
bright the Blade stays in silence while sound drives brightness. The
toggle switch turns sound on and off: it decides at power-on and
whenever it's flipped, and serial m still works in between. The unit's
LEDs stay off. It runs from the Grove port's 3.3 V as it is.
Power on and hold the Dong still for a second. Ignition extends the Blade like a lightsaber while it learns which way is down, then it fades into the clouds. Re-level (hold C) points the clouds' reference heading where you are pointing. After a minute at Rest the Blade dims and the screen turns off; picking the Dong up wakes them. Picking up means turning it about 20° or carrying it for 2 s, so a bump of the table doesn't.
Sound drives brightness: in the room's usual quiet the Blade glows at 15%
(the quiet glow; knob 7 on an 8-Encoder unit changes it), and it brightens
to full with the loudest sounds it has heard lately. A steady hum (a fan, a
crowd) fades back to quiet within about 15 s, so beats and shouts over it
still show. Serial m or the knob unit's switch turns this off and on.
The Dong has to know how its IMU sits relative to the Blade. Hold C for 5
seconds (on the bare board C is USR1), or send c over serial; calibration
starts as the press bar fills:
- Point the Blade at the ceiling and hold. Every Line shows a bright tip; green fills from the Hilt as the hold registers (~2 s). Normal hand shake is fine.
- Hold the Blade level with Line 0 on top and hold. Only Line 0 lights.
On the bare board, pretend it is mounted in a Dong: the Blade sticks straight out of the screen's right edge (away from the USB cable, as the Blade view draws it) and Line 0 faces out of the screen. So pose 1 is the right edge up with the screen facing you, and pose 2 is lying flat, screen up. Check it with TIP in the screen's header: about +90 in pose 1, 0 lying flat.
The result is saved in flash and Ignition runs again. A square-mounted board
snaps to the exact axes, so roughly held poses are fine; an angled mount is
kept as measured. Until calibrated, the profile's imuAxes is used.
Toolchain: PlatformIO + pioarduino (Arduino core 3.3.x) — see ADR 0004.
uv tool install platformio==6.2.0pio test -e nativepio run -e xiao -t uploadAll logic lives in lib/core as plain C++17 and is tested on
the PC; src/ is thin hardware glue. After flashing, open the
serial console and type ?.
The firmware's version is in VERSION, as major.minor. Every pull request raises it, usually the minor number (0.1 to 0.2); the GitHub check Version fails a PR whose VERSION isn't above the base branch's. Two PRs open at once both start from the same number, so whichever merges second raises it again when it is rebased.
The build compiles in the version and the git commit (git describe, with
-dirty for uncommitted changes) through
tools/pio_version.py. The serial console shows them
at boot and at the top of ? and i, and IMU recordings carry fw= in their
header:
# kalman-dong 0.1 (28604ac) ready; ? for help
With the Dong plugged in over USB:
uv run tools/capture_imu.py swing-figure8 --seconds 60Recordings land in test/data/ and are replayed through the motion tracker by
pio test -e native -f test_replay.
The Cloud Field is anchored by the Dong's orientation (ADR 0002), so the orientation filter is what makes the clouds stay put in the air. It fuses the IMU's two sensors, read about 817 times a second:
- the gyro is smooth and quick but drifts, and
- the accelerometer knows which way is down, but only when the Dong isn't being swung.
A filter only does that fusion. MotionTracker does the rest around it: it learns the gyro's bias whenever the Dong is at Rest and hands the filter bias-corrected rates, and it runs Ignition, Re-level, Travel and the look-ahead to when the Pixels light. With six axes nothing can correct heading (the turn around vertical): it drifts slowly, and shows as the clouds turning and in the ROT readout's turn number. A magnetometer would fix that (docs/research/imu-options.md).
| Filter | How it works | Cost per sample on the board |
|---|---|---|
| Mahony (default) | Integrates the gyro and nudges tilt toward the measured gravity: 2 gains (kp 1/s, ki 0.02/s²). Trusts gravity fully within 0.5 m/s² of 1 g and not at all beyond 2.5 m/s², so swing forces don't tip it. |
~100 µs |
| Kalman | Error-state Kalman filter over orientation and the gyro bias MotionTracker hasn't caught. Trusts the accelerometer less the further it strays from 1 g, and skips readings too unlikely for its own uncertainty (innovation gate), which catches swings that pass through 1 g pointing nowhere near down. | ~200 µs |
Both are in kd/orientation_filter.h. Each IMU sample has 1224 µs, of which the I²C read takes about 550.
On real hand-held recordings (test/data/) the two tie on tilt, but Mahony holds heading far better at rest:
| Recording | Measure | Mahony | Kalman |
|---|---|---|---|
| still, 30 s | heading drift | 0.07° | 1.0° |
| slow sweeps | tilt error at quiet moments, mean | 4.6° | 4.5° |
| fast swings | tilt error at quiet moments, mean | 1.1° | 0.95° |
Differences under about half a degree of tilt are noise in this yardstick. Heading drift is the error a wielder actually sees (the clouds slowly turn), and nothing corrects it, so it decides. Mahony also costs half the CPU and has four settings to tune instead of nine.
In simulation (pio test -e native -f test_motion -v prints the report) the
Kalman filter holds tilt 13× better when the gyro bias shifts mid-swing, and
somewhat better through big swings, but slightly worse under hand jitter.
That bias shift is what a warming gyro does, so it is the case to revisit
with a better IMU or a magnetometer.
- Switch live: serial
fcycles the filters;inames the one running and its cost (imu us: ... track). - Change a build's default: add
-DKD_FILTER=Kalmanto itsbuild_flagsin platformio.ini.
To add a filter:
- Implement
OrientationFilter(kd/orientation_filter.h):name(),reset(orientation),update(gyro, accel, dt)andorientation(). It gets body-frame rates in rad/s with MotionTracker's bias already removed, and specific force in m/s² that reads +g upward at rest; it returns body to world, Z up. Leave bias at Rest, Re-level and heading reference to MotionTracker. - Register it in kd/filter_bank.h: a
member, an entry in the constructor's list, and one more in
kCount. It then shows up inf, in-DKD_FILTER=, and in the replay of every recording. - Test it: add its type to
FILTERSin test/test_motion/test_main.cpp, so every motion test runs against it, and tocompare()there, so the simulated report shows it. - Compare it:
- in simulation:
pio test -e native -f test_motion -v; - on real recordings:
uv run tools/bringup.py compare(orpio test -e native -f test_replay -v); - on the board:
funtil it runs, thenifor its cost, which has to stay well inside the 1224 µs per sample.
- in simulation:
- To make it the default, change
kDefaultFilterin kd/filter_bank.h and the test "Mahony is the default filter" in test/test_filter_bank, and update Why Mahony above.
The xiao_screen build draws the four Lines on the display (landscape: Hilt
on the left, tip on the right, Line 0 on top) exactly as they would go to the
Pixels, after brightness and the Power Budget. The board stands in for the
Hilt; the right edge of the screen is the direction the Blade points.
At the tip end, a header reads upright with the Blade pointing up, as a grid of x, y and z:
- POS is how far the Dong has moved through the clouds (Travel plus Push), in centimetres, wrapping every 10 m to -500..499.
- ROT is how it has turned since Ignition or the last Re-level, as roll, tip up and turn in percent of a half turn (90° = 50, 180° = 100). Positive is rolled clockwise looking out along the Blade, tip raised, and turned clockwise seen from above. The turn is where heading drift, and so the orientation filter, shows.
While a button is held the header shows what letting go will do instead, and setting changes show there briefly. The Status page (C quick) is upright in the same pose, with the same grid at the top. The screen turns off after 60 s of lying still with no button pressed, and comes back when the board is picked up (turned about 20° or moved for 2 s) or a button is pressed.
The quickest route is the guided bring-up script: it flashes, checks rates and filter costs, runs Mounting Calibration and verifies it, records swings and replays them through both filters, printing PASS/FAIL as it goes.
uv run tools/bringup.py allSteps can also run one at a time (check, mount, record NAME,
compare); see uv run tools/bringup.py --help. Manually:
pio run -e xiao_screen -t uploadThen open the serial console and type ?.
- Boot and IMU. The Blade view comes up right away (C shows the Status page).
ishould name the chip (LSM6DS3 / LSM6DS3TR-C / LSM6DSO) and report about 833 Hz IMU and up to 240 fps. - Mounting. Run Mounting Calibration (hold C for 5 s, see above). The header's TIP should then read about +90 with the Blade at the ceiling and about 0 with it level.
- Ignition. Reset with the board still: the bars extend from the left in about a second. Reset while shaking it: still done within two seconds.
- Slicing. With the clouds frozen (the default; B quick cycles frozen, 1x, 2x), sweep the board left and right and tilt it up and down: the colors change as you move through the clouds and return exactly when you come back to the same pointing direction.
- Girth. Hold B for 5 s to step Girth; at 0 all four bars match. At 0.4 m, roll the board 90 deg about the Blade: the colors move one bar over (what was on TOP shows on L after a clockwise roll seen from the Hilt).
- Buttons. See the table above; while a button is down the header shows what letting go will do.
- Settings survive power cycles. Change something, wait 3 s, unplug, plug back in.
- Rest. Lay the board down: after 60 s the bars dim to 20%; knock on the table and they stay dim; pick it up and they brighten instantly. Leave it a few minutes and watch for heading drift (the clouds slowly turning).
- Drift data. Record the board lying still and some waving for the
replay tests:
uv run tools/capture_imu.py bench-still --seconds 120. - Sound.
llogs the microphone: a quiet room sits at Loudness 0 and the bars at 15% (the Status page's bottom row shows it too). Talk or clap near the board: the bars brighten at once and fade back within a second. - LED outputs (optional). A scope or logic analyzer on GPIO38 (data) and GPIO39 (clock) should show the SK9822 stream while the display keeps updating; a short strip on those pads works too (mind the 3.3 V logic).
Screen colors are gamma-encoded to approximate what the Pixels emit; they will still look more saturated than LEDs under fur.
Running on the bare board: LSM6DS3TR-C at 817 Hz, Ignition learns the gyro bias, Rest detection works. Frame rates measured on the board (LED output running, no LEDs attached):
| env | render | frame | fps |
|---|---|---|---|
xiao_screen |
2.2 ms | 2.9 ms | 240 (the cap) |
xiao_lightswarm |
2.9 ms | 3.7 ms | 240 (the cap) |
xiao_2m |
3.7 ms | 4.6 ms | 217 |
Hardware lessons so far:
- FastLED 3.10.8 crashed the S3 and 3.9.20 bit-bangs (3.5 ms per frame), so Pixels go out through our own SPI2 DMA driver (ADR 0005); the display owns SPI3.
- Seeed_GFX sends one pixel per SPI call; the Blade view bypasses it with one burst per bar (~21 ms per screen).
- The S3 has no double-precision FPU and no float divide: keep
double, division,std::floorand function-local statics out of per-sample code (together they cost 2x in rendering). - Only GPIO1-20 can read analog, and all of those are taken on this carrier except the board's own JST LiPo divider (GPIO10). Pack sensing for the 2 m build needs a 1S pack on the JST or an I2C fuel gauge.
- Real gyro noise is ~0.03 rad/s sample to sample; Rest detection works on
smoothed readings (see
test/data/bench-desk-20s.csv).
IMU cost per sample on the board (budget 1224 us at 817 Hz): I2C read ~550 us, tracking ~100 us with Mahony and ~200 us with Kalman (see Orientation filters).
Still to check, on a real strip: LED colors, signal quality at 4 MHz without a level shifter (the 1 m prototype), and current draw.
