fix(widget): assign default row ids in row order

[Yigtwxx]

Problem

When row_id isn't given, the widget adds __row_index__ with DEFAULT nextval('row_id_sequence'). DuckDB fills that default in parallel per row group (122,880 rows), so on larger data frames the ids stop following the row order. With 300k rows, 299,648 rows got a different id than their position.

The neighbors column from compute_projection stores row positions (and the docstring says to use the zero-based row index), so in the widget the nearest neighbors point to the wrong points once the data frame has more than 122,880 rows.

Fix

Number the rows in the CREATE TABLE ... AS SELECT with row_number() OVER (), which keeps the original row order (duckdb docs). Like before, the column is only added when the data frame doesn't have it already. Added a test with 200k rows.

[domoritz]

Could you fix https://github.com/apple/embedding-atlas/blob/main/packages/viewer/src/app/FileViewer.svelte#L74 and https://github.com/apple/embedding-atlas/blob/main/packages/backend/src/streamlit/streamlit.ts#L48 as well?

[Yigtwxx]

done, same row_number approach in both files.
fyi duckdb-wasm runs single threaded here so the old nextval already gave the right order (checked 300k rows). new version copies the table once, peak wasm memory 260 -> 540 MB for a 234 MB table (alter was ~312). ok to keep?

[domoritz]

Oh, extra memory usage is an issue. Let's not do it in wasm for sure. How's memory usage with the native version? Does it spill to disk?

[Yigtwxx]

ok, reverted the wasm part, FileViewer and streamlit are back to nextval.
native is different: widget.py already copies the dataframe into a temp table, row_number is added in that same query so no extra copy. 2M rows: peak rss +546 MB old vs +561 MB new, duckdb's own memory 276 vs 251 MB.
spill: yes. with memory_limit=60MB both finish and write ~85-100 MB to .tmp. with the default limit nothing spilled.