feat(etl): carry playlist removal history through the migration

[rickyrombo]

Stacked on #481 (which merged while this was in flight, so this is now rebased directly onto main).

The gap

playlist_tracks.is_removed is a tombstone. The primary key is (playlist_id, track_id), so taking a track out of a playlist flips the flag instead of deleting the row — the row survives as the record that the track was once a member, and when it left.

The migration cannot produce one. updatePlaylistTracks recognizes a removal as a transition: a track present in playlist_contents before an update and absent after it. A migration replays a snapshot — one Playlist/Create carrying the playlist's final contents, with no earlier state to differ from. The "was present, now absent" loop has nothing to iterate, so only the insert loop runs, and it hardcodes is_removed = false.

Measured on a production clone: 20,763 tombstones across 3,917 playlists that cannot be reproduced at all (24,579 rows over 4,064 playlists source-wide; the smaller figures are the subset whose contents are byte-identical across two snapshots).

They are not bookkeeping. They are the input to tracks.playlists_previously_containing_track, which the API's usdc_purchase check reads to decide whether someone who bought an album still has access to a track that later left it (AudiusProject/api#1014). 19,441 tracks in the source carry such a record. Migrated data would carry none, and every one of those purchases would quietly stop covering the track it was bought for.

The shape

The removals ride on the Create as state, matching the established pattern (trackState, playlistState, userState: production passes the zero value, only the migration fills it in). No second Update transaction per playlist.

• cmd/genesis-writer/entities_playlist.go — preloads the source's is_removed = true rows, a table the writer did not read at all until now, and emits them under a new removed_tracks metadata key: [{"track_id", "created_at", "updated_at"}], where created_at is when the track joined and updated_at is when it left. 24,579 rows is small enough to hold in memory.
• migration.go — migratedPlaylistCreate parses them into playlistState.RemovedTracks.
• playlist_create.go — insertPlaylistAndRouteWithState passes them through. Zero value writes nothing, so production is unaffected.
• playlist_tombstones.go (new) — inserts the junction rows with is_removed = true and the source timestamps, then feeds the reverse index through the same updateTrackPlaylistIndex the production removal path uses, so the two cannot disagree. The record shape stays {"<playlist_id>": {"time": <unix epoch>}} — that is the API contract.

Two things the source's timestamps decide rather than the block: updated_at on the junction row, and the removal epoch in the reverse index, which is compared against a purchase date. Stamping either at replay time would move who is entitled. Because tracks that left at different moments cannot share one timestamp, the reverse index is written once per distinct removal time rather than once per playlist.

A track appearing in both the contents and the removals is a contradiction in the source; the live row wins, so a replay can never mark a present track removed.

Merge note

playlist_tracks.go is untouched — the new work lives in a new file and calls the existing updateTrackPlaylistIndex. This keeps it clear of the concurrent change converting that file's updated_at = now() statements to block time.

Testing

Four DB-backed tests in playlist_tombstones_test.go:

test	asserts
TestMigratedPlaylistCreate_CarriesRemovalHistory	tombstone row with source created_at/updated_at, live track unaffected, reverse index carries {"<pid>": {"time": <source epoch>}}
TestPlaylistCreate_WritesNoTombstones	a production create writes none
TestMigratedPlaylistCreate_LiveMembershipBeatsTombstone	a track in both wins as live, gains no removal record
TestMigratedPlaylistCreate_PerTrackRemovalTimes	two departures at different times keep their own timestamps

ok  github.com/OpenAudio/go-openaudio/pkg/etl                            9.699s
ok  github.com/OpenAudio/go-openaudio/pkg/etl/processors                 0.206s
ok  github.com/OpenAudio/go-openaudio/pkg/etl/processors/entity_manager 95.446s

Negative check — with the insertPlaylistTrackTombstones call removed:

--- FAIL: TestMigratedPlaylistCreate_CarriesRemovalHistory
    removed track has no playlist_tracks row: no rows in result set
--- FAIL: TestMigratedPlaylistCreate_PerTrackRemovalTimes
    track 2006305 removal time = 0, want 1767916800
    track 2006304 removal time = 0, want 1714608000

The other two stay green by design — they assert what must not happen.

gofmt -l and go vet ./... clean in pkg/etl (three pre-existing gofmt hits unrelated to this change), go build ./cmd/genesis-writer/ clean.

🤖 Generated with Claude Code