feat(audit): Implement retention policy for audit events#1025
Conversation
9cf44bd to
2555837
Compare
93e0824 to
7d1b2aa
Compare
7d1b2aa to
59c33a6
Compare
…nd adaptive deleter cadence
…pping A cutoff newer than the min-retention floor (e.g. publisher window 300 < audit floor 400, a future date, or a misconfig) was refused outright, so the marker did nothing — audit events were never expired. Clamp such a cutoff back to the floor instead: still delete everything older than the floor, never expire anything younger. Removes the cross-repo equality coupling and the boundary clock-skew foot-gun. Metric renamed cutoff_too_recent → cutoff_clamped. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
59c33a6 to
519686d
Compare
loadez
left a comment
loadez
left a comment
There was a problem hiding this comment.
Two things I want to flag before we enable the deleter in prod. None of them block the merge since everything is off by default.
1. Disabling the feature flag does not stop deletion after rows are marked.
check_feature_enabled only gates the marking. If we enable audit_logs_retention for the wrong org, the marker sets expires_at, and turning the flag off later does not unmark anything. The Deleter only checks expires_at <= now (deleter.ex:79 / queries.ex:135). So the 15 day grace is the only abort window, and the recovery is to null expires_at by hand in prod. Maybe re-check the org flag in the delete path? Or at least document that the flag only gates marking.
2. The initial marking sweep is unbounded inside one AMQP message.
batch_mark recurses the whole backlog of the org synchronously inside handle_message. The first sweep on a big org (400 days of events, batches of 1000) can pass the RabbitMQ consumer timeout, then the channel closes, the message is redelivered and the sweep restarts. With retry_limit: 10 this can repeat a few times. It is idempotent so we don't lose anything, but maybe cap the batches per message and requeue, or pace the marking through a worker like the Deleter already does.
The rest of the safety design looks good to me. The floor clamp, the changeset excluding expires_at, everything off by default.
3 participants
📝 Description
Applies the system-wide 400-day data retention policy to audit logs, reusing the same
usage.apply_organization_policyevent that Zebra and Plumber consume: aPolicyMarkerconsumer stamps events older than the cutoff with anexpires_at(clamped to a hard 400-day minimum so retention can never be shortened below policy), and a periodicDeleterremoves expired rows using an adaptive cadence to drain backlogs.Processing is gated per organization by the new
audit_logs_retentionfeature flag, off by default and fail-closed, so retention can be rolled out gradually without risking premature deletion.Includes the supporting schema migrations (
expires_atcolumn and indexes) and dependency bumps to address CVEs.✅ Checklist
This change requires documentation updateN/A