Repository navigation
Add watchlist sources and rebuild rules when a source value changes - #716
Conversation
An authored empty value list is a null check. Expanding a placeholder against an empty list produced that, so a feed with no values would match events whose field is missing.
A refresh that re-fetched every source could loop. Cache-first rebuilds use the stored value and fetch only misses, and a failed rebuild leaves the running engine in place. NATS push payloads are stored in that cache.
Refreshes updated the cache and left compiled rules stale until a full reload. A referenced value that actually differs now requests a cache-first rebuild, and a full reload still wins when both are pending.
A watchlist is a push-model source with a size limit and an optional default lifetime. Fetch fields and the ids resolve and cache are rejected so the indicator routes stay unambiguous.
Pushes update an in-memory set that resolves to a sorted list of unexpired values. A change that would pass the size limit is rejected as a whole, and a blank, oversized, or control-character value is refused.
Operators can add, replace, and delete indicators without rewriting a source file, and a successful change asks the daemon to rebuild rules from the new value.
A restart keeps pushed indicators, drops ones that have expired, and a failed write leaves the in-memory set unchanged.
A pushed indicator changes what a live daemon matches, a lookup enricher reads the same set, and a restart keeps the indicators that have not expired.
The indicator routes call the method on the router, so the free function was unused and failed a warnings-denied build.
A 10k push and rebuilds at 1k, 10k, and 100k show whether a large indicator set should stay a placeholder list or move to a lookup enricher.
Malformed indicator JSON must be rejected without panicking, and a set that is accepted stays sorted and unique.
Operators can see how to push indicators, which permissions and limits apply, and that a referenced source value now rebuilds rules without a full reload.
The watchlist error path boxed its response so the result stays small, and the docs now count the new fuzz harness and include a valid rule example.
Signed-off-by: Meghdad Shamsaei <meghdadshamsaei@gmail.com>
mostafa
left a comment
There was a problem hiding this comment.
Requesting changes. The head commit does not compile, and three correctness problems reproduce against the code as written.
There is a lot here that is right. The RBAC and audit mappings are complete and covered by the route test. The cap check in add and replace is atomic. A failed rebuild keeps the loaded engine, and there is a test for it. The empty placeholder fix is correct, and disabling it fails empty_placeholder_expansion_matches_nothing. The metrics are preregistered, and the bench is honest that 10k and 100k measure the refusal. The end-to-end tests poll instead of sleeping on the positive paths.
The CI workflows on this head are still waiting for approval, so nothing has built the merge from main yet. To review the rest, I repaired the merge locally. With that repair, fmt, clippy, the workspace tests, doc, and the no-default-features check all pass.
Blockers (details inline):
- The merge from
mainleftengine.rsandprocessor.rsunparseable, and the cache-first fetch mode is lost on the pathmainrefactored. - The rebuild trigger loses source changes, because the result channel keeps only the latest result. In my probe, an expired indicator kept matching in 5 of 5 runs.
- Concurrent writes can roll back other requests' acknowledged changes, or leave the state database out of step with memory. Both reproduce in a deterministic test.
- A watchlist referenced by a
%name%placeholder accepts more than 4096 indicators. The push returns 200, the rebuild fails, and after a restart with--state-dbthe daemon refuses to start. scripts/docs-audit/check_sources.pyfails withvars.yml rsigma.metrics.names should be 111. This PR adds three metrics, butdocs/_data/vars.ymlstill saysnames: 108andalways: 103, probably because both sides of the merge bumped the count to 108. The three metrics are not feature-gated, so the values should be 111 and 106. The test plan says this script passes. Onmainit does.
Non-blocking (inline): the audit body limit undercuts the documented 4 MiB, an already-expired item is counted as added, and short_ttl_stops_matching can time out on a slow runner.
Compare every referenced cache value so a later result cannot hide an earlier change, serialize indicator writes with a per-source rollback, and cap a pipeline-referenced watchlist at the placeholder limit.
Both crates were marked unmaintained with no replacement, which fails cargo deny for direct workspace dependencies.
mostafa
left a comment
There was a problem hiding this comment.
Requesting changes for two remaining items. Every item from the last review is fixed, and the fixes hold up under the same probes. One of the remaining items is a bug I missed in the first round.
Verified fixed:
- Merge repair. The merge is repaired, and the fetch mode now reaches
resolve_pipeline_sources. fmt, clippy, workspace tests, doc, and the no-default-features check all pass locally, and CI is green. - Lost source changes. I reran the two-watchlist expiry reproduction, and the expired indicator stops matching in 5 of 5 runs, versus 0 of 5 before. The new end-to-end test covers it, and capturing values before the load closes the window after a rebuild.
- Concurrent writes. Writes are now serialized, and rollback is per source. Removing the lock from the
POSThandler fails botha_failed_write_does_not_roll_back_another_sourceandoverlapping_writes_persist_the_set_memory_keeps, so those tests guard the fix. - Placeholder cap. A referenced watchlist gets 422 past 4096. In my two-batch reproduction, the second push of 2,500 now returns
watchlist would exceed 4096 indicators, and no rebuild fails. - Docs and smaller items.
vars.ymlis fixed, andcheck_sources.pypasses. The audit body limit is documented. A pastexpires_atis rejected, with a test.short_ttl_stops_matchingno longer races. - Advisory ignores. The
deny.tomlignores match RUSTSEC-2026-0543 and RUSTSEC-2026-0542, andmainfailscargo denywithout them. That fix isn't specific to watchlists, so it might be better on its own PR somainturns green regardless of this one. That is non-blocking and the maintainers' call.
Blockers (details inline):
- With
--state-db,DELETE /api/v1/sources/{source_id}/indicatorsalways returns 400, because the audit middleware dropsDELETEbodies. - The 4096 cap is computed only at startup. A hot reload that makes a watchlist referenced brings back the original failure, where pushes return 200 and rebuilds fail. A later restart then silently trims the set.
The audit trail was dropping DELETE bodies, and a hot reload that started referencing a watchlist left its cap at the configured maximum.
mostafa
left a comment
There was a problem hiding this comment.
Approving. Both remaining items are fixed and verified against a build of this head.
- Audited
DELETE. With--state-db,DELETEremoves the indicator and the stored row, and it is audited with a digest. A bodylessDELETEstill gets 400. Putting back the oldDELETEshort-circuit failsdelete_with_state_db_removes_the_stored_indicator. - Reload cap. The cap follows reloads in both directions. After a reload adds the reference, the push past 4096 gets 422. After a reload removes it, the same push succeeds, and no rebuild fails. A restart with more than the cap saved warns with the number left out (904 in my run), reports 4,096 restored, and starts. Removing the cap refresh after a full reload fails
a_reload_changes_the_cap_when_a_watchlist_becomes_referenced.
fmt, clippy, workspace tests, doc, the no-default-features check, and CI are all green.
One non-blocking wording nit for whenever it's convenient: the CHANGELOG and http-api.md say the audit trail "keeps" the DELETE body, but it records a SHA-256 digest of the body (payload_digest), not the body itself.
Part of #685
Summary
type: watchlistsource holds indicators pushed over the daemon API.POST,PUT, andDELETEon/api/v1/sources/{source_id}/indicatorsneedsources:write;GETneedssources:read.--state-db, indicators are written before the HTTP response and restored on startup.--clear-statedeletes them.Breaking changes
POST /api/v1/reload.SourceTypeneeds theWatchlistvariant.Test plan
cargo fmt --all -- --checkcargo clippy --workspace --all-targets --all-features --locked -- -D warningscargo test --workspace --all-features --lockedcargo doc --workspace --all-features --locked --no-depscargo +1.88.0 check --workspace --all-features --lockedcargo check --manifest-path fuzz/Cargo.toml --bin fuzz_watchlist_requestcd docs && npm run docs:build && npm run docs:validatepython3 scripts/docs-audit/check_sources.pypython3 scripts/docs-audit/check_binary.py --bin target/debug/rsigma