Skip to content

Add watchlist sources and rebuild rules when a source value changes - #716

Merged
mostafa merged 19 commits into
timescale:mainfrom
SecurityEnthusiast:feat/source-watchlists
Nov 17, 2026
Merged

mostafa merged 19 commits into
timescale:mainfrom
SecurityEnthusiast:feat/source-watchlists

Conversation

@SecurityEnthusiast

Copy link
Copy Markdown
Contributor

Part of #685

Summary

  • A type: watchlist source holds indicators pushed over the daemon API. POST, PUT, and DELETE on /api/v1/sources/{source_id}/indicators need sources:write; GET needs sources:read.
  • With --state-db, indicators are written before the HTTP response and restored on startup. --clear-state deletes them.
  • A referenced source value change rebuilds rules from the cache. A full reload still fetches fresh values. An empty watchlist matches nothing.
  • Lists larger than 4096 placeholder combinations belong in a lookup enricher.

Breaking changes

  • Rules that reference a dynamic source now change when that source's value changes, without POST /api/v1/reload.
  • An exhaustive match on SourceType needs the Watchlist variant.

Test plan

  • [*] cargo fmt --all -- --check
  • [*] cargo clippy --workspace --all-targets --all-features --locked -- -D warnings
  • [*] cargo test --workspace --all-features --locked
  • [*] cargo doc --workspace --all-features --locked --no-deps
  • [*] cargo +1.88.0 check --workspace --all-features --locked
  • [*] cargo check --manifest-path fuzz/Cargo.toml --bin fuzz_watchlist_request
  • [*] cd docs && npm run docs:build && npm run docs:validate
  • [*] python3 scripts/docs-audit/check_sources.py
  • [*] python3 scripts/docs-audit/check_binary.py --bin target/debug/rsigma

An authored empty value list is a null check. Expanding a placeholder
against an empty list produced that, so a feed with no values would
match events whose field is missing.
A refresh that re-fetched every source could loop. Cache-first rebuilds
use the stored value and fetch only misses, and a failed rebuild leaves
the running engine in place. NATS push payloads are stored in that cache.
Refreshes updated the cache and left compiled rules stale until a full
reload. A referenced value that actually differs now requests a
cache-first rebuild, and a full reload still wins when both are pending.
A watchlist is a push-model source with a size limit and an optional
default lifetime. Fetch fields and the ids resolve and cache are rejected
so the indicator routes stay unambiguous.
Pushes update an in-memory set that resolves to a sorted list of unexpired
values. A change that would pass the size limit is rejected as a whole, and
a blank, oversized, or control-character value is refused.
Operators can add, replace, and delete indicators without rewriting a source file, and a successful change asks the daemon to rebuild rules from the new value.
A restart keeps pushed indicators, drops ones that have expired, and a failed write leaves the in-memory set unchanged.
A pushed indicator changes what a live daemon matches, a lookup enricher reads the same set, and a restart keeps the indicators that have not expired.
The indicator routes call the method on the router, so the free function was unused and failed a warnings-denied build.
A 10k push and rebuilds at 1k, 10k, and 100k show whether a large indicator set should stay a placeholder list or move to a lookup enricher.
Malformed indicator JSON must be rejected without panicking, and a set that is accepted stays sorted and unique.
Operators can see how to push indicators, which permissions and limits apply, and that a referenced source value now rebuilds rules without a full reload.
The watchlist error path boxed its response so the result stays small, and the docs now count the new fuzz harness and include a valid rule example.

@mostafa mostafa left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Requesting changes. The head commit does not compile, and three correctness problems reproduce against the code as written.

There is a lot here that is right. The RBAC and audit mappings are complete and covered by the route test. The cap check in add and replace is atomic. A failed rebuild keeps the loaded engine, and there is a test for it. The empty placeholder fix is correct, and disabling it fails empty_placeholder_expansion_matches_nothing. The metrics are preregistered, and the bench is honest that 10k and 100k measure the refusal. The end-to-end tests poll instead of sleeping on the positive paths.

The CI workflows on this head are still waiting for approval, so nothing has built the merge from main yet. To review the rest, I repaired the merge locally. With that repair, fmt, clippy, the workspace tests, doc, and the no-default-features check all pass.

Blockers (details inline):

  1. The merge from main left engine.rs and processor.rs unparseable, and the cache-first fetch mode is lost on the path main refactored.
  2. The rebuild trigger loses source changes, because the result channel keeps only the latest result. In my probe, an expired indicator kept matching in 5 of 5 runs.
  3. Concurrent writes can roll back other requests' acknowledged changes, or leave the state database out of step with memory. Both reproduce in a deterministic test.
  4. A watchlist referenced by a %name% placeholder accepts more than 4096 indicators. The push returns 200, the rebuild fails, and after a restart with --state-db the daemon refuses to start.
  5. scripts/docs-audit/check_sources.py fails with vars.yml rsigma.metrics.names should be 111. This PR adds three metrics, but docs/_data/vars.yml still says names: 108 and always: 103, probably because both sides of the merge bumped the count to 108. The three metrics are not feature-gated, so the values should be 111 and 106. The test plan says this script passes. On main it does.

Non-blocking (inline): the audit body limit undercuts the documented 4 MiB, an already-expired item is counted as added, and short_ttl_stops_matching can time out on a slow runner.

Comment thread crates/rsigma-runtime/src/engine.rs Resolved
Comment thread crates/rsigma-cli/src/daemon/server.rs Resolved
Compare every referenced cache value so a later result cannot hide an earlier change, serialize indicator writes with a per-source rollback, and cap a pipeline-referenced watchlist at the placeholder limit.
Both crates were marked unmaintained with no replacement, which fails cargo deny for direct workspace dependencies.

@mostafa mostafa left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Requesting changes for two remaining items. Every item from the last review is fixed, and the fixes hold up under the same probes. One of the remaining items is a bug I missed in the first round.

Verified fixed:

  • Merge repair. The merge is repaired, and the fetch mode now reaches resolve_pipeline_sources. fmt, clippy, workspace tests, doc, and the no-default-features check all pass locally, and CI is green.
  • Lost source changes. I reran the two-watchlist expiry reproduction, and the expired indicator stops matching in 5 of 5 runs, versus 0 of 5 before. The new end-to-end test covers it, and capturing values before the load closes the window after a rebuild.
  • Concurrent writes. Writes are now serialized, and rollback is per source. Removing the lock from the POST handler fails both a_failed_write_does_not_roll_back_another_source and overlapping_writes_persist_the_set_memory_keeps, so those tests guard the fix.
  • Placeholder cap. A referenced watchlist gets 422 past 4096. In my two-batch reproduction, the second push of 2,500 now returns watchlist would exceed 4096 indicators, and no rebuild fails.
  • Docs and smaller items. vars.yml is fixed, and check_sources.py passes. The audit body limit is documented. A past expires_at is rejected, with a test. short_ttl_stops_matching no longer races.
  • Advisory ignores. The deny.toml ignores match RUSTSEC-2026-0543 and RUSTSEC-2026-0542, and main fails cargo deny without them. That fix isn't specific to watchlists, so it might be better on its own PR so main turns green regardless of this one. That is non-blocking and the maintainers' call.

Blockers (details inline):

  1. With --state-db, DELETE /api/v1/sources/{source_id}/indicators always returns 400, because the audit middleware drops DELETE bodies.
  2. The 4096 cap is computed only at startup. A hot reload that makes a watchlist referenced brings back the original failure, where pushes return 200 and rebuilds fail. A later restart then silently trims the set.

Comment thread crates/rsigma-cli/src/daemon/audit.rs Resolved
Comment thread crates/rsigma-cli/src/daemon/server.rs Resolved
The audit trail was dropping DELETE bodies, and a hot reload that started referencing a watchlist left its cap at the configured maximum.

@mostafa mostafa left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approving. Both remaining items are fixed and verified against a build of this head.

  • Audited DELETE. With --state-db, DELETE removes the indicator and the stored row, and it is audited with a digest. A bodyless DELETE still gets 400. Putting back the old DELETE short-circuit fails delete_with_state_db_removes_the_stored_indicator.
  • Reload cap. The cap follows reloads in both directions. After a reload adds the reference, the push past 4096 gets 422. After a reload removes it, the same push succeeds, and no rebuild fails. A restart with more than the cap saved warns with the number left out (904 in my run), reports 4,096 restored, and starts. Removing the cap refresh after a full reload fails a_reload_changes_the_cap_when_a_watchlist_becomes_referenced.

fmt, clippy, workspace tests, doc, the no-default-features check, and CI are all green.

One non-blocking wording nit for whenever it's convenient: the CHANGELOG and http-api.md say the audit trail "keeps" the DELETE body, but it records a SHA-256 digest of the body (payload_digest), not the body itself.

@mostafa
mostafa merged commit 824f254 into timescale:main Nov 17, 2026
3 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants