Repository navigation
feat: poll TAXII collections and STIX bundles as dynamic sources - #722
Conversation
Envelope parsing now applies the same nesting, string-length, and object-count limits as bundle parsing, so a TAXII page cannot bypass the bounds a bundle already enforces.
Watchlists need the exact values inside a pattern. simple_observables returns those values only when every comparison is an independent equality or IN list, and reports why a pattern cannot be flattened.
A filter lets a caller refuse destinations such as loopback. IP literals are checked on every request, DNS answers that fail the filter are dropped, and redirects and proxies are disabled so a later hop cannot bypass the check.
Pipeline config can name a TAXII collection or a STIX bundle. The parser rejects mixed auth, unknown observable keys, watch or push refresh on TAXII, and extract on a STIX format. Credentials are environment variable names.
Workspace clippy rejects needless returns, default construction, and imports that only one feature compiles.
Workspace builds that unify serde_json preserve_order pretty-print insertion order, which failed the golden text compare.
A pooled wiremock server let one poll test answer another. Envelope limit checks are exported only with the taxii feature, and the metrics docs link the 11-metric heading.
mostafa
left a comment
There was a problem hiding this comment.
Requesting changes. Six blockers: two red gates, a TAXII poll that can loop forever, feed values acting as Sigma wildcards, the cap counting indicators that emit nothing, and a TAXII indicator set that never shrinks.
What is right: the parser validation is thorough (exactly one auth method, unknown stix keys rejected, format: stix refused on command and NATS, watch/push refused on TAXII), the observable extractor rejects every shape a value list cannot represent, the egress filter covers both IP literals and resolved names with redirects and proxies disabled, and the feature gating keeps rstix out of default builds. I mutated the object-count cap, version ordering, the literal-host filter, the DNS answer filter, and the envelope value limits, and each mutation failed its test. The lifecycle golden matches the fixture when recomputed by hand.
Blockers:
cargo fmt --all -- --checkfails in two files this PR adds (inline).npm run docs:buildfails on three broken anchors (inline on the heading that moved them).TaxiiClient::poll_objectshas no page or object bound and loops forever on a server that keeps sayingmore: trueor keeps answering 416. Nothing wraps the resolve in an outer timeout, so this hangs daemon startup.- STIX values reach rules through
value_placeholderswith*and?live, so an exact STIX comparison becomes a wildcard. - An indicator whose comparisons are all unmapped is kept, counted as active, and takes a slot under
max_indicators, which can evict indicators that do emit values. - The TAXII
IndicatorSetnever drops an expired or revoked indicator, so memory grows with feed churn regardless ofmax_indicators, andrsigma_stix_source_skipped_totalrecounts the whole set on every poll.
Non-blocking: the CHANGELOG breaking note misses the new stix field on SourceType::File and SourceType::Http, and one test mutates process environment while sibling tests run in parallel.
Everything else passed locally: clippy, cargo test --workspace --all-features --locked, cargo doc, the no-default-features check, cargo check for stix-source alone on rsigma-runtime and on the CLI, check_sources.py, check_binary.py, and docs:validate. The fork's CI workflows have not run yet, so only the CLA and StepSecurity checks are reported on this head.
A TAXII poll stops when the page budget, a repeated cursor, or a repeated 416 would otherwise run without end. Array values escape Sigma wildcards, indicators that emit nothing do not take a cap slot, and revoked or expired ids are retained only as a modified time.
mostafa
left a comment
There was a problem hiding this comment.
Requesting changes for one remaining blocker: a TAXII collection that holds more indicator objects than max_indicators plus one page never loads. Everything else from the last review is fixed, and I checked each one in the code and with the original probes.
cargo fmt --checkandnpm run docs:buildnow pass.- Unbounded poll: a server that always answers 416 now fails after 1 request (it was 18,306 requests in 5 seconds), a page that repeats the cursor fails with
RepeatedPaginationCursor, and an endlessmore: truestops at the budget. - Wildcards: with a
format: stixfile source onengine daemon,http://bad.example/a.php?id=1no longer matchesa.phpXid=1and a*domain no longer matchesgoogle.com. Only the exact values match, and theindicatorskeys stay raw. - An indicator with no configured observable no longer takes a slot: the cap probe now keeps
198.51.100.7. - Retention and counting: expired and revoked ids become capped tombstones, the poll state is taken instead of cloned, and five polls of one expired indicator each now count 5, not 15.
- The CHANGELOG names the new
stixfield with its migration, and the env-var test has its own binary.
I mutated the 416 repeat check, the cursor repeat check, the budget check, the escaping, the empty-selection skip, and the tombstone check, and each mutation failed a test. Locally, fmt, clippy, cargo test --workspace --all-features --locked, cargo doc, the no-default-features check, rsigma-runtime --features stix-source --all-targets, check_sources.py, check_binary.py, docs:build, and docs:validate all pass. CI is green except the coarse regression gate, which is still pending.
A collection larger than one poll keeps the pages it applied and continues from the last added-date. Indicators cut by the cap stay while the retained set is under twice max_indicators, and PollBudget is constructed rather than built as a struct literal.
mostafa
left a comment
There was a problem hiding this comment.
Requesting changes. The budget stop now keeps progress and resumes, which fixes the large-collection blocker from the last round: collection_larger_than_the_budget_loads_across_polls covers that case, and reverting the cursor advance makes it fail. Both earlier non-blocking points are addressed too. Indicators cut by the cap are kept in reserve and refill freed slots, and PollBudget is non-exhaustive with PollBudget::new. Formatting, clippy, the full test suite, docs, the no-default-features and stix-source checks, the docs audit, and the docs site build and validation all pass locally. CI is green apart from the coarse regression gate, which is still running.
Two blockers remain, both in the new resume path. A budget stop that lands inside a run of objects sharing one date_added skips the rest of that run forever, and with the default refresh: once a collection larger than one poll loads only partly with no signal. Two non-blocking notes cover the cost of the per-page lifecycle scan and three untested branches.
A budget stop now sets the cursor one microsecond before the last page's X-TAXII-Date-Added-Last, because added_after is exclusive and the next page can share that date. A stop that cannot move the cursor fails, and every budget stop logs a warning. A failed poll keeps the pages it applied with the old cursor instead of cloning the set to roll back, and the per-page lifecycle pass is gone because the object budget already bounds one poll.
|
I pushed three commits that settle how the poll budget behaves. My last three reviews pulled this in different directions, so here is where it lands:
New tests cover a budget stop inside a shared added-date, a run of one date longer than the budget, a missing header, the cursor fallback when the last page has no header, a failure after an applied page, and the object budget against a server that ignores |
|
Thanks @mostafa |
|
@SecurityEnthusiast Thank you for your contribution! 🙏 |
Summary
Part of #685.
type: taxiipolls a TAXII 2.1 collection.format: stixon a file or HTTP source reads a bundle. Both need thestix-sourcefeature. Without it the daemon exits 3 and names the feature.indicatorsmaps each value to at most 16 metadata objects. An indicator is kept when its pattern type isstix, it is not revoked, it is inside its valid window, its confidence is unset or at leastmin_confidence, and the pattern is a simple comparison. Versions of one id keep the greatestmodified. A newer revoked version removes it. Expiry happens on the first refresh aftervalid_until.added_afterfrom the last page'sX-TAXII-Date-Added-Last. A failed poll leaves the previous value and cursor in place. The cursor is not saved across a restart. The client uses the daemon egress policy, follows no redirects, and uses no proxy. Credentials are environment variable names, read when the resolver is built.Pattern::simple_observablesextracts the values. TAXII envelope parsing enforces the same depth, string, and object-count limits as bundle parsing.${source.ti.ipv4}is the address list.rsigma_stix_source_indicators,rsigma_stix_source_objects_total, andrsigma_stix_source_skipped_totalreport the active set, objects received, and skip reasons.taxii syncand thetype: stixenricher are unchanged. This path does not write a local store.Breaking changes. Exhaustive matches on
SourceTypeandDataFormatmust handleTaxiiandStix. A TAXII envelope that exceeds the parse limits now fails.Test plan
cargo fmt --all -- --checkcargo clippy --workspace --all-targets --all-features --locked -- -D warningscargo test --workspace --all-features --lockedcargo doc --workspace --all-features --locked --no-deps/Users/meghdadshamsaei/.cargo/bin/cargo +1.88.0 check --workspace --all-features --lockedpython3 scripts/docs-audit/check_sources.py