Skip to content

feat: poll TAXII collections and STIX bundles as dynamic sources - #722

Merged
mostafa merged 20 commits into
timescale:mainfrom
SecurityEnthusiast:feat/stix-sources
Nov 19, 2026
Merged

mostafa merged 20 commits into
timescale:mainfrom
SecurityEnthusiast:feat/stix-sources

Conversation

@SecurityEnthusiast

Copy link
Copy Markdown
Contributor

Summary

Part of #685.

  • A dynamic source can turn STIX 2.1 indicators into a value pipelines and lookup enrichers already consume. type: taxii polls a TAXII 2.1 collection. format: stix on a file or HTTP source reads a bundle. Both need the stix-source feature. Without it the daemon exits 3 and names the feature.
  • The value is a JSON object. Every configured observable kind is present as a sorted, deduplicated array, and indicators maps each value to at most 16 metadata objects. An indicator is kept when its pattern type is stix, it is not revoked, it is inside its valid window, its confidence is unset or at least min_confidence, and the pattern is a simple comparison. Versions of one id keep the greatest modified. A newer revoked version removes it. Expiry happens on the first refresh after valid_until.
  • The first TAXII poll reads the collection. Later polls send added_after from the last page's X-TAXII-Date-Added-Last. A failed poll leaves the previous value and cursor in place. The cursor is not saved across a restart. The client uses the daemon egress policy, follows no redirects, and uses no proxy. Credentials are environment variable names, read when the resolver is built.
  • Pattern::simple_observables extracts the values. TAXII envelope parsing enforces the same depth, string, and object-count limits as bundle parsing.
sources:
  - id: ti
    type: taxii
    url: https://taxii.example/api1/
    collection: 91a7b528-80eb-42ed-a74d-c6fbd5a26116
    refresh: 15m
    page_size: 500
    auth:
      bearer_token_env: TAXII_TOKEN
    stix:
      min_confidence: 50
      observables: [ipv4, domain]

${source.ti.ipv4} is the address list. rsigma_stix_source_indicators, rsigma_stix_source_objects_total, and rsigma_stix_source_skipped_total report the active set, objects received, and skip reasons.
taxii sync and the type: stix enricher are unchanged. This path does not write a local store.
Breaking changes. Exhaustive matches on SourceType and DataFormat must handle Taxii and Stix. A TAXII envelope that exceeds the parse limits now fails.

Test plan

  • [*] cargo fmt --all -- --check
  • [*] cargo clippy --workspace --all-targets --all-features --locked -- -D warnings
  • [*] cargo test --workspace --all-features --locked
  • [*] cargo doc --workspace --all-features --locked --no-deps
  • [*] /Users/meghdadshamsaei/.cargo/bin/cargo +1.88.0 check --workspace --all-features --locked
  • [*] python3 scripts/docs-audit/check_sources.py

Envelope parsing now applies the same nesting, string-length, and object-count limits as bundle parsing, so a TAXII page cannot bypass the bounds a bundle already enforces.
Watchlists need the exact values inside a pattern. simple_observables returns those values only when every comparison is an independent equality or IN list, and reports why a pattern cannot be flattened.
A filter lets a caller refuse destinations such as loopback. IP literals are checked on every request, DNS answers that fail the filter are dropped, and redirects and proxies are disabled so a later hop cannot bypass the check.
Pipeline config can name a TAXII collection or a STIX bundle. The parser rejects mixed auth, unknown observable keys, watch or push refresh on TAXII, and extract on a STIX format. Credentials are environment variable names.
Workspace clippy rejects needless returns, default construction, and imports that only one feature compiles.
Workspace builds that unify serde_json preserve_order pretty-print insertion order, which failed the golden text compare.
A pooled wiremock server let one poll test answer another. Envelope limit checks are exported only with the taxii feature, and the metrics docs link the 11-metric heading.

@mostafa mostafa left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Requesting changes. Six blockers: two red gates, a TAXII poll that can loop forever, feed values acting as Sigma wildcards, the cap counting indicators that emit nothing, and a TAXII indicator set that never shrinks.

What is right: the parser validation is thorough (exactly one auth method, unknown stix keys rejected, format: stix refused on command and NATS, watch/push refused on TAXII), the observable extractor rejects every shape a value list cannot represent, the egress filter covers both IP literals and resolved names with redirects and proxies disabled, and the feature gating keeps rstix out of default builds. I mutated the object-count cap, version ordering, the literal-host filter, the DNS answer filter, and the envelope value limits, and each mutation failed its test. The lifecycle golden matches the fixture when recomputed by hand.

Blockers:

  1. cargo fmt --all -- --check fails in two files this PR adds (inline).
  2. npm run docs:build fails on three broken anchors (inline on the heading that moved them).
  3. TaxiiClient::poll_objects has no page or object bound and loops forever on a server that keeps saying more: true or keeps answering 416. Nothing wraps the resolve in an outer timeout, so this hangs daemon startup.
  4. STIX values reach rules through value_placeholders with * and ? live, so an exact STIX comparison becomes a wildcard.
  5. An indicator whose comparisons are all unmapped is kept, counted as active, and takes a slot under max_indicators, which can evict indicators that do emit values.
  6. The TAXII IndicatorSet never drops an expired or revoked indicator, so memory grows with feed churn regardless of max_indicators, and rsigma_stix_source_skipped_total recounts the whole set on every poll.

Non-blocking: the CHANGELOG breaking note misses the new stix field on SourceType::File and SourceType::Http, and one test mutates process environment while sibling tests run in parallel.

Everything else passed locally: clippy, cargo test --workspace --all-features --locked, cargo doc, the no-default-features check, cargo check for stix-source alone on rsigma-runtime and on the CLI, check_sources.py, check_binary.py, and docs:validate. The fork's CI workflows have not run yet, so only the CLA and StepSecurity checks are reported on this head.

Comment thread docs/content/reference/metrics.md Resolved
Comment thread crates/rstix/src/taxii/client.rs Resolved
Comment thread CHANGELOG.md Resolved
A TAXII poll stops when the page budget, a repeated cursor, or a repeated 416 would otherwise run without end. Array values escape Sigma wildcards, indicators that emit nothing do not take a cap slot, and revoked or expired ids are retained only as a modified time.

@mostafa mostafa left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Requesting changes for one remaining blocker: a TAXII collection that holds more indicator objects than max_indicators plus one page never loads. Everything else from the last review is fixed, and I checked each one in the code and with the original probes.

  • cargo fmt --check and npm run docs:build now pass.
  • Unbounded poll: a server that always answers 416 now fails after 1 request (it was 18,306 requests in 5 seconds), a page that repeats the cursor fails with RepeatedPaginationCursor, and an endless more: true stops at the budget.
  • Wildcards: with a format: stix file source on engine daemon, http://bad.example/a.php?id=1 no longer matches a.phpXid=1 and a * domain no longer matches google.com. Only the exact values match, and the indicators keys stay raw.
  • An indicator with no configured observable no longer takes a slot: the cap probe now keeps 198.51.100.7.
  • Retention and counting: expired and revoked ids become capped tombstones, the poll state is taken instead of cloned, and five polls of one expired indicator each now count 5, not 15.
  • The CHANGELOG names the new stix field with its migration, and the env-var test has its own binary.

I mutated the 416 repeat check, the cursor repeat check, the budget check, the escaping, the empty-selection skip, and the tombstone check, and each mutation failed a test. Locally, fmt, clippy, cargo test --workspace --all-features --locked, cargo doc, the no-default-features check, rsigma-runtime --features stix-source --all-targets, check_sources.py, check_binary.py, docs:build, and docs:validate all pass. CI is green except the coarse regression gate, which is still pending.

Comment thread crates/rstix/src/taxii/client.rs Resolved
Comment thread crates/rstix/src/taxii/client.rs Resolved
A collection larger than one poll keeps the pages it applied and continues from the last added-date. Indicators cut by the cap stay while the retained set is under twice max_indicators, and PollBudget is constructed rather than built as a struct literal.

@mostafa mostafa left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Requesting changes. The budget stop now keeps progress and resumes, which fixes the large-collection blocker from the last round: collection_larger_than_the_budget_loads_across_polls covers that case, and reverting the cursor advance makes it fail. Both earlier non-blocking points are addressed too. Indicators cut by the cap are kept in reserve and refill freed slots, and PollBudget is non-exhaustive with PollBudget::new. Formatting, clippy, the full test suite, docs, the no-default-features and stix-source checks, the docs audit, and the docs site build and validation all pass locally. CI is green apart from the coarse regression gate, which is still running.

Two blockers remain, both in the new resume path. A budget stop that lands inside a run of objects sharing one date_added skips the rest of that run forever, and with the default refresh: once a collection larger than one poll loads only partly with no signal. Two non-blocking notes cover the cost of the per-page lifecycle scan and three untested branches.

A budget stop now sets the cursor one microsecond before the last page's X-TAXII-Date-Added-Last, because added_after is exclusive and the next page can share that date. A stop that cannot move the cursor fails, and every budget stop logs a warning. A failed poll keeps the pages it applied with the old cursor instead of cloning the set to roll back, and the per-page lifecycle pass is gone because the object budget already bounds one poll.
@mostafa

mostafa commented Nov 19, 2026

Copy link
Copy Markdown
Member

I pushed three commits that settle how the poll budget behaves. My last three reviews pulled this in different directions, so here is where it lands:

  • The per-poll budget stays, because it bounds the time and memory of one poll.
  • A budget stop keeps what it applied and moves the cursor one microsecond before the last page's X-TAXII-Date-Added-Last. added_after is exclusive, so resuming from the date itself skipped objects that share it. This replaces my earlier suggestion to resume from that date, and also my suggestion to store the server's next token, which servers may expire between refreshes. TaxiiTimestamp::adjust_micros is the new rstix helper for it.
  • A budget stop that cannot move the cursor (no header, or one added-date covering more objects than the budget) fails, and every budget stop logs a warning. The docs now say that refresh: once loads only one poll, so a large collection needs a duration.
  • A failed poll keeps the pages it applied with the old cursor, so the poll no longer clones the whole set to roll back. The next poll reads those pages again and the upsert ignores the repeats.
  • The per-page tombstone_terminal pass is gone. I had asked for page-by-page pruning, but the object budget already bounds one poll and build prunes at the end, so the pass only added cost (about 5 s on a first poll of 100,000 indicators).

New tests cover a budget stop inside a shared added-date, a run of one date longer than the budget, a missing header, the cursor fallback when the last page has no header, a failure after an applied page, and the object budget against a server that ignores limit. Each one fails when its branch is reverted. Formatting, clippy, the workspace tests, docs, the feature checks, the docs audit, and the docs site build and validation pass locally.

@mostafa
mostafa merged commit bf544c3 into timescale:main Nov 19, 2026
34 checks passed
@SecurityEnthusiast

Copy link
Copy Markdown
Contributor Author

Thanks @mostafa

@mostafa

mostafa commented Nov 19, 2026

Copy link
Copy Markdown
Member

@SecurityEnthusiast Thank you for your contribution! 🙏

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants