Repository navigation
Magika 2.0: custom rules, and one fast pipeline for the CLI, Python and C #1537
Description
This issue tracks the work that puts Magika's rules and speed in the library, where every front end gets them, instead of in the CLI alone. Each PR below is self-contained and says Part of this issue.
Design
-
The low-level library API stays the real API:
Builder/Runtime;FeaturesOrRuled::extract_file/extract_content;Session::identify_features_batch.
Integrators build their own pipelines on it.
-
Custom rules live in
Options(Options::custom_rules, a cheaply clonedRules), so extraction applies them in any pipeline.- Order: custom rules, then built-in rules, then the model, each when enabled.
- Veto: custom rules veto the model the same way built-in rules do since Veto ML output when rules have no false negatives #1529. A content type whose enforced rules are all
class = "full"is vetoed when none of them matched.
-
Shared orchestration for our front ends lives in
magika::pipeline, behind apipelinefeature that is off by default, and uses the public API only.Engine: identifies from any thread, on the CPU at once and on the GPU once it's ready.Pipeline: parallel walk, read, batch, infer and reorder.- Front ends: the CLI, Python and C use these helpers instead of their own copies.
PRs in progress
| PR | What | Status |
|---|---|---|
| #1529 | Veto ML output when rules have no false negatives (@ia0); also reads the zip tail | In review (Yanick); prerequisite of #1530 |
| #1530 | Custom rules before the built-in rules: library Options::custom_rules, CLI --rules-file / --rules-check, Python rules= / rules_files=, C magika_rules_new; includes two fixes to #1529 (GPU tests; veto when features were extracted without rules) |
Ready, stacked on #1529 |
| #1531 | Engine and Pipeline in magika::pipeline (feature off by default); CLI, Python and C use them; C magika_engine_identify_paths |
Ready, stacked on #1530 |
| #1532 | Prepare CPU plans beyond the smallest on first use; fixes #1497 | ✅ Merged |
| #1533 | python: accept raw binary streams in identify_stream; fixes #1403, carries #1426 forward with its author |
Ready |
| #1476 | python: reject a bare str or bytes in identify_paths (@kaluli123123); fixes #1472 |
Ready, verified on main |
| #1475 | python: propagate the real error for unseekable streams (@kaluli123123); fixes #1473 | Ready, verified on main |
- Veto ML output when rules have no false negatives #1529
- Identify files with custom rules before the built-in rules #1530
- Share the CLI's fast pipeline with Python and C through the library #1531
- Prepare CPU plans beyond the smallest on first use #1532
- python: accept raw binary streams in identify_stream #1533
- python: reject a bare str or bytes in identify_paths (#1472) #1476
- python: propagate the real error for unseekable streams in identify_stream (#1473) #1475
Merged earlier, from splitting #1447
| PR | What |
|---|---|
| #1462 | Add the magika-rules crate |
| #1463 | Detect recursive directory cycles in the CLI |
| #1464 | Harden the CLI against bad limits, special files and pipeline errors |
| #1465 | Read small files once during feature extraction |
| #1466 | Fix tract convolution padding and check every GPU batch plan at startup |
| #1468 | Restore the reference conformance tests and qualify GPU decisions |
| #1470 | Link the C library on macOS |
| #1471 | Add 62 content types to the knowledge base |
| #1484 | Look up content types by label |
| #1486 | Fix release scripts on macOS and stop persisting CI checkout tokens |
| #1487 | Identify files with format rules in the library and CLI |
| #1489 | Start the tract runtime without parsing NNEF |
| #1490 | Identify on the CPU while the GPU is prepared |
| #1491 | Update the CLI README with BSD sed too |
| #1492 | Compile the bundled rules when the crate is built |
| #1494 | Evaluate zip and PE facts in magika-rules |
| #1496 | Build a rules regex only where a match can start |
Closed
| PR | Why |
|---|---|
| #1447 | The reference PR for the split; every lane is merged or listed here |
| #1495 | Its labels landed with #1494, and its zip tail read lands with #1529 |
| #1501 | Implemented #1497 from outside the maintainers; #1532 landed instead |
| #1426, #1412 | Same fix for #1403; carried forward in #1533 |
| #1407 | The Python symlink race no longer exists since symlinks are handled in Rust |
| #1379, #1362, #1317 | They targeted the Python fallback CLI, removed with #1483 (#1243 stays open) |
Next
- Misdetection sweep, once Veto ML output when rules have no false negatives #1529 and Identify files with custom rules before the built-in rules #1530 land: rules now cover docx, pptx, xlsb, pgp, heif, avif and gguf, so [Misdetection] DOCX file misdetected as spreadsheetml (XLSX) or application/zip #1326, [Misdetection] pptx file misdetected as xlsb #1259, [Misdetection] ASCII-armored GPG key file misdetected as PEM certificate #1338, GPG key type #1339, [Misdetection] HEIF Image file misdetected as MP4 media file #1439, [Misdetection] AVIF image file misdetected as MP4 video file #1233 and gguf files aren't recognized #1430 are worth rechecking with validated samples.
- Contributor sample PRs for Add more "basic" tests samples to cover supported content types #662 (Add basic test samples for cpp, go, shell, java, sql, and xml content types #1417, test: add basic samples for gif pickle powershell and ttf #1394, Add basic test samples for 10 missing content types (fixes #662) #1393, test: enable Windows python suite #1372, feat(tests): add sample test files for Bash, Go, Java, SAS, and XML #1367, Add Test Cases for Various File Types #1366, Add multiple sample files to test data #1346, test: add basic properties and env samples (#662) #1307): checking each sample with a format validator and importing the proven ones.
- Startup: CLI start pays about 1.5 ms for linking Metal even when it never uses the GPU #1493 (about 1.5 ms of Metal linking at startup) stays open.
Rules: preparation and packaging
- Authoring: authors edit
rust/rules/rulesets/{full,partial,notworking}/*.yar. - Generation:
rust/sync.shregenerates the committedsrc/bundled.rsand the rule labels, whichrust/genchecks against the knowledge base YAML. - Freshness: a test fails if
bundled.rsis stale. - Shipping: the rules are compiled into the CLI, the Python extension and the C library, so no rule file is read at runtime.
- Custom rules: they use the same validation, and are compiled once when the options are built.
Testing
- In every PR:
rust/test.sh(library, CLI, rules, C, generator),rust/changelog.sh, and the Python suite against the PyO3 build. - Rules PRs: the rules dataset gate (0 false positives).
- Performance: front-end changes report interleaved before/after timings for 1 to 3,000 files.
🤖 Generated with Claude Code
Activity