Omni Extract Bench

This repository contains a way to benchmark extraction systems using our deterministic field-matching scorer and dataset as well as your own. Concretely, this repository holds:

• the scorer, which works on its own: a prediction its ground truth and its schema as input, the scoring as output;
• the benchmark: run over our dataset on Hugging Face, or your own; and
• the systems: different registered systems that you can easily evaluate.

Contents:

1. How scoring works: what field matching compares, and how;
2. Install: the scorer alone, or with the adapters and the dataset;
3. Run the benchmark: one system on the dataset, from a config;
4. Grade: a run again, or predictions of your own;
5. Score on your own: the scorer on its own, in code or from the command line;
6. Predict one document: one document and its schema through one system;
7. Results: every system we ran on our dataset;
8. Tests; and
9. License.

docs/API.md documents every public function and command in more depth.

How scoring works

Field matching is deterministic. It works in four stages:

• Normalize the document;
• Flatten the prediction and the ground truth to addresses (the path to a value, such as line_items[0].qty), each mapped to its scalar value;
• Normalize each scalar value, so "1,240.00" and 1240.0 are the same number; and
• Pair the addresses of the two documents:
  • If unambiguous, pair exactly; and
  • If ambiguous (the items of an array), Hungarian match on the items' content, recursively for nested arrays.

A predicted address with no matching ground-truth address is a false positive and a ground-truth address with no matching predicted address is a false negative. Accuracy is the matched addresses over the addresses either document used. The full specification is docs/METRIC_SPEC.md.

Install

uv pip install omni-extract-bench                 # the scorer, and grading a dataset folder on disk
uv pip install 'omni-extract-bench[harness]'      # + the adapters, to get predictions
uv pip install 'omni-extract-bench[benchmark]'    # + huggingface_hub, to download our dataset

Run the benchmark

A run is one system on one dataset. It works in three stages:

1. Plan: check the config, the system's keys and the dataset's files before any call;
2. Predict: send each document to the system and save each answer in the run's folder; and
3. Grade: score the saved answers. Grading calls no system, so it can be done again for free.

A config

A run is one config: a YAML file with every key written out. oeb init writes one for a registered system, every option at its default, and on our dataset:

oeb init datalab > datalab.yaml

dataset: hf://datasets/datalab-to/omni_extract_bench   # a folder, or a Hugging Face dataset
manifest: manifest.parquet                             # which of its documents, with which files
scorers:
  field_matching: {}
rows:                    
  sample: 0
  limit: 0
  only: null
timeout: 1800.0          # seconds
workers: 0               # documents in flight at once; 0 for the system's default
system:
  datalab:
    mode: balanced
    ...

oeb systems lists every system you can run; oeb systems <name> shows its options and the key it needs.

--set path=value changes one value the file has, read as YAML, and refuses a path it hasn't. This runs Datalab in accurate mode on one document:

export DATALAB_API_KEY=...
oeb run --config datalab.yaml --set system.datalab.mode=accurate --set rows.limit=1 --out runs/

oeb run shows the plan and asks before calling any system.

Run folder shape

runs/datalab-20261026-185908-1517/     
  config.yaml                         # resolved config; point to this to rerun
  settings.json                       # + launch information (run's record), resume uses this
  records/<id>.json                   # response from system in full (cost, raw response)
  predictions/<id>.json               # prediction portion of resopnse
  field_matching/summary.json         # summary of scores 
  field_matching/scores.jsonl         # scores (line-per-document)
  field_matching/details/<id>.jsonl   # detailed info (verdicts) per document

docs/API.md has more information, including how to run on your own dataset.

Resume

You can resume a run by giving a path to its folder.

oeb resume runs/<run>                                         # continue where it stopped
oeb resume runs/<run> --retry-failed                          # also ask again for timeouts, etc.
oeb resume runs/<run> --retry-failed --set timeout=3600       # increase timeout

Grade

oeb run grades its run when it finishes. To grade a run again, or a folder of your own predictions:

oeb grade runs/datalab-20261021-160012-3f9a        # a run knows its own dataset
oeb grade mine/ --dataset my-dataset/              # your predictions on your own dataset

Grading is fast on most documents, but field matching on a very large one takes long. On our dataset, long__real_ofac_ssi_full alone takes about 2 minutes, and all 620 documents take about 7 to 8 minutes of CPU, spread over up to 8 processes. Each document's score is kept in the run folder, so grading again only scores the documents whose prediction or gold changed.

Score on your own

The scorer works on its own, with no dataset and no network.

git clone https://github.com/datalab-to/omni_extract_bench && cd omni_extract_bench

from omni_extract_bench import score

schema = {
  "type": "object",
  "properties": {
    "invoice_id": {
      "type": "string",
      "description": "The invoice number as printed on the document."
    },
    "invoice_date": {
      "type": "string",
      "description": "Date of issue, ISO 8601 (YYYY-MM-DD)."
    },
    "total_due": {
      "type": "number",
      "description": "Total amount payable, in the invoice currency."
    },
    "purchase_order": {
      "type": "string",
      "description": "Buyer's purchase order number, where the invoice cites one."
    },
    "line_items": {
      "type": "array",
      "description": "One entry per billed line.",
      "items": {
        "type": "object",
        "properties": {
          "sku": {"type": "string", "description": "Stock code for the item."},
          "description": {"type": "string", "description": "Item description as printed."},
          "qty": {"type": "integer", "description": "Units billed."},
          "unit_price": {"type": "number", "description": "Price per unit."}
        }
      }
    }
  }
}

prediction = {
  "invoice_id": "INV-4417",
  "invoice_date": "03/31/2024",
  "total_due": "1,240.00",
  "purchase_order": "PO-88231",
  "currency": "USD",
  "line_items": [
    {"sku": "BX-2201", "description": "Washer, 8mm", "qty": 100, "unit_price": 9.4},
    {"sku": "AX-9910", "qty": 24, "unit_price": 12.05},
    {"sku": "ZZ-0000", "description": "Freight surcharge", "qty": 1, "unit_price": 45.0}
  ]
}

ground_truth = {
  "invoice_id": "INV-4417",
  "invoice_date": "2024-03-31",
  "total_due": 1240.0,
  "line_items": [
    {"sku": "AX-9910", "description": "Hex bolt, M8", "qty": 24, "unit_price": 12.5},
    {"sku": "BX-2201", "description": "Washer, 8mm", "qty": 100, "unit_price": 9.4}
  ]
}

result = score(prediction, ground_truth, schema)
result["accuracy"]    # 0.5294 -- matched addresses / addresses either document used
result["precision"]   # 0.5625
result["recall"]      # 0.8182

The same from the command line:

oeb score --pred data/pred.json --gold data/gold.json --schema data/schema.json

{
  "accuracy": 0.5294117647058824,
  "precision": 0.5625,
  "recall": 0.8181818181818182,
  "f1": 0.6666666666666666,
  "total": 17,
  "matched": 9,
  "misread": 1,
  "unfound": 1,
  "fabricated": 1,
  "invented_item": 4,
  "invented_field": 1,
  "asserted": 16,
  "addresses_found": 0.5882352941176471,
  "addresses_read_right": 0.9,
  "gold_rows": 2,
  "pred_rows": 3,
  "matched_rows": 2,
  "matching_exact": true,
  "approximated": [],
  "skipped_open_maps": []
}

To see where a prediction went wrong, ask for the verdicts: one per address, with its raw and normalized values on both sides and its result.

result = score(prediction, ground_truth, schema, verdicts=True)
print(result["verdicts"][0])
# Verdict(address=(('k', 'currency'),), gold_raw=None, pred_raw='USD', gold_canon=None,
#         pred_canon='usd', result='invented_field')

From the command line, --details adds them:

oeb score --pred data/pred.json --gold data/gold.json --schema data/schema.json --details

{
  "accuracy": 0.5294117647058824,
  ...
  "verdicts": [
    {
      "address": [["k", "currency"]],
      "gold_raw": null,
      "pred_raw": "USD",
      "gold_canon": null,
      "pred_canon": "usd",
      "result": "invented_field"
    },
    ...

Predict one document

With [harness] installed, each vendor is an adapter class. harness.system makes one from a system name and its options and harness.predict sends one document under the same rules a run uses (one time budget, retries, failures recorded) and returns its record:

from pathlib import Path

from omni_extract_bench import harness, score
from omni_extract_bench.harness import contract

datalab = harness.system("datalab", {"mode": "accurate"})
record = harness.predict(datalab, contract.Question(Path("invoice.pdf"), schema))
record["output"]    # the extracted object
record["raw"]       # the answer as the system gave it
record["cost"]      # what it cost, its wall time and attempts

score(record["output"], ground_truth, schema)

This sends the schema as you give it. A run first applies the benchmark's conventions to the schema (task.Extract().input), so use oeb predict to send exactly what a run sends.

oeb predict --system datalab --doc invoice.pdf --schema schema.json --options '{"mode": "accurate"}'
oeb predict --system datalab --doc invoice.pdf --schema schema.json --options '{"datalab": {"mode": "accurate"}}'

Results

Every system on our dataset, 620 documents. Accuracy is the matched addresses over the addresses either document used. The other columns are the share of addresses that fell short in each way, so a row sums to 100.

system	setting	accuracy	misread	unfound	fabricated	invented item	invented field
Datalab	accurate	93.85	3.10	1.64	0.73	0.69	0.00
Datalab	balanced	93.48	2.93	2.05	0.77	0.75	0.02
Reducto	deep_extract v2	93.47	3.23	1.59	0.99	0.71	0.00
Claude	Opus 5	90.96	2.33	4.91	1.22	0.58	0.00
Extend	extraction_performance	90.17	3.23	1.69	0.88	4.01	0.00
LlamaExtract	agentic_plus	87.94	3.85	2.53	5.14	0.54	0.00
Gemini	3.7 Flash	86.79	2.89	8.10	1.71	0.51	0.00
GPT	5.6 Sol	83.85	2.98	11.88	0.63	0.66	0.00
Mistral	OCR 4.1	76.78	6.11	12.73	1.87	0.88	1.63
Azure Content Understanding	gpt-4.1-mini	61.08	6.15	27.80	1.06	3.28	0.63

• misread: the address matches, but the value doesn't;
• unfound: a ground-truth address the system didn't produce;
• fabricated: the schema asks for it and the ground truth omits it, but the system produced it;
• invented item: an address under an array item that has no partner in the ground truth; and
• invented field: an address the schema never offered.

Tests

uv run --group test --extra benchmark pytest -n auto

License

Apache 2.0, see LICENSE. All code here is Datalab's own.