Build hybrid data pipeline (bootstrap + synthetic distillation)

[espetro]

1. Bootstrap: translate Mind2Web/Online-Mind2Web (CC BY 4.0) + WebArena trajectories into our schema.
2. Synthetic distillation: record real trajectories (nanobrowser or agent-browser/camofox-browser harness), label/correct via teacher LLM into our schema.
3. Hold out eval slice.

[espetro]

Pipeline Status

Bootstrap, map, split, and distill scaffold committed in .

Run to build training data:

# Requires ~6.7GB download + Mind2Web_AXT variant (bounding boxes)
uv run python -m data.pipeline.cli bootstrap --out data/raw/mind2web-axtree --axtree

# Map Mind2Web rows → HTLM records (one JSONL)
uv run python -m data.pipeline.cli map-mind2web     --src data/raw/mind2web-axtree     --out data/processed/mind2web-htlm.jsonl

# Deterministic train/eval split (task-grouped, stable across runs)
uv run python -m data.pipeline.cli split     --in data/processed/mind2web-htlm.jsonl     --train data/processed/train.jsonl     --eval data/processed/eval.jsonl

# Synthetic distillation (requires TEACHER_API_KEY)
uv run python -m data.pipeline.cli distill \
    --in data/processed/eval_raw.jsonl \
    --out data/distilled/teacher-eval.jsonl

See data/pipeline/README.md for full documentation.

Remaining work: Online-Mind2Web and WebArena require live browser env — deferred.