Add mteb/api FastAPI service and HF Space Dockerfile

[Samoed]

A FastAPI service that powers the leaderboard frontend. New package under mteb/api/.

Module layout (mteb/api/)

Module	Role
app.py	App factory: GZip + CORS + Prometheus middleware, mounts router under /v1 and infra_router at root, lifespan-driven warmup, /og static mount with 1-day Cache-Control.
routes.py	All HTTP endpoints. Cached endpoints serve pre-built bytes via _cached_json (handles 304, gzip negotiation, Cache-Control); uncached ones return pydantic schemas.
schemas.py	Pydantic response models. snake_case in Python, camelCase over the wire (matches leaderboardv2/src/lib/types.ts).
adapters.py	Memoised wrappers around Schema.from_* constructors so each benchmark/task/model pays construction cost once.
aggregators.py	Pure builders (build_benchmark_summary, build_benchmark_per_language, build_benchmark_leaders, build_model_scores, build_task_scores) — turn long polars frames into schema objects.
frames.py	Process-wide long-results polars frames, loaded once. Sits below cache so aggregators can depend on it without dragging in the bytes cache.
cache.py	CacheLayer generic: single-flight per-key locks + LRU store + Prometheus labels. Holds the warm serialised bytes routes hand out.
serialization.py	Off-thread schema → JSON bytes + gzip pair, used by the bytes cache.
warmup.py	Lifespan-time warmup: builds frames, prewarms schema caches, optionally preloads every summary in the background (semaphore-capped).
metrics.py	Prometheus middleware + counters/histograms/in-flight gauge; cache-outcome counter; /metrics renderer.
otel.py	OTLP HTTP tracing setup — one span per request, W3C traceparent propagation. No-op unless OTEL_EXPORTER_OTLP_ENDPOINT is set.
icons.py	Benchmark icon proxy + cache, so the browser gets immutable 1-year cache instead of upstream's 5-minute one.
settings.py	pydantic-settings knobs: CORS_ORIGINS, PRELOAD, CACHE_REPO, OG_DIR, PREWARM_MAX_WORKERS, PRELOAD_CONCURRENCY, HTTP_MAX_AGE, DISK_CACHE, log level, OTEL vars.
static/favicon.png	Shipped as package data.

Endpoint map

Data routes under /v1, infra at root.

/health                                   GET   liveness
/metrics                                  GET   prometheus scrape
/robots.txt, /favicon.ico, /og/*          GET   static / proxied assets
/v1/icon/{name:path}                      GET   benchmark icon proxy

/v1/benchmarks                            GET   flat list
/v1/benchmarks/menu                       GET   nested menu tree
/v1/benchmarks/{name}                     GET   single benchmark metadata
/v1/benchmarks/{name}/scores              GET   full summary (legacy alias: /summary)
/v1/benchmarks/{name}/per-language        GET   per-language rows
/v1/benchmarks/{name}/leaders             GET   slim home-page leader tiles

/v1/tasks                                 GET   flat list
/v1/tasks/{name}                          GET   single task metadata
/v1/tasks/{name}/scores                   GET   per-task scores

/v1/models                                GET   flat list
/v1/models/{name}                         GET   single model metadata
/v1/models/{name}/scores                  GET   per-model scores

Request flow

request → CORSMiddleware → GZipMiddleware → PrometheusMiddleware → router
            │
            ├── cached endpoints ──→ cache.CacheLayer (single-flight)
            │                          │ miss → aggregators.build_* → serialize_schema (off-thread)
            │                          └ hit  → cached (bytes, gzip, etag)
            │                                    → Response with 304 / Vary / Cache-Control
            │
            └── uncached endpoints → adapters.*_to_schema → pydantic-core JSON

Long-frame source of truth lives in frames.py (loaded once at startup or first request; persisted to ~/.cache/mteb/leaderboard/, invalidated by HF dataset commit SHA).

[Copilot]

Pull request overview

This PR introduces a new mteb.api FastAPI service to expose leaderboard data over HTTP, plus tooling and packaging changes to support deployment (including an HF Space–oriented Dockerfile) and pre-rendered Open Graph (OG) image generation/serving.

Changes:

• Added mteb/api FastAPI app with cached JSON endpoints, warmup/preload, Prometheus metrics, icon proxying, and optional OpenTelemetry tracing.
• Added OG hero-card HTML template + Playwright-based generator, and mounted generated PNGs under /og in the API service.
• Refactored leaderboard/summary table building and related display logic to support the API’s canonical column naming and metadata handling; added language label helpers and moved leaderboard parquet path onto ResultCache.

Reviewed changes

Copilot reviewed 36 out of 39 changed files in this pull request and generated 3 comments.

Show a summary per file

File	Description
tests/test_benchmarks/test_get_benchmarks.py	Updates the “display_on_leaderboard” test fixture benchmark name.
scripts/og-template/template.html	Adds a parameterized HTML template used for OG card rendering.
scripts/generate_og_images.py	Adds Playwright-based OG image renderer with incremental hashing and manifest output.
scripts/bench_leaderboard.py	Switches parquet path lookup to ResultCache.leaderboard_parquet_path.
scripts/bench_api.py	Adds a stdlib-only benchmark script for measuring API endpoint latency/size/gzip/ETag behavior.
scripts/bench_api_inproc.py	Adds an in-process benchmark using httpx ASGITransport for stable perf comparisons.
pyproject.toml	Adds api and og extras, packages mteb.api static assets, and enables Ruff FAST rules.
mteb/results/benchmark_results.py	Extends result frame handling (incl. trained-on flag) and adds frame-splitting helpers.
mteb/leaderboard/table.py	Adjusts summary/per-task/per-language styling; adds model short-name/link wrapping and type header humanization.
mteb/leaderboard/figures.py	Updates model-name parsing and task-type column detection consistent with canonical column names.
mteb/leaderboard/app.py	Removes unused typing import and uses cache.leaderboard_parquet_path.
mteb/languages/iso_mappings.py	Adds language_label() helper with script display aliases and caching.
mteb/languages/init.py	Exposes language_label from the languages package.
mteb/cache/result_cache.py	Adds leaderboard_parquet_path property to decouple consumers from the Gradio leaderboard module.
mteb/benchmarks/benchmarks/benchmarks.py	Adds new_version metadata and sets specific benchmark aggregation behavior.
mteb/benchmarks/benchmark.py	Adds BenchmarkAggregation, summary-table refactor (pivot reuse), and benchmark-specific aggregation config.
mteb/benchmarks/_leaderboard_menu.py	Introduces HOME_BENCHMARK_ENTRIES for the new API/menu surface.
mteb/benchmarks/_create_table.py	Introduces SummaryTable wrapper and refactors summary/per-task/per-language builders and metadata attachment.
mteb/api/warmup.py	Adds startup warmup orchestration and optional background preload for summary/per-language caches.
mteb/api/settings.py	Adds pydantic-settings based environment configuration for the API service.
mteb/api/serialization.py	Adds shared JSON+gzip+ETag serialization primitives used by caches/routes.
mteb/api/schemas.py	Adds pydantic response models matching frontend types, including language labeling and icon proxy behavior.
mteb/api/routes.py	Adds FastAPI route handlers (cached bytes responses, icons, favicon, metrics, menu/bench/task/model endpoints).
mteb/api/README.md	Documents API install/run, endpoints, CORS, and observability setup.
mteb/api/otel.py	Adds optional OTEL tracing setup/instrumentation for FastAPI.
mteb/api/metrics.py	Adds Prometheus middleware and registry-scoped metrics for requests/caches/entities.
mteb/api/icons.py	Adds cached icon proxying with timeouts and negative caching.
mteb/api/frames.py	Adds hub/disk-cache loading and split/unified polars-frame management for API aggregators.
mteb/api/cache.py	Adds single-flight caches for schemas and pre-serialized bytes per endpoint.
mteb/api/app.py	Adds FastAPI application factory, middleware, route mounting, and /og static mount.
mteb/api/aggregators.py	Adds builders for summary/task/model/per-language/leaders payloads.
mteb/api/adapters.py	Adds cached adapters around schema construction and a threaded prewarm routine.
mteb/api/init.py	Exposes create_app() entrypoint for the API package.
Makefile	Adds serve-api target and includes the api extra in test installs.
Dockerfile	Replaces the previous single-stage image with a multi-stage build for API runtime + OG builder.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Samoed]

generate_og_images_mpl.py is for generating the same benchmark "preview" as generate_og_images, but using only Matplotlib. Currently in Docker, generate_og_images is used, but it renders from a browser with Playwright. I'm using it because it looks a bit nicer than Matplotlib

[KennethEnevoldsen]

Let me start with some general comments before I go deeper.

packaging: Do we want the API in the mteb package

while I agree that we want the API in this repo (so we can test the two jointly), I would probably consider making it into a second package within the repo called mteb-api.

This made be think that we could also factor out models into a packages (mteb-models) to isolate our dependency hell outside of mteb. Not for this PR though.

leaderboard cli:

For the current leaderboard CLI how do we want it to behave? should the leaderboard CLI run the current leaderboard? if so we probably want to clean up gradio.

[KennethEnevoldsen]

? do we want use this as the favicon?

[Samoed]

I'm not sure. I used this to differentiate during development between api and leaderboard

[KennethEnevoldsen]

How big is this once built?

[Samoed]

~6 GB.

• 1 GB source image.
• 2 GB pip dependencies
• Results 250 mb
• Open graph images 750mb

[KennethEnevoldsen]

why is this changed?

[Samoed]

I changed list of "visible" benchmarks with new menu. Mteb multilingual is "primary" benchmark, so it's not listed in new object. We can change this, but I'm not sure how to handle primary benchmarks

[KennethEnevoldsen]

would be nice to combine all of these scripts into one folder

[KennethEnevoldsen]

delete this file?

[Samoed]

I think this could help for future api modifications to not have performance degradation

[KennethEnevoldsen]

superseded_by? (for consistency)

[KennethEnevoldsen]

if we have this here - we might even be able to deprecate the RTEBBenchmark object

[KennethEnevoldsen]

maybe some of the other ones as well

[Samoed]

Yes, we can do this

[Samoed]

Updated to use aggregations directly

[KennethEnevoldsen]

Does some of these belong in the API rather than in mteb core?

[Samoed]

I think we can reuse this in future somewhere. I don't think that mapping from lang codes to langname is api specific

[KennethEnevoldsen]

why is this function changed in this PR?

[Samoed]

I changed list of visible benchmarks. From GP_BENCHMARKS + RTEB_BENCHMARK to

mteb/mteb/benchmarks/benchmark.py

Line 60 in 30757af

entries = HOME_BENCHMARK_ENTRIES

benchmark. I changed this to keep previous behavior

[Samoed]

while I agree that we want the API in this repo (so we can test the two jointly), I would probably consider making it into a second package within the repo called mteb-api.

I think we can keep as is for now like previous leaderboard. We can refactor it in future

For the current leaderboard CLI how do we want it to behave? should the leaderboard CLI run the current leaderboard? if so we probably want to clean up gradio.

I think we should keep old leaderboard. I think some people can fork mteb to create their propriete benchmarks and to see scores they could use gradio implmenetation. I don't think that this is possible to run new leaderboard from our package because it's js app