Skip to content

Add provider-agnostic text extraction (OCR / document parsing) capability #268

Open
#269
Open
Add provider-agnostic text extraction (OCR / document parsing) capability#268
#269

Description

@saarnilauri
saarnilauri
opened on Jul 29, 2026

Proposal: Text Extraction Capability (OCR / Document Parsing)

Summary

Add a first-class, provider-agnostic text extraction capability to the PHP AI Client, so that provider packages can register document-parsing/OCR models the same way they register text generation and image generation models today.

The design is validated against five concrete provider APIs with very different shapes:

Provider API style Sync/Async Native output Validation
Mistral OCR (mistral-ocr-*) dedicated POST /v1/ocr sync markdown per page + image bboxes Tested (working PoC connector, live API)
LlamaParse (LlamaCloud) upload → job → poll (/api/v1/parsing/*; v2 exists) async only markdown/text/structured items per page Tested (working PoC connector, live API)
Google Cloud Document AI processor :process / :batchProcess both Document JSON (blocks, anchors, bboxes) Researched with AI (official docs)
AWS Textract DetectDocumentText / AnalyzeDocument + Start/Get jobs both flat Blocks[] graph (bboxes, tables, KV) Researched with AI (official docs)
Qwen2.5-VL / Qwen3-VL / qwen-vl-ocr chat completions with image input sync prompt-shaped text (md/HTML/LaTeX/JSON) Researched with AI (official docs)

The goal is that each of these can ship as an independent connector package implementing one shared interface, exactly as ai-provider-for-mistral does for text generation today.

Motivation

Document text extraction is one of the most common AI tasks in CMS contexts (ingesting PDFs into posts, indexing uploads for search/RAG, accessibility). Today the SDK cannot model it:

  1. CapabilityEnum has no extraction-shaped value; ProviderRegistry::findModelsMetadataForSupport() therefore can never discover such models.
  2. Extraction results (pages, markdown, bounding boxes, extracted images, page counts) do not fit the GenerativeAiResult envelope (candidates + finish reasons + messages). Flattening a multi-page structured response into a single model-message candidate loses the structure that is the entire point of dedicated OCR endpoints.
  3. The prompt paradigm doesn't fit: the primary input is a document, not a message list, and options like page ranges have no home in PromptBuilder.

Downstream provider plugins are blocked on this (e.g. ai-provider-for-mistral cannot expose mistral-ocr-latest); the alternative, provider-specific side APIs outside the SDK, defeats the SDK's core value of capability-agnostic model discovery.

Why not just send the file to a text generation model?

The SDK can already attach a document to a chat prompt (inputModalities: [document]) and ask an LLM to "transcribe this". That path is a complement, not a substitute, because dedicated extraction endpoints are fundamentally more reliable for getting a document's actual text out:

  • Hallucination. A generative model produces text conditioned on the document; nothing constrains its output to be a faithful transcription. On long PDFs especially, LLMs are known to silently skip pages, paraphrase, "fix" numbers and names, or fabricate plausible-looking content for low-quality scans. Extraction pipelines transcribe what is on the page, and several (Google, AWS, optionally Mistral) attach per-word/per-block confidence scores so uncertainty is reported instead of papered over.
  • Length limits. A long PDF quickly exhausts a chat model's context window, and output token limits cap how much text can come back in one completion. Extraction endpoints process documents page by page (Mistral OCR: up to ~1,000 pages per request; Textract/Document AI batch: thousands of pages) with no prompt-window coupling.
  • Determinism and structure. Extraction returns machine-verifiable structure, page indices, bounding boxes, tables, embedded images, that a free-text completion cannot guarantee and that flattening into a chat message destroys.
  • Cost. Per-page extraction pricing (e.g. $4/1,000 pages for Mistral OCR) is orders of magnitude cheaper at document scale than paying per input+output token to have an LLM re-emit an entire document.

For ingestion pipelines (search indexing, RAG, archiving, accessibility), where fidelity to the source is the requirement, extraction is the correct tool, and the SDK currently cannot express it. Document QnA via chat remains the right tool for reasoning about a document, and the two compose: extract first, then feed clean markdown to a text generation model.

Completing the RAG pipeline with embedding generation

The SDK gained embedding generation in 1.4.0, but embedding models take text (list<MessagePart>), they cannot read a PDF. Text extraction is the missing first stage of that pipeline: without it, a consumer who wants to embed their document library must leave the SDK to get the text out, which reintroduces exactly the per-provider integration work the SDK exists to remove.

With this capability the full document-to-vector flow becomes SDK-native and provider-agnostic:

$result = AiClient::document($pdfUrl)->extractTextResult();

foreach ($result->getPages() as $page) {
    $embedding = AiClient::input($page->getMarkdown())->generateEmbedding();
    // store vector with page number for citation-accurate retrieval
}

The extraction result is also a better embedding input than any ad-hoc alternative: per-page (and optionally per-block) segmentation gives natural chunk boundaries that respect document structure instead of arbitrary character offsets; markdown preserves headings and tables that improve embedding quality; and page numbers/bounding boxes carried alongside each chunk enable citation-accurate retrieval, the retrieved passage can point back to the exact page and region of the source document. Layout-aware chunking is precisely what Google's Layout Parser markets itself for in RAG contexts; this proposal makes that pattern available uniformly across providers.

Precedent: embedding generation (1.4.0) faced the same mismatch and established the pattern this proposal follows: a new capability value, a standalone model interface, a dedicated result DTO implementing ResultInterface, a separate builder (AiClient::input() → EmbeddingBuilder), a dedicated ModelRequirements factory, and its own events. Text extraction is the same kind of "not prompt-shaped" capability.

Naming

Recommend TEXT_EXTRACTION = 'text_extraction' over OCR:

  • "OCR" implies raster input; LlamaParse and Document AI Layout Parser parse born-digital DOCX/PPTX/HTML with no optical step.
  • It parallels existing verb-object names (text_generation, image_generation, embedding_generation).
  • It leaves room for the same capability to cover future "document parsing" providers without a misleading name.

(Naming is genuinely bikesheddable; DOCUMENT_TEXT_EXTRACTION is the verbose alternative. The rest of this proposal uses TEXT_EXTRACTION.)

Design principles (derived from the API survey)

  1. Input is a File. The existing File DTO already models the two transport forms every provider accepts in its sync path: remote URL and inline base64 (Mistral document_url/data-URI, LlamaParse source_url/upload, Google rawDocument.content, Textract Document.Bytes, Qwen image_url/data-URI). Provider-side file references (S3/GCS/pre-uploaded file IDs) are provider implementation details, reachable via customOptions in v1.
  2. Normalize to pages of markdown; keep structure optional. Every provider has a per-page notion and can produce text per page. Markdown is the richest common text format (Mistral and LlamaParse emit it natively; Google/Textract adapters synthesize it from blocks; Qwen is prompted into it). Bounding boxes, tables, and extracted images exist only in some providers, so they are optional fields, and model metadata declares what a model can do via supported options.
  3. Never discard provider fidelity. The raw decoded provider payload rides along in additionalData, so consumers who need Textract Blocks relationships or the full Document AI JSON aren't blocked by the normalization.
  4. Sync interface first; async as a follow-up that reuses the existing operations track. Mistral and Qwen are sync; Google and Textract have sync paths with limits; LlamaParse is async-only but short-lived (an adapter can poll internally). A TextExtractionOperationModelInterface mirroring the existing generateXOperation() pattern is specified but explicitly deferred (the operations track has no concrete implementation in-tree yet).
  5. Auth stays out of the capability. Google needs OAuth service accounts and AWS needs SigV4, neither fits RequestAuthenticationMethod::apiKey(). That is an orthogonal, pre-existing gap tracked in Support provider-owned custom request authentication #237 (provider-owned custom request authentication); this proposal depends on it only for those two connectors, not for the capability itself.

Proposed API

1. Capability enum value

src/Providers/Models/Enums/CapabilityEnum.php, add one constant plus the magic-method docblock lines (no factory body needed; AbstractEnum::__callStatic provides textExtraction() and isTextExtraction() automatically):

/**
 * Text extraction (OCR / document parsing) capability.
 *
 * @since n.e.x.t
 */
public const TEXT_EXTRACTION = 'text_extraction';
 * @method static self textExtraction() Creates an instance for TEXT_EXTRACTION capability.
 * @method bool isTextExtraction() Checks if the capability is TEXT_EXTRACTION.

ProviderRegistry discovery works with zero further changes, ModelRequirements::areMetBy() is capability-generic.

2. Model interface

src/Providers/Models/TextExtraction/Contracts/TextExtractionModelInterface.php (standalone interface, like all capability contracts; concrete models also implement ModelInterface via AbstractApiBasedModel):

interface TextExtractionModelInterface
{
    /**
     * Extracts text and structure from a document.
     *
     * Extraction options (page selection, image/bounding-box inclusion,
     * output format) are provided via the model's ModelConfig, consistent
     * with how generation options are handled.
     *
     * @since n.e.x.t
     *
     * @param File $document The document to process (remote URL or inline data).
     * @return TextExtractionResult The structured extraction result.
     */
    public function extractTextResult(File $document): TextExtractionResult;
}

Options ride on ModelConfig (house style, capability interfaces take only the payload; setConfig() carries everything else), which also makes them discoverable/matchable through SupportedOption metadata.

3. Result DTOs

Following the EmbeddingResult precedent: a dedicated result implementing ResultInterface (NOT candidate-based), living in src/Results/DTO/. All DTOs extend AbstractDataTransferObject with KEY_* constants, toArray()/fromArray()/getJsonSchema(), and deep __clone.

class TextExtractionResult implements ResultInterface
{
    public function getId(): string;
    /** @return list<ExtractedPage> */
    public function getPages(): array;
    public function getPageCount(): int;                 // pages processed (billing-relevant for per-page providers)
    public function getTokenUsage(): TokenUsage;         // zeros for page-priced providers; real for VLM-based extraction
    public function getProviderMetadata(): ProviderMetadata;
    public function getModelMetadata(): ModelMetadata;
    public function getAdditionalData(): array;          // MUST include the raw provider payload under 'raw'
    public function toText(): string;                    // convenience: all pages' markdown joined
    public function toMarkdown(): string;                // alias emphasizing format
}

class ExtractedPage extends AbstractDataTransferObject
{
    public function getPageNumber(): int;                // 1-based, normalized across providers
    public function getMarkdown(): string;               // markdown (may be plain text for text-only providers)
    /** @return list<ExtractedBlock> */
    public function getBlocks(): array;                  // optional; empty when unsupported/not requested
    /** @return list<ExtractedImage> */
    public function getImages(): array;                  // optional; empty when unsupported/not requested
    public function getDimensions(): ?PageDimensions;    // null when the provider doesn't report them
}

class ExtractedBlock extends AbstractDataTransferObject
{
    public function getType(): TextExtractionBlockTypeEnum; // PARAGRAPH | HEADING | TABLE | LIST | IMAGE | OTHER
    public function getText(): string;
    public function getBoundingBox(): ?BoundingBox;
    public function getConfidence(): ?float;             // 0–1; null when the provider doesn't score
}

class ExtractedImage extends AbstractDataTransferObject
{
    public function getId(): string;
    public function getFile(): ?File;                    // inline base64 File when returned, else null
    public function getBoundingBox(): ?BoundingBox;
}

class BoundingBox extends AbstractDataTransferObject
{
    // Normalized coordinates in the 0–1 range, origin top-left.
    // Rationale: Textract is natively normalized; Google provides both;
    // pixel-native providers (Mistral) divide by page dimensions, which
    // they always return. Pixel values are recoverable via PageDimensions.
    public function getLeft(): float;
    public function getTop(): float;
    public function getWidth(): float;
    public function getHeight(): float;
}

class PageDimensions extends AbstractDataTransferObject
{
    public function getWidth(): int;    // pixels
    public function getHeight(): int;   // pixels
    public function getDpi(): ?int;
}

TokenUsage note: ResultInterface requires it; page-priced providers return new TokenUsage(0, 0, 0) and the meaningful unit is getPageCount(). VLM-based connectors (Qwen) populate real token counts. Both are honest; neither overloads the other's field.

4. ModelConfig / OptionEnum additions

OptionEnum reflects ModelConfig::KEY_* constants automatically, so adding config keys is the entire change. Proposed minimal set:

ModelConfig::KEY_EXTRACTION_PAGES          = 'extractionPages';         // list<int>, 1-based page selection
ModelConfig::KEY_EXTRACTION_INCLUDE_IMAGES = 'extractionIncludeImages'; // bool
ModelConfig::KEY_EXTRACTION_INCLUDE_BLOCKS = 'extractionIncludeBlocks'; // bool (bounding boxes / layout)

Deliberately reused rather than duplicated:

  • outputMimeType, text/markdown (default) vs text/plain; a connector for a provider with HTML table output could advertise text/html.
  • outputSchema, schema-driven structured extraction (Mistral document_annotation_format with JSON schema, Textract QUERIES/FORMS mapping, Document AI custom extractors, qwen-vl-ocr key_information_extraction.result_schema). A model advertising SupportedOption(OptionEnum::outputSchema()) under this capability means "can return schema-shaped extraction", the result's additionalData['structuredData'] carries it in v1.
  • inputModalities, declares what documents a model accepts: [document], [image], or both. This is how the Qwen connector honestly declares images-only (no native PDF input; rasterization is out of scope), while Mistral/LlamaParse/Google declare [document, image].
  • customOptions, escape hatch for provider-specific knobs (Mistral image_min_size, LlamaParse tier, Textract FeatureTypes, Google processor selection, DashScope ocr_options.task).

5. Fluent API

Text extraction gets its own builder, exactly as embeddings did (per ARCHITECTURE.md's rationale: prompt-oriented parameters don't apply).

AiClient additions:

public static function document($document = null, ?ProviderRegistry $registry = null): TextExtractionBuilder;

public static function extractTextResult($document, $modelOrConfig = null, ?ProviderRegistry $registry = null): TextExtractionResult;
public static function extractText($document, $modelOrConfig = null, ?ProviderRegistry $registry = null): string;

src/Builders/TextExtractionBuilder.php:

$result = AiClient::document('https://example.com/report.pdf')
    ->fromPages([1, 2, 3])
    ->includingImages()
    ->includingBlocks()
    ->usingProvider('mistral')            // via the shared ModelResolutionTrait
    ->extractTextResult();                // TextExtractionResult

$markdown = AiClient::document($file)->extractText(); // shorthand → string

The builder:

  • accepts File|string (URL, data URI, or local path, File already normalizes all three);
  • uses ModelResolutionTrait (usingModel(), usingProvider(), usingModelPreference(), …) unchanged;
  • builds requirements via a new ModelRequirements::fromExtractionData(File $document, ModelConfig $config): capability textExtraction(), RequiredOption(inputModalities, [document|image]) chosen from the file's MIME type, plus required options for any set config keys;
  • validates the resolved model instanceof TextExtractionModelInterface and throws the standard 'Model "%s" does not support text extraction.' otherwise;
  • dispatches BeforeExtractTextEvent / AfterExtractTextEvent (mirroring the embedding events);
  • exposes isSupported(): bool for feature detection.

6. Abstract base class

src/Providers/ApiBasedImplementation/AbstractApiBasedTextExtractionModel.php, mirrors the existing OpenAI-compatible generation bases: holds metadata/config, template method flow prepareRequest(File): Request → authenticate → send → throwIfNotSuccessful() → parseResponseToTextExtractionResult(Response): TextExtractionResult. Concrete connectors override the two abstract ends.

Unlike text generation there is no dominant wire format to ship a shared "compatible" implementation for, each connector implements its own request/parse pair. That is fine; the shared value is in the result normalization and discovery, not the HTTP shape.

7. Async operations (specified, deferred)

TextExtractionOperationModelInterface::extractTextOperation(File $document): TextExtractionOperation plus a TextExtractionOperation DTO (id, OperationStateEnum, ?TextExtractionResult), mirroring the existing generateXOperation() contracts and GenerativeAiOperation. Deferred because the operations track has no concrete provider-side implementation or builder exposure anywhere in the SDK yet; text extraction shouldn't be the pioneer.

Until then, async-only providers poll internally behind the sync interface with a configurable timeout (LlamaParse jobs on typical documents complete in seconds; this is what its own SDKs do). Connectors SHOULD expose the timeout via customOptions and throw a clear RuntimeException on expiry.

Connector mapping

How each target provider implements TextExtractionModelInterface:

Concern Mistral OCR LlamaParse Google Document AI AWS Textract Qwen-VL / qwen-vl-ocr
Request POST /v1/ocr with document_url/data-URI upload or source_url → POST /api/v2/parse → poll GET /api/v2/parse/{id} processors/{id}:process with base64 rawDocument AnalyzeDocument/DetectDocumentText with Document.Bytes chat completion, image content + extraction prompt / ocr_options
Auth Bearer key (supported today) Bearer key (supported today) OAuth service account, needs #237 SigV4, needs #237 Bearer key (DashScope) or none (self-hosted vLLM)
Pages → ExtractedPage native pages[].markdown v2 expand=markdown,items per page synthesize markdown from pages[].blocks/paragraphs + textAnchor synthesize from LINE/LAYOUT_* blocks grouped by Page one page per input image; text as returned
Blocks/bboxes include_blocks paragraph bboxes (pixel → normalize by dimensions) layout items boundingPoly (already normalized) Geometry.BoundingBox (already normalized) only qwen-vl-ocr advanced_recognition (rotated rects → axis-aligned approximation, or omit)
Images include_image_base64 → ExtractedImage image download URLs → fetch → inline File none (empty) none (empty) none (empty)
Model discovery GET /v1/models (mistral-ocr-*) fixed tier list, hardcoded metadata list processors as "models" single hardcoded model entry (+ adapters later) hardcoded / GET /v1/models on self-hosted
Usage usage_info.pages_processed → getPageCount() page count from result pages length DocumentMetadata.Pages real TokenUsage from completion

The Qwen connector is the important stress test: it proves the capability is about the contract, not the transport, a chat-completions-backed extractor and a dedicated-endpoint extractor are interchangeable to consumers, which is precisely the provider-agnostic philosophy. It also demonstrates why inputModalities must be declarable per model ([image] only).

Proof of concept

The proposed API has been implemented end to end and validated against two live providers with intentionally different API shapes:

  • SDK capability, saarnilauri/php-ai-client @ feature/text-extraction-poc (fork of this repo): CapabilityEnum::TEXT_EXTRACTION, TextExtractionModelInterface, the result DTOs (TextExtractionResult, ExtractedPage, ExtractedImage, BoundingBox, PageDimensions), ModelRequirements::fromExtractionData(), TextExtractionBuilder, and the AiClient::document() / extractTextResult() / extractText() entry points, with unit tests.
  • Sync connector, saarnilauri/ai-provider-for-mistral @ feature/text-extraction-poc: Mistral's dedicated POST /v1/ocr endpoint. Covers URL and inline (data URI) input, page selection and other options via custom-options passthrough, pixel→normalized bounding box conversion, and embedded image decoding.
  • Async connector, saarnilauri/ai-provider-for-llamaparse (main): LlamaParse's job-based API (upload → poll → fetch result) hidden behind the synchronous interface with internal polling, multipart file upload, and hardcoded model metadata (LlamaParse has no model-list endpoint, parsing modes are exposed as model entries).

Both connectors pass the same integration test suite against the real APIs, capability discovery, multi-page remote PDF extraction, local PDF upload, and local image extraction, with only the provider ID differing between the suites, which demonstrates the provider-agnostic claim in practice.

Running the PoC locally

The connector packages consume the SDK branch via a Composer path repository with a temporary version pin:

"require-dev": { "wordpress/php-ai-client": "dev-feature/text-extraction-poc" },
"repositories": [ { "type": "path", "url": "../php-ai-client", "options": { "symlink": true } } ]

This means the three checkouts must be sibling directories (e.g. dev/php-ai-client, dev/ai-provider-for-mistral, dev/ai-provider-for-llamaparse), with the SDK checkout on the feature/text-extraction-poc branch. The pin is PoC-only and reverts to a released version constraint once the capability ships in the SDK.

Integration tests require MISTRAL_API_KEY / LLAMAPARSE_API_KEY in each connector's .env and are run with composer test:integration. Extracted markdown and embedded images are written to tests/integration/extractions/ for inspection.

Findings from the PoC

  • The embedding-generation pattern (separate builder + dedicated result type) transferred cleanly; no changes to ProviderRegistry, ModelResolver, or option matching were needed beyond the new requirements factory.
  • MIME inference needs an explicit escape hatch: extensionless document URLs (e.g. https://arxiv.org/pdf/1805.04770) cannot be typed automatically, which is why withDocument() accepts an optional $mimeType.
  • Providers differ in how strictly they validate declared vs. actual file content: Mistral silently accepted an AVIF image mislabeled as PNG, while LlamaParse hard-failed the parsing job. Content sniffing (magic bytes) in the File DTO would harden this for all capabilities.
  • Async-behind-sync polling is workable for LlamaParse-scale jobs (seconds to ~1 minute for a 15-page PDF), supporting the decision to defer the operation interface.

Out of scope (v1)

  • PDF rasterization for image-only models (Qwen), consumer or connector concern; the model honestly declares inputModalities: [image].
  • First-class table / key-value DTOs, tables arrive as markdown/HTML inside page text (all providers can deliver that); a structured ExtractedTable normalization across Textract CELL graphs, Document AI pages.tables, and LlamaParse items is a follow-up once two connectors ship.
  • First-class structured annotation results, reuse outputSchema + additionalData['structuredData'] in v1; promote to a typed result field later.
  • Async operation wiring (specified above, deferred).
  • Streaming, no target provider streams extraction results.
  • Provider-side storage inputs (S3/GCS URIs, pre-uploaded file IDs), via customOptions in v1.
  • Cost estimation, pricing units diverge irreconcilably (per page vs per credit vs per token).

Rollout plan

  1. SDK core (this issue): CapabilityEnum::TEXT_EXTRACTION, TextExtractionModelInterface, result DTOs, ModelConfig/OptionEnum keys, ModelRequirements::fromExtractionData(), TextExtractionBuilder, AiClient::document()/extractTextResult()/extractText(), events, AbstractApiBasedTextExtractionModel, unit tests (DTO fromArray(toArray($x)) roundtrips, builder validation, requirements matching, JSON schemas).
  2. Reference connector: ai-provider-for-mistral registers mistral-ocr-latest (simplest API: sync, bearer auth, markdown-native, list-models discovery), proves the contract end to end.
  3. Second connector, different shape: LlamaParse (async-only, no model listing), validates internal polling and hardcoded metadata; then Qwen (chat-transport), validates the VLM path.
  4. Auth-dependent connectors: Google Document AI and AWS Textract, gated on Support provider-owned custom request authentication #237.
  5. Follow-ups: async operations, table normalization, typed structured-annotation results.

All additions use @since n.e.x.t, PHP 7.4-compatible code, and are purely additive (no BC breaks).

Open questions

  1. Capability name: text_extraction (recommended) vs ocr vs document_text_extraction?
  2. Builder entry point name: AiClient::document() (recommended, parallels AiClient::input()) vs AiClient::extract()?
  3. Should ExtractedPage::getMarkdown() be named getText() with format governed by outputMimeType, to avoid baking a format into the API name?
  4. Should pageCount live on TextExtractionResult (proposed) or inside TokenUsage as a new nullable field usable by other per-unit-priced capabilities?
  5. Multi-document batching in one call (Document AI batch, Textract async), out of scope, or should the interface accept list<File> from day one? (Proposed: single File; batching via multiple calls.)

References

Use of AI Tools

This proposal and the proof-of-concept implementations were drafted with the assistance of Claude Code (Anthropic), used for researching the provider APIs, analyzing the SDK architecture, and writing code and prose. All work was done with a human in the loop: the design direction, scope decisions, and API trade-offs were made or reviewed by the author, and the proof of concept was verified by the author against the live Mistral and LlamaParse APIs (including real integration test runs and inspection of the extracted output).

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions