ModelMetadata has no context-window concept, and TokenLimitReachedException is never thrown

[georgestephanis]

Summary

ModelMetadata has no concept of a model's context window (maximum input/total token count), and TokenLimitReachedException is declared in the SDK but never thrown anywhere in it. There's currently no way for a provider, a PromptBuilder consumer, or a plugin built on top of the SDK to know how much room a model has for a given prompt, or to be told proactively that a prompt won't fit — the only way to find out is to send the request and let the underlying API reject it (or silently truncate/misbehave, depending on the server).

Current behavior

• ModelMetadata (src/Providers/Models/DTO/ModelMetadata.php) exposes id, name, supportedCapabilities, and supportedOptions — nothing describing the model's context window or max input tokens.
• ModelConfig::maxTokens (→ OptionEnum::maxTokens()) only ever maps to the output-side max_tokens request parameter. It's a per-request cap on generation length, unrelated to how much input the model can accept.
• TokenLimitReachedException (src/Common/Exception/TokenLimitReachedException.php) exists, with a docblock describing it as being thrown "when token usage exceeds the model's context window or a configured maximum," and carries a getMaxTokens() accessor — but grepping the SDK, it is never actually thrown by any provider implementation or by PromptBuilder. It's a defined contract with no implementation.
• PromptBuilder builds the message list to send without any token estimation or trimming.

Why this matters

Callers that maintain their own conversation history (chat UIs, agents, plugins) have no way to:

1. Discover a model's context window size from the SDK, so they could truncate/summarize history themselves.
2. Get a clear, typed exception before firing a request that's too large, rather than an opaque provider-specific HTTP error.

This came up while working on an OpenAI-compatible-server connector plugin: we let site owners configure a "context length" for their self-hosted model, but the SDK gives us nowhere to plug that in on the input side — it can only be forwarded to the server as a best-effort hint (e.g. Ollama's num_ctx), not used by the SDK itself to warn or trim before sending.

Relationship to #193

#193 ("Missing Required Parameters in Tool Calls When Token Limit Reached") is adjacent but describes a different failure mode: output-side truncation, where generation stops mid-tool-call because finish_reason hit the length cap, and asks for TokenLimitReachedException to be thrown reactively when that happens.

This issue is about the input side: knowing a model's context window ahead of time as metadata, and optionally being able to catch an over-budget prompt before it's ever sent. Both problems converge on the same TokenLimitReachedException class, so a design for "when should this exception actually fire" should probably consider both at once — but they're distinct enough (reactive output-truncation vs. proactive input-budgeting) that I didn't want to conflate them in one issue.

Proposed solution (open to alternatives)

1. Add a context-window field to ModelMetadata (e.g. contextWindow or maxInputTokens), populated by providers that know it (static per-model tables for hosted providers; left null for providers, like OpenAI-compatible self-hosted servers, where it's genuinely unknowable/configurable).
2. Give callers a way to check a prompt against it — either a helper on ModelMetadata/PromptBuilder that estimates whether a set of messages fits, or having TokenLimitReachedException thrown proactively when the caller supplies/knows the limit and the assembled prompt exceeds it.
3. Document the distinction between ModelConfig::maxTokens (output cap) and this new context-window concept clearly, since the naming is easy to conflate (we ran into exactly this confusion while building against the SDK).

Happy to help scope or contribute an implementation once there's agreement on direction — this surfaced from real usage, not a hypothetical.

AI disclosure

This issue was raised collaboratively between a human maintainer and Claude (Anthropic's AI coding assistant), working together in a Claude Code session against a WordPress plugin that consumes this SDK. Claude traced the plugin's context_length setting to confirm there was no corresponding SDK-side concept, read the relevant SDK source (ModelMetadata, ModelConfig, TokenLimitReachedException, PromptBuilder) to verify the gap, and searched this repository's existing issues (finding #193 as the closest related-but-distinct report) before drafting this write-up. The human reviewed and directed the investigation and approved filing this issue.

[the-hercules]

Hi @georgestephanis Thanks for the detailed write-up , I traced through the SDK and this all checks out. Breaking the proposal into its three parts, they land in very different places difficulty-wise:

1. Add a context-window field to ModelMetadata , straightforward.
This is a clean, additive change: a nullable int threaded through the constructor, toArray/fromArray, getJsonSchema, and a getter. Backward-compatible, no behavior change for anyone not using it. One naming decision to settle up front, since the two options mean different numbers:

• contextWindow = total input + output budget
• maxInputTokens = input side only

Worth picking deliberately, because any "does it fit?" logic later depends on which one this is.

Also worth flagging: the concrete providers (OpenAI, Google, etc.) now live in separate repos, so this repo can add the field but can't populate the actual per-model numbers , those tables belong in the provider repos. So this PR would be plumbing + null defaults, with real values following in the provider repos.

3. Document maxTokens (output cap) vs. the new context-window concept , straightforward.
Agreed this is easy and worth doing alongside (1). We hit the same naming confusion. ModelConfig::maxTokens maps directly to the output-side max_tokens request param, so calling the new thing anything with "maxTokens" in it would make the collision worse , another point for contextWindow.

2. Proactively check a prompt against the window , this is the part that needs a design decision.

The blocker here is token counting, and I don't think the issue pins down a mechanism. The proposal mentions "a helper that estimates whether a set of messages fits" and "thrown when the caller supplies/knows the limit," but even the second option still needs something to measure the assembled prompt against that limit , and that's the hard bit:

• A real token count needs the model's own tokenizer (each model tokenizes differently), and the SDK ships none. Bundling tokenizers per provider is a large scope.
• The obvious shortcut , chars ÷ 4 or similar , gives false confidence: it'll pass prompts that actually overflow and reject ones that would've fit. Arguably worse than no check.

So before writing any code for (2), how do you picture the counting working? A few directions I can see:

1. SDK estimates internally , pull in a tokenizer (big dependency/scope), or ship a documented rough heuristic with explicit "estimate only" caveats.
2. Caller supplies the count , the SDK only compares/throws when the caller passes in a token count it already has (e.g. from its own tokenizer), and never guesses. Smallest, safest.
3. Metadata only for now , ship (1) + (3), expose the number, and let callers do their own budgeting; defer any SDK-side checking until there's a counting story.

I'd also want (2)'s exception semantics to line up with #193, since both converge on TokenLimitReachedException , same message/getMaxTokens() conventions for the input-side (proactive) and output-side (reactive) cases.

My suggestion: land (1) + (3) now as a self-contained, low-risk PR (real value on its own , callers get the number), and settle the counting question for (2) separately. Happy to open that PR if the field name and this split sound right. What's your thinking on the counting approach?

[georgestephanis]

@the-hercules Realistically I'd lean toward a relatively basic token estimator in core, and let individual providers or plugins override it with their own if desired -- which feels reasonable.

So that feels similar to 1+3, with a hook for folks to BYOB for 2?

[the-hercules]

Hi @georgestephanis

Opened a PR for the input-side half of this: #264

It follows the direction we landed on in this thread, a basic estimator in core, overridable (BYOB), scoped to the proactive/input side so it doesn't conflate with the reactive/output side in #193.

What was implemented

1. ModelMetadata::getContextWindow(): ?int — a new optional field for a model's context window (total input + output tokens). Nullable and fully backward compatible; providers populate it where known, null where it isn't (e.g. self-hosted, user-configurable servers).
2. TokenCounterInterface + a default HeuristicTokenCounter — a rough, dependency-free chars / 4 estimate. Overridable per request via PromptBuilder::usingTokenCounter() so anyone with an accurate, model-specific tokenizer (a server tokenize endpoint, a bundled BPE library, a provider count-tokens API) can bring their own.
3. Proactive check — PromptBuilder now throws the (previously never-thrown) TokenLimitReachedException before a request is sent, when the resolved model exposes a context window and the estimate exceeds it. It's a no-op when the context window is unknown, so nothing changes for existing callers until providers populate the field.
4. Docs — clarified contextWindow (input budget) vs ModelConfig::maxTokens (output cap) in docblocks, ARCHITECTURE.md, and GLOSSARY.md, since that naming overlap is exactly what tripped us up.

Assumptions I'd like you to sanity-check

• Naming/semantics: contextWindow = total tokens (input + output), matching published model specs — not input-only.
• Check math: estimate = message text + system instruction, plus the configured output maxTokens when set, compared against the total window.
• Always-on but advisory: the check runs whenever a context window is known. Because a heuristic can be wrong, it can occasionally false-positive; the escape hatch is usingTokenCounter(). The docblock states a passing check is not a guarantee the provider will accept the request.
• Default estimator is deliberately crude: strlen($text) / 4 (byte length, no mbstring dependency), rounded up. Non-text parts (files, tool calls/responses) are not counted. It's meant to catch grossly oversized prompts, not to be accurate.
• Not capability-gated: the check runs for any model exposing a contextWindow (only text is counted). Happy to restrict it to text generation if you'd prefer.
• Override is per-call only (usingTokenCounter()); I did not add registry-level/provider propagation (the heavier HttpTransporter-style parity) — left as a possible follow-up.
• No real per-model numbers are shipped here — those belong in the provider packages; core stays null.

Relationship to #193

This PR is only the proactive/input side. #193 is the reactive/output side (truncation detected via finish_reason == length) and is not implemented here. This PR throws TokenLimitReachedException; I've written the exception and its docblock to cover both cases, so #193 can reuse the same class when it's picked up — the length finish reason already flows through to Candidate::getFinishReason()->isLength(), so no new plumbing is needed there. Happy to pick up #193 as a follow-up once the approach here looks right to you.