MessagePart cannot represent source attribution, so every provider drops citations

[juanlentino]

MessagePart (src/Messages/DTO/MessagePart.php) can carry exactly seven things:
channel, type, thoughtSignature, text, file, functionCall,
functionResponse. None of them can hold a citation.

All three first-party providers return source attribution, in three different
shapes (per vendor documentation):

Provider	Field
Anthropic	citations on text blocks — cited_text, document_index, document_title, typed location
OpenAI	annotations — type: "url_citation", url, title, start_index, end_index, plus sources
Gemini	citationMetadata.citationSources and groundingMetadata; annotations/url_citation on the Search-grounding surface

And all three provider packages currently drop it. Verified by reading the source:
AnthropicTextGenerationModel::parseResponseContentMessagePart()'s case 'text'
keeps only $partData['text']; GoogleTextGenerationModel and
OpenAiTextGenerationModel contain no citation/grounding/annotation handling at all.

This is a gap rather than a data-loss bug — no provider currently requests
citations, so nothing is being lost in production today. But the parse paths are
not ready, and no provider can be made ready while the DTO has nowhere to put the
result.

Suggested shape: thoughtSignature is the precedent — an opaque,
provider-supplied, round-trippable field already accepted on MessagePart. A
citation carrier could follow it, normalising the three vendor shapes into one
representation (source identifier/URI and optional title).

Related: #268 (provider-agnostic text extraction) — same theme of getting
structured meaning out of documents.

Happy to open a PR once maintainers settle the shape; the API-surface decision is
yours.

[the-hercules]

@juanlentino I agree that we need a normalized way to handle citations across all providers.

To address the API-surface blocker you mentioned, I am currently implementing a normalized Citation object. Here is the approach I am taking:

1. New Citation DTO: I've created a Citation class to serve as a uniform representation across all vendors. It contains two properties:

   • uri (string): The source identifier (e.g., URL or document index).
   • title (string|null): An optional title for the source.

2. Updating MessagePart: I have extended MessagePart to accept an optional array of these Citation objects via a new $citations property, similar to how thoughtSignature was implemented. This allows any message part (especially text blocks) to carry its source attribution cleanly.

3. Provider Parsers (Next Steps): With the DTO foundation in place, the next step is to update the parsing logic in AnthropicTextGenerationModel, GoogleTextGenerationModel, and OpenAiTextGenerationModel to extract the respective citation data (citations, citationMetadata, annotations) and map them into the new Citation DTO.

I'm working on preparing a PR with the core DTO updates now. Let me know if you have any thoughts on this shape!

[juanlentino]

Thanks for picking this up. The DTO direction is right, and the two properties you chose are where I'd push back before the parsers get written: as proposed, uri + title keeps the identity of the source while dropping the two fields that make a citation checkable.

All three vendors anchor a citation to a span of the generated text. Anthropic returns a typed location (char, page, or content-block ranges), OpenAI's url_citation carries start_index and end_index, and Google's citationMetadata carries startIndex and endIndex. Anthropic additionally returns cited_text, the quoted source passage itself. If the normalized object holds neither the span nor the quote, the flattening this issue describes moves up a layer instead of going away: a caller can render a source list but can no longer say which sentence rests on which source.

One smaller thing on uri holding "URL or document index": those are different reference kinds. A URL points outside the conversation, while an index refers into the request's own documents array. Collapsing them into one string means every consumer has to sniff the format to know what it is holding. Explicit fields keep it honest, something like:

class Citation {
    public ?string $url;          // remote source
    public ?int $documentIndex;   // request-supplied document
    public ?string $title;
    public ?int $startIndex;      // span in the generated text
    public ?int $endIndex;
    public ?string $quotedText;   // Anthropic cited_text; null elsewhere
}

(Google's license has no counterpart in the other two, so it could ride an optional field or wait.)

There is one fork worth settling before touching the parsers. Anthropic does not annotate a single text block: it splits the text into several blocks, each carrying its own citations. OpenAI and Google instead annotate index ranges over one string. So the normalization has to choose between citations as ranges over a part's own text (parts stay stable, but concatenation needs index bookkeeping) and part-splitting at citation boundaries as the normal form (native for Anthropic, part-multiplying for the other two). Which way are you leaning?

[zahidgh]

The range question is important enough to make explicit in the core contract before provider parsers land. My recommendation would be:

• Keep citations attached to the MessagePart that owns the generated text.
• Define startIndex/endIndex as offsets into that part's text (rather than a concatenated conversation string).
• Keep source identity typed instead of overloading one uri: nullable url and documentIndex, plus optional title and quotedText.
• Treat vendor-only fields (for example Google's license metadata) as optional or provider-scoped data rather than dropping the common span/source fields.

Anthropic can map each cited text block to its own part without index translation. OpenAI and Gemini can either attach ranges to one text part or split parts at annotation boundaries; either way, part-local offsets remain deterministic for consumers. The parser should validate that offsets are non-negative and ordered, since an invalid range should not silently produce a misleading citation.

If maintainers agree on part-local ranges and typed source identity, the DTO can be implemented independently of the provider wire formats while keeping citations checkable for callers.

[the-hercules]

@juanlentino @zahidgh Thanks for the discussion and guidance on the architecture here!

I've updated the PR with the agreed-upon direction:

1. Expanded Citation DTO: The DTO has been updated to include typed source identity and part-local ranges, independent of provider wire formats:
   • ?string $url (remote source)
   • ?int $documentIndex (request-supplied document)
   • ?string $title
   • ?int $startIndex (part-local byte offset into the owning MessagePart)
   • ?int $endIndex (part-local byte offset into the owning MessagePart)
   • ?string $quotedText (for excerpts like Anthropic's cited_text)
2. Validation: Added safeguards to ensure that when startIndex and endIndex are provided, they are non-negative and properly ordered (startIndex <= endIndex).
3. Tests updated: Thoroughly expanded the unit tests for Citation and MessagePart to verify JSON serialization, hydration, and cloning behavior with the new shape.

Since we are aligned on anchoring citations as ranges on a part rather than splitting parts at boundaries, the DTO foundation is now fully in place. The next step will be to handle the provider-specific parsing logic (Anthropic, OpenAI, Google) to map their respective wire shapes into this normalized DTO.