[maxim-smotrov]

🎯 What problem does this PR solve?

SDK doesn't provide support for EmbeddingGemma 2 for multi-modal embeddings. The addon already added support in #5237.

📝 How does it solve it?

• Add ordered inputs batches with text, file-path media, and base64 media while preserving the existing text calls.
• Resolve the projector through the model lifecycle and expose context, media, and projector settings.
• Update the wire contract, Python and Kotlin bindings, examples, documentation, and e2e catalog.
• Keep this capability library-only. CLI OpenAI-compatible embeddings remain text-only.
• Registry entries are tracked separately in QVAC-27164 feat[mod]: add EmbeddingGemma 2 to the registry #5463.

🧪 How was it tested?

• Full local inference, SDK, and e2e build passed.
• Inference lint/type checks and focused Bare tests passed, 13 tests and 67 assertions.
• SDK lint/type checks, unit tests, and contract freshness checks passed.
• Python generation check and Kotlin contract/JVM tests passed.
• E2e formatting, type checks, and catalog checks passed.
• Real-model e2e runs remain pending registry ingestion of the entries in QVAC-27164 feat[mod]: add EmbeddingGemma 2 to the registry #5463.

💥 Breaking Changes

The generated Kotlin and Python EmbedRequest types change because the shared request schema now represents two alternatives: EmbedTextRequest with required text, or EmbedInputsRequest with required inputs. Each alternative forbids the other input field in the schema. This retains the explicit choice between text and ordered multimodal inputs instead of making both fields optional and relying only on runtime validation. The generators are unchanged.

The TypeScript embed() overloads preserve precise result types: text: string returns one vector, while text: string[] and inputs return a batch of vectors. These overloads do not cause the generated-client migration; the request schema union does. Existing TypeScript text calls remain supported.

Kotlin

Direct construction of EmbedRequest now requires a request variant wrapper. The text value type is renamed from EmbedRequestText to EmbedTextRequestText.

BEFORE:

import io.tether.qvac.sdk.generated.schema.EmbedRequest
import io.tether.qvac.sdk.generated.schema.EmbedRequestText

val request = EmbedRequest(
    modelId = "m",
    text = EmbedRequestText.Variant1("hi"),
    requestId = "embed-1",
)

AFTER:

import io.tether.qvac.sdk.generated.schema.EmbedRequest
import io.tether.qvac.sdk.generated.schema.EmbedTextRequest
import io.tether.qvac.sdk.generated.schema.EmbedTextRequestText

val request = EmbedRequest.EmbedTextRequest(
    EmbedTextRequest(
        modelId = "m",
        text = EmbedTextRequestText.Variant1("hi"),
        requestId = "embed-1",
    ),
)

For text batches, use EmbedTextRequestText.Variant2(texts) inside the same wrapper. For multimodal batches, use EmbedRequest.EmbedInputsRequest wrapping an EmbedInputsRequest. When inspecting a request typed as EmbedRequest, select the variant and read its fields through .value.

The existing QvacEmbeddings.embed(modelId, text) and embed(modelId, texts) convenience methods remain supported and need no caller migration.

Python

The existing keyword constructor still works. EmbedRequest is now a Pydantic RootModel containing the selected request variant, so direct .model_id, .text, and .request_id access must move under .root.

BEFORE:

from tetherto.qvac_sdk import EmbedRequest

request = EmbedRequest(model_id="m", text="hi", request_id="embed-1")
model_id = request.model_id
text = request.text
request_id = request.request_id

AFTER:

from tetherto.qvac_sdk import EmbedRequest

request = EmbedRequest(model_id="m", text="hi", request_id="embed-1")
model_id = request.root.model_id
text = request.root.text
request_id = request.root.request_id

For the inputs variant, read request.root.inputs after selecting that variant. Continue passing the outer request to embed(transport, request); do not replace it with request.root. Wire serialization remains flat, with no root or value field added to the RPC payload.

🔌 API Changes

Load the model with modelConfig.projectionModelSrc, then pass text and media inputs:

const { embedding } = await embed({
  modelId,
  inputs: [
    'task: search result | query: an elephant',
    [{ media: { type: 'filePath', value: '/path/to/elephant.jpg' } }]
  ]
})

The result contains one vector per input.

[github-actions]

License compliance — clean

No new dependency license findings in this PR.

Warn-only (shadow) mode — this check does not block merges yet.

Updated automatically by the canonical license compliance workflow.

NOTICE presence (advisory)

Missing NOTICE (advisory, does not block):

• ./docs/website
• ./packages/asr-ggml/benchmarks/server
• ./packages/fabric/test/integration
• ./packages/llm-llamacpp/benchmarks/performance
• ./packages/llm-llamacpp/benchmarks/server
• ./packages/inference-addon-cpp/mobile
• ./packages/sdk-e2e
• ./packages/embed-llamacpp/benchmarks/performance
• ./packages/embed-llamacpp/benchmarks/server
• ./packages/vla-ggml/sim/server
• ./.github/actions/release-merge-guard
• ./.github/bare-toolchain

[github-actions]

Review Status

Current Status: ❌ PENDING
Approvals so far: none

Pending reviews: Needs 1 Management or Team Lead, and 1 more from Management, Team Lead, or Member.

[greptile-apps]

Confidence Score: 5/5

[Medium impact] The PR appears safe to merge; no new actionable issue remains.

Summary

Adds ordered text, image, and audio batches to embed while preserving existing TypeScript text calls.

• Embedding calls can return vectors for ordered text and media inputs.
• Embedding models can load a projector and use its media settings.

Diagram

sequenceDiagram
  participant Caller
  participant Worker
  participant Registry
  participant Addon
  Caller->>Worker: embed(text or inputs)
  Worker->>Registry: Acquire model slot
  Worker->>Worker: Read or decode media
  Worker->>Addon: Run ordered inputs
  Addon-->>Worker: Embedding vectors
  Worker->>Registry: Finish cleanup and release slot
  Worker-->>Caller: Single vector or batch

Loading

_{Reviews (6) · Last reviewed commit: "Merge branch 'main' into feat/sdk-embedd..." · Reviewed by Greptile}