[maxim-smotrov]

🎯 What problem does this PR solve?

SDK doesn't provide support for EmbeddingGemma 2 for multi-modal embeddings. The addon already added support in #5237.

📝 How does it solve it?

• Add ordered inputs batches with text, file-path media, and base64 media while preserving the existing text calls.
• Resolve the projector through the model lifecycle and expose context, media, and projector settings.
• Update the wire contract, Python and Kotlin bindings, examples, documentation, and e2e catalog.
• Keep this capability library-only. CLI OpenAI-compatible embeddings remain text-only.
• Registry entries are tracked separately in QVAC-27164 feat[mod]: add EmbeddingGemma 2 to the registry #5463.

🧪 How was it tested?

• Full local inference, SDK, and e2e build passed.
• Inference lint/type checks and focused Bare tests passed, 13 tests and 67 assertions.
• SDK lint/type checks, unit tests, and contract freshness checks passed.
• Python generation check and Kotlin contract/JVM tests passed.
• E2e formatting, type checks, and catalog checks passed.
• Real-model e2e run on macOS.

💥 Breaking Changes

The generated Kotlin and Python EmbedRequest types now represent two alternatives. EmbedTextRequest requires text and forbids inputs. EmbedInputsRequest requires inputs and forbids text.

A single object with optional text and inputs would keep the old constructors, but its Zod refinement requiring exactly one field would not appear in the exported JSON Schema. Generated clients could then accept both fields or neither. With the union, Kotlin callers must select a variant with a required input field, and Python rejects both or neither during model validation, before RPC. The cost is the generated-client migration below. Existing QvacEmbeddings.embed convenience methods remain unchanged.

The TypeScript embed() overloads preserve precise result types: text: string returns one vector, while text: string[] and inputs return a batch of vectors. These overloads do not cause the generated-client migration; the request schema union does. Existing TypeScript text calls remain supported.

Kotlin

Direct construction of EmbedRequest now requires a request variant wrapper. The text value type is renamed from EmbedRequestText to EmbedTextRequestText.

BEFORE:

import io.tether.qvac.sdk.generated.schema.EmbedRequest
import io.tether.qvac.sdk.generated.schema.EmbedRequestText

val request = EmbedRequest(
    modelId = "m",
    text = EmbedRequestText.Variant1("hi"),
    requestId = "embed-1",
)

AFTER:

import io.tether.qvac.sdk.generated.schema.EmbedRequest
import io.tether.qvac.sdk.generated.schema.EmbedTextRequest
import io.tether.qvac.sdk.generated.schema.EmbedTextRequestText

val request = EmbedRequest.EmbedTextRequest(
    EmbedTextRequest(
        modelId = "m",
        text = EmbedTextRequestText.Variant1("hi"),
        requestId = "embed-1",
    ),
)

For text batches, use EmbedTextRequestText.Variant2(texts) inside the same wrapper. For multimodal batches, use EmbedRequest.EmbedInputsRequest wrapping an EmbedInputsRequest. When inspecting a request typed as EmbedRequest, select the variant and read its fields through .value.

The existing QvacEmbeddings.embed(modelId, text) and embed(modelId, texts) convenience methods remain supported and need no caller migration.

Python

The existing keyword constructor still works. EmbedRequest is now a Pydantic RootModel containing the selected request variant, so direct .model_id, .text, and .request_id access must move under .root.

BEFORE:

from tetherto.qvac_sdk import EmbedRequest

request = EmbedRequest(model_id="m", text="hi", request_id="embed-1")
model_id = request.model_id
text = request.text
request_id = request.request_id

AFTER:

from tetherto.qvac_sdk import EmbedRequest

request = EmbedRequest(model_id="m", text="hi", request_id="embed-1")
model_id = request.root.model_id
text = request.root.text
request_id = request.root.request_id

For the inputs variant, read request.root.inputs after selecting that variant. Continue passing the outer request to embed(transport, request); do not replace it with request.root. Wire serialization remains flat, with no root or value field added to the RPC payload.

🔌 API Changes

Load the model with modelConfig.projectionModelSrc, then pass text and media inputs:

const { embedding } = await embed({
  modelId,
  inputs: [
    'task: search result | query: an elephant',
    [{ media: { type: 'filePath', value: '/path/to/elephant.jpg' } }]
  ]
})

The result contains one vector per input.

[github-actions]

License compliance — clean

No new dependency license findings in this PR.

Warn-only (shadow) mode — this check does not block merges yet.

Updated automatically by the canonical license compliance workflow.

NOTICE presence (advisory)

Missing NOTICE (advisory, does not block):

• ./docs/website
• ./packages/asr-ggml/benchmarks/server
• ./packages/fabric/test/integration
• ./packages/llm-llamacpp/benchmarks/performance
• ./packages/llm-llamacpp/benchmarks/server
• ./packages/inference-addon-cpp/mobile
• ./packages/sdk-e2e
• ./packages/embed-llamacpp/benchmarks/performance
• ./packages/embed-llamacpp/benchmarks/server
• ./packages/vla-ggml/sim/server
• ./.github/actions/release-merge-guard
• ./.github/bare-toolchain

[github-actions]

Review Status

Current Status: ❌ PENDING
Approvals so far: none

Pending reviews: Needs 1 Management or Team Lead, and 1 more from Management, Team Lead, or Member.

[greptile-apps]

Confidence Score: 5/5

[Medium impact] The PR appears safe to merge; no new actionable issues remain from this review.

Summary

Adds ordered text, image, and audio embedding batches while keeping existing TypeScript text calls supported.

• Embedding calls turn ordered text and media inputs into vectors.
• SDK callers can choose text or multimodal embedding requests.
• The end-to-end catalog now covers EmbeddingGemma 2 inputs.

Diagram

%%{init: {'theme': 'neutral'}}%%
flowchart LR
  A["loadModel with projectionModelSrc"] --> B["Resolve model and projector"]
  B --> C["Create embedding model"]
  C --> D["embed with text or inputs"]
  D --> E["Admit one request per model"]
  E --> F["Read media and run embedding"]
  F --> G["Return vectors in input order"]

Loading

_{Reviews (15) · Last reviewed commit: "test: isolate embed client mock behavior" · Reviewed by Greptile}

[github-actions]

QVAC E2E — base run recorded

Base: run 41661064462 · suite smoke · sha c367ca9

Platform	Passed	Total	Failed
android/android	187	213	0
desktop/linux	213	213	0
desktop/macos	210	213	0
desktop/windows	213	213	0
ios/ios	183	213	0

All recorded platforms are green — nothing for test-e2e-rerun-failed to re-run.

[github-actions]

QVAC E2E — tests changed by this PR

This smoke run additionally covers 89 test(s) that are not tagged smoke, adding roughly 59.1 min.

Catalog: 624 tests · smoke: 124 · touched by this PR: 129 (40 already in smoke)

Changed file	Matched via	Tests
packages/inference/src/handlers/assess-model-fit	sdk handler	26
packages/sdk/src/client/api/embed.ts	sdk api	85
catalog/embeddinggemma2-tests.ts	definitions (whole file)	4
catalog/platform-skips.ts	import graph	18
catalog/test-definitions.ts	definitions (whole file)	18
runners/js/shared/executors/embeddinggemma2-executor.ts	executor pattern	4

89 added test id(s)

model-load-llm-load-mode-none
model-load-llm-legacy-no-mmap-rejected
model-load-llm-prompt-cache-keys
model-load-llm-cache-checkpoints-out-of-range-rejected
model-load-embedding-rank-pooling-rejected
model-reload-llm
embed-long-text
embed-empty-text
embed-similarity
embed-unicode
embed-very-short
embed-code-snippet
embed-multilingual
embed-special-chars
embed-numbers-only
embed-semantic-similarity
embeddinggemma2-image-retrieval
embeddinggemma2-audio-deterministic
embeddinggemma2-text-parity
embeddinggemma2-reject-corrupt-media
vector-index-remove-contains
vector-index-write-load
sharded-model-hash-validation
sharded-model-backward-compatibility
sharded-model-progress
sharded-model-resume
sharded-model-cancellation
sharded-model-batch-inference
sharded-model-long-text-inference
sharded-model-llm-completion
sharded-model-llm-reload
sharded-model-llm-missing-shards
http-sharded-embed-progress
http-archive-embed-load
http-archive-embed-progress
http-archive-embed-inference
model-fit-probe-llm
model-fit-probe-embeddings
model-fit-probe-parakeet
model-fit-probe-whisper
model-fit-probe-bci
model-fit-probe-tts-chatterbox
model-fit-probe-audiogen
model-fit-probe-diffusion
model-fit-probe-diffusion-esrgan
model-fit-probe-upscaler
model-fit-probe-ocr
model-fit-probe-translation
error-invalid-response-type
error-model-load-failed
… and 39 more

These changed files map to no test, so nothing was added for them — review by hand if they affect behaviour:

• packages/inference/src/plugins/builtin/llamacpp-embedding
• catalog/resource-table.ts
• catalog/resources/resource-table.json
• catalog/resources/skip-matrix.json
• runners/js/desktop/consumer.ts
• runners/js/electron/consumer.ts
• runners/js/mobile/consumer.ts

[github-actions]

QVAC E2E — windows — ✅ all tests passed (213/213, 427s)

Config: suite=smoke · filter=(none) · exclude=(none)
Inference: branch:c367ca9660995045b895d40bc70813a584cf29e2:@qvac/inference@0.23.0 · Test-suite: branch:c367ca9660995045b895d40bc70813a584cf29e2:@qvac/test-suite@0.13.0
View run · Artifacts: reports

[github-actions]

QVAC E2E — linux — ✅ all tests passed (213/213, 399s)

Config: suite=smoke · filter=(none) · exclude=(none)
Inference: branch:c367ca9660995045b895d40bc70813a584cf29e2:@qvac/inference@0.23.0 · Test-suite: branch:c367ca9660995045b895d40bc70813a584cf29e2:@qvac/test-suite@0.13.0
View run · Artifacts: reports

[github-actions]

QVAC E2E — macos — ✅ all tests passed (210/213, 437s)

Config: suite=smoke · filter=(none) · exclude=(none)
Inference: branch:c367ca9660995045b895d40bc70813a584cf29e2:@qvac/inference@0.23.0 · Test-suite: branch:c367ca9660995045b895d40bc70813a584cf29e2:@qvac/test-suite@0.13.0
View run · Artifacts: reports

[github-actions]

QVAC E2E — ios — ✅ all tests passed (183/213, 1192s)

Config: suite=smoke · filter=(none) · exclude=(none)
Inference: branch:c367ca9660995045b895d40bc70813a584cf29e2:@qvac/inference@0.23.0 · Test-suite: branch:c367ca9660995045b895d40bc70813a584cf29e2:@qvac/test-suite@0.13.0
Device pool: iPhone 17 Pro - iOS 26 — ☀️ highly available · Billing: UNMETERED
View run · Artifacts: reports · Device Farm logs

[github-actions]

QVAC E2E — android — ✅ all tests passed (187/213, 3942s)

Config: suite=smoke · filter=(none) · exclude=(none)
Inference: branch:c367ca9660995045b895d40bc70813a584cf29e2:@qvac/inference@0.23.0 · Test-suite: branch:c367ca9660995045b895d40bc70813a584cf29e2:@qvac/test-suite@0.13.0
Device pool: Google Pixel 9 Pro - Android 15 — ☀️ highly available · Billing: UNMETERED
View run · Artifacts: reports · Device Farm logs

[lauripiisang]

The way we do e2e is a bit different now so the implementation should be changed to match there, plus the schemas could use titles so they get better naming in generated code.

[lauripiisang]

These definitions carry no test data: params is {}, nothing reads the type: 'string' expectation, and there are no steps. The inputs, asset names, the 768 dimension, the cosine thresholds (0.999, 0.99999) and the error pattern all live in embeddinggemma2-executor.ts. So the tests only run where that executor is registered: the Python leg is skipped in platform-skips.ts, the Kotlin runner cannot run them, and no e2e test sends the regenerated Python EmbedRequest to a real model.

New e2e tests are written as declarative definitions now:

• packages/sdk-e2e/README.md, "Developing new tests": prefer a declarative body; an executor is only for bodies that cannot be data.
• .agents/skills/qv-sdk-e2e-create/SKILL.md (/qv-sdk-e2e-create): "Body and placement" says the same. "Model-output strategy" names cosine similarity against a reference vector as the case for a named: assertion implemented in both clients, and lists type as the last-resort fallback. references/scaffolding.md has the definition template.
• catalog/rerank-tests.ts is the closest existing example: its ranking steps are built from params and checked with compare … named: 'greaterThan', and RerankExecutor reads the same params on mobile.

The existing steps can express every check here:

• asset with form: 'path' for elephant.jpg, cat.jpg, sample-hi.wav (kinds image / audio; the JS and Python runners both resolve them)
• call: { method: 'embed', params: { modelId: '$model', inputs: [...] } } with { media: { type: 'filePath', value: '$elephant' } } parts
• project on embedding[0] … embedding[3]
• callError + errorMatches for the corrupt WAV (the addon rejects it with code UnsupportedMedia), then a second call for the recovery check

Missing named checks, added to runners/js/shared/step-bindings.ts and runners/python/qvac_e2e/assertions.py:

• assertion unitVectors (with: { dimensions: 768, tolerance: 1e-3 })
• comparison cosineAtLeast (with: { min }) for the audio determinism and text parity tests
• comparison moreSimilarThan (with: { than: '$v3' }) for the retrieval ranking

const retrievalSteps = (queries: [string, string], pairs: Array<[number, number, number]>): Step[] => [
  { useModel: { deps: ['embeddinggemma2'], as: 'model' } },
  { asset: { kind: 'image', file: 'elephant.jpg', form: 'path', as: 'elephant' } },
  { asset: { kind: 'image', file: 'cat.jpg', form: 'path', as: 'cat' } },
  {
    call: {
      method: 'embed',
      params: {
        modelId: '$model',
        inputs: [
          ['title: none | text: ', { media: { type: 'filePath', value: '$elephant' } }],
          ['title: none | text: ', { media: { type: 'filePath', value: '$cat' } }],
          ...queries
        ]
      },
      as: 'response'
    }
  },
  { project: { from: '$response', path: 'embedding', as: 'vectors' } },
  { assert: { on: '$vectors', named: 'lengthIs', with: { length: 4 } } },
  { assert: { on: '$vectors', named: 'unitVectors', with: { dimensions: 768 } } },
  ...[0, 1, 2, 3].map((i): Step => ({
    project: { from: '$response', path: `embedding[${i}]`, as: `v${i}` }
  })),
  ...pairs.map(([anchor, match, other]): Step => ({
    compare: { left: `$v${anchor}`, right: `$v${match}`, named: 'moreSimilarThan', with: { than: `$v${other}` } }
  }))
]

With declarative bodies, the desktop-python rule in platform-skips.ts and its four skip-matrix.json entries can go. Mobile has no step interpreter, so either keep the executor for mobile only and have it read params from the definition (as RerankExecutor.ranking does), or skip mobile with a reason.