Skip to content

mtmd: preserve tiled chunk geometry and reference resize rounding - #457

Merged
gianni-cor merged 2 commits into
temp-11739from
codex/mtmd-grid-rounding-release
Nov 3, 2026
Merged

gianni-cor merged 2 commits into
temp-11739from
codex/mtmd-grid-rounding-release

Conversation

@DmitryMalishev

@DmitryMalishev DmitryMalishev commented Oct 30, 2026 •

Copy link
Copy Markdown

Task: QVAC-27614

🎯 What problem does this PR solve?

  • Fixes Issue 2 — Cached tiled-image geometry is lost. Fabric's server serializes image chunks into placeholders without preserving grid_x/grid_y. Some valid Qwen-style multi-grid images then have incorrect token counts and decoder positions, causing HTTP 500 / invalid input batch failures. Runtime/addon PR #5489 integrates this engine fix through the shared Fabric package.
  • Fixes Issue 12 — Image resize rounding differs from the reference processor. Half-way dimensions round upward instead of using ties-to-even, potentially adding an image-token row or column. Runtime/addon PR #5489 delivers the corrected runtime and adds an addon-level boundary regression.
  • Issue numbers refer to the reported issue list. Issue 3 — SDK completion stop reasons is addressed independently in SDK PR #5516.

📝 How does it solve it?

  • The b11739 addon compatibility changes and temporary overlay are maintained in QVAC #5331. Runtime/addon Update README.md for llava ggml-org/llama.cpp#5489 needs those adaptations for end-to-end validation against this rebased engine; changing only its source pin is insufficient.
  • Serialize and restore both tile-grid dimensions. Bump the chunk format from version 3 to 4 and its minimum accepted version to 4. The new base already uses version 3 for per-image suffix/any-resolution metadata; versions 2 and 3 lack the token-grid fields and must be rejected.
  • Test 1/2/3/4-tile round trips, placeholder token counts and decoder positions, version-2/version-3 rejection and truncated input. Keep the C API assertions active in release builds.
  • Use integer ties-to-even rounding in the shared image-sizing helper. Cover half-way and adjacent dimensions, pixel budgets, existing sizing callers and all four floating-point rounding modes.
  • Target temp-11739 (upstream b11739 integration), rebased directly onto base 14ecfda0c (merged ci : drop duplicate permissions block in pre-tokenizer-hashes #480). After this merges, publish the updated vcpkg port and replace the temporary overlay in QVAC #5489 with its official dependency before merging that integration PR. Public model/addon APIs are unchanged.

🧪 How was it tested?

  • Current head 97159252e is rebased directly onto temp-11739 at 14ecfda0c, including #480. Only five mtmd implementation/test files differ from the base. git diff --check passed.
  • This final rebase changes only the inherited pre-tokenizer workflow relative to tested c3edc86c2; the mtmd implementation and regression files are identical. That verified source passed all three focused mtmd targets under ASan+UBSan with leak detection and halt-on-error; cache tests reject versions 2 and 3.
  • Fresh current-head CI is running: CPU, server, CUDA, Vulkan, Metal, Apple and sanitizers. Pending or queued jobs are not claimed green.
  • The inherited duplicate-permissions workflow error is corrected by merged ci : drop duplicate permissions block in pre-tokenizer-hashes #480. The verified archive contains one top-level permissions block; the current push has no corresponding invalid-workflow failure.
  • QVAC #5331 now combines the b11739 native API adaptations with this exact engine pin and verified SHA-512 (overlay commit). All 51 local overlay/runtime policy tests passed; Fabric prebuild and C++/desktop end-to-end validation is running.
  • Historical server counterfactual/replay on old-base 4fe4ded85 passed 100 sequential GPU requests, 14 batched controls, four CPU grid cases, two resize-boundary requests, slot restore and stale-cache rejection. Those results and the old addon overlay do not certify the new base. SDK stop-reason PR server : fix system prompt cli ggml-org/llama.cpp#5516 remains independent.

💥 Breaking Changes

  • Serialized mtmd chunk format changes from version 3 to 4; versions 2 and 3 are rejected. Existing persisted media-chunk caches must be recreated; old chunks are rejected rather than restored with incorrect geometry.

AI assistance: Codex assisted implementation and validation; human maintainer review is required.

@DmitryMalishev
DmitryMalishev changed the base branch from temp-11259 to temp-11739 November 3, 2026 12:56
@DmitryMalishev
DmitryMalishev force-pushed the codex/mtmd-grid-rounding-release branch from 4fe4ded to c3edc86 Compare November 3, 2026 12:57
@DmitryMalishev
DmitryMalishev force-pushed the codex/mtmd-grid-rounding-release branch from c3edc86 to 9715925 Compare November 3, 2026 13:48
@gianni-cor
gianni-cor merged commit c1ef8d4 into temp-11739 Nov 3, 2026
35 of 43 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants