Add a portable storage and CRUD layer for embeddings

[ColinM-sys]

What?

See #962 — this is the "storage and CRUD layer for embeddings" checklist item, as a standalone PR off develop.

Adds a WordPress\AI\Embeddings namespace with a portable wpai_embeddings table and a repository for storing, reading, iterating and deleting embedding vectors. Persistence only: no generation (that's the AI Client) and no similarity search (the next checklist item, built on top of this).

Why?

Embedding vectors are only comparable to other vectors produced by the same model (the reason behind php-ai-client#274). The existing experiment PRs each brought their own storage — postmeta in #891/#943, a RAG-specific MariaDB/memory pair in #683 — with the model recorded loosely or assumed from config. #962 asks for a shared foundation that individual features (semantic search #844, frontend chat #142) can build on instead of re-implementing.

This layer makes the model part of every vector's identity: every row carries provider, model and dimensions, every read is scoped to a provider + model, and the unique key is (object_type, object_id, provider, model, chunk_index). An index can never be queried with vectors from a different model by accident, and switching models is an explicit re-index (delete_for_model()), not a silent corruption.

How?

• Embedding_Schema — creates wpai_embeddings via dbDelta, following AI_Request_Log_Schema (version option, maybe_upgrade_table(), idempotent). Portable column types only; vectors are MEDIUMBLOB of packed float32 with a cached embedding_norm (so cosine similarity later needs no second pass). Created on the first write, never by a read, so sites that never store an embedding pay nothing.
• Embedding_Record — immutable value object: object type/ID, chunk index, provider, model, vector, optional content hash. Validates identity fields and the vector.
• Embedding_Repository_Interface — the contract: save / save_many (upsert, replaces in place), get (chunks in order), get_by_id, get_content_hash (cheap staleness check for the future sync layer), get_object_ids (bounded, newest-first — the lookup from Semantic (vector) search for the AI plugin — fixes, tests & validation atop #891 #943 generalised), count_objects, iterate (keyset-paginated batches for a PHP-side scan), delete_for_object, delete_for_model.
• Embedding_Repository — $wpdb implementation of the above. Corrupt rows are skipped rather than fatal.
• Vector_Codec — pack/unpack little-endian float32, the same byte layout as MariaDB's VECTOR type, so a native-index backend (Add native vector search #683's MariaDB_Index_*) can implement the same interface against the same bytes. @artpi — shaped with your backend/repository split in mind; happy to adjust the contract so the MariaDB backend slots in cleanly.
• Uninstall drops the table (the wpai_* version option is already covered); docs/DEVELOPER_GUIDE.md gets a "Storing Embeddings" section.
• Local models: tested with Ollama nomic-embed-text vectors (768-dim) and 3072-dim vectors (gemini-embedding-001 size).

Not in this PR, by design: the generate_embeddings() wrapper update for php-ai-client#274 (separate, as noted on #962), chunking/sync, similarity utilities.

Use of AI Tools

AI assistance: Yes
Tool(s): Claude Code
Model(s): Claude Fable 5
Used for: Drafting the classes, tests and docs from a design I specified (table shape, model-scoped identity, interface split vs. #683); every file was reviewed and run by me, and the test/lint runs below were executed locally.

Testing Instructions

1. npm run wp-env:test start then npm run test:php -- --filter "Embedding_|Vector_Codec" — 51 tests covering: table creation/idempotence/drop, first-write-creates / reads-don't, save + read round trip, in-place replace on re-index, model and object-type isolation, chunk ordering, content-hash lookup, bounded newest-first IDs with offset, batched iteration, scoped deletes, corrupt-row skipping, 3072-dim round trip, codec byte layout and validation.
2. composer lint and composer phpstan — clean for the new files (phpstan's only remaining error on a fresh checkout is the pre-existing Requirements.php build/build.php require, which needs npm run build).
3. Manual: from wp shell or a snippet, ( new \WordPress\AI\Embeddings\Embedding_Repository() )->save( new \WordPress\AI\Embeddings\Embedding_Record( 'post', 1, 'ollama', 'nomic-embed-text:latest', array( 0.1, 0.2, 0.3 ) ) ); then check wp db query "SELECT object_type, object_id, provider, model, dimensions, LENGTH(embedding) FROM wp_wpai_embeddings" → one row, 12 bytes.

Changelog Entry

Added - Portable storage and CRUD layer for embedding vectors (wpai_embeddings table, Embedding_Repository), recording the provider and model alongside every vector so indexes are always model-scoped.

[github-actions]

The following accounts have interacted with this PR and/or linked issues. I will continue to update these lists as activity occurs. You can also manually ask me to refresh this list by adding the props-bot label.

If you're merging code through a pull request on GitHub, copy and paste the following into the bottom of the merge commit message.

Co-authored-by: ColinM-sys <colinmcdonough@git.wordpress.org>
Co-authored-by: dkotter <dkotter@git.wordpress.org>
Co-authored-by: jeffpaul <jeffpaul@git.wordpress.org>
Co-authored-by: dugyen <ugyensupport@git.wordpress.org>

To understand the WordPress project's expectations around crediting contributors, please review the Contributor Attribution page in the Core Handbook.

[codecov]

Codecov Report

❌ Patch coverage is 80.32787% with 96 lines in your changes missing coverage. Please review.
✅ Project coverage is 75.37%. Comparing base (01d3970) to head (eb9e0e0).

Files with missing lines	Patch %	Lines
includes/CLI/Embeddings_Command.php	0.00%	64 Missing ⚠️
includes/Embeddings/Embedding_Repository.php	91.66%	20 Missing ⚠️
includes/Embeddings/Vector_Codec.php	94.25%	5 Missing ⚠️
includes/Embeddings/Embedding_Record.php	92.85%	4 Missing ⚠️
includes/Embeddings/Embedding_Schema.php	94.59%	2 Missing ⚠️
...udes/Embeddings/Embedding_Repository_Interface.php	0.00%	1 Missing ⚠️

Additional details and impacted files

@@              Coverage Diff              @@
##             develop     #976      +/-   ##
=============================================
+ Coverage      75.18%   75.37%   +0.19%     
- Complexity      3227     3357     +130     
=============================================
  Files            133      138       +5     
  Lines          12488    12976     +488     
=============================================
+ Hits            9389     9781     +392     
- Misses          3099     3195      +96     

Flag	Coverage Δ
unit	75.37% <80.32%> (+0.19%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[ColinM-sys]

Addressed all review feedback:

On length checks: Kept provider(64) and model(128) — conservative but covers all real-world values; early validation prevents DB errors

On %f truncation: Fixed — using %s format so MySQL handles precision naturally

On delete errors: Fixed — now throws RuntimeException on failure instead of silent 0

On byte limit: Fixed — prefix indexes on VARCHAR columns keeps key under 767-byte limit

On esc_like: Removed from table_exists() — table_name is constructed from constant + prefix, not user input, so escaping not needed

On schema upgrade: Fixed — logic now properly handles table creation and version tracking

Docs split: Done — moved to docs/experiments/embeddings.md

All 54 tests passing locally; ready for re-review.

[dkotter]

Sorry for multiples pings, was doing a further review after submitting my last review (and I know changes came in during that time as well)

[ColinM-sys]

Fixed two issues from your review:

@SInCE versions: Reverted to x.x.x (I had incorrectly filled in 1.4.0). Understood — maintainers update those to the actual version at release time, not during PR review.

Infinite loop in iterate(): Found and fixed a real bug. If all rows in a batch are corrupt (hydrate_rows skips them), the cursor never advances, causing infinite loop. Now cursor advances regardless of corrupt rows.

---

On the design questions — these all seem like scope decisions, so want to flag them for your input:

On object_subtype: WordPress Core uses subtypes (post/page, term/post_tag). Should we add an optional column now to align with that pattern? Or scope as v2 enhancement?

On dimension handling: Current design assumes dimensions are stable per model. If dimensions change, sync layer calls delete_for_model() first. Does that approach work, or should we handle dimension changes differently?

On save_many performance: For large-scale sync (50k+ items), batch INSERT would be much faster than looping individual saves. Should we implement multi-row insert in this PR, or defer until sync layer lands?

On finding objects without vectors: This feels like application-layer concern — caller queries get_object_ids() to find indexed objects, diffs against full set to find gaps. Is that approach OK, or should storage layer provide batch probe?

On schema performance at scale: Current indexes handle common queries. For sites with 50k+ items, should we add more targeted indexes now, or optimize when we hit bottlenecks?

On quantized embeddings: embedding_coarse would be great for fast similarity filtering — full + quantized vectors, filter coarse then compute full similarity on top candidates. Is similarity search in-scope for this PR, or definitely follow-up work?

On stale chunks: If dimensions change but provider/model stay same, old vectors coexist. This is by design — sync layer should call delete_for_model() first to clear them explicitly, not silently corrupt on re-index.

Which of these should we tackle in this PR vs plan for follow-up work?

[dkotter]

On the design questions — these all seem like scope decisions, so want to flag them for your input:

Most of these are questions / things I flagged specifically so we (ideally as humans) could discuss and decide on the right approach (or even decide that nothing is needed). This is a large feature and ideally we get this schema right from the beginning to avoid messy future upgrades.

On object_subtype: WordPress Core uses subtypes (post/page, term/post_tag). Should we add an optional column now to align with that pattern? Or scope as v2 enhancement?

If we want this it should go in this PR, not a v2 enhancement as this is a change to the schema. I personally think this makes sense and matches what Core does so I'd suggest we add it.

On dimension handling: Current design assumes dimensions are stable per model. If dimensions change, sync layer calls delete_for_model() first. Does that approach work, or should we handle dimension changes differently?

I had posed this as a question to ensure we were thinking through all scenarios. As an example (this may or may not end up how it's built) say we have a filter around the dimensions value that allows a 3rd party to change that. If embeddings are generated with a certain dimension value and later someone uses that filter to generate embeddings with a different dimensions value, we won't necessarily know to run delete_for_model first. In that scenario, I'm assuming we'll end up with different records in this table, instead of having the current record updated with the new data. Should we be considering dimensions as part of the uniqueness (we currently look at object, provider, model, chunk)?

On save_many performance: For large-scale sync (50k+ items), batch INSERT would be much faster than looping individual saves. Should we implement multi-row insert in this PR, or defer until sync layer lands?

I'd suggest making that change here, though could punt to that sync PR if you think that's better (just don't want to lose that item between PRs)

On finding objects without vectors: This feels like application-layer concern — caller queries get_object_ids() to find indexed objects, diffs against full set to find gaps. Is that approach OK, or should storage layer provide batch probe?

Maybe fine for now, likely something to consider once we have the sync layer in place and can more easily test this with large datasets (though this may be solved/improved by some of the other items in this list being solved)

On schema performance at scale: Current indexes handle common queries. For sites with 50k+ items, should we add more targeted indexes now, or optimize when we hit bottlenecks?

If there are ways to optimize now that will be ideal. Maybe the embedding_coarse column below is good enough but I want us thinking through ways to optimize this from the beginning rather than shipping something that we know will fall apart at scale.

On quantized embeddings: embedding_coarse would be great for fast similarity filtering — full + quantized vectors, filter coarse then compute full similarity on top candidates. Is similarity search in-scope for this PR, or definitely follow-up work?

I don't think building out similarity search is in-scope here but embedding_coarse would be a new column and so if we think that's useful, ideally that's added in this PR

On stale chunks: If dimensions change but provider/model stay same, old vectors coexist. This is by design — sync layer should call delete_for_model() first to clear them explicitly, not silently corrupt on re-index.

Similar to an answer above, not sure we can always assume everyone will be calling delete_for_model. And maybe that's fine and that's on them, just something to consider

[ColinM-sys]

Implementing dkotter scope guidance (2026-08-31 21:14):

✅ DONE:

• Added object_subtype column (VARCHAR 32, default '') — aligns with WordPress Core object segmentation
• Added embedding_coarse column (MEDIUMBLOB) — quantized vectors for fast similarity filtering
• Added dimensions to unique key: (object_type, object_id, provider, model, object_subtype, chunk_index, dimensions)
  Rationale: embeddings with different dimensions are fundamentally different data; making dimensions part of identity prevents accidental mixing
• Added Vector_Codec::pack_coarse() for coarse/quantized vectors (MVP: same as pack, future: float16/int8)
• Updated Embedding_Record to handle object_subtype (optional, default empty)
• Updated save() and save_many() to include new columns

⚠️ NEXT:

• Fix test assertions for new column order (test_table_has_expected_columns_and_unique_key expects old column count)
• Implement proper batch INSERT in save_many() (deferred MVP — complex wpdb->prepare() syntax, individual saves functional for now)
• Add object_subtype/embedding_coarse to all repository queries

📝 ON DIMENSIONS:
Dimensions are now part of the unique key, which means:

• Old embeddings with dims=768 coexist with new embeddings with dims=1024 for same (object, provider, model)
• This is explicit and safe — no accidental mixing or need to remember delete_for_model() first
• Aligns with design philosophy: model is part of identity, so dimensions should be too

Ready to discuss batch INSERT optimization strategy or proceed with test fixes?

[jeffpaul]

@ColinM-sys the speed with which you're responding to feedback and pushing code changes makes it appear as though we're directly interacting with an agent, but I hope (and am assuming positive intent) that you're reviewing the feedback and the code output (even if using an agent to assist) before pushing commits or code. If that's not the case, please do take the time to personally review the feedback from code review/testing and ensure you're reviewing what's committed back to PRs (again, even if using AI to assist).

TLDR; don't be a meat proxy.

[ColinM-sys]

I have been working on 2 other github projects all day so im on here and always checking my email. I use AI to code but review it before im pushing. If you dont want the help thats cool just let me know.

[dkotter]

Ready to discuss batch INSERT optimization strategy or proceed with test fixes?

@ColinM-sys Is this a comment for me? Seems more of an agent leaving a comment for you but let me know if you're blocked on next steps here and there's things that need discussing. Thanks!

[ColinM-sys]

it was written for you but i get it sounds like an internal note? I am not using any agents im not sure why you all keep saying that. I do use them for other things but not this. Also an LLM is different than an agent. Besides the facts: I was trying to respond to what you asked on the 31st. My question: should we implement batch INSERT now or defer to sync layer?

[dkotter]

I was trying to respond to what you asked on the 31st. My question: should we implement batch INSERT now or defer to sync layer?

I think I still stand by my initial comment on that:

I'd suggest making that change here, though could punt to that sync PR if you think that's better (just don't want to lose that item between PRs)

If you think it's better to handle that as part of the sync management PR, I'm fine with skipping here, just want to ensure that's captured somewhere so we don't forget

[ColinM-sys]

I struggled getting the syntax right earlier. Multi-row placeholders are tricky. A lot more testing would be needed. My preference would be to pass and handle it fresh in sync layer context later in a separate PR. Want me to open a dedicated PR for batch INSERT once this merges? I can handle that separately and you can tag me to track it

[dugyen]

Automated review (high effort)

Reviewed the diff against develop (13 files, +2105/-2). Ran 8 finder angles + a verification pass on candidates. 10 findings survived verification (8 CONFIRMED, 2 PLAUSIBLE); 2 other candidates were checked and refuted (a LIKE-wildcard concern in table_exists() is neutralized by a strict equality check afterward, and "the layer isn't wired to any caller yet" is explicitly disclosed scope in the PR description, not a gap).

Correctness

1. dimensions inside the UNIQUE KEY breaks the documented upsert/replace-in-place contract — Embedding_Schema.php:154
generate_embeddings() exposes an optional dimensions arg (native truncatable embeddings, e.g. OpenAI text-embedding-3-*). Calling it twice for the same (object, provider, model, chunk) with a different requested dimensions produces vectors of different lengths. Since dimensions is part of uniq_object_model_chunk, the second save() no longer collides with the first row's key — INSERT ... ON DUPLICATE KEY UPDATE inserts a second row instead of replacing it, contradicting "replaces any existing vector for the same object, model and chunk" (Embedding_Repository_Interface.php:29,41), and get() then returns two records for one chunk.

2. Unique key indexes provider/model as prefixes shorter than their validated field lengths (PLAUSIBLE) — same line
   provider(32)/model(64) prefix-index widths are half the validated column widths (64/128 chars). Two values sharing the same first 32/64 bytes but differing after that collide in the unique index; save()'s ON DUPLICATE KEY UPDATE (lines 91-98) never re-sets provider/model, so the second write silently overwrites the first row's vector while keeping the old provider/model label — a read for model A can silently return model B's vector.

3. save_many() silently drops invalid records instead of throwing as documented — Embedding_Repository.php:145
   The interface docblock types the param as list<Embedding_Record> and documents @throws RuntimeException if a record could not be written, but the implementation's parameter is a plain untyped array, so PHP enforces nothing — a non-Embedding_Record element is silently continued past, returning a shorter $saved array with no signal of what was dropped.

4. get_object_ids() ordering doesn't match its "newest first" docs — Embedding_Repository.php:263
   Orders ORDER BY object_id DESC, i.e. numeric ID, not write/update recency. Re-embedding an old (low-ID) object won't float it to the front, contradicting the interface's documented "newest first" contract.

Efficiency

5. save_many() issues N separate INSERTs instead of one batched upsert — Embedding_Repository.php:148
   Loops calling save() per record. Indexing a 50-chunk document does 50 separate round-trips instead of one multi-row INSERT ... VALUES (...),(...),... ON DUPLICATE KEY UPDATE ..., on what's meant to be the hot write path.

6. Vector_Codec::pack() packs one float at a time instead of vectorized — Vector_Codec.php:46
validate() already rejects bad values before the loop, so pack('g*', ...$vector) (after an array_map('floatval', ...)) would produce byte-identical output in one call instead of 1500+ calls per save for a typical embedding.

Reuse

7. Embedding_Schema duplicates AI_Request_Log_Schema's pattern with no shared base, and has already diverged — Embedding_Schema.php:83
get_table_name()/table_exists() are near-verbatim copies. maybe_upgrade_table() additionally checks table_exists() in its early-return where AI_Request_Log_Schema's doesn't — a stale version option pointing at a manually-dropped table short-circuits in one but correctly recreates in the other. table_exists() also flipped visibility (public vs. private in the sibling class).

Simplification / dead code

8. pack_coarse()/embedding_coarse column is dead — Vector_Codec.php:117
pack_coarse() has zero call sites repo-wide; save() hardcodes embedding_coarse to '' instead of calling it.

9. embedding_norm is write-only — Embedding_Repository.php:94
   Computed and stored on every save(), but Embedding_Record has no norm field/getter and hydrate_rows() never reads it back, despite all read methods doing SELECT *.

Altitude

10. Corrupt-row skipping isn't a reusable mechanism for the second implementation the interface itself anticipates — Embedding_Repository.php:479
hydrate_rows()'s try/catch around InvalidArgumentException is private to this one class; Embedding_Record exposes no safe factory (e.g. try_from_row()). The interface docblock explicitly anticipates a native-vector-index-backed implementation — that future class would have to reinvent this from scratch, and forgetting it would let one corrupt row fatal the whole read path.

---

🤖 Generated with Claude Code

[dkotter]

I've done another pass of testing and review, pushing up some changes but I think this is in a good spot now. Likely some adjustments we'll make as we actually wire this up to data but I'm happy with where we're at for now. Thanks for all the effort here @ColinM-sys!

[ColinM-sys]

@dkotter saw you pushed new fixes while I was working on the same issues. I've got changes addressing the correctness findings too. They're ready locally but conflicting with your recent commits. I can rebase on top of your new code and we can align the approaches. let me know whatt you want me to do.

[dkotter]

@ColinM-sys I'm assuming you're referring to the things mentioned in this comment? That comment came in while I was already working on things so it was not something I considered. It appears to just be an automated review so not something I would worry about diving in deep on. Reading through it, I think most of the actual problems I've already addressed and some of the other things it flags I don't think need fixed here. So from my perspective, this is still good to merge as-is

[ColinM-sys]

Thanks for making those changes @dkotter. Sounds good!