Repository navigation
Conversation
3 of 5 tasks
Vad1mo
force-pushed
the
feat/model-sync-option-a
branch
from
November 5, 2026 08:12
4ecec77 to
4595753
Compare
Add a `huggingface` registry type that lets pull replication and proxy cache copy Hugging Face Hub models into Harbor as ModelPack model-spec artifacts, without a dedicated sync framework. The adapter synthesizes a byte-deterministic OCI manifest from the Hub API for a pinned commit: one raw layer per file, LFS sha256 reused as the layer digest, non-LFS files hashed and checked against their git blob id, file classification frozen from modctl. The same commit always yields the same digest, so replication's digest-equality skip and proxy cache's local/remote comparison work unchanged. Digest-to-file lookups go through lib/cache scoped per registry (Redis in core, in-process memory in jobservice) with a stateless rebuild fallback, because proxy cache creates a new adapter per request. The type is source-only: replication policies with a Hugging Face destination are rejected. Refs: goharbor/community#297 Signed-off-by: Vadim Bauer <vb@container-registry.com>
…s, propagate lookup errors Signed-off-by: Vadim Bauer <vadim@8gears.com>
Add a generic "file" replication filter that limits which files a source adapter packs into an artifact. The policy's filter becomes model.SourceOptions, which the copy flow hands to the source adapter through the optional adapter.SourceConfigurer interface and serializes with the source resource, so the transfer configures its own adapter the same way. Adapters that do not implement the interface fail instead of ignoring the options. The Hugging Face adapter selects files with the filter before hashing and synthesis. The digest depends on the commit and the selected files only, so the same filter always yields the same digest and a filter that drops files yields a different one. Signed-off-by: Vad1mo <vadim@8gears.com>
Vad1mo
force-pushed
the
feat/model-sync-option-a
branch
from
November 5, 2026 08:50
4595753 to
e7675b8
Compare
Add the optional adapter.ArtifactResolver interface. Resolve reports what a copy of one source reference would produce without copying: the revision, the moving tags pointing at it, license, task, the file list with class, size, sha256 and whether the file filter selects it, matched totals, and the manifest digest of the copy. The Hugging Face adapter implements it with the same snapshot, hashing and synthesis code as the copy, so the previewed digest is the copied digest. The Hub's pipeline_tag is read into the snapshot as the task. A live test behind the hubintegration build tag resolves against the public Hub. Signed-off-by: Vad1mo <vadim@8gears.com>
Add a revision_tag flag to replication policies. With it, every copy also gets the immutable tag sha-<first 12 hex of the source revision> next to its moving tags, so a scheduled pull of main keeps every imported revision addressable. The flag travels with the other source options to the source adapter; policy validation accepts it only for sources whose registry info declares supported_revision_tag. The Hugging Face adapter appends the revision tag after the tag filter and resolves a revision tag back to the ref head it names. The column is added idempotently in the 2.17.0 schema migration. Signed-off-by: Vad1mo <vadim@8gears.com>
Add source_only to the registry info. Policy validation now rejects any registry that declares it as a replication destination instead of checking for the Hugging Face type, and the registry info API returns the flag so a UI can hide such registries from destination pickers. The local Harbor is never probed, and a registry whose capabilities cannot be read is accepted as before. Signed-off-by: Vad1mo <vadim@8gears.com>
Signed-off-by: Vad1mo <vadim@8gears.com>
Signed-off-by: Vad1mo <vadim@8gears.com>
Vad1mo
force-pushed
the
feat/model-sync-option-a
branch
from
November 5, 2026 09:18
e7675b8 to
5921d3d
Compare
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## main #24318 +/- ##
==========================================
+ Coverage 74.24% 74.86% +0.62%
==========================================
Files 1096 1122 +26
Lines 103614 109122 +5508
Branches 3079 3194 +115
==========================================
+ Hits 76926 81699 +4773
- Misses 26319 27023 +704
- Partials 369 400 +31
Flags with carried forward coverage won't be shown. Click here to find out more.
🚀 New features to boost your workflow:
|
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Thank you for contributing to Harbor!
Comprehensive Summary of your change
Draft for the design discussion in goharbor/community#297. Not meant to be merged as is. This is Option A of two prototypes compared in #297 (comment); Option B is #24319.
Option A: a separate Administration > Model Sync product surface, modelled on the proposal and the model-poc PoC, running on the shared backend:
model_sync_policytable (0200_2.17.0_schema.up.sql) and/model-sync/*API, same paths and shapes as the PoC, plus exclude filtersMODEL_SYNCbut execute through the existing replication copy flow, transfer, job and execution/task tables. No second packer or transfer.Stack
This branch contains three layers. Please review only the last one:
filefilter with include/exclude,ArtifactResolverfor copy previews, optionalsha-<12>revision tag,source_onlyinRegistryInfofeat(modelsync): Add Model Sync Policies on the Replication Copy Flow,feat(portal): Add Model Sync Administration PageGitHub can't stack PRs from a fork onto #24008, so the base is
mainand the diff includes layers 1 and 2.Verified
HuggingFaceTB/SmolLM2-135M@main, include*.json:sha256:2dc8fb68cff37ff1…with tagsmainandsha-93efa2f097d5; rerun skipped with the same digest. Same result as Option B.pkg/reg/...,controller/replication/...,controller/registry/...,server/v2.0/handler/...,controller/modelsync/...,pkg/modelsync/...; DB-tagged tests against Postgres with all migrations appliedng build, model-sync karma specs 23/23Limitations
org/**DestRepositoryoverride because its namespace rewrite always keeps the source's last path componentScreenshots
A1: Model Sync entry in Administration
A2: Policy list
A3: Policy form with resolved preview (include *.json, exclude onnx/, .bin)*
A4: Executions
A5: Task list with log link
A6: Resulting model artifact. The raw descriptor JSON is today's artifact view; the Model panel in Option B is independent of the import path and would work here too
A7: Rerun skipped with the same digest
Issue being fixed
Related: goharbor/community#297, depends on #24008
Please indicate you've done the following: