Repository navigation
feat(replication): Model Sync Option B, Capability-Driven Replication and Registry UI for Models - #24319
Draft
Vad1mo wants to merge 11 commits into
Draft
feat(replication): Model Sync Option B, Capability-Driven Replication and Registry UI for Models#24319Vad1mo wants to merge 11 commits into
Vad1mo wants to merge 11 commits into
Conversation
Vad1mo
force-pushed
the
feat/model-sync-option-b
branch
from
November 5, 2026 08:12
bba19af to
be00d91
Compare
Add a `huggingface` registry type that lets pull replication and proxy cache copy Hugging Face Hub models into Harbor as ModelPack model-spec artifacts, without a dedicated sync framework. The adapter synthesizes a byte-deterministic OCI manifest from the Hub API for a pinned commit: one raw layer per file, LFS sha256 reused as the layer digest, non-LFS files hashed and checked against their git blob id, file classification frozen from modctl. The same commit always yields the same digest, so replication's digest-equality skip and proxy cache's local/remote comparison work unchanged. Digest-to-file lookups go through lib/cache scoped per registry (Redis in core, in-process memory in jobservice) with a stateless rebuild fallback, because proxy cache creates a new adapter per request. The type is source-only: replication policies with a Hugging Face destination are rejected. Refs: goharbor/community#297 Signed-off-by: Vadim Bauer <vb@container-registry.com>
…s, propagate lookup errors Signed-off-by: Vadim Bauer <vadim@8gears.com>
Add a generic "file" replication filter that limits which files a source adapter packs into an artifact. The policy's filter becomes model.SourceOptions, which the copy flow hands to the source adapter through the optional adapter.SourceConfigurer interface and serializes with the source resource, so the transfer configures its own adapter the same way. Adapters that do not implement the interface fail instead of ignoring the options. The Hugging Face adapter selects files with the filter before hashing and synthesis. The digest depends on the commit and the selected files only, so the same filter always yields the same digest and a filter that drops files yields a different one. Signed-off-by: Vad1mo <vadim@8gears.com>
Vad1mo
force-pushed
the
feat/model-sync-option-b
branch
from
November 5, 2026 08:50
be00d91 to
59379cd
Compare
Add the optional adapter.ArtifactResolver interface. Resolve reports what a copy of one source reference would produce without copying: the revision, the moving tags pointing at it, license, task, the file list with class, size, sha256 and whether the file filter selects it, matched totals, and the manifest digest of the copy. The Hugging Face adapter implements it with the same snapshot, hashing and synthesis code as the copy, so the previewed digest is the copied digest. The Hub's pipeline_tag is read into the snapshot as the task. A live test behind the hubintegration build tag resolves against the public Hub. Signed-off-by: Vad1mo <vadim@8gears.com>
Add a revision_tag flag to replication policies. With it, every copy also gets the immutable tag sha-<first 12 hex of the source revision> next to its moving tags, so a scheduled pull of main keeps every imported revision addressable. The flag travels with the other source options to the source adapter; policy validation accepts it only for sources whose registry info declares supported_revision_tag. The Hugging Face adapter appends the revision tag after the tag filter and resolves a revision tag back to the ref head it names. The column is added idempotently in the 2.17.0 schema migration. Signed-off-by: Vad1mo <vadim@8gears.com>
Add source_only to the registry info. Policy validation now rejects any registry that declares it as a replication destination instead of checking for the Hugging Face type, and the registry info API returns the flag so a UI can hide such registries from destination pickers. The local Harbor is never probed, and a registry whose capabilities cannot be read is accepted as before. Signed-off-by: Vad1mo <vadim@8gears.com>
…y Info
POST /registries/{id}/preview resolves what a copy of a repository reference
with an optional file filter would produce, for any adapter that implements
adapter.ArtifactResolver; others answer 400. RegistryInfo.supported_preview is
derived from that interface, so adapters declare nothing new.
FilterStyle gains optional label, placeholder and description so a source can
name its filters in its own terms, and the adapter pattern gains source_only so
clients can leave such registries out of destination pickers without creating
an adapter per registry. The Hugging Face adapter uses both and fixes its access
key, so the endpoint form only asks for the token.
Signed-off-by: Vad1mo <vadim@8gears.com>
Signed-off-by: Vad1mo <vadim@8gears.com>
The form reads everything model-specific from the source's registry info, never from the registry type: the file filter gets include/exclude inputs when the source lists the file filter, the revision tag option appears when the source declares supported_revision_tag, a preview panel appears when it declares supported_preview, and filters show the adapter's own label, placeholder and help. Registries whose type is source_only are left out of the destination picker. Sources without these declarations render exactly as before. Signed-off-by: Vad1mo <vadim@8gears.com>
…attern A fixed access key (AccessKeyTypeFix) is prefilled and read-only, and the secret is a text area only for AccessSecretTypeFile, instead of checking for the GCR type. Hugging Face endpoints thus only ask for a token. Source-only providers say so under the provider select. Signed-off-by: Vad1mo <vadim@8gears.com>
A Model section lists licenses, source URL, source revision and the model-spec config (architecture, format, parameters, precision, quantization) instead of the raw descriptor/config/modelfs JSON in the overview. The Files tab picks icons by file class, the artifact list marks models with a badge, and the quantization label no longer reads the misspelled config.puantization key. Signed-off-by: Vad1mo <vadim@8gears.com>
Vad1mo
force-pushed
the
feat/model-sync-option-b
branch
from
November 5, 2026 09:18
59379cd to
6ff7eba
Compare
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## main #24319 +/- ##
==========================================
+ Coverage 74.24% 74.51% +0.27%
==========================================
Files 1096 1113 +17
Lines 103614 105747 +2133
Branches 3079 3156 +77
==========================================
+ Hits 76926 78800 +1874
- Misses 26319 26557 +238
- Partials 369 390 +21
Flags with carried forward coverage won't be shown. Click here to find out more.
🚀 New features to boost your workflow:
|
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Thank you for contributing to Harbor!
Comprehensive Summary of your change
Draft for the design discussion in goharbor/community#297. Not meant to be merged as is. This is Option B of two prototypes compared in #297 (comment); Option A is #24318.
Option B: no new policy type. Model imports are ordinary replication rules, and the existing screens are extended by capabilities, with no
huggingfacecheck in the portal:POST /registries/{id}/previewandsupported_previewin registry info, available for any adapter that implementsArtifactResolversource_onlyregistries are hidden from push destinationsgoogle-gcrspecial case, and Quay also declares the fixed access-key pattern, so Quay's access key becomes read-only. Needs a decision.config.puantizationtypo in the label rowStack
This branch contains three layers. Please review only the last one:
filefilter with include/exclude,ArtifactResolverfor copy previews, optionalsha-<12>revision tag,source_onlyinRegistryInfofeat(registry): Add Copy Preview API and Form Capabilities to Registry InfoGitHub can't stack PRs from a fork onto #24008, so the base is
mainand the diff includes layers 1 and 2.Verified
HuggingFaceTB/SmolLM2-135M@main, include*.json, revision tag on:sha256:2dc8fb68cff37ff1…with tagsmainandsha-93efa2f097d5; rerun skipped with the same digest. Same result as Option A. Hugging Face absent from push destinations.pkg/reg/...,controller/replication/...,controller/registry/...,server/v2.0/handler/...ng buildfor every portal commit; touched karma specs 45/45Limitations
Screenshots
B1: Hugging Face endpoint, token only
B2: Rule form with Hugging Face source: Model, Revision, Files
B3: Preview inside the rule form
B4: Rule form with Docker Hub source, unchanged
B5: Executions succeeded
B6: Model artifact with the Model panel
Issue being fixed
Related: goharbor/community#297, depends on #24008
Please indicate you've done the following: