RFC-0008: MVP Skill Registry - #26
Conversation
Metadata-first registry for AI agent skills with versioning, lifecycle management, trace integration, and package manager plugin interface. Phase 1 covers skills and skill bundles (skills-only). Phase 2 (RFC-0009, separate PR) will extend bundles with subagents, hooks, and MCP server references. Replaces the closed PR mlflow#10 with a phased approach per Slack discussion with Databricks maintainers. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Accept changes from Codex review: SDK namespace to mlflow.genai.*, plugin import section, automatic harness instrumentation, install via package manager. Trim auto instrumentation to essentials. Remove register_skill idempotency and align with MCP Server Registry behavior (fail if version exists), citing register_mcp_server() in mlflow/mlflow#23696 as rationale. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Fill in rfc_pr link, fix "The two entity types:" grammar, replace all RFC-0009 references with "follow-up RFC", and change non-skill member lists to examples (e.g., subagents, MCP server references) since the set may change before that RFC is written. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
…ntics Resolve both open investigation items: harness argument is now required on install commands, reproducibility depends on the package manager plugin (APM has full lockfile, Lola has version constraints). Drop Phase 3 from adoption strategy. Add agentskills.io citation. Clarify that bundles can contain non-skill content which is pulled and installed but does not receive individual registry entries in Phase 1. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Add missing --harness claude-code to both install-bundle examples. Fix Phase 2 phrasing to say "add registry entries for" instead of "extend bundles to include" non-skill members. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Add mlflow-skills.lock resolution lock for reproducible cross-machine installation. Package manager plugin interface now returns PackageManagerInstallResult with harness-local skill names so trace manifests are accurate even when plugins rename skills. Monolithic bundle install passes bundle_path to the plugin so non-skill content is installed as a unit. Fix span annotation from "registry" to "workspace" and "skipped" to "unregistered" wording. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Add Open questions section per RFC template. Note that structured security scan metadata is valuable but should be addressed as a cross-registry capability shared across all registries, not as a skill-specific feature. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
| MLflow discovers and registers the plugin's skills as members of a | ||
| monolithic bundle. It preserves the Git source on the bundle and warns | ||
| about subagents, hooks, and MCP configurations that are not registered | ||
| in Phase 1. | ||
|
|
There was a problem hiding this comment.
and warns about subagents, hooks, and MCP configurations that are not registered
in Phase 1.
I see these warnings are provided in various cases (like import). It feels unnecessary because it is expected that a skill api / option would register only skills.
There was a problem hiding this comment.
I could argue it either way, but Yuki specifically asked for the warnings, and they seem somewhat useful to me.
|
|
||
| #### Automatic harness instrumentation | ||
|
|
||
| Phase 1 extends the Claude Code autologger to recognize skill |
There was a problem hiding this comment.
It would be nice if the RFC also points out the plan for the others in a future enhancement section, which, if any, should be prime candidates for auto log support.
There was a problem hiding this comment.
Broadly speaking, I think the process would be something like this:
- Start with the list of harness agents that have auto-loggers at https://mlflow.org/docs/latest/genai/tracing/integrations/#coding-agents--long-running-agents
- Prioritize based on popularity, openness, importance to whatever contributor wants to do the work, etc.
In addition to supporting the harness agents, you might want to support the SDKs at https://mlflow.org/docs/latest/genai/tracing/integrations/#agent-frameworks-python and https://mlflow.org/docs/latest/genai/tracing/integrations/#agent-frameworks-typescript . Not all of those have a "skill" concept, and you can't autolog skill use if the SDK that you're using doesn't have a concept of a skill. (You could still log it yourself if you layer a skill concept on top of the SDK you're using of course).
This seems like a long, hard conversation to have to get any sort of meaningful conclusions. I'd prefer to keep it out of the RFC because I'm a little concerned that resolving this debate would take a bunch of time that would slow down work on this RFC in order to pre-decide things for future enhancements. Furthermore, I think a lot of things will change between now and the time that someone starts working on future enhancements, so whatever we decide now might turn out to be irrelevant by that time anyway.
|
|
||
| ### Manifest writing and discovery | ||
|
|
||
| Installation commands write or update the manifest after all requested |
There was a problem hiding this comment.
One thing that caught me by surprise was the decision to have MLflow locally materialize the skill or skull-bundle. I think this is a good way to ensure MLflow retains domain ownership over resolution semantics, instead of trying to squeeze registry behavior into another tool's package-resolution abstraction.
I think it would be good to explicitly mention the justification somewhere.
I'd imagine there are some tradeoffs here as well, for example if a package manager has features / capabilities around remote source semantics. If you agree, consider adding a small note about it.
There was a problem hiding this comment.
Yes, I agree this needs some more thought. We'll sort it out in the upcoming installation RFC now that we've decided to defer installation.
mprahl
left a comment
mprahl
left a comment
There was a problem hiding this comment.
Looks great! I left a few comments.
Replace all "Phase 1" qualifiers with "this RFC" or remove where unnecessary. Replace "Phase 2" and "follow-up RFC" references with explicit links to RFC-0009: Extended Skill Bundles (PR mlflow#27). Restructure Adoption strategy from phased rollout into a description of what this RFC delivers plus a Future improvements subsection. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Avoids confusion with the existing `mlflow skills` CLI group that inspects bundled Assistant skills. Updated all CLI command references and narrative text in both the main RFC and implementation details. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Bundle, import, and introspect commands now live under `mlflow skills-registry bundles` instead of using flat prefixed names (e.g., `bundles create-version` instead of `create-bundle-version`). Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
All members are skills in this RFC. RFC-0009 can add the column with a default of 'skill' via a standard schema migration when it introduces non-skill member types. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Add `list-package-managers` CLI command and SDK function - Use fixed `.mlflow-skills/` directory for cached content instead of temporary directories - Note that plugins may write native lockfiles alongside cached content for MLflow-free collaboration - Add `check_requirements()` preflight method to plugin interface - Include tracking server URL in `mlflow-skills.lock` so `--from-lock` can connect without separate configuration Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
…te RFC Remove all install commands, package manager plugin interface, mlflow-skills.lock resolution lock, and install_count from RFC-0008. This content will be covered in a separate RFC per reviewer feedback. User journeys now use pull + manual harness installation. Pull and plugin import remain in scope. Saved removed content to .local/deferred-install.md for the new RFC. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
|
This push removes install commands, the package manager plugin interface, the Fully moot threads (will be addressed in the new RFC):
I marked these as Resolved since the content is no longer in this RFC. All feedback will be tracked and addressed in the new RFC. -- Bill Murdock (with assistance from Claude Code) |
Per reviewer feedback from mprahl: add agent setup integration, MCP server skill search, and skill signature/trusted publisher support as future improvements. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Skill names are already human-readable (e.g., code-review), so a separate display_name adds complexity without clear benefit. Can be reintroduced later if needed. Softened UI "consistent with" to "modeled on" since MCP Server Registry still has display_name. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
These are local working notes that should not be in the PR. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
| # Explicit: all fields specified | ||
| mlflow.genai.register_skill( | ||
| name="code-review", | ||
| source_type="git", |
There was a problem hiding this comment.
Optional: Does it make sense to have a generic source field that maps to different classes such as GitSource, OCISource, etc. so that the required fields for each type is clear?
| 1. Register individual skill versions pointing to their sources: | ||
| ```bash | ||
| # Minimal: name and source type inferred | ||
| mlflow skills register \ |
There was a problem hiding this comment.
Optional: similar to my previous comment, it may be nice to have mlflow skills register git or mlflow skills register oci so that you know which args are required.
…e, unify content_path Drop content_digest from MVP scope. Make source_type server-inferred from the source value rather than user-specified. Unify content_path into source (local paths detected by absence of :// scheme). Use controlled artifact path for MLflow storage with create-then-upload flow. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
| source_type: SkillSourceType | None = None | ||
| source: str | None = None | ||
| subpath: str | None = None | ||
| status: SkillStatus = SkillStatus.DRAFT |
There was a problem hiding this comment.
@B-Step62, @jwm4 , do you think defaulting the status to active makes sense? This would reduce the cognitive load for a user since unlike an MCP server that is essentially an application, I think a skill that gets published is almost always going to be active unless they are specific evaluation runs comparing tweaks to skills.
| harnesses. Discovers plugins via the entrypoint mechanism.""" | ||
|
|
||
|
|
||
| def search_skill_traces( |
There was a problem hiding this comment.
@jwm4 Can you checkout this comment? I don't think we should add a different search trace API for skills. The filter string can handle this well.
There was a problem hiding this comment.
@B-Step62 , since this iteration of the RFC is no longer including how traces should be linked to skills. I think we should either:
- Add a manual way to link skills to a trace like MLflow does for prompts.
- We remove trace searching via skill usage from the RFC and add it in the follow up PR.
There was a problem hiding this comment.
FWIW, I've taken a note on this topic to make sure I address it in the trace linking PR. I haven't thought through that one carefully enough to know exactly where it's going to land, but I'll be sure to take into account this discussion and specifically the concern about having a separate search trace API for skills.
|
@jwm4 @HumairAK @B-Step62 , just bumping this comment here: |
| ## Create a skill bundle | ||
|
|
||
| ```python | ||
| mlflow.genai.create_skill_bundle_version( |
There was a problem hiding this comment.
Just curious, since skills within a bundle can be individually versioned, and the same common skill can be potentially consumed by many bundles, do we intend to have a simple way to update all affected bundles to include/update the new skill version?
There was a problem hiding this comment.
Good question. I just added two user journeys that address this for both cases:
Monolithic agent plugins (created via import): You can re-import from the updated source. Import matches discovered skills to existing members by comparing plugin-relative directory paths against the #subpath fragments in the previous agent plugin version. Matching subpaths get new skill versions; new subpaths become new skills; removed subpaths are omitted from the new version. So mlflow agent-plugins import --source ...@v2.0.0 is all you need.
Assembled agent plugins (manually composed): You register the new skill version, then create a new agent plugin version with the updated member reference. The skill detail view shows agent plugin memberships, so you can find which agent plugins include a given skill. There's no bulk "update all agent plugins that use this skill" operation in the MVP, but aliases give you a safe rollout mechanism: agents using skills:/pr-workflow@production keep resolving to the old version until you explicitly update the alias.
See the "Update an imported agent plugin" and "Update an assembled agent plugin after a member skill changes" journeys in the latest push.
Posted by Bill Murdock with assistance from Claude Code
| and `search_skill_traces()`, is part of the MVP scope and will be | ||
| added to this RFC as a separate PR. The trace manifest mechanism | ||
| depends on the installation RFC below. | ||
| - **Installation and package manager integration.** Harness-specific |
There was a problem hiding this comment.
I think this will be really useful, and can be a simple way for users to integrate skills/bundles quickly to specific harnesses (without manual installation), since another evaluation they could perform could be using the same skill bundle version for different harnesses, and comparing which AI harnesses performs better
There was a problem hiding this comment.
I agree. We originally had this in scope for the MVP. I think it will be high priority for post-MVP.
There was a problem hiding this comment.
Yup agreed, let's discuss on this in the follow-up RFC and target to ship it soon.
|
@jwm4 I think the rename of skill bundles to "agent plugins" and this comment are all that are remaining from what I can see: It may be worth having Claude double check for open threads that don't have a response yet. |
|
@mprahl Good idea. Here's my read on threads that don't have a definitive conclusion yet. Please let me know if I'm missing any or have the status wrong. Updated status report (2026-08-06):
Summary: 5 items done, 3 deferred/aligned without code changes needed, 4 still open (8, 9, 10, 13). Items 8 and 9 are related (typed sources + CLI surface). Item 11 is agreed but not yet in the RFC. New item: There is an emerging open standard for agent plugins at agent-plugins.org (spec repo). I'd like to work on aligning this RFC with that effort. Posted by Bill Murdock with assistance from Claude Code |
Add server-assigned UUID PKs (skill_id, skill_bundle_id) to all entities while keeping names unique per workspace via unique constraint, following the GatewayEndpoint pattern. Update all schema tables, dataclasses, store methods, SDK functions, REST endpoints, and code examples to use UUID parameters. Trim UI section from detailed card/filter/detail-view specs to a short paragraph deferring layouts and card designs to UI mocks. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
…etails Rename entity types, DB tables, SDK functions, CLI commands, REST endpoints, code examples, and prose from "skill bundle" to "agent plugin" per reviewer consensus. CLI commands move from "mlflow skills bundles" to "mlflow agent-plugins". Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
When name is omitted and the server cannot access the source (e.g., private repos requiring client-side credentials), registration fails with an error. Name must be provided explicitly for private sources. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Remove the manual monolithic agent plugin example that called register_skill() without a source, which does not produce the intended embedded skill. Monolithic agent plugins are created through import_plugin(). Clarify that git source_type inference checks for .git before the @ref portion, not at the end of the full source string. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
…kinds - Import into an existing monolithic agent plugin now matches skills by subpath against the previous version's member list, enabling plugin updates without name-based matching. - New user journey for updating an assembled agent plugin when a member skill publishes a new version. - All versions of a given agent plugin must be the same kind (monolithic or assembled); the server rejects mismatches. - New embedded skills from re-import use the agent plugin version number. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Drop skill_id/agent_plugin_id UUIDs in favor of a composite primary key. Organization defaults to empty string and scopes ownership so the same skill name can exist under different organizations. Add validation rules: reserve _ as empty-org placeholder, forbid purely numeric names and URI-significant characters. Update all schema, dataclasses, store interface, SDK, REST paths, URIs, and code examples across both files. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Skill registration is typically a publish action on known-good content. Users who need a review gate can set status=draft explicitly. Add divergence note explaining the difference from RFC-0004. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Allow callers to create agent plugin versions as draft when they need review-before-publish. Update evaluation and CI journeys to create draft versions before promotion. Add status to store interface, SDK function, and REST request model. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Make the Agent Plugins format (agent-plugins.org) the canonical package representation for AgentPluginVersion. Each version stores an immutable plugin.json manifest, uses string versions from the manifest, and adds SemVer-aware latest resolution. Import auto-detects Agent Plugins v1, Claude Code, and generic formats. Introduces agent-plugins:/ URI scheme, register_agent_plugin() convenience API, and free-text search across manifest metadata. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
|
Status update (2026-08-11) Major change: Agent Plugins alignmentThe latest push aligns the
Fluent API / load_skillReplied on this thread: Remaining open items
Items 8 and 9 were both marked "Optional" by Matt. Items 11 and 13 are related (both about how embedded skill locations are represented in member references). Posted by Bill Murdock with assistance from Claude Code |
6 participants
Summary
Add a Skill Registry to MLflow: a governed, metadata-first registry for AI agent skills.
This is a restructured Phase 1 of the work originally proposed in PR #10, narrowed per Slack discussion with Databricks maintainers to focus on skills and skill bundles only.
mlflow.skill_context()creates SKILL spans with registry coordinatesPhase 2 (RFC-0009, separate PR) will extend bundles with subagents, hooks, and MCP server references.
Changes from PR #10
Files
rfcs/0008-mvp-skill-registry/0008-mvp-skill-registry.md: Main RFCrfcs/0008-mvp-skill-registry/implementation-details.md: DB schema, store interface, REST API, SDK, CLI, package manager plugin interfaceTest plan
Posted by Bill Murdock with assistance from Claude Code.