Roadmap: the spec pipeline, v6 → v7 → v8

[DerManoMann]

Pinned. This body is the current plan and gets updated as things ship; the comments below are the history of how it got here.

swagger-php 6 has two ways of turning PHP into an OpenAPI document. The classic pipeline is what everybody uses today: OpenApi\Attributes (or docblock annotations), the Generator, processors. The spec pipeline is new in 6.5: a separate, much thinner set of attributes under OpenApi\Spec, collected into a flat Specification, enriched by augmenters and compiled per OpenAPI version (3.0, 3.1, 3.2). In between sits hybrid mode - it scans your existing classic attributes and runs the result through the new pipeline. All three are described at Processing modes.

This is the plan for getting from the first to the second over three majors. The plan itself lives in ROADMAP.md; this issue is the discussion and the tick list. Nothing breaks in 6.x - everything new is opt-in, and a @deprecated marker names the version that removes the thing, so anything marked for 8.0 keeps working through v7 unchanged.

The problem:
The classic attributes do way too much - capture data, validate, nest, merge, serialize and branch on the OpenAPI version. Context grew to share state across all of that. Adding a field, supporting a new spec version or extending the library from the outside is all harder than it should be.

The idea:
Attributes are plain typed constructors and nothing else. Validation, type resolution, version handling etc. happen in stages you can hook into: Builder::withAugmenters(), withResolver(), withAttributeFactory() (which is where attribute translators are registered) and setCompiler().

What that buys:

• Programmatic use, without scanning. Spec attributes are ordinary value objects with typed properties: $spec->add(new Spec\Info(...), new Spec\Schema(...)) compiles straight to a PHP array. No more toJson() + json_decode() to merge with something else. The classic annotation objects were never usable this way - every property is untyped and defaults to an UNDEFINED sentinel, so reading one back means a sentinel check on every field.
• Mistakes reported instead of absorbed. date is not an OpenAPI type, and classic quietly rewrites type: 'date' to type: 'string', format: 'date' - so the attribute stays wrong and the next person copies it. The new pipeline says Schema has unknown type "date" instead. Running hybrid over a real codebase turned up three of them.
• OpenAPI 3.1 by default, with 3.0 and 3.2 as compiler targets instead of isVersion() branches inside every attribute.
• Less for integrations to reinvent. Component lookup and traversal are library API (ComponentIndex, Walker), and there is no _context to fabricate.

Version plan

v6 - now (6.10.0)

• spec pipeline (6.5), Builder as the shared entry point, three modes
• hybrid mode runs real-world classic codebases unchanged (6.10.0)
• docblock annotation support marked @deprecated, removed in 8.0, with a runtime deprecation when one is parsed (docs(Roadmap): say when classic goes, and mark what the file says is marked #2216) - the README had said so since 4.8, the code agrees now
• Generator::UNDEFINED / Generator::isDefault() re-targeted to 8.0 (docs(Roadmap): say when classic goes, and mark what the file says is marked #2216) - every classic custom processor uses them, so they go with classic
• a "why spec?" page in the guide, and spec framed properly on the homepage and in the README (docs(Guide): add a why-spec page, and stop three places underselling it #2217)
• docs for the programmatic, no-scanning use above - it leads the why page (docs(Guide): add a why-spec page, and stop three places underselling it #2217)
• a migration story for OpenApi\Attributes -> OpenApi\Spec worked out on a real project first, and whatever part of it turns out to be mechanical packaged as rector rules. Property is the awkward one: classic takes 61 constructor arguments because it extends Schema, spec takes 4 and holds a Schema, so that rewrite is a split rather than a rename
• a reference migration of a real open-source project, published with its output diff
• a seam to contribute pre-built attributes to a Specification before resolution - what framework integrations (routing, forms, serializer metadata) need to feed the pipeline without scanning
• declare the attribute API frozen and drop the beta label in that minor

Only the last item touches compatibility. Classic attributes, the Generator and the processors are deliberately not marked in v6 - a stable API does not get deprecated in favour of one still labelled beta.

v7 - hybrid by default, classic deprecated

• hybrid becomes the default mode: existing OpenApi\Attributes keep working, output goes through the new pipeline
• classic stays available via Mode::CLASSIC / --mode classic for anyone who needs the old Generator API
• all classic code - annotations, attributes, the related pipeline code - plus the bridge and Builder::setMode() marked @deprecated, removed in 8.0
• everything whose marker says 7.0 is removed (LegacyTypeResolver, the relocation shims)
• the v7 migration guide says, with the version, that v8 moves the spec attributes to OpenApi\Attributes (see below) - no surprises

v8 - classic removed

• classic pipeline, OpenApi\Annotations, docblock annotation support and the Generator are gone
• the spec attributes move to OpenApi\Attributes. "Spec" is really a mode name, and with one pipeline left it means nothing - OpenApi\Attributes is the better home. For code written against OpenApi\Spec that is one use line per file (use OpenApi\Attributes as OA; where you had use OpenApi\Spec as OA;), and a rector rule can do it - that hop really is a rename, the classes either side are the same
• if you are still on the classic OpenApi\Attributes at that point: the class names stay but the constructors differ (classic Property takes type:, spec Property takes schema:), so you get unknown-named-parameter errors at generation time rather than missing classes. The v8 migration guide will lead with that; the rector set is the fix

(My July notes below had the namespace move as v9, optional icing. It is v8 now, because there is no reason to make people wait a major for the better name.)

What this means for you

You use OpenApi\Attributes or annotations in your own code. Nothing to do in 6.x. If you are curious, run --mode hybrid (or Builder::setMode(Mode::HYBRID)) over your codebase - the output should be the same, and any warnings it prints are real. When you want the new attributes, the rector set is a 6.x item above; until then the spec attributes guide has the differences.

You maintain a package wrapping swagger-php (a bundle, a framework integration). Hybrid is your compatibility path. The pre-resolve contribution seam above is what lets non-attribute metadata into the pipeline, and it is the item where I would most like to hear what you need. The hooks that exist today are at Extension points.

You build documents in code and never scan. Spec attributes as value objects plus the Compiler do that today; the docs for it are on the list.

You want the smallest possible migration. Stay on classic through v7. The rector set exists before v8 does.

Still to decide

• Default output version at v7. Hybrid defaults to OpenAPI 3.1.0 today, classic to 3.0.0. Flipping the default mode flips the default version for everyone unless v7 keeps 3.0.0 as the default. If your consumer is 3.0-only, say so here.
• OpenApi\Spec kept as deprecated aliases in v8, so the namespace move is optional for one more major?
• Anything else you would need before moving a real project - comments welcome.

[miken32]

I just spent the week learning more than I care to about the OpenAPI spec to document an API. One thing that would have helped make this package easier to use is more use of enums. E.g. a type parameter can only be string, number, array, object, etc. Those should be enumerated rather than just strings.

Another suggestion would be finding shortcuts that can move away from the annotations just being 1:1 mappings to OpenAPI entries. For example: properties: ['foo' => 'string'] for simple declarations, rather than properties: [new OA\Property(property: 'foo', type: 'string')]. Or schema: 'date-string' in place of schema: new OA\Schema(type: 'string', format: 'date').

[DerManoMann]

Enums are definitely a good idea although migration is definitely an issue. As a v7 item that might be doable.

Shortcuts are also good. I've been thinking about the coupling on/off also, in particular since supporting different spec versions with a single set of attributes becomes really awkward when property types change, etc...

So, all good things, thanks @miken32

[DerManoMann]

Is it time to drop docblock support and focus on attributes?

[DerManoMann]

As for shortcuts - maybe the openapi-extras annotations could/should be merged into swagger-php..?

[DerManoMann]

With annotations dropping it would be a good time to re-architecture the internals. in paticular getting rid of the current Context, separating validation and serialization and improving version support (3.0.0, 3.1.0, 3.2.0, ...)
This would also help simplify extending and customizing attributes in a cleaner, less confusing way (merge, nesting rules, etc)

[DerManoMann]

Had lots of spare time on the weekend - there might be a new pipeline coming in v6 still with both current annotations and attributes deprecated. v7 will then be to fine tune and make the new attributes the default, v8 will finally drop annotations and the current set of attributes.

Main design goal is to decouple attributes from the spec - they will be very thin DTOs with everything else happening outside. This also resolved my other gripes with the current architecture in that serialization, logging and validation can be extracted into separate components.

[DerManoMann]

Another update with more of a general overview on where this is heading:

The problem:
The current annotations/attributes do way too much. capture data, validate, serialize and more. This lead to more overloading of the Context class to share things into the attribute tree without using globals (too much).

The idea:
A new set of attributes that are thin and leave validation, serialization, error handling and version support (3.0.x, 3.1.x, 3.2.x, ...) to other parts of the system.

The solution:
v6 introduces a new processing chain alongside the current one. Everything is additive/opt-in - nothing breaks.

One thing that is becoming clear is that the current Generator nad processors contract is no longer suiteable. The won't be any more merging, tracking of attributes in the Analysis and the Context.

New processing chain:

• Thin DTO attributes (OpenApi\Spec\*) - no merge, no nesting rules, no version awareness
• An Assembler collects DTOs into a flat Specification container (the new Analysis)
• Augmenters enrich (type resolution, docblocks, etc - same role as processors today)
• Compilers produce version-specific output (3.0, 3.1, 3.2) - all the isVersion() branching moves here

Start of the planned changes has happened already - some of the generic parts of the current system have already been extracted (the new Utils namespace) and will be re-used. In fact, most of the augment and type handling code will be re-used.

Since the Generator is build around the Analyser/Analysis/processors chain, a new, shared entrypoint is needed. This will be a new fluid Builder style class that will support both the current Generator options and also the new processing chain.
After that, nothing has to change for anyone until we hit v8 :)

Version plan:

• v6: both processing chains, builder ships, deprecation notices on old entry points
• v7: new chain default, classic still available, old namespaces removed
• v8: classic gone (this means current annotations and attributes)
• v9: it might be nice to switch the new attributes back to the OpenApi\Attributes namespace once they old ones are removed in v8, but that is totally optional icing.

Deprecations introduced in v6 will stay functional through v7 inclusive - no cliff.

The new attributes will be loosely shaped around the 3.1 spec, but since serialization is decoupled there is more room for convenience classes, shortcuts and things similar to the Controller attribute in openapi-extras.

The new system should also vastly reduce the overhead for downstream projects as there is no tree and references to manage when modifying / adding attributes.

@DarkaOnLine @DjordyKoert

[DerManoMann]

Quite a few things I am not quite sure about - for example the version changes and deprecations, but I figured adding new stuff in v6 would be safe enough. Maybe v6 will be a very long version and we drop the old bits already in v7?

[DerManoMann]

• drop LegacyTypeResolver (v7)

[DerManoMann]

• mark all classis only code as deprecated, including annotations and classic attributes

[DerManoMann]

Rewrote the body of this issue into the actual roadmap for the spec pipeline across v6/v7/v8 and pinned it. I'll keep it current from here - the checklists get ticked as things ship, and the 6.x release notes will link back. Everything above this comment stays as it was.

Two changes compared with my July notes: the namespace move (OpenApi\Spec -> OpenApi\Attributes) is now v8 rather than v9, and there is a short "still to decide" list at the bottom. The default output version at v7 is the one I'd like input on - if you depend on 3.0.0 output by default, now is the time to say so.