diff --git a/dev/agent-skills/flink-agents-dev/SKILL.md b/dev/agent-skills/flink-agents-dev/SKILL.md new file mode 100644 index 000000000..7405e51f8 --- /dev/null +++ b/dev/agent-skills/flink-agents-dev/SKILL.md @@ -0,0 +1,321 @@ +--- +name: flink-agents-dev +description: Use when building, scaffolding, modifying, debugging, converting, or verifying Apache Flink Agents applications, including Flink Agents YAML, Workflow Agent, ReAct Agent, Actions, Resources, MCP servers, vector stores, runtime skills, Python, or Java. +--- + +# Developing Flink Agents Applications + +## Core Model + +Model every application as four connected parts: + +1. **Resources**: named models, prompts, tools, skills, vector stores, MCP servers, + and connections. +2. **Actions**: event handlers that use Resources and emit events. +3. **Orchestration**: trigger conditions and emitted-event graph. +4. **Implementation**: Python/Java functions, types, and runner. + +Skills, vector stores, MCP, RAG, and memory are combinations of these parts. + +The Agent API language and custom Action/Tool language do not constrain every +Resource implementation. Chat-model connections/setups, embedding-model +connections/setups, and vector stores support Python/Java bridging in both +directions in the bundled snapshot. For those Resource types, offer every +target-version implementation supported natively or through the bridge, regardless +of whether the application uses YAML, direct Python, or direct Java. Treat the +selected implementation's language as part of that Resource choice; do not add a +separate application-wide cross-language confirmation. Do not generalize this rule +to Resource types that lack a verified bridge. + +## Source Authority + +This skill must work from its installed directory; never assume the user cloned the +Flink Agents repository. Resolve bundled paths relative to this `SKILL.md`. + +Inspect the target version and conventions. Use sources in this order: + +1. Target application code, dependency metadata, installed package/JAR APIs, and tests. +2. Version-matching Flink Agents schema, docs, examples, or source when available. +3. This skill's references and [bundled YAML schema](assets/yaml-schema.json) as + the offline baseline. + +The bundled schema and patterns describe the Flink Agents revision that published +this skill. If the target dependency differs, prefer its matching contracts and +state any compatibility assumption that could not be verified. + +Do not invent APIs, versions, or commands from memory. + +Before presenting code as runnable, resolve every nontrivial import, constructor, +method, descriptor argument, and dependency coordinate against those sources. If a +contract cannot be resolved, either use a documented alternative or label the +fragment as pseudocode and state exactly what remains unresolved. Never create an +adapter or wrapper merely to make a speculative API look complete. + +## Definition Strategy + +| Situation | Recommendation | +|---|---| +| New Workflow Agent or rewired event graph | Offer YAML plus Python/Java implementation files | +| Existing YAML application | Preserve and extend YAML | +| Existing programmatic Agent | Preserve its current API unless conversion is requested | +| `ReActAgent`, unsupported YAML surface, or explicit code-only request | Direct Python or Java API | + +For a new application, present YAML, direct Python API, and direct Java API as an +explicit choice after versions are confirmed. A recommendation explains the +tradeoff; it is not permission to select the API for the user. + +YAML does not declare an agent `type`. Do not add `type: workflow` or +`type: react`; `type` selects an implementation language where the schema allows it. + +YAML does not choose the business implementation language either. For a new YAML +application, ask the user to choose Python or Java before generating files. For an +existing application, detect and preserve the language from build metadata, source +files, function references, and explicit YAML `type` fields. Do not default to +Python merely because omitted YAML `type` fields currently resolve to Python. + +Workflow Agents already include built-in chat, tool-call, and context-retrieval +Actions. A model reasoning/tool loop can therefore remain YAML-defined. Choose +`ReActAgent` only when the user explicitly wants that programmatic abstraction, the +existing application already uses it, or a required ReAct-specific surface is not +available through YAML. + +## Scaffolding Boundary + +Complete framework-owned wiring, but do not invent business behavior. Unless the +user explicitly asks for an implementation and supplies its contract, generate only +an importable or compilable signature skeleton for each custom Action, Tool, domain +client, data transformation, and runtime Skill. Preserve user-provided names, +parameters, types, and descriptions. When those details are absent, derive a stable +name from the stated capability and use a minimal neutral signature such as one +opaque string request and string result; mark unresolved input fields and result +types with focused TODOs and fail explicitly with `NotImplementedError` or +`UnsupportedOperationException`. + +Do not author domain rules, REST endpoints, diagnostic procedures, prompts, data +models, Tool results, runtime Skill instructions, or test doubles from a high-level +application idea. Do not propose a standard REST API, MCP server, or mock backend as +the implementation of a custom Tool. Do not ask the user to lock a business input +identity schema, deployment platform, service API, authentication design, or log/ +metric backend merely to scaffold the application. Generate the neutral skeleton +and leave those choices as TODOs for the user. Built-in Flink Agents Actions are +framework behavior and need no generated implementation. + +## Interaction Discipline + +For a new or empty project, use sequential decision gates. Ask only the current +gate, wait for the answer, and then continue. Never send one proposed baseline that +bundles versions, API, implementation language, runtime version, Resource +implementations, business backends, and mock behavior. + +Present every closed set of choices through a native structured single-select UI +when one is available. Before the first closed gate, identify the current host from +explicit system and tool context, then read exactly one matching interaction +adapter: + +- Codex: [platforms/codex.md](references/platforms/codex.md) +- Claude Code: [platforms/claude-code.md](references/platforms/claude-code.md) +- Gemini CLI: [platforms/gemini-cli.md](references/platforms/gemini-cli.md) +- Qoder: [platforms/qoder.md](references/platforms/qoder.md) +- unknown or unsupported host: [platforms/generic.md](references/platforms/generic.md) + +Treat the tools exposed in the current session and mode as authoritative. A platform +adapter may name an optional host tool, but the core workflow must not require that +tool, install host-specific metadata, or add host configuration to the generated +application. Follow any mode prerequisite defined by the selected adapter before +falling back. In particular, the Codex adapter pauses and asks the user to enter +Plan mode before the first gate so Codex can expose its native selector, then pauses +again after the final gate and asks the user to return to Default execution mode +before any implementation work. If the matching native tool remains unavailable +after the adapter's prerequisite or errors, follow the adapter's fallback rule. +Never use an open-ended text question when the valid options are already known. When +a gate has a recommendation, place it first and label it `(Recommended)`, but do not +preselect or continue without the user's answer. The YAML implementation-language +gate intentionally has no recommendation: present Python and Java as equal peer +options with parallel descriptions and no `(Recommended)` label. + +Use this order: + +1. Ask for the Flink Agents version. Wait. Then offer only compatible Flink versions + and wait for the Flink choice. Do not mention Python/JDK versions, an Agent API, + a model provider, or business architecture in these version questions. +2. Ask the user to choose YAML API, direct Python API, or direct Java API. Wait. +3. Only when YAML is selected, ask whether custom Actions, function Tools, and the + Flink entry point use Python or Java. Resource implementations are selected + independently at their own gates. Direct Python or Java API already determines + the application-code choice. Do not recommend, preselect, or imply a preference + for either YAML implementation language. Only now resolve a compatible Python or + JDK version. +4. Inventory the Resources required by the user's stated design. Resolve real + descriptor-backed framework integrations one at a time and wait before moving to + the next integration. Assign a deterministic internal name, ask for its + implementation class or documented alias, and stop the interview for that + Resource. Inspect the target-version constructor/descriptor and generate every + mandatory configuration key as `TODO_REQUIRED_` for the user to fill. + For chat-model connections/setups, embedding-model connections/setups, and vector + stores, do not filter candidates by the Agent API or application-code language. + Present all verified Python and Java implementations, label each candidate's + implementation language when needed for disambiguation, and generate the matching + bridge declaration and runtime dependencies after selection. + Do not ask for model identifiers, endpoints, credential values/mechanisms, + provider options, Skill source paths/URLs/packages, or other Resource arguments. + Custom Actions, function Tools, domain clients, Prompt content, runtime Skill + instructions, business input schemas, and backend platforms are not integration + gates: scaffold them without a business interview. Do not ask the user to name a + single Resource or repeat a generated reference. Ask about naming only when + multiple Resources need semantic disambiguation, an existing external reference + constrains the name, or the user requested a naming convention. +5. As soon as the confirmed design first requires a Python runtime, pause before + environment creation or dependency installation. Unless the existing project + already declares its environment unambiguously, inspect compatible local Python + executables/environments and ask whether to reuse one of them or create a + project-local `.venv`. Wait for the choice and use the selected interpreter for + every install, import, test, and local run. This conditional gate fires + immediately after the choice that introduces Python; do not delay it until the + end of the Resource interview. +6. After every framework decision is confirmed, complete any post-interview handoff + required by the selected host adapter. Once the host is in an execution-capable + mode, design the Action graph and generate the project. + +For Codex, completing the gates does not itself authorize implementation in Plan +mode. Follow the Codex adapter's execution-mode handoff, stop for the user to switch +back, and begin file edits only after the session continues in Default mode. Other +hosts follow their own adapter and need no Codex-specific mode prompt. + +Do not infer OpenAI, Ollama, any model name, an environment-variable credential, +Skill distribution, MCP, a vector store, or a domain-service protocol. After a +framework implementation is selected, do not run a second configuration interview. +Generate a declaration or builder scaffold that names every verified mandatory +argument and leaves its value explicit for the user. For YAML string fields, use a +clear placeholder such as `TODO_REQUIRED_API_KEY`; for direct APIs, generate a +compilable factory skeleton that lists the required arguments and fails explicitly +until they are filled. If the user already supplied a value or explicitly requested +plaintext in local YAML, use that instruction; keep supplied secrets out of tracked +files and output. Never claim `${ENV_VAR}` is interpolated unless the target loader +or provider actually implements it. + +The conditional Python-environment gate is dependency management, not Resource +configuration. It still runs when a selected Resource introduces Python, but it asks +only which concrete compatible Python environment to use, never provider values. + +## Build Workflow + +1. Inspect the project, build metadata, existing definitions, and confirmed user + requirements. Preserve explicit existing decisions. Read + [application patterns](references/application-patterns.md). +2. For a new project, complete the version gate in + [local development](references/local-development.md): Flink Agents first, then a + compatible Flink version. Do not generate files or combine later decisions into + this question. +3. Complete the API gate: YAML, direct Python, or direct Java. For YAML, read + [YAML patterns](references/yaml-patterns.md) and use the + [bundled YAML schema](assets/yaml-schema.json) unless a target-version schema is + available. +4. If YAML was selected, complete the implementation-language gate. Then read + [Python patterns](references/python-patterns.md) or + [Java patterns](references/java-patterns.md) for the application code. When a + bridge-supported Resource uses the other language, also read that language's + cross-language Resource section; selecting that Resource is already explicit + confirmation. +5. Inventory each Resource and assign stable names automatically. Ask only which + documented implementation alias/class to use for actual framework integrations. + For bridge-supported types, build this choice from both Python and Java + implementations instead of the application-code language alone. Inspect the + selected implementation, add its integration and bridge dependencies, and + scaffold all mandatory configuration keys without asking for their values. + Generate custom + Tool/Action/Prompt/runtime-Skill business scaffolds without asking the user to + choose a domain platform or complete an input contract; generate and record all + cross-references without asking the user to repeat internal identifiers. Never + select a provider, model, endpoint, credential source, business backend, or + integration from the application domain alone. +6. When runtime Skills are requested, preserve an explicit source in an existing + application. For a new application, generate a minimal runtime `SKILL.md` business + scaffold plus a source-configuration TODO that lists the valid `paths`, `urls`, + Python `package`, and Java `classpath` forms. Do not ask for loading paths or + distribution, and never inspect, copy, or offer to reuse Skills installed in the + coding-agent host, such as a local `flink-diag`; coding-agent Skills and Flink + runtime Skills are different artifacts. Read + [YAML patterns](references/yaml-patterns.md#runtime-skills). +7. Draw the framework Action graph from input through Resource interactions to + output and errors. Use TODO-marked opaque payloads where business event fields or + branching semantics are unspecified; do not turn those TODOs into more gates. +8. Use only documented built-in Actions. Generate a resolvable, correctly typed + signature skeleton for every other Action and reference it as + `:`; leave its business body for the user unless they + explicitly requested implementation. +9. Resolve the concrete API calls and dependency coordinates before generating the + complete executable project. For Java, generate a Maven project with + `flink-agents-api`, `flink-agents-plan`, `flink-agents-runtime`, and only the + integrations actually used; declare all Flink Agents and Flink dependencies as + `provided`. Whenever the design requires Python, generate source and dependency + files, resolve the Python environment choice when not already declared, and + install the resolved Flink Agents, PyFlink, and integration dependencies into the + selected existing environment or project-local `.venv`. Read + [local development](references/local-development.md). Preserve versions already + selected by the target project; never guess one. +10. Connect the Agent to a Flink DataStream or Table through the public factory backed + by `RemoteExecutionEnvironment`. Local validation submits that same remote-style + job to a MiniCluster. Never use a local Agents environment, a no-argument factory, + `from_list`/`to_list`, or their Java equivalents. +11. Run the checks in [verification](references/verification.md) before claiming + the application is valid or runnable. + +## Required Output + +- Complete framework files or edits, including every custom function signature and + a remote-style Flink job entry point. +- A runnable Maven project for Java and, whenever Python is required, pinned Python + dependency input plus the user-selected existing environment or populated + project-local `.venv`. +- User-confirmed Flink Agents and Flink versions. Never silently choose the bundled + recommended versions for a new project. +- A user-confirmed YAML, direct Python, or direct Java API choice for every new + application. +- A user-confirmed Python or Java application-code language for every new YAML + application, reflected consistently in project layout, custom Action/Tool function + references, and the Flink entry point. Do not use it to filter bridge-supported + Resource implementations. +- For applications using runtime Skills, a user-fillable Skill business scaffold + and source-configuration TODO. Preserve existing source configuration, but do not + ask a new-project user to choose distribution or reuse coding-agent host Skills. +- Resource declarations built from user-selected implementation aliases/classes, + including independent Resource implementation language and bridge wiring where + supported, with every verified mandatory provider key present and marked for user + input. + Keep model, endpoint, credential, and optional provider values unresolved instead + of interviewing for them or choosing defaults. +- Deterministic Resource names and references generated by the coding agent. Do not + require the user to name ordinary Connection, Setup, Prompt, Skill container, + VectorStore, or MCP Resource identifiers when there is no ambiguity. +- When the user chooses plaintext credentials for local testing, a loader-compatible + local YAML that is excluded from version control and actually used by the local + run command. Do not replace this explicit choice with programmatic registration. +- Explicit user-fillable business skeletons for custom Actions, Tools, prompts, + runtime Skills, domain clients, secrets, endpoints, and external data. Keep + framework wiring and signatures concrete instead of replacing them with a + checklist or fabricated business implementation. +- A final consolidated `User must provide` list for unresolved business input + fields, platform clients, authentication, queries, response mapping, and domain + behavior. These items must not block project scaffolding or become Agent workflow + decisions unless the user explicitly asks for their implementation. +- Exact commands that match the target repository's build tooling; no guessed + package, Flink, provider, model, or plugin versions. +- Framework snippets whose imports and API calls resolve in the target version. + Business skeletons must import or compile, fail explicitly when invoked, and be + labeled user-fillable rather than runnable behavior. +- Evidence separated into schema, load/compile, tests, and runtime. +- A clear statement for every check that was not run or requires an external service. + +## Quick Reference + +| Task | Read | +|---|---| +| Present closed choices in the current coding agent | One matching file under [platforms/](references/platforms/generic.md) | +| Select Agent/API shape; design Resources and event graph | [application-patterns.md](references/application-patterns.md) | +| Author or review YAML; resolve names and functions | [yaml-patterns.md](references/yaml-patterns.md) | +| Scaffold runtime Skill business and source TODOs | [runtime Skills](references/application-patterns.md#scaffold-runtime-skills) | +| Scaffold Python Actions, Tools, types, or runner | [python-patterns.md](references/python-patterns.md) | +| Scaffold Java Actions, Tools, resources, or runner | [java-patterns.md](references/java-patterns.md) | +| Select versions, generate dependencies, and submit to MiniCluster | [local-development.md](references/local-development.md) | +| Validate schema, references, imports, compilation, and execution claims | [verification.md](references/verification.md) | +| Validate YAML without a source checkout | [bundled YAML schema](assets/yaml-schema.json) | diff --git a/dev/agent-skills/flink-agents-dev/assets/yaml-schema.json b/dev/agent-skills/flink-agents-dev/assets/yaml-schema.json new file mode 100644 index 000000000..183cc7ac8 --- /dev/null +++ b/dev/agent-skills/flink-agents-dev/assets/yaml-schema.json @@ -0,0 +1,533 @@ +{ + "$defs": { + "ActionSpec": { + "additionalProperties": false, + "description": "An action references a user function and its trigger conditions.\n\n``function`` is written as ``:`` \u2014 the\ncolon separates the Python module (or Java class FQN) from the\nattribute path inside it.\n\n``trigger_conditions`` carries one or more strings. Each is either an\nevent-type name (bare identifier) or a future condition-expression\nform \u2014 the runtime classifies the string when it loads the plan.\n\nAction signatures are fixed (``(Event, RunnerContext)``), so there is\nno ``parameter_types`` knob \u2014 Python doesn't need it, and the Java\naction signature is determined by the action contract.", + "properties": { + "config": { + "anyOf": [ + { + "additionalProperties": true, + "type": "object" + }, + { + "type": "null" + } + ], + "default": null, + "title": "Config" + }, + "function": { + "anyOf": [ + { + "type": "string" + }, + { + "type": "null" + } + ], + "default": null, + "title": "Function" + }, + "name": { + "title": "Name", + "type": "string" + }, + "trigger_conditions": { + "items": { + "type": "string" + }, + "minItems": 1, + "title": "Trigger Conditions", + "type": "array" + }, + "type": { + "anyOf": [ + { + "enum": [ + "python", + "java" + ], + "type": "string" + }, + { + "type": "null" + } + ], + "default": null, + "title": "Type" + } + }, + "required": [ + "name", + "trigger_conditions" + ], + "title": "ActionSpec", + "type": "object" + }, + "AgentSpec": { + "additionalProperties": false, + "description": "One agent inside a YAML file's ``agents:`` list.\n\nHolds the agent's own resources and actions. Resources/actions declared\nat the file level (siblings of ``agents:``) are merged in by the loader.", + "properties": { + "actions": { + "items": { + "anyOf": [ + { + "$ref": "#/$defs/ActionSpec" + }, + { + "type": "string" + } + ] + }, + "title": "Actions", + "type": "array" + }, + "chat_model_connections": { + "items": { + "$ref": "#/$defs/DescriptorSpec" + }, + "title": "Chat Model Connections", + "type": "array" + }, + "chat_model_setups": { + "items": { + "$ref": "#/$defs/DescriptorSpec" + }, + "title": "Chat Model Setups", + "type": "array" + }, + "description": { + "anyOf": [ + { + "type": "string" + }, + { + "type": "null" + } + ], + "default": null, + "title": "Description" + }, + "embedding_model_connections": { + "items": { + "$ref": "#/$defs/DescriptorSpec" + }, + "title": "Embedding Model Connections", + "type": "array" + }, + "embedding_model_setups": { + "items": { + "$ref": "#/$defs/DescriptorSpec" + }, + "title": "Embedding Model Setups", + "type": "array" + }, + "mcp_servers": { + "items": { + "$ref": "#/$defs/DescriptorSpec" + }, + "title": "Mcp Servers", + "type": "array" + }, + "name": { + "title": "Name", + "type": "string" + }, + "prompts": { + "items": { + "$ref": "#/$defs/PromptSpec" + }, + "title": "Prompts", + "type": "array" + }, + "skills": { + "items": { + "$ref": "#/$defs/SkillsSpec" + }, + "title": "Skills", + "type": "array" + }, + "tools": { + "items": { + "$ref": "#/$defs/ToolSpec" + }, + "title": "Tools", + "type": "array" + }, + "vector_stores": { + "items": { + "$ref": "#/$defs/DescriptorSpec" + }, + "title": "Vector Stores", + "type": "array" + } + }, + "required": [ + "name" + ], + "title": "AgentSpec", + "type": "object" + }, + "DescriptorSpec": { + "additionalProperties": true, + "description": "Schema for any ResourceDescriptor-backed resource.\n\nRequired: ``name`` and ``clazz``. ``type`` selects the implementation\nlanguage (``\"python\"`` or ``\"java\"``; ``None`` means Python). All\nremaining fields are forwarded verbatim to ``ResourceDescriptor`` as\nkwargs (or as the Java wrapper's kwargs when ``type: java``); the\nforwarding and language-aware wrapping is done by ``loader._build_descriptor``.", + "properties": { + "clazz": { + "title": "Clazz", + "type": "string" + }, + "name": { + "title": "Name", + "type": "string" + }, + "type": { + "anyOf": [ + { + "enum": [ + "python", + "java" + ], + "type": "string" + }, + { + "type": "null" + } + ], + "default": null, + "title": "Type" + } + }, + "required": [ + "name", + "clazz" + ], + "title": "DescriptorSpec", + "type": "object" + }, + "InjectedArg": { + "description": "Declarative source binding for a framework-injected tool parameter.", + "properties": { + "key": { + "anyOf": [ + { + "type": "string" + }, + { + "type": "null" + } + ], + "default": null, + "title": "Key" + }, + "source": { + "$ref": "#/$defs/ToolParameterSource", + "default": "sensory_memory" + } + }, + "title": "InjectedArg", + "type": "object" + }, + "MessageRole": { + "description": "Role of a message in a chat conversation.", + "enum": [ + "system", + "user", + "assistant", + "tool" + ], + "title": "MessageRole", + "type": "string" + }, + "PackageSkillSpec": { + "additionalProperties": false, + "description": "A single ``package`` skill source entry: a Python package name plus a\nresource path relative to that package's root.", + "properties": { + "package": { + "title": "Package", + "type": "string" + }, + "resource": { + "title": "Resource", + "type": "string" + } + }, + "required": [ + "package", + "resource" + ], + "title": "PackageSkillSpec", + "type": "object" + }, + "PromptMessage": { + "additionalProperties": false, + "description": "One message in a multi-turn prompt template.", + "properties": { + "content": { + "title": "Content", + "type": "string" + }, + "role": { + "$ref": "#/$defs/MessageRole", + "default": "user" + } + }, + "required": [ + "content" + ], + "title": "PromptMessage", + "type": "object" + }, + "PromptSpec": { + "additionalProperties": false, + "description": "Declarative prompt: either a single ``text`` template or a list of\nrole-tagged ``messages``. Exactly one of the two fields must be set.", + "properties": { + "messages": { + "anyOf": [ + { + "items": { + "$ref": "#/$defs/PromptMessage" + }, + "type": "array" + }, + { + "type": "null" + } + ], + "default": null, + "title": "Messages" + }, + "name": { + "title": "Name", + "type": "string" + }, + "text": { + "anyOf": [ + { + "type": "string" + }, + { + "type": "null" + } + ], + "default": null, + "title": "Text" + } + }, + "required": [ + "name" + ], + "title": "PromptSpec", + "type": "object" + }, + "SkillsSpec": { + "additionalProperties": false, + "description": "Declarative Skills resource: one or more skill sources grouped by scheme.\n\nEach list below maps to a skill source scheme:\n\n- ``paths`` \u2014 ``local`` scheme: directories or ``.zip`` files\n- ``urls`` \u2014 ``url`` scheme: ``http(s)`` URLs pointing to a ``.zip``\n- ``classpath`` \u2014 ``classpath`` scheme (Java-only at runtime): resource\n paths on the Java classpath\n- ``package`` \u2014 ``package`` scheme (Python-only at runtime): resources\n inside installed Python packages, given as ``{package, resource}`` pairs\n\nAt least one of the four must be non-empty. ``classpath`` is exposed on\nPython for YAML schema parity with Java \u2014 it deserializes successfully\nbut ``SkillManager`` on Python will fail at load time because Python does\nnot register a ``classpath`` handler.", + "properties": { + "classpath": { + "items": { + "type": "string" + }, + "title": "Classpath", + "type": "array" + }, + "name": { + "title": "Name", + "type": "string" + }, + "package": { + "items": { + "$ref": "#/$defs/PackageSkillSpec" + }, + "title": "Package", + "type": "array" + }, + "paths": { + "items": { + "type": "string" + }, + "title": "Paths", + "type": "array" + }, + "urls": { + "items": { + "type": "string" + }, + "title": "Urls", + "type": "array" + } + }, + "required": [ + "name" + ], + "title": "SkillsSpec", + "type": "object" + }, + "ToolParameterSource": { + "description": "Source for a framework-injected tool parameter.", + "enum": [ + "config", + "sensory_memory", + "short_term_memory" + ], + "title": "ToolParameterSource", + "type": "string" + }, + "ToolSpec": { + "additionalProperties": false, + "description": "Points ``function:`` at a callable tool.\n\n``function`` is written as ``:`` \u2014 the\ncolon separates the Python module (or Java class FQN) from the\nattribute path inside it. For Python, the right side may be a\nnested ``Class.method``.\n\n``parameter_types`` is required when ``type: java`` and is forbidden\notherwise (Python tools are reflected from the callable signature).\nThe list contains one string per declared parameter of the Java\nmethod, in declaration order \u2014 the loader uses it to disambiguate\noverloaded methods on the Java class. Each string is one of:\n\n- A Java primitive name: one of ``boolean``, ``byte``, ``short``,\n ``int``, ``long``, ``float``, ``double``, ``char``.\n- A fully-qualified Java reference type (including boxed\n primitives), e.g. ``java.lang.Double``, ``java.lang.String``,\n ``java.util.List``.\n\nGeneric type arguments are not part of the JVM method descriptor\nand must not be included (``java.util.List``, not\n``java.util.List``).", + "properties": { + "function": { + "anyOf": [ + { + "type": "string" + }, + { + "type": "null" + } + ], + "default": null, + "title": "Function" + }, + "injected_args": { + "additionalProperties": { + "anyOf": [ + { + "$ref": "#/$defs/InjectedArg" + }, + { + "additionalProperties": true, + "type": "object" + } + ] + }, + "title": "Injected Args", + "type": "object" + }, + "name": { + "title": "Name", + "type": "string" + }, + "parameter_types": { + "anyOf": [ + { + "items": { + "type": "string" + }, + "type": "array" + }, + { + "type": "null" + } + ], + "default": null, + "title": "Parameter Types" + }, + "type": { + "anyOf": [ + { + "enum": [ + "python", + "java" + ], + "type": "string" + }, + { + "type": "null" + } + ], + "default": null, + "title": "Type" + } + }, + "required": [ + "name" + ], + "title": "ToolSpec", + "type": "object" + } + }, + "additionalProperties": false, + "description": "Top-level YAML document.\n\nAgents are declared under ``agents:``. The block is optional, so a\nfile may carry only shared infrastructure (chat-model connections,\nvector stores, ...) \u2014 useful for splitting a topology file from an\ninfrastructure file that can be swapped per environment. Resources\nand actions declared at the same level as ``agents:`` are shared:\nresources are registered on the environment; actions can be\nreferenced from any agent by name string.", + "properties": { + "actions": { + "items": { + "$ref": "#/$defs/ActionSpec" + }, + "title": "Actions", + "type": "array" + }, + "agents": { + "items": { + "$ref": "#/$defs/AgentSpec" + }, + "title": "Agents", + "type": "array" + }, + "chat_model_connections": { + "items": { + "$ref": "#/$defs/DescriptorSpec" + }, + "title": "Chat Model Connections", + "type": "array" + }, + "chat_model_setups": { + "items": { + "$ref": "#/$defs/DescriptorSpec" + }, + "title": "Chat Model Setups", + "type": "array" + }, + "embedding_model_connections": { + "items": { + "$ref": "#/$defs/DescriptorSpec" + }, + "title": "Embedding Model Connections", + "type": "array" + }, + "embedding_model_setups": { + "items": { + "$ref": "#/$defs/DescriptorSpec" + }, + "title": "Embedding Model Setups", + "type": "array" + }, + "mcp_servers": { + "items": { + "$ref": "#/$defs/DescriptorSpec" + }, + "title": "Mcp Servers", + "type": "array" + }, + "prompts": { + "items": { + "$ref": "#/$defs/PromptSpec" + }, + "title": "Prompts", + "type": "array" + }, + "skills": { + "items": { + "$ref": "#/$defs/SkillsSpec" + }, + "title": "Skills", + "type": "array" + }, + "tools": { + "items": { + "$ref": "#/$defs/ToolSpec" + }, + "title": "Tools", + "type": "array" + }, + "vector_stores": { + "items": { + "$ref": "#/$defs/DescriptorSpec" + }, + "title": "Vector Stores", + "type": "array" + } + }, + "title": "YamlAgentsDocument", + "type": "object" +} diff --git a/dev/agent-skills/flink-agents-dev/references/application-patterns.md b/dev/agent-skills/flink-agents-dev/references/application-patterns.md new file mode 100644 index 000000000..527c999c1 --- /dev/null +++ b/dev/agent-skills/flink-agents-dev/references/application-patterns.md @@ -0,0 +1,458 @@ +# Application Patterns + +## Contents + +- [Portable Baseline](#portable-baseline) +- [Run a Gated Interview](#run-a-gated-interview) +- [Present Closed Choices Portably](#present-closed-choices-portably) +- [Choose the API and Agent Form](#choose-the-api-and-agent-form) +- [Choose the Implementation Language](#choose-the-implementation-language) +- [Resource Inventory](#resource-inventory) +- [Resolve Resources One at a Time](#resolve-resources-one-at-a-time) +- [Scaffold Runtime Skills](#scaffold-runtime-skills) +- [Design the Action Graph](#design-the-action-graph) +- [Built-in Action Boundaries](#built-in-action-boundaries) +- [Separate Framework and Business Code](#separate-framework-and-business-code) +- [Minimal Layouts](#minimal-layouts) + +## Portable Baseline + +Do not require a Flink Agents source checkout. This installed skill contains: + +- `references/application-patterns.md`: Agent selection, Resources, Actions, and graph design; +- `references/yaml-patterns.md`: complete YAML structure and naming rules; +- `references/python-patterns.md`: Python Action, Tool, event, and runner contracts; +- `references/java-patterns.md`: Java function, Resource, classpath, and runner contracts; +- `references/local-development.md`: Maven/venv generation and local Flink execution; +- `references/verification.md`: schema, reference, compile, and runtime checks; +- [`assets/yaml-schema.json`](../assets/yaml-schema.json): a machine-readable offline YAML contract. + +These files are a coherent snapshot and are sufficient for the default workflow. +When the target application pins another Flink Agents version, inspect that installed +Python package, Java dependency/source JAR, released docs, or source tag for changed +constructors and provider arguments. Prefer the target-version contract over the +snapshot and report the mismatch; never require cloning the repository as a setup +step. + +## Run a Gated Interview + +An empty workspace provides no evidence for language, API, provider, credentials, +Skill source, or business integrations. Do not answer that uncertainty with a +combined recommended baseline. Complete and wait at each gate: + +| Gate | Ask now | Do not ask or assume yet | +|---|---|---| +| Versions | Flink Agents, then compatible Flink | API, Python/JDK, Resources, business backend | +| API | YAML, direct Python, or direct Java | YAML implementation language, providers, credentials | +| Language/runtime | Python or Java as equal choices only for YAML; then compatible interpreter/JDK | Resource implementations | +| Resources | One descriptor-backed framework implementation; generate its required arguments as TODOs | Another integration or any custom business body | +| Python environment | When Python first becomes required: compatible existing environment or project `.venv` | Dependency installation before the choice | +| Application | Action graph and custom signatures | Unspecified domain logic or test doubles | + +Skip a gate only when the user already answered it or an existing project declares +it unambiguously. A recommended option must be presented inside its own gate and +must not silently answer later gates. + +## Present Closed Choices Portably + +For version, API, YAML implementation language, runtime, and framework Resource +implementation gates, the valid set is known. Use the single host adapter selected +by `SKILL.md`. +Platform-specific tool names belong only in `references/platforms/`; they must not +leak into the generated application or become a prerequisite for using the core +skill. Treat the current session's actual tool contract as authoritative instead of +assuming a tool exists because of the host product name. + +Follow the selected adapter's mode prerequisite before deciding that its native +tool is unavailable. For Codex, pause before the first gate and ask the user to +enter Plan mode with `/plan` or `Shift+Tab`; do not render a text menu while Codex +is still in Default mode. When the matching native tool remains unavailable after +the adapter prerequisite or rejects the call, use the generic portable fallback +without repeatedly retrying the unavailable tool: + +```text +Select the API: +1. YAML API (Recommended) +2. Direct Python API +3. Direct Java API + +Reply with 1, 2, or 3. +``` + +Keep one decision gate per selector. When that gate has a recommendation, put the +recommended option first and label it, but do not mark it selected or proceed on +timeout/silence. The YAML implementation-language gate has no recommended option. +For a target-version list of Resource aliases/providers, offer every implementation +verified as compatible with the selected API either natively or through a supported +cross-language bridge. Do not filter chat-model, embedding-model, or vector-store +implementations by the application-code language. Include a custom class option only +when the framework supports one. Ask free-form text only for genuinely open values +such as a custom class name, model identifier, endpoint, or user-defined Tool +signature. Generate internal Resource names unless the user requested a naming +convention or multiple Resources cannot be distinguished from confirmed context. + +## Choose the API and Agent Form + +After the version pair is confirmed, ask the user to choose: + +| API choice | Consequence | +|---|---| +| YAML API | Declarative Workflow Agent; ask Python or Java implementation language next | +| Direct Python API | Python implementation and PyFlink entry point | +| Direct Java API | Java implementation and Java Flink entry point | + +YAML is the recommended option for a new Workflow Agent because the schema can +validate the wiring, but do not select it without the user's answer. Direct API does +not imply `ReActAgent`; both Python and Java can define Workflow Agents +programmatically. + +Use a Workflow Agent when the user needs an explicit event graph, deterministic +stages, custom state transitions, branching, joins, or direct Resource access. + +Workflow Agents can also provide a model reasoning/tool loop using the built-in +chat and tool-call Actions. Add the built-in context-retrieval flow when reasoning +needs retrieved context. Do not switch to `ReActAgent` merely because the +application uses model reasoning, Tools, RAG, MCP Tools, or runtime Skills. + +Use the programmatic `ReActAgent` when the user explicitly requests that +opinionated abstraction, the existing application already uses it, or a required +ReAct-specific surface is unavailable through YAML. The YAML schema has no +`type: react` discriminator. Do not simulate one with an unsupported field. + +Preserve an existing definition style unless the user requests conversion or the +current style blocks the requested capability. + +## Choose the Implementation Language + +YAML defines the Agent graph but does not make custom code language-neutral. Only +after the user selects YAML, ask them to choose one implementation language before +creating files: + +| Choice | Generated contracts | +|---|---| +| Python | Python module references for custom Actions/Tools, a user-selected Python environment, and a PyFlink entry point | +| Java | Java class/method references for custom Actions/Tools, Tool `parameter_types`, Maven, and a Java Flink entry point | + +Present these as two equal peer options. Use parallel factual descriptions, do not +append `(Recommended)` to either label, do not preselect either language, and do not +describe list order as a preference. The user must make the choice even when one +language happens to be installed locally. + +Do not mention or choose a Python/JDK version before this language choice. Do not +choose Python because the coding agent runs in Python or because missing YAML `type` +defaults to Python. Do not choose Java merely because Flink runs on the JVM. The +selected language constrains application source layout, runtime compatibility, and +local tooling, but it does not constrain bridge-supported chat-model, +embedding-model, or vector-store implementations. Resolve the compatible interpreter +or JDK immediately after the application-code language is confirmed. + +The Python environment is a separate conditional gate. It also applies when the +application code is Java but a selected Resource implementation or verification +step requires Python. Detect compatible local Python executables and environments, +then ask whether to reuse a specific detected environment or create a project-local +`.venv`. Do not create an environment or install dependencies before the user +chooses. Preserve an existing project environment when project metadata declares it +unambiguously. + +For an existing application, detect and preserve its language from `pom.xml`, Python +dependency metadata, source layout, function references, and explicit YAML `type` +fields on custom Actions and Tools. If those application-code signals conflict, +surface the conflict and confirm the boundary with the user. A chat-model, +embedding-model, or vector-store implementation in the other language is an +independent bridged Resource choice, not an application-language conflict and not a +reason for another confirmation gate. + +## Resource Inventory + +After the API and language gates, build a name table before writing Actions or YAML. +Include only Resources required by the user's stated design; do not add a model, +Skills, MCP, vector store, or Tool because it appeared in an example: + +| Resource | Typical references | +|---|---| +| `chat_model_connections` | `chat_model_setups[].connection` | +| `chat_model_setups` | `ChatRequestEvent.model`; runner/Agent configuration | +| `prompts` | `chat_model_setups[].prompt` | +| `tools` | `chat_model_setups[].tools`; direct `ToolRequestEvent` | +| `skills` | available Skill sources; individual names in chat model `skills` | +| `embedding_model_connections` | `embedding_model_setups[].connection` | +| `embedding_model_setups` | vector-store `embedding_model` argument | +| `vector_stores` | Action lookup; `ContextRetrievalRequestEvent` | +| `mcp_servers` | dynamically discovered MCP prompts and Tools | + +Treat each name as an API. Keep spelling and case identical at declaration and +every reference. Separate a Skills Resource name from the individual Skill names +discovered from its `SKILL.md` files. + +For a new application, naming is coding-agent work rather than a user decision. +Generate deterministic `snake_case` identifiers and wire every reference to them. +Use these defaults when there is one Resource of that role: + +| Resource | Default name | +|---|---| +| Chat-model connection | `chat_model_connection` | +| Chat-model setup | `chat_model` | +| Embedding-model connection | `embedding_model_connection` | +| Embedding-model setup | `embedding_model` | +| Prompt | `_prompt` | +| Tool | snake_case of the confirmed function or capability | +| Skills Resource | `runtime_skills` | +| Vector store | `vector_store` | +| MCP server | `mcp_server` | + +When there are multiple Resources of one role, derive a meaningful provider or +purpose stem from already confirmed information. Ask the user for naming input only +when that information cannot disambiguate them, an existing external reference +requires an exact name, or the user explicitly requested a naming convention. +Preserve existing names during modifications. Report generated names, but do not +pause for confirmation. + +## Resolve Resources One at a Time + +Do not ask the user to approve a complete Resource stack. Select one Resource from +the inventory, resolve it fully, and wait before moving to the next one. Follow its +dependency edges: configure a connection before the setup that names it, and +configure the Prompt, Tools, or Skills before finalizing a setup that references +them. + +For each Resource, generate its name and references. Ask only which documented +framework implementation/alias to use; inspect that target-version implementation +and generate all mandatory configuration fields as TODOs. Function Tools, Prompt +content, runtime Skill instructions, and Skill source configuration are scaffolds, +not provider-integration interviews: + +| Resource | Generate | Ask for | +|---|---|---| +| Chat/embedding connection | Stable name plus every mandatory provider field marked `TODO_REQUIRED_*` | Documented `clazz`/alias only | +| Chat/embedding setup | Stable name, connection reference, and mandatory setup fields marked `TODO_REQUIRED_*` | Documented `clazz`/alias only | +| Prompt | Purpose-derived name and TODO scaffold | Nothing unless the user already supplied content or explicitly requests prompt implementation | +| Function Tool | Capability-derived name, minimal neutral signature, and failing body | Nothing unless the user already supplied a signature or explicitly requests implementation | +| Skills | `runtime_skills`, minimal Skill metadata, TODO instructions, and source-configuration TODO | Nothing; do not interview for source paths or business instructions | +| Vector store | Stable name, embedding-setup reference, and mandatory backend fields marked `TODO_REQUIRED_*` | Documented implementation only | +| MCP server | Stable name and mandatory transport/authentication fields marked `TODO_REQUIRED_*` | Documented implementation only | + +Choosing a Resource implementation determines which integration artifact and +arguments are valid. For chat-model connections/setups, embedding-model +connections/setups, and vector stores, build the candidate set from both Python and +Java implementations supported by the target version. Label otherwise ambiguous +choices with their implementation language, for example `Ollama (Python)` and +`Ollama (Java)`. Selecting one candidate is sufficient confirmation of the Resource +language; do not ask a separate cross-language question. Read the matching +target-version docs or installed API after the user names the implementation; do not +propose OpenAI, Ollama, a model name, or a provider alias from the application +domain. + +Generate the corresponding bridge form instead of rewriting the provider into the +application language. In YAML, set each descriptor's `type` from that Resource's +implementation language. In direct Python, use the documented Java wrapper and +`java_clazz` metadata for a Java implementation. In direct Java, use the documented +Python wrapper and `pythonClazz` metadata for a Python implementation. Add the +selected implementation artifact and bridge runtime requirements to the generated +project. Keep connection and setup implementations compatible with one another. + +Do not ask for values after the implementation is selected. Put every verified +mandatory key in the generated declaration and use an unmistakable placeholder, +for example `model: TODO_REQUIRED_MODEL` or `api_key: TODO_REQUIRED_API_KEY`. +Include concise comments that identify required types or constraints when the +target-version API exposes them. Do not invent optional values. If the required +arguments cannot be verified, add `TODO_VERIFY_REQUIRED_PROVIDER_ARGUMENTS` and +report the documentation gap instead of asking the user to design the provider. + +An API-key placeholder is not a secret and may remain in a tracked template. If the +user explicitly supplies a real value or asks to keep plaintext in local YAML, put +the value only in an ignored local file and redact output. The lack of general +`${ENV_VAR}` interpolation does not justify another interview; simply document it +and leave the supported credential field for the user to fill. + +A custom Tool, Action, or domain client is business code. When its contract is not +already supplied, derive a name from the user's stated capability and generate a +minimal framework-compatible skeleton using opaque inputs and outputs. Do not ask +the user to choose structured identity fields, Flink/VVR/VVP platform variants, +service endpoints, authentication, log/metric APIs, or response schemas merely to +finish scaffolding. Record each unknown as a TODO and consolidate it in the final +`User must provide` list. Do not generate a fake or mock implementation unless the +user explicitly requests that behavior. + +## Scaffold Runtime Skills + +Runtime Skill instructions and deployment are user-owned. A coding-agent Skill +installed in Codex, Claude Code, Qoder, Gemini CLI, or another host is not a Flink +runtime Skill artifact. Never inspect host Skill directories, ask whether to reuse a +local Skill such as `flink-diag`, or copy its content into the generated application. + +Preserve an existing YAML source field or `Skills` factory. For a new application, +do not ask how Skills will be loaded. Generate: + +- a minimal runtime `SKILL.md` with capability-derived metadata and a TODO body; +- a `runtime_skills` Resource/source scaffold that clearly requires user input; +- a final TODO listing the supported source forms and deployment requirements. + +The supported forms are reference information for the user's later edit, not +selection options in the scaffolding interview: + +| Source form | YAML source | Direct API | Deployment contract | +|---|---|---|---| +| Bundle with the application | Python `package`; Java `classpath` | Python `Skills.from_package`; Java `Skills.fromClasspath` | Version the Skills with the Python package/wheel or application JAR and install/deploy that artifact on the runtime | +| TaskManager-local path | `paths` | Python `Skills.from_local_dir`; Java `Skills.fromLocalDir` | Every TaskManager must resolve the same directory or ZIP path with the same contents | +| Versioned HTTP(S) ZIP | `urls` | Python `Skills.from_url`; Java `Skills.fromUrl` | Every TaskManager must reach the URL; the ZIP top level contains Skill subdirectories | + +For a YAML scaffold, use a visibly unresolved schema-shaped template such as: + +```yaml +skills: + - name: runtime_skills + # TODO(required): replace this placeholder with paths, urls, package, or classpath. + paths: [TODO_REQUIRED_SKILL_SOURCE] +``` + +The placeholder keeps the required declaration visible; it is not a selected +distribution mode and is not runnable. For a direct API scaffold, generate a +compilable factory/helper that lists the valid factories and throws +`NotImplementedError` or `UnsupportedOperationException` until the user chooses one. +Do not package, mount, download, or activate the runtime Skill on the user's behalf +unless they later provide the source configuration or explicitly ask for it. + +Once the user fills the source, multiple fields may coexist only when explicitly +intended. The implementation language removes invalid choices (`package` is +Python-only and `classpath` is Java-only), while `paths` and `urls` can bridge +languages. Validate loader order and deployment constraints at that time. +YAML loaders append them in `paths`, `urls`, `classpath`, `package` order. Runtime +registration is last-wins for duplicate Skill frontmatter names, so avoid duplicate +names instead of relying on an implicit override or fallback chain. Prefer immutable, +versioned URLs because the YAML contract has no checksum field. A path that works in +a local MiniCluster proves only single-machine availability, not cluster-wide +TaskManager visibility. + +## Design the Action Graph + +Write the framework graph before code. This is an internal design checklist, not a +business requirements interview. Use explicit user requirements when available; +otherwise generate stable stage/event names and TODO-marked opaque payloads without +asking blocking questions: + +| Item | Record | +|---|---| +| Trigger | A documented built-in alias or generated custom event type | +| Input | Confirmed attributes, or an opaque payload with a TODO for the domain schema | +| State | Only explicitly requested memory; otherwise no invented domain state | +| Resources | Exact generated or preserved Resource names | +| Emission | Framework stage transition; unresolved business fields remain TODOs | +| Failure | Explicit failure for skeleton code; domain recovery remains a TODO | + +Multiple Actions may listen to the same event. Account for fan-out, duplicate +outputs, joins, and correlation state explicitly. Normally emit `OutputEvent` only +after the final response; it bypasses further Action routing and reaches downstream +immediately. + +## Built-in Action Boundaries + +The official docs currently identify these built-in behaviors: + +| Built-in Action | Listens for | Emits | +|---|---|---| +| `chat_model_action` | `ChatRequestEvent`, `ToolResponseEvent` | `ChatResponseEvent` or `ToolRequestEvent` | +| `tool_call_action` | `ToolRequestEvent` | `ToolResponseEvent` | +| `context_retrieval_action` | `ContextRetrievalRequestEvent` | `ContextRetrievalResponseEvent` | + +These Actions are added automatically when an AgentPlan is compiled. Do not invent +YAML declarations or implementation functions for them. Invoke them through their +documented Events and configured Resources. + +The reasoning/tool loop is: + +```text +ChatRequestEvent -> chat_model_action -> ToolRequestEvent +ToolRequestEvent -> tool_call_action -> ToolResponseEvent +ToolResponseEvent -> chat_model_action -> ChatResponseEvent or another ToolRequestEvent +``` + +Start the loop from a custom Action by emitting `ChatRequestEvent`. Handle the final +`ChatResponseEvent` in another custom Action, usually by emitting `OutputEvent`. +Context retrieval can run before or between reasoning stages by emitting +`ContextRetrievalRequestEvent` and handling `ContextRetrievalResponseEvent`. + +Every Action listed in YAML is otherwise a custom Action and needs a concrete, +resolvable function signature. If a purported built-in is not documented for the +target version, treat it as custom until source or tests prove otherwise. + +## Separate Framework and Business Code + +Generate framework-owned code completely: + +- project structure, dependency metadata, YAML sections, names, and references; +- built-in Action wiring and Resource declarations whose arguments the user chose; +- typed custom Action and Tool signatures with the target-version imports; +- runtime Skill directories and valid frontmatter when the user requests them; +- the Flink DataStream/Table entry point and MiniCluster submission path. + +Scaffold business-owned code unless the user explicitly requests its implementation +and provides enough behavior to implement it: + +- custom Action and Tool bodies; +- service clients, REST paths, queries, authentication, and response semantics; +- domain models and transformations not specified by the user; +- prompt policy, diagnosis or decision procedures, and Tool result interpretation; +- runtime Skill instructions and allowed commands; +- simulated services, fake Tool results, and test doubles. + +A function scaffold must import or compile and expose the exact signature referenced +by YAML, but its body should contain a focused TODO and raise +`NotImplementedError` or `UnsupportedOperationException`. A runtime Skill scaffold +contains its confirmed `name` and `description`, followed by a TODO for the user; do +not expand a one-line business goal into a runbook. Do not add tests that pretend +placeholder business behavior is implemented, and do not replace the placeholder +with a mock merely to make a behavior test pass. + +When no custom Tool signature was supplied, use the least committal supported shape, +normally one `String`/`str` request and one `String`/`str` result. Name the function +from the stated capability, document that the request/result contracts are TODOs, +and keep the body explicitly failing. This is a scaffold convention, not a claim +that the eventual business API should use strings. + +## Minimal Layouts + +Python YAML application: + +```text +app/ +├── .venv/ # only when selected; generated locally, never committed +├── requirements.txt +├── .gitignore +├── agent.yaml +├── actions.py +├── tools.py # when the Agent declares local Tools +├── types.py +├── main.py +├── resources/ +│ └── skills//SKILL.md # `paths` distribution only +└── tests/ +``` + +Java YAML application: + +```text +app/ +├── pom.xml +└── src/ + ├── main/java/com/example/agent/{Actions,Types,Main}.java + ├── main/resources/ + │ ├── yaml/agent.yaml + │ └── skills//SKILL.md # `classpath` only + └── test/java/com/example/agent/ +``` + +For Python `package` distribution, generate an installable package and place Skills +under that package's configured package-data path instead of the flat `resources/` +directory. For Java `classpath` distribution, keep Skills under +`src/main/resources` so Maven includes them in the application JAR. For `urls`, do +not copy the ZIP into the application artifact. Apply the user's confirmed choice +consistently in YAML, project metadata, packaging, and verification. + +Adapt to the target repository rather than forcing these layouts. Keep a streaming +file source pointed at input data only; do not point it at a parent directory that +also contains Skill or YAML assets. + +For every new application, generate the actual dependency project and a remote-style +Flink entry point. Follow `local-development.md`; an Agent definition without a +MiniCluster submission path is incomplete. When business functions are still +placeholders, keep deployment-only and behavior verification claims separate. diff --git a/dev/agent-skills/flink-agents-dev/references/java-patterns.md b/dev/agent-skills/flink-agents-dev/references/java-patterns.md new file mode 100644 index 000000000..9d9bb884d --- /dev/null +++ b/dev/agent-skills/flink-agents-dev/references/java-patterns.md @@ -0,0 +1,184 @@ +# Java Patterns + +## Contents + +- [Match the Installed API](#match-the-installed-api) +- [Cross-language Descriptor Resources](#cross-language-descriptor-resources) +- [YAML-referenced Implementations](#yaml-referenced-implementations) +- [Function Tools](#function-tools) +- [Java-loaded YAML](#java-loaded-yaml) +- [Programmatic ReActAgent](#programmatic-reactagent) +- [Java Checks](#java-checks) + +## Match the Installed API + +Inspect the target `pom.xml` or Gradle build, Java version, Flink version, Flink Agents +version, and existing examples. Reuse those versions and scopes. Do not copy versions +from a different branch or release. If no dependency metadata exists, use the +supported choices in `local-development.md` and obtain the user's selection before +writing the POM or source files. + +The patterns below are the bundled offline baseline. When target dependencies are +available, inspect the installed Flink Agents JARs, source JARs, and build metadata +for changed signatures or provider integrations. A source checkout is optional. + +## Cross-language Descriptor Resources + +A direct Java Agent may use Python implementations of chat-model +connections/setups, embedding-model connections/setups, and vector stores. Do not +filter Resource provider choices to Java implementations merely because the Agent, +custom Actions, or entry point are Java. + +After the user selects a Python implementation, build the target-version documented +descriptor with the corresponding Java-side Python wrapper and `pythonClazz` set to +the selected Python implementation FQN. Install the matching Python integration +package in the TaskManager Python environment and include the Python bridge runtime. +Do not translate the provider into a Java implementation or ask for a separate +cross-language confirmation. Verify the wrapper map before applying this pattern to +another Resource type. + +## YAML-referenced Implementations + +Java custom Actions use a public static method with the fixed Event and +`RunnerContext` parameters. YAML supplies those parameter types automatically; do +not add `parameter_types` to an Action. When business behavior is not supplied, +generate compilable skeletons rather than inventing event transformations: + +```java +public static void processInput(Event event, RunnerContext ctx) throws Exception { + throw new UnsupportedOperationException( + "TODO: implement the application-specific input Action"); +} + +public static void processChatResponse(Event event, RunnerContext ctx) throws Exception { + throw new UnsupportedOperationException( + "TODO: implement the application-specific response Action"); +} +``` + +Implement event construction only after the user supplies the required behavior. +Confirm constructors against the target version and format with the target build. +Reference the methods with `com.example.agent.Actions:processInput` and +`com.example.agent.Actions:processChatResponse`. For an inner class, use `$` on the +left side, for example `com.example.Outer$Actions:processInput`. + +## Function Tools + +Java Tool methods are public static methods. YAML must provide one ordered Java type +per declared parameter so overloaded methods can be resolved: + +```yaml +tools: + - name: lookupOrder + type: java + function: com.example.agent.Tools:lookupOrder + parameter_types: [java.lang.String] +``` + +Use primitive names or fully qualified reference types. Omit generic arguments: +write `java.util.List`, not `java.util.List`. Keep method annotations and +parameter metadata consistent with the target version's Tool documentation. + +Generate a matching method skeleton by default: + +```java +public static String queryLogs(String request) { + throw new UnsupportedOperationException( + "TODO: define the request contract and connect the log backend"); +} +``` + +Do not invent clients, endpoints, request/response fields, or error handling unless +the user explicitly supplied that business contract. If only the capability is +known, do not stop to ask for identity fields, platform variants, service APIs, or +authentication. Use `String request` and `String` result as neutral placeholder +boundary types, keep the body explicitly failing, and list the unresolved contract +after scaffolding. This placeholder is not a recommendation for the final domain +API. + +## Java-loaded YAML + +Missing `type` defaults to Python even in the Java loader. Set `type: java` on every +Java Action, Tool, chat/embedding descriptor, and vector store. Prompts and Skills +use their own schema rather than a language `type`. + +Runtime Skill behavior and source configuration are user-owned. Preserve an existing +`classpath`, `paths`, or `urls` source. For a new application, generate a minimal +`SKILL.md` TODO scaffold and list `Skills.fromClasspath(...)`, +`Skills.fromLocalDir(...)`, and `Skills.fromUrl(...)` in a factory TODO; do not ask +the user to select one and do not package or load a source speculatively. Keep the +factory compilable but explicitly failing until configured. + +Never inspect the coding-agent host's Skill directories or offer to reuse a local +Skill such as `flink-diag`. Those files are business instructions for another host, +not Java classpath assets. See `yaml-patterns.md#runtime-skills` for the fields the +user can fill later. + +Provider aliases and arguments are language-specific. For example, a provider may +use different aliases or camelCase arguments in Java. Copy them from the matching +Java docs or examples; schema validation alone cannot verify forwarded descriptor +arguments. + +That rule refers to the selected Resource implementation language, not the Java +application language. For a Python Resource selected by a Java application, use the +Python alias/class and Python argument contract together with the Java-side wrapper +and `pythonClazz` metadata. + +### MCP Limitation + +In the current source, Java `AgentPlan` rejects MCP servers registered through +`Agent.addResource`, while Java `YamlLoader` maps YAML `mcp_servers` to that path. +Do not generate a Java-loaded YAML application with MCP and describe it as runnable. +Use the documented programmatic `@MCPServer` static method instead, or confirm in the +target source that this restriction no longer exists. Keep the whole Agent in a +supported definition style; do not invent wrapper Agents, loader result accessors, +or YAML-to-MCP adapters. + +When this limitation requires the programmatic form, still generate every custom +Action and Tool signature whose framework contract is confirmed. Unknown provider +coordinates, endpoints, and business behavior remain explicit user-fillable +placeholders rather than fabricated implementation or an architecture checklist. + +## Programmatic ReActAgent + +Do not choose `ReActAgent` merely for a reasoning/tool loop; Workflow Agents already +provide that loop through built-in Actions. When the decision rules do select the +programmatic abstraction, construct it from the user-confirmed chat-model descriptor, +optional Prompt, and optional POJO class or `RowTypeInfo` output schema. This is shape +pseudocode; resolve each value after the Resource interview: + +```java +ReActAgent agent = + new ReActAgent( + configuredChatModelDescriptor, + confirmedPromptOrNull, + confirmedOutputClassOrNull); +``` + +Build `configuredChatModelDescriptor` after the user selects its implementation. +Generate its verified mandatory arguments as TODOs rather than asking for values. +Register every Resource it references under the exact name before `.apply(agent)`; +do not introduce a provider, model, Prompt, or Tool solely to complete the example. + +## Java Checks + +- Validate YAML before compilation. +- Compile the exact module with Maven/Gradle and the configured JDK. +- Resolve every Java function reference to a public static method. +- Check each Tool's `parameter_types` count and order against reflection. +- Check every Action uses the framework's fixed Event/`RunnerContext` contract. +- For a Python chat-model, embedding-model, or vector-store implementation, verify + the Java descriptor wrapper, `pythonClazz`, installed Python integration package, + and Python resource adapter path. Do not reject it because the application code is + Java. +- Resolve every generated framework method and constructor in target source, then + compile it. Label unverified fragments as pseudocode rather than runnable code. +- Run focused unit tests. Only after the user fills a runtime Skill source: inspect + the JAR for configured classpath Resources, or verify `paths`/`urls` deployment + preconditions without treating local access as cluster proof. +- Do not claim a Flink job ran from a successful compile or JAR inspection. +- Follow `local-development.md` to generate the Maven project with the API, plan, + runtime, and required integration artifacts in `provided` scope, then submit the + bounded remote-style Flink job to a local MiniCluster. Use the public + `AgentsExecutionEnvironment.getExecutionEnvironment(env)` factory backed by + `RemoteExecutionEnvironment`; never use a local Agents environment or list APIs. diff --git a/dev/agent-skills/flink-agents-dev/references/local-development.md b/dev/agent-skills/flink-agents-dev/references/local-development.md new file mode 100644 index 000000000..a7cd75524 --- /dev/null +++ b/dev/agent-skills/flink-agents-dev/references/local-development.md @@ -0,0 +1,506 @@ +# Project Generation and MiniCluster Validation + +## Contents + +- [Delivery Contract](#delivery-contract) +- [Resolve and Confirm a Compatible Version Set](#resolve-and-confirm-a-compatible-version-set) +- [Generate a Java Maven Project](#generate-a-java-maven-project) +- [Generate a Python Project and Select an Environment](#generate-a-python-project-and-select-an-environment) +- [Handle Local Plaintext Credentials](#handle-local-plaintext-credentials) +- [Scaffold Runtime Skill Configuration](#scaffold-runtime-skill-configuration) +- [Connect the Agent through RemoteExecutionEnvironment](#connect-the-agent-through-remoteexecutionenvironment) +- [Submit to a Local MiniCluster](#submit-to-a-local-minicluster) + +## Delivery Contract + +Do not stop after generating an Agent class or YAML definition. For a new +application, also generate and verify its executable project: + +- Java: a complete Maven project with `pom.xml`, sources, Resources, a Flink job + entry point, and a configured local run command; +- Python: source files, pinned dependency input, `.gitignore`, a user-selected + existing Python environment or project-local `.venv`, installed dependencies, and + a Flink job entry point; +- both: a `StreamExecutionEnvironment`, the public Agents factory backed by + `RemoteExecutionEnvironment`, a keyed DataStream/Table, a sink, and an + `AgentsExecutionEnvironment.execute(...)` call. + +For existing applications, preserve their package manager and layout, but provide +the same runnable path. "Local" changes only the Flink deployment target: the job +is still built through `RemoteExecutionEnvironment` and submitted to a MiniCluster. +Never substitute the removed local Agents APIs. + +## Resolve and Confirm a Compatible Version Set + +For an existing application, resolve versions from its lockfile, Maven metadata, +installed package, or `FLINK_HOME`, preserve them, and report what was detected. For +a new application, do not create dependency files, a virtual environment, or source +files until all sequential gates are complete. Versions are the first gate; do not +choose or mention the implementation language before it. + +Offer choices rather than asking an open-ended version question. The publication +snapshot, mirrored from `tools/install.sh`, is: + +| Component | Installer-supported versions | New-project choices | Recommended choice | +|---|---|---|---| +| Flink Agents | `0.3.0`, `0.2.1`, `0.2.0`, `0.1.1`, `0.1.0` | `0.3.0`, `0.2.1`, `0.1.1` | `0.3.0` | +| Flink | `2.2.1`, `2.1.3`, `2.0.2`, `1.20.5` | `2.2.1`, `2.1.3`, `2.0.2`, `1.20.5` | `2.2.1` | + +For a new project, offer only the highest supported patch in each Flink Agents +minor line. Older patches such as `0.2.0` and `0.1.0` remain valid when an existing +project pins them or the user explicitly requests one, but they add no useful +choice to the default new-project menu. + +Render the Flink Agents row through the interaction adapter selected by `SKILL.md`. +If the native tool limits the number of options, use the adapter's hierarchy or +paging rule so every version remains selectable. First satisfy any mode prerequisite +defined by that adapter. If the native tool remains unavailable afterward, use a +numbered fallback instead of asking the user to type a version string: + +```text +Select the Flink Agents version: +1. 0.3.0 (Recommended) +2. 0.2.1 +3. 0.1.1 + +Reply with 1, 2, or 3. +``` + +After that answer, present the compatible Flink versions through a separate native +selector or numbered list. Never put both version decisions into one multi-select +control: they are ordered single-choice gates because the first filters the second. + +Flink Agents `0.1.x` supports Flink `1.20` only in this snapshot. Flink Agents +`0.2.x` and `0.3.x` publish artifacts for all listed Flink minors. Use a +target-version installer, release metadata, or compatibility matrix when available +because it overrides this bundled snapshot. + +Present the recommended choice first, but do not select it because the user is +silent. Ask only for the Flink Agents version and wait. Then show only the Flink +versions compatible with that answer and wait again. These questions must not also +propose YAML/Python/Java APIs, Python/JDK versions, model providers, Resource +configuration, Skill loading, business backends, or mocks. Summarize only the +confirmed Flink Agents/Flink pair before moving to the API gate. + +Do not describe a bundled recommendation as "latest", "preview", or preferable to +an unlisted Flink release unless target-version release metadata was actually +checked and that comparison is needed for the user's request. The installer choices +are a compatibility menu, not permission to make broader release claims. + +After the API and implementation language are confirmed, resolve the language +runtime separately. When the design first requires Python, inspect available +interpreters and environments and keep only versions compatible with the already +selected Flink pair; do not assume Python 3.12. In this snapshot, Python 3.12 +requires Flink Agents `0.3+` and Flink `2.1+`; older combinations require Python +3.10 or 3.11. Then ask whether to reuse one specific compatible environment or +create a project-local `.venv` from a compatible interpreter. This conditional gate +also fires when Java application code selects a Python Resource. For Java, detect or +ask for a compatible JDK only after Java application code is selected. Keep these +values aligned: + +- the exact Flink patch version used for local execution; +- the Flink Agents Java or Python package version; +- every Flink Agents module and integration version; +- the user-confirmed Python or Java application-code language; +- each selected Resource implementation language and bridge dependency; +- the Java and Python versions supported by that Flink Agents release. + +The selected Flink binary may require a newer Java runtime; use the higher +requirement from Flink and Flink Agents. Do not leave guessed versions or unresolved +version placeholders in the generated project. + +Resolve integration dependencies from each Resource's implementation language, not +from the application language. A Java application using a bridge-supported Python +chat model, embedding model, or vector store still needs the corresponding Python +package in every TaskManager environment. A Java implementation selected by a +Python application still needs its Java integration JAR on the runtime classpath. +Generate both sides of that dependency contract and the target-version bridge +runtime; do not hide the foreign-language dependency or replace the Resource. + +## Generate a Java Maven Project + +Generate at least: + +```text +agent-app/ +├── pom.xml +└── src/ + ├── main/java//{Actions,Tools,Main}.java + ├── main/resources/agent.yaml + └── test/java// +``` + +Use the Flink Agents application modules directly. Always add +`flink-agents-api`, `flink-agents-plan`, and `flink-agents-runtime`; add only the +integration artifacts required by the Resources declared in the application. Do +not use a dist artifact or `flink-agents-ide-support` as the application's compile +contract. + +Declare every Flink Agents and Flink dependency with `provided` scope. The target +deployment must supply the matching Flink Agents modules, integrations, and Flink +runtime; `provided` is a packaging contract, not an automatic cluster install. The +generated forked local-run configuration includes these dependencies through +Maven's compile classpath. + +```xml + + RESOLVE_EXACT_FLINK_VERSION + RESOLVE_EXACT_AGENTS_VERSION + RESOLVE_COMPATIBLE_JAVA_RELEASE + RESOLVE_MAIN_CLASS + 3.14.1 + 3.6.3 + + + + + org.apache.flink + flink-agents-api + ${flink-agents.version} + provided + + + org.apache.flink + flink-agents-plan + ${flink-agents.version} + provided + + + org.apache.flink + flink-agents-runtime + ${flink-agents.version} + provided + + + org.apache.flink + flink-streaming-java + ${flink.version} + provided + + + org.apache.flink + flink-clients + ${flink.version} + provided + + + org.apache.flink + flink-table-api-java-bridge + ${flink.version} + provided + + + + + + + org.apache.maven.plugins + maven-compiler-plugin + ${maven-compiler-plugin.version} + + ${maven.compiler.release} + + + + org.codehaus.mojo + exec-maven-plugin + ${exec-maven-plugin.version} + + ${java.home}/bin/java + compile + + -classpath + + ${main.class} + + + + + +``` + +For each selected integration, add its target-version artifact with the same +Flink Agents version and `provided` scope. Resolve its actual artifact from +target-version metadata only after the user confirms the Resource implementation; +do not select an integration from an example and do not add every integration +preemptively. The Table bridge is required even for the default DataStream job +because the `AgentsExecutionEnvironment` API and runtime constructor reference +`StreamTableEnvironment`. Add the matching Table planner only for Table API jobs, +and add connector dependencies only for the selected sources and sinks. Declare +those Flink artifacts as `provided` too. + +Generate a complete POM, not just the fragment. The bundled plugin versions above +are a verified baseline; preserve a compatible newer version already selected by +the target project. Run the main class in a forked JVM with compile classpath scope: + +```bash +mvn clean compile exec:exec +``` + +Do not use `exec:java` as the default. Its in-process plugin ClassLoader can make a +Flink MiniCluster fail while deserializing operator factories. `exec:exec` starts a +normal JVM with the generated dependency classpath. + +Keep `compile` in the forked local-run +configuration: Maven's compile classpath includes `provided` dependencies, so the +same POM supports both local smoke tests and cluster-oriented packaging without a +second dependency profile. + +## Generate a Python Project and Select an Environment + +Generate at least: + +```text +agent-app/ +├── .gitignore +├── requirements.txt +├── agent.yaml +├── actions.py +├── tools.py +└── main.py +``` + +Pin the resolved versions in `requirements.txt`: + +```text +flink-agents==RESOLVE_EXACT_AGENTS_VERSION +apache-flink==RESOLVE_EXACT_FLINK_VERSION +``` + +Include any application-specific dependencies used by Actions or Tools. Before +installing them, inspect compatible local Python executables and environments. If +the existing project does not already declare one unambiguously, present a closed +choice that names concrete paths, for example: + +1. `Create .venv with /path/to/python3.11 (Recommended)` +2. `Reuse /path/to/existing/environment/bin/python` + +If several compatible existing environments or base interpreters exist, use the +host adapter's hierarchy so the user selects an exact executable. Do not create the +environment, install dependencies, or silently use the active shell Python before +the answer. + +For a project-local environment, create and populate it with the selected base +interpreter: + +```bash + -m venv .venv +.venv/bin/python -m pip install -r requirements.txt +.venv/bin/python -m pip check +``` + +For an existing environment, do not create `.venv`; use the exact selected +executable for installation and verification: + +```bash + -m pip install -r requirements.txt + -m pip check +``` + +On Windows, use the selected executable path such as `.venv\Scripts\python`. Add +`.venv/` to `.gitignore` only when it is created, and always ignore caches, logs, +and local secrets. Use the selected executable for every later import, test, build, +and local job command. If an existing interpreter rejects installation because it +is externally managed or read-only, report that result and return to the environment +choice instead of bypassing its protection or silently creating a venv. + +The `flink-agents` Python package carries the common and Flink-version-specific +Flink Agents JARs. Constructing `AgentsExecutionEnvironment` registers those JARs +with the Flink pipeline, so a Python application must not copy another Flink Agents +dist JAR into the project. The generated Action modules must still be importable +from the working directory or installed application package. + +## Handle Local Plaintext Credentials + +Do not ask the user to choose a credential mechanism while scaffolding a Resource. +Generate required credential keys with `TODO_REQUIRED_*` values. Plaintext YAML is +still a valid local-testing mechanism when the user explicitly requests it or +provides a value; do not reject that instruction or replace it with Java/Python +`addResource` wiring. Put a real value only in a clearly local file such as +`config/agent.local.yaml`, add its exact path to `.gitignore`, and make the local +Java or Python entry point load that file. Keep a tracked secret-free example only +when it helps the user reproduce the layout. + +If the project is a Git worktree, verify the protection before running: + +```bash +git check-ignore -v config/agent.local.yaml +git status --short +``` + +The first command must identify the intended ignore rule, and the secret-bearing +file must not appear as untracked or staged in the second command. If the workspace +does not use Git, state that repository-level protection could not be verified. + +Do not ask for the value. When the user supplies one without prompting, write it +only to the ignored local file. Do not place it in a shell command, generated test +fixture, tracked example, Maven configuration, dependency file, console output, +diff excerpt, or final report. Warn once that the file contains plaintext and is for +local testing; then continue with the verification that its filled configuration +permits. + +## Scaffold Runtime Skill Configuration + +For an existing application, preserve and validate its configured runtime Skill +source. For a new application, do not ask the user to choose distribution and do +not inspect or reuse coding-agent host Skills. Generate a runtime `SKILL.md` TODO +shell and an unresolved source declaration/factory TODO. The following table tells +the user what they can fill later: + +| Choice | Generated project requirement | +|---|---| +| Bundle with Java application | Use YAML `classpath` or `Skills.fromClasspath`; place Skill directories under `src/main/resources/` and verify the built JAR contains them | +| Bundle with Python application | Use YAML `package` or `Skills.from_package`; generate an installable Python package, include the Skill tree as package data, build or install it into the selected Python environment, and verify the resource is readable from that installed package | +| TaskManager-local path | Use YAML `paths` or the language's local-dir factory; accept directories or ZIPs and document the path that every TaskManager must mount or provision | +| Versioned HTTP(S) ZIP | Use YAML `urls` or the language's URL factory; require a ZIP whose top level contains Skill directories and document TaskManager network access | + +Do not copy a local `flink-diag` or any Skill found under Codex, Claude Code, Qoder, +Gemini CLI, or another coding-agent installation. The generated runtime Skill shell +belongs to the user's application and contains only capability-derived metadata plus +a TODO body. + +Do not package, mount, or download the runtime Skill until the user fills the source. +If the user later chooses bundled Python Skills, use a package layout such as: + +```text +agent-app/ +├── pyproject.toml +└── src/app/ + ├── __init__.py + └── resources/skills//SKILL.md +``` + +Then configure the selected build backend to include every Skill Markdown file, script, +and reference as package data. Install the application package with the same +selected Python executable used to run the job, for example with +` -m pip install -e .`, then verify the configured `package` and +`resource` pair through the installed package-resource API. Deploy that package to +every Python worker in the target cluster; installation in the selected local +environment proves only local availability. Do not point `package` at an uninstalled +source directory. + +Do not treat implementation language as the distribution decision: Java and Python +may both use `paths` or `urls`. Use those portable schemes for an intentional +cross-language source. If multiple YAML source fields are explicitly combined, +preserve loader order `paths`, `urls`, `classpath`, `package` and reject duplicate +Skill names rather than depending on last-wins replacement. + +A local MiniCluster shares one machine and can validate artifact contents, ZIP +shape, and local resolution. It cannot prove that a distributed cluster mounts the +same path or allows every TaskManager to reach a URL. Report those as deployment +requirements unless they were verified in the target cluster environment. Prefer +immutable, versioned URLs; the current YAML contract has no checksum field. + +## Connect the Agent through RemoteExecutionEnvironment + +Both Python and Java must create a Flink `StreamExecutionEnvironment` first, then +pass it to the public `AgentsExecutionEnvironment` factory. In the supported API, +that factory creates `RemoteExecutionEnvironment`; use the public factory instead +of importing the runtime implementation directly. + +Never call the Python factory without `env`, and never use a local Agents +environment, `from_list`/`to_list`, or Java list equivalents. Local validation means +the remote-style Flink job is submitted to the MiniCluster selected by the local +`StreamExecutionEnvironment`. + +Use a bounded source so the validation command terminates. Apply a YAML Agent by its +declared name; apply a programmatic Agent by instance. + +Python shape: + +```python +from pathlib import Path + +from pyflink.common import Types +from pyflink.datastream import StreamExecutionEnvironment + +from flink_agents.api.execution_environment import AgentsExecutionEnvironment + + +env = StreamExecutionEnvironment.get_execution_environment() +env.set_parallelism(1) +agents_env = AgentsExecutionEnvironment.get_execution_environment(env) +agents_env.load_yaml(Path(__file__).with_name("agent.yaml")) + +input_stream = env.from_collection( + ["local smoke test"], type_info=Types.STRING() +) +output_stream = ( + agents_env.from_datastream(input_stream, key_selector=lambda value: value) + .apply("agent_name") + .to_datastream() +) +output_stream.print() +agents_env.execute("Flink Agent MiniCluster Validation") +``` + +Java shape: + +```java +StreamExecutionEnvironment env = + StreamExecutionEnvironment.getExecutionEnvironment(); +env.setParallelism(1); +AgentsExecutionEnvironment agentsEnv = + AgentsExecutionEnvironment.getExecutionEnvironment(env); +agentsEnv.loadYaml(Paths.get("src/main/resources/agent.yaml")); + +DataStream input = env.fromElements("local smoke test"); +DataStream output = + agentsEnv + .fromDataStream( + input, (KeySelector) value -> value) + .apply("agent_name") + .toDataStream(); +output.print(); +agentsEnv.execute("Flink Agent MiniCluster Validation"); +``` + +Adapt types, key selection, Agent name, and paths to the generated application. +Use a key that is stable for all events belonging to one Agent invocation. A +continuous file or message source may be added separately, but it must not replace +the terminating smoke-test path. + +The non-empty examples above are behavior smoke tests and require implemented +business functions. When custom Action or Tool bodies are intentionally still +scaffolds, do not invoke them with fabricated behavior. A deployment-only check may +use a typed empty bounded source, such as Python +`env.from_collection([], type_info=Types.STRING())` or Java +`env.fromCollection(Collections.emptyList(), Types.STRING)`, while preserving the +Agent operator and sink. Report that this validates job construction, submission, +and deployment only; it does not validate Action, Tool, model, or output behavior. + +## Submit to a Local MiniCluster + +Python, using the environment selected earlier: + +```bash + -c "import flink_agents; import pyflink" + main.py +``` + +Java: + +```bash +mvn dependency:tree +mvn clean compile exec:exec +``` + +Running these commands directly uses Flink's local MiniCluster while retaining the +same `RemoteExecutionEnvironment` integration used for cluster submission. Before a +behavior smoke test, configure only the user-confirmed model/backend and credential +mechanism. Do not generate or select a test double to fill an unspecified business +implementation. If the user has not implemented the business bodies or configured +the external Resource, run deployment-only validation and state what blocks behavior +or integration validation. + +Report evidence at the correct level: + +- deployment validation: the bounded job was submitted to the MiniCluster and + finished; an empty source is acceptable when business functions are scaffolds; +- behavior validation: non-empty input invoked the implemented business functions, + produced the expected sink record, and finished; +- integration validation: the external model, MCP server, vector store, or service + was actually reached. + +A compile, YAML load, or successful deployment-only job is not proof of business or +external integration behavior. diff --git a/dev/agent-skills/flink-agents-dev/references/platforms/claude-code.md b/dev/agent-skills/flink-agents-dev/references/platforms/claude-code.md new file mode 100644 index 000000000..4db7afb81 --- /dev/null +++ b/dev/agent-skills/flink-agents-dev/references/platforms/claude-code.md @@ -0,0 +1,23 @@ +# Claude Code Interaction Adapter + +Use this adapter only when explicit host context identifies Claude Code. + +When `AskUserQuestion` is present in the current tool catalog, call it for closed +decision gates. Ask exactly one question for the current gate, use its option list, +set `multiSelect` to `false`, and wait for the answer before continuing. When a gate +has a recommendation, put it first and label it `(Recommended)`. The YAML +implementation-language gate has no recommendation; present Python and Java with +parallel descriptions and no recommendation label. + +If the live schema cannot fit all valid choices in one question, preserve every +choice with a meaningful hierarchy or short paged selectors. Use the complete +numbered fallback when splitting would obscure the choices. + +Tool availability depends on the active Claude Code surface and configuration. If +`AskUserQuestion` is absent or errors, read and follow [generic.md](generic.md). +Do not add it to allowed tools, modify Claude Code settings, retry it repeatedly, +or silently select a default. The generated Flink Agents project must not contain +Claude Code configuration. + +Follow the argument schema exposed by the running Claude Code version. Do not copy +a stale parameter shape from this reference when the live tool contract differs. diff --git a/dev/agent-skills/flink-agents-dev/references/platforms/codex.md b/dev/agent-skills/flink-agents-dev/references/platforms/codex.md new file mode 100644 index 000000000..575b3dbd7 --- /dev/null +++ b/dev/agent-skills/flink-agents-dev/references/platforms/codex.md @@ -0,0 +1,110 @@ +# Codex Interaction Adapter + +Use this adapter only when explicit host context identifies Codex. + +## Capability Check + +The current session and mode's tool contract is authoritative. Codex exposes its +native closed-choice UI through `request_user_input` in Plan mode. Before the first +decision gate, check the explicit collaboration mode and the available tool +contract. + +When the session is not in Plan mode, respond with one short message in the user's +language that conveys: + +> Flink Agents setup has several required choices. Switch Codex to Plan mode with +> `/plan` or `Shift+Tab`, then continue this request so I can present one native +> selector at a time. + +Stop after that message. Do not present a numbered choice list, begin project +generation, or silently choose defaults. When the user continues in Plan mode, +resume at the first unresolved gate without repeating already supplied facts. + +When Plan mode is active and `request_user_input` is exposed and callable, use it +for every closed decision gate. Follow [generic.md](generic.md) only when the user +explicitly declines or cannot enter Plan mode, or when Plan mode is active but the +tool is still absent or returns an availability error. Do not retry an unavailable +tool or invent an alias. + +## Return to Execution Mode + +After the last required decision gate is answered, do not start generating files, +installing dependencies, or running verification while Codex remains in Plan mode. +Respond with one short message in the user's language that conveys: + +> All required Flink Agents choices are confirmed. Switch Codex back to Default +> (execution) mode with the mode selector, or press `Shift+Tab` until Default mode +> is selected. Then continue so I can generate the project and run verification. + +Stop after that message. When the user continues in Default mode, resume directly +at the build workflow with the confirmed decisions. Do not ask them to enter Plan +mode again, repeat completed selectors, or require a summary of their previous +answers. Return to Plan mode only if a genuinely new unresolved closed decision is +discovered before implementation can safely continue. + +## Native Single Select + +Call `request_user_input` with exactly one question for the current gate and wait +for its result before doing more work. Omit `autoResolutionMs` because every gate +requires an explicit answer. Use stable `snake_case` IDs, a short header, concise +labels, and one-sentence descriptions. When a gate has a recommendation, put it +first and suffix its label with `(Recommended)`. The YAML implementation-language +gate has no recommendation: give Python and Java parallel descriptions and suffix +neither label. + +For example, the API gate maps to: + +```text +request_user_input( + questions=[ + { + "header": "Agent API", + "id": "agent_api", + "question": "Select the API for this Flink Agents application.", + "options": [ + { + "label": "YAML API (Recommended)", + "description": "Use schema-validated declarative workflow wiring." + }, + { + "label": "Direct Python API", + "description": "Build the application programmatically in Python." + }, + { + "label": "Direct Java API", + "description": "Build the application programmatically in Java." + } + ] + } + ] +) +``` + +Follow the argument schema exposed by the running Codex version if it differs from +this snapshot. + +## Option Limits + +The bundled Codex contract accepts two or three options per question. Preserve all +choices with a short hierarchy rather than dropping options or immediately using a +text list. + +The default new-project Flink Agents menu fits in one native selector: + +1. `0.3.0 (Recommended)` +2. `0.2.1` +3. `0.1.1` + +Do not offer older patches from those minor lines unless an existing project pins +one or the user explicitly requests it. For the four bundled Flink versions, offer +`2.2.1 (Recommended)`, `2.1.3`, and `Older Flink`, then resolve the last choice to +`2.0.2` or `1.20.5`. Apply compatibility filtering before building the hierarchy. + +If compatibility filtering leaves one valid value but the live tool requires at +least two options, offer `Use (Recommended)` and `Change previous choice`. +The second option returns to the preceding gate; it is not another version. + +For another option set, use meaningful release/provider families when they are +unambiguous. Otherwise use short paged native selectors. If hierarchy or paging +would alter or obscure the choices, follow the complete numbered list in +[generic.md](generic.md). diff --git a/dev/agent-skills/flink-agents-dev/references/platforms/gemini-cli.md b/dev/agent-skills/flink-agents-dev/references/platforms/gemini-cli.md new file mode 100644 index 000000000..4be5f13fb --- /dev/null +++ b/dev/agent-skills/flink-agents-dev/references/platforms/gemini-cli.md @@ -0,0 +1,20 @@ +# Gemini CLI Interaction Adapter + +Use this adapter only when explicit host context identifies Gemini CLI. + +When the `ask_user` communication tool is present in the current tool catalog, call +it with one single-select question for the current gate and wait for the answer. +Provide concise option descriptions, put a recommended option first only when the +gate has one, and keep multi-select disabled. The YAML implementation-language gate +has no recommendation; present Python and Java with parallel descriptions and no +recommendation label. + +If the live schema cannot fit all valid choices in one question, preserve every +choice with a meaningful hierarchy or short paged selectors. Use the complete +numbered fallback when splitting would obscure the choices. + +Gemini CLI can exclude interactive tools in non-interactive or ACP-style surfaces. +If `ask_user` is absent or errors, read and follow [generic.md](generic.md). Do not +confuse the communication tool with an approval-policy decision of the same name, +change Gemini settings, or silently choose a default. Follow the live tool schema +when its arguments differ from this bundled snapshot. diff --git a/dev/agent-skills/flink-agents-dev/references/platforms/generic.md b/dev/agent-skills/flink-agents-dev/references/platforms/generic.md new file mode 100644 index 000000000..3c5eb3a79 --- /dev/null +++ b/dev/agent-skills/flink-agents-dev/references/platforms/generic.md @@ -0,0 +1,34 @@ +# Generic Interaction Adapter + +Use this adapter when the host is unknown, no dedicated adapter exists, or a +dedicated adapter's native question tool is unavailable. + +## Closed Choices + +When this adapter was selected directly, use a structured single-select question +tool only if the current tool catalog explicitly exposes it and its argument +contract is available. Do not guess a tool name or argument shape. When another +adapter reached this file because its native tool failed, skip native discovery and +go directly to the numbered list; do not retry the failed tool. + +Otherwise render all valid choices as a numbered list: + +```text +Select the API: +1. YAML API (Recommended) +2. Direct Python API +3. Direct Java API + +Reply with 1, 2, or 3. +``` + +Put one option on each line. When a gate has a recommended option, keep it first and +label it, but do not preselect it. The YAML implementation-language gate has no +recommendation: list Python and Java with parallel descriptions and no +`(Recommended)` label. Stop after the current question and wait for an explicit +answer. Do not combine gates, continue on silence, or replace a known enumeration +with an open-ended question. + +For a non-interactive or headless run, emit the same numbered question and stop. +The caller must resume or rerun with the selected value; never choose a default to +keep automation moving. diff --git a/dev/agent-skills/flink-agents-dev/references/platforms/qoder.md b/dev/agent-skills/flink-agents-dev/references/platforms/qoder.md new file mode 100644 index 000000000..9d418d24c --- /dev/null +++ b/dev/agent-skills/flink-agents-dev/references/platforms/qoder.md @@ -0,0 +1,16 @@ +# Qoder Interaction Adapter + +Use this adapter only when explicit host context identifies Qoder. + +The bundled publication snapshot does not assume a stable Qoder structured-question +tool name or argument contract. Inspect the current session's tool catalog. If it +explicitly exposes a structured single-select question tool, call it according to +the live schema for one gate at a time and wait for the answer. + +For the YAML implementation-language gate, present Python and Java as equal peer +options with parallel descriptions. Do not mark either language as recommended or +preselect one. + +Otherwise read and follow [generic.md](generic.md). Do not guess that a tool from +Codex, Claude Code, or Gemini CLI exists in Qoder, and do not add Qoder-specific +metadata to the generated Flink Agents project. diff --git a/dev/agent-skills/flink-agents-dev/references/python-patterns.md b/dev/agent-skills/flink-agents-dev/references/python-patterns.md new file mode 100644 index 000000000..633a9a555 --- /dev/null +++ b/dev/agent-skills/flink-agents-dev/references/python-patterns.md @@ -0,0 +1,182 @@ +# Python Patterns + +## Contents + +- [Match the Installed API](#match-the-installed-api) +- [Cross-language Descriptor Resources](#cross-language-descriptor-resources) +- [YAML-referenced Implementations](#yaml-referenced-implementations) +- [Function Tools](#function-tools) +- [Resource Access and Events](#resource-access-and-events) +- [Runtime Skill Packaging](#runtime-skill-packaging) +- [Programmatic ReActAgent](#programmatic-reactagent) +- [Python Checks](#python-checks) + +## Match the Installed API + +Inspect the target application's dependency version, imports, tests, and existing +Agents code. Confirm uncertain constructors and provider parameters in the matching +docs or source. Do not pin guessed Flink, Flink Agents, provider, or model versions. +If the target has no dependency metadata, use the supported choices in +`local-development.md` and obtain the user's selection before writing dependency or +source files. + +The patterns below are the bundled offline baseline. When a target environment is +available, inspect the installed `flink_agents` package and its metadata for changed +signatures or provider integrations. A source checkout is optional. + +## Cross-language Descriptor Resources + +A direct Python Agent may use Java implementations of chat-model connections/setups, +embedding-model connections/setups, and vector stores. Do not filter Resource +provider choices to Python implementations merely because the Agent, custom Actions, +or entry point are Python. + +After the user selects a Java implementation, build the target-version documented +descriptor with the corresponding Python-side Java wrapper and `java_clazz` set to +the selected Java implementation FQN. Add the matching Java integration artifact to +the job/runtime classpath as well as the Python bridge dependencies. Do not translate +the provider into a Python implementation or ask for a separate cross-language +confirmation. This bridge does not establish support for arbitrary Resource types; +verify the wrapper map in the target version. + +## YAML-referenced Implementations + +Custom Actions use the fixed `(Event, RunnerContext) -> None` contract. When the +user has not supplied their business behavior, generate importable skeletons rather +than inferring event transformations or domain policy: + +```python +from flink_agents.api.events.event import Event +from flink_agents.api.runner_context import RunnerContext + + +def process_input(event: Event, ctx: RunnerContext) -> None: + """TODO: Map an InputEvent to the application's first event.""" + raise NotImplementedError( + "Implement the application-specific input Action" + ) + + +def process_chat_response(event: Event, ctx: RunnerContext) -> None: + """TODO: Validate the model response and emit the application output.""" + raise NotImplementedError( + "Implement the application-specific response Action" + ) +``` + +Implement an event transformation only when the user explicitly requests it and its +input/output behavior is known. Then adapt each event constructor only after reading +its signature in the target version. For custom event types, emit +`Event(type="...", attributes={...})` and use the same type string in YAML +`trigger_conditions`. + +Treat each constructor keyword and property as a versioned contract. Import or +compile the generated module against the target environment before describing it as +runnable; otherwise label the example as pseudocode and identify the unresolved +symbol. + +YAML references these as: + +```yaml +function: app.actions:process_input +function: app.actions:process_chat_response +``` + +The module must be importable from the runner's environment. Static methods use +`module:Class.method`. Generate the module and callable before claiming the YAML +loads. + +## Function Tools + +Use typed parameters and a useful docstring so the framework can derive the Tool +schema. Keep framework-owned values out of the model schema with documented injected +arguments rather than hidden globals. If the user stated a capability but did not +supply its business contract, do not stop to interview for domain identity fields, +platform variants, endpoints, or authentication. Generate a neutral skeleton: + +```python +def query_logs(request: str) -> str: + """TODO: Define the diagnostic request and result contracts.""" + raise NotImplementedError("Connect the user-selected log backend") +``` + +`NotImplementedError` is the default when the user has not provided the business +integration. Do not invent an HTTP client, endpoint, query, authentication scheme, +response shape, or fallback merely to make the Tool look complete. `request: str` +and `str` are placeholder boundary types, not a recommended domain API. Keep all +Flink Agents wiring around the signature complete and list the unresolved contract +for the user after scaffolding. + +## Resource Access and Events + +Use `ctx.get_resource(name, ResourceType.)` with an exact declared name. For +vector retrieval, prefer the documented built-in flow when it fits: + +1. Send `ContextRetrievalRequestEvent` using the target version's constructor. +2. Handle `ContextRetrievalResponseEvent` in a custom Action. +3. Read its documents and send the next event. + +For direct queries, construct `VectorStoreQuery` and call the retrieved vector +store. Do not mix direct and event-driven retrieval accidentally. + +## Runtime Skill Packaging + +Runtime Skill behavior and source configuration are user-owned. Preserve an existing +`package`, `paths`, or `urls` source. For a new application, generate a minimal +`SKILL.md` TODO scaffold and list `Skills.from_package(...)`, +`Skills.from_local_dir(...)`, and `Skills.from_url(...)` in a factory TODO; do not +ask the user to select one and do not package or load a source speculatively. + +The factory helper must remain importable and fail explicitly until configured. +Never inspect Python environment, Codex, Claude Code, Qoder, or other host Skill +directories to find reusable business content. A host `flink-diag` Skill is not a +Flink runtime Skill dependency. See `yaml-patterns.md#runtime-skills` for the fields +the user can fill later. + +## Programmatic ReActAgent + +Do not choose `ReActAgent` merely for a reasoning/tool loop; Workflow Agents already +provide that loop through built-in Actions. When the decision rules do select the +programmatic abstraction, construct it from the user-confirmed chat-model descriptor, +optional Prompt, and optional Pydantic or `RowTypeInfo` output schema. This is shape +pseudocode; resolve each value after the Resource interview: + +```python +from flink_agents.api.agents.react_agent import ReActAgent + +agent = ReActAgent( + chat_model=configured_chat_model_descriptor, + prompt=confirmed_prompt_or_none, + output_schema=confirmed_output_schema_or_none, +) +``` + +Build `configured_chat_model_descriptor` only after the user selects its +implementation. Generate its verified mandatory arguments as TODOs rather than +asking for values. Register every Resource it references under the exact name before +`.apply(agent)`; do not introduce a provider, model, Prompt, or Tool solely to +complete the example. + +## Python Checks + +- Parse and build YAML with the Flink Agents loader. +- Import every left-side module in `function` references and resolve each qualname. +- Inspect or test each Action signature and emitted event constructor. +- Run focused pytest tests with the target repository's environment. +- Set the repository-required `PYTHONPATH` before Python-facing or cross-language + tests when working in the Flink Agents source checkout. +- Exercise provider integrations only when their services and credentials are + available; label skipped integration checks. +- For a Java chat-model, embedding-model, or vector-store implementation, verify the + Python descriptor wrapper, `java_clazz`, selected Java integration JAR, and Java + resource adapter path. Do not reject it because the application code is Python. +- Only after the user fills a runtime Skill source: for bundled Skills, inspect the + built wheel or installed package and load the configured package-data resource; + for `paths` or `urls`, verify the corresponding deployment preconditions without + treating local access as cluster proof. +- Follow `local-development.md` to ask whether to reuse a compatible existing + Python environment or create `.venv`, install dependencies only after that choice, + and submit the bounded remote-style Flink job to a local MiniCluster. Always pass a + `StreamExecutionEnvironment` to `AgentsExecutionEnvironment.get_execution_environment`; + never use the no-argument factory, a local Agents environment, `from_list`, or + `to_list`. diff --git a/dev/agent-skills/flink-agents-dev/references/verification.md b/dev/agent-skills/flink-agents-dev/references/verification.md new file mode 100644 index 000000000..b22f9a92b --- /dev/null +++ b/dev/agent-skills/flink-agents-dev/references/verification.md @@ -0,0 +1,331 @@ +# Verification + +## Contents + +- [1. Version and Source](#1-version-and-source) +- [2. YAML Schema](#2-yaml-schema) +- [3. Static Reference Graph](#3-static-reference-graph) +- [4. Action and Event Graph](#4-action-and-event-graph) +- [5. Language Checks](#5-language-checks) +- [6. Resource-specific Checks](#6-resource-specific-checks) +- [7. Runtime Evidence](#7-runtime-evidence) +- [Final Report Template](#final-report-template) + +Verify in layers. A later layer does not replace an earlier one. + +## 1. Version and Source + +- Identify the target Flink Agents, Flink, language, and provider versions from + dependency metadata or installed packages. +- For a new project, record the user's explicit Flink Agents and Flink choices. A + bundled recommended version without user confirmation is not valid evidence. +- Record the user's explicit YAML, direct Python, or direct Java API choice after + the version pair. A combined recommended baseline is not evidence for any of + these independent decisions. +- Confirm the workflow selected one matching host adapter before the first closed + gate. Each gate must use its available native single-select tool or the generic + numbered fallback; an unavailable native tool must not be retried in a loop. + Reject open-ended version/API/language questions when the valid options were + known, and reject a preselected recommended value without a user response. +- For a new YAML project, record the user's explicit Python or Java implementation + choice after the YAML selection. An omitted YAML `type` default is not evidence + of user intent. Record a Python/JDK choice only after the implementation language + is known. +- When runtime Skills are requested for a new project, confirm that source and + business implementation remain explicit TODOs rather than interview gates. For an + existing configured source, preserve and validate its deployment topology. +- Require one resolved version set across the Maven Flink Agents modules and + integrations or Python package, the selected Flink patch release, and the local + Java/Python runtime. +- Use a matching target-version schema and API contract when available. Otherwise, + use the [bundled YAML schema](../assets/yaml-schema.json) as the offline baseline + and record that compatibility assumption. +- Record any API assumption that could not be confirmed. Do not generate a guessed + version merely to make a dependency file look complete. + +## 2. YAML Schema + +Validate every generated or modified YAML file against the matching schema. If +`check-jsonschema` is already available: + +```bash +check-jsonschema --schemafile /assets/yaml-schema.json path/to/agent.yaml +``` + +Resolve `` from the installed `SKILL.md` location. Do not assume the +current working directory is a Flink Agents source checkout. Replace the bundled +schema path with a target-version schema when one is available. + +Otherwise use another real JSON Schema validator that accepts YAML. A YAML parser, +formatter, or linter only proves syntax and is not a schema substitute. + +When working in the Flink Agents source checkout, the Python typed loader is a useful +additional check: + +```bash +cd python +python - path/to/agent.yaml <<'PY' +from pathlib import Path +import sys + +from flink_agents.api.yaml.loader import build_agents + +agents, shared_resources, shared_actions = build_agents(Path(sys.argv[1])) +print("agents:", sorted(agents)) +print("shared actions:", sorted(shared_actions)) +PY +``` + +Use the repository's configured environment (`uv run --no-sync`, activated virtual +environment, or equivalent) instead of assuming system Python has the dependencies. + +## 3. Static Reference Graph + +Check every name edge explicitly: + +- chat setup -> connection; +- chat setup -> Prompt; +- chat setup -> local or MCP Tool names; +- chat setup -> individual runtime Skill names; +- embedding setup -> embedding connection; +- vector store -> embedding setup; +- Action implementation -> Resource names passed to context lookups; +- emitted `ChatRequestEvent.model` -> chat setup; +- YAML shared Action string -> top-level Action; +- Python/Java `function` reference -> real callable/method. + +Check uniqueness within each file. Across files loaded into the same execution +environment, multiple YAML loads accumulate and duplicate Agent or shared Resource +names fail. Shared Actions and their string references are file-scoped rather than +registered globally. + +## 4. Action and Event Graph + +For each YAML Action, classify it: + +| Classification | Required evidence | +|---|---| +| Documented built-in behavior | Target-version docs/source identify its Event contract | +| Custom Python Action | Importable callable with `(Event, RunnerContext) -> None` | +| Custom Java Action | Public static method with Event and `RunnerContext` parameters | + +Trace every path from `input` to `OutputEvent`. Check fan-out, correlation state, +custom event type strings, error paths, and whether any input can terminate without +an output. Verify event constructor arguments against the installed API rather than +memory. + +For scaffolding, distinguish intended edges from implemented edges. Verify every +custom callable exists with the right signature, list each TODO business body, and +do not claim the path emits an event until that body is implemented. Reject invented +domain logic, service calls, prompts, runtime Skill instructions, and tests that +assert behavior the user did not specify. + +## 5. Language Checks + +Python: + +- Require pinned `flink-agents` and `apache-flink` dependency input. +- Require a recorded user choice between a compatible existing Python environment + and a project-local `.venv`, unless existing project metadata already made the + environment unambiguous. +- Run the exact selected Python executable for every install, import, test, and job; + do not require `.venv` when the user selected an existing environment. +- Run `pip check` with that executable. +- Import every function module from the same working directory/import path as the + runner. +- Resolve nested qualnames. +- Run focused pytest tests for Actions, Tools, serialization, and graph branches. +- Check the runner's input/output types, key selector, and packaging of assets. +- When `package` Skills are selected, require an installable application package, + include the full Skill tree as package data, install it into the selected Python + environment, and resolve the configured resource from the installed package. +- Require `StreamExecutionEnvironment.get_execution_environment()` followed by + `AgentsExecutionEnvironment.get_execution_environment(env)`, which is backed by + `RemoteExecutionEnvironment`. Reject the no-argument Agents factory, + `from_list`/`to_list`, and local Agents environment APIs. + +Java: + +- Require a complete Maven project with `flink-agents-api`, `flink-agents-plan`, + `flink-agents-runtime`, and only the integration artifacts actually used. +- Require explicit Flink streaming, client, and Table bridge dependencies. Require + the matching Table planner when the job uses Table API, and selected connector + dependencies for non-built-in sources or sinks. +- Require `provided` scope on every Flink Agents and Flink dependency. Do not use a + dist artifact or `flink-agents-ide-support` as an application dependency. +- Run `mvn dependency:tree` and inspect the resolved versions before compilation. +- Run the remote-style main class against a local MiniCluster in a forked JVM, for + example through the generated `exec:exec` configuration; do not treat `exec:java` + as equivalent evidence. +- Compile with the target Maven/Gradle command and configured JDK. +- Require `type: java` for Java YAML implementations. +- Match each Tool's `parameter_types` to the method parameters in order. +- Inspect the built JAR for YAML and, when `classpath` Skills are selected, the + configured Skill resource tree. +- Require `AgentsExecutionEnvironment.getExecutionEnvironment(env)`, which is backed + by `RemoteExecutionEnvironment`. Reject local Agents environment and list APIs. + +Cross-language: + +- Treat the selected API/application-code language and each Resource implementation + language as independent dimensions. +- For chat-model connections/setups, embedding-model connections/setups, and vector + stores, selecting a provider implementation is sufficient confirmation of its + language. Do not require an additional application-wide cross-language question. +- Confirm the Resource type supports the bridge in the target-version API, not only + in the matching YAML docs; direct Python and direct Java APIs support these bridges + too. +- Confirm the generated descriptor uses the correct wrapper plus `java_clazz` or + `pythonClazz` metadata, or the equivalent target-version factory. +- Confirm the selected language's integration artifact and bridge runtime are + available to every TaskManager. +- Run the repository's cross-language tests or an equivalent focused test. +- Do not assume MCP, Skill source schemes, or arbitrary providers bridge languages. + +## 6. Resource-specific Checks + +All Resources: + +- Each Resource was requested by the user or required by a confirmed reference; + reject Resources inferred only from the application domain or a sample. +- Each descriptor-backed Resource uses the user-selected implementation class or + documented alias. Its declaration contains every mandatory target-version + argument as a typed/commented `TODO_REQUIRED_*` placeholder; reject additional + configuration interviews or invented optional values. +- New-project Resource names are deterministic and all references use them + consistently. Do not require user confirmation for an unambiguous generated name; + require naming input only for collisions, external contracts, or an explicit + naming convention. Existing names remain unchanged unless renaming was requested. +- Model names, endpoints, authentication, and optional provider settings are either + explicitly supplied by the user or left as clear TODOs. Reject assumed provider + values and reject scaffolding that blocks on collecting these values. +- No tracked file contains a supplied secret. Plaintext in a user-selected local + YAML is valid when the file is ignored, the local entry point actually loads it, + and output/reporting redacts the value. In a Git worktree, run `git check-ignore` + and inspect `git status --short` before runtime validation. +- Do not treat `${ENV_VAR}` in YAML as secret injection unless the target + loader/provider explicitly resolves it. Do not require programmatic Resource + registration when the user selected a literal value in ignored local YAML. +- Resource references are resolved in dependency order, and the required + integration artifact is selected only after the implementation is known. + +Custom Tools, Actions, and domain clients: + +- Preserve supplied names/signatures/types/descriptions. When they are absent, + verify that capability-derived neutral signatures compile or import and their + bodies fail explicitly; missing business contracts are TODOs, not failed gates. +- Reject guessed Flink REST endpoints, service protocols, domain transformations, + and mock/test implementations unless the user explicitly requested and specified + them. +- Reject scaffolding workflows that require the user to choose business identity + fields, Flink/VVR/VVP platforms, log/metric interfaces, business authentication, + or response schemas before files are generated. + +Runtime Skills: + +- Each Skill directory contains valid `SKILL.md` frontmatter. +- A user-fillable Skill scaffold contains only capability-derived metadata and a + clear TODO; + it must not contain invented domain instructions or commands. +- For a new project, the source is an explicit TODO and no coding-agent host Skill + was inspected, copied, or offered for reuse. Do not require a path, URL, package, + classpath, or distribution answer before scaffolding. +- Once the user fills it, the YAML field or direct `Skills` factory matches the source: + bundled Python uses `package`/`from_package`, bundled Java uses + `classpath`/`fromClasspath`, TaskManager-managed files use + `paths`/`from_local_dir`/`fromLocalDir`, and remote ZIP distribution uses + `urls`/`from_url`/`fromUrl`. +- A `package` source is Python-only and its resource is present in the installed + package/wheel on every Python worker; a `classpath` source is Java-only and its + resource is present in the application JAR or runtime classpath. +- For a filled source, every `paths` directory or ZIP is provisioned at a resolvable path on every + TaskManager. A local MiniCluster check is not cluster-wide path evidence. +- For a filled source, every `urls` value is HTTP(S), points to a ZIP with Skill directories at its top + level, and is reachable from every TaskManager. Prefer immutable, versioned URLs; + do not claim cluster connectivity from a client-side download. +- Cross-language Skill sources use `paths` or `urls` and are tested across the + confirmed bridge. +- If fields are intentionally combined, account for loader order `paths`, `urls`, + `classpath`, `package`; reject duplicate Skill names rather than relying on + last-wins replacement. +- Chat model `skills` entries match individual Skill names, not the Skills Resource + name. +- `allowed_commands` contains only commands actually required by the enabled Skills. + +MCP: + +- Endpoint/auth configuration matches the language-specific docs. +- Discovered prompt/Tool names exist on the target server. +- Static checks do not claim MCP connectivity. + +Vector stores and embedding models: + +- Provider and language are supported together. +- Connection/setup/vector-store name references resolve. +- Collection/index, dimensions, and backend arguments match the provider. +- Tests distinguish schema/load checks from live backend queries. + +File sources: + +- Point streaming sources at input files/directories only. +- Exclude YAML, Skill, and other Resource assets from recursive input enumeration. + +## 7. Runtime Evidence + +Provide exact commands and name the user-confirmed configuration/credential +mechanism without reproducing a secret. Mention environment variables only when the +user selected them and the generated wiring resolves them. Label evidence precisely: + +- `schema valid`: a JSON Schema validator completed successfully; +- `loads`: the Flink Agents loader built the definitions; +- `imports`: Python references resolved; +- `compiles`: Java build completed; +- `tests pass`: name the command and result; +- `MiniCluster deployment passed`: the remote-style bounded Flink job was submitted + locally and finished; an empty typed source proves deployment only; +- `behavior smoke passed`: non-empty input invoked implemented business functions, + produced the expected sink output, and the job terminated successfully; +- `integration verified`: the external model/MCP/vector service was actually reached. + +Never upgrade `compiles` to MiniCluster deployment, deployment-only evidence to +behavior evidence, or `schema valid` to `provider configuration works`. State +skipped checks and the exact missing implementation, service, credential, +dependency, or environment. + +## Final Report Template + +```text +Changed: +- + +Selected versions: +- Flink Agents ; Flink + +Selected API: +- + +Implementation language: +- + +Python environment, when required: +- + +Selected Resource implementations: +- + +Runtime Skill source: +- +- + +User must provide: +- +- +- +- + +Verified: +- : + +Not run: +- : +``` diff --git a/dev/agent-skills/flink-agents-dev/references/yaml-patterns.md b/dev/agent-skills/flink-agents-dev/references/yaml-patterns.md new file mode 100644 index 000000000..c8c676312 --- /dev/null +++ b/dev/agent-skills/flink-agents-dev/references/yaml-patterns.md @@ -0,0 +1,276 @@ +# YAML Patterns + +## Contents + +- [Contract First](#contract-first) +- [Canonical Workflow Shape](#canonical-workflow-shape) +- [Section Rules](#section-rules) +- [Language Selection](#language-selection) +- [Name-resolution Pass](#name-resolution-pass) + +## Contract First + +Use the [bundled YAML schema](../assets/yaml-schema.json) as the offline contract +before generating YAML. If the target application provides a schema for its pinned +Flink Agents version, use that matching schema instead. The schema defines structure; +target-version provider metadata defines the additional arguments forwarded by +Resource descriptors. + +The valid top-level sections are: + +```text +agents, actions, chat_model_connections, chat_model_setups, +embedding_model_connections, embedding_model_setups, prompts, tools, +skills, vector_stores, mcp_servers +``` + +Top-level Resources and Actions are shared. The same sections nested under an +`agents[]` entry belong to that Agent. There is no `resources:` wrapper and no Agent +`type` field. + +## Canonical Workflow Shape + +This example deliberately shows only the Action structure. It contains no model, +Prompt, Tool, Skill, MCP server, or vector store because those Resources must come +from the staged user interview. Generate `app.actions` with matching skeletons; do +not infer what either Action emits. + +```yaml +agents: + - name: application_agent + + actions: + - name: process_input + function: app.actions:process_input + trigger_conditions: [input] + type: python +``` + +Add downstream Actions only after their trigger events are confirmed. A skeleton's +existence proves only that the YAML reference resolves; it does not prove an event +is emitted. + +## Section Rules + +### Actions + +An inline Action requires `name`, non-empty `trigger_conditions`, and a valid +`function` for custom behavior. `config` is optional. An Agent may also reference a +top-level shared Action by a bare string. + +```yaml +actions: + - name: shared_input + function: app.actions:shared_input + trigger_conditions: [input] + +agents: + - name: first_agent + actions: [shared_input] +``` + +Event aliases include `input`, `output`, `chat_request`, `chat_response`, +`tool_request`, `tool_response`, `context_retrieval_request`, and +`context_retrieval_response`. Use the exact `EVENT_TYPE` string for custom events. + +### Function References + +Use exactly one colon: + +```yaml +# Python top-level function +function: app.actions:process_input + +# Python static method +function: app.actions:ReviewActions.process_input + +# Java static method +function: com.example.agent.Actions:processInput + +# Java nested class +function: com.example.Outer$Actions:processInput +``` + +The left side is a Python module or Java class FQN; the right side is the qualname. +Do not use a single dotted path in place of the colon. + +### Prompts and Tools + +A Prompt has exactly one of `text` or `messages`. Template substitution uses +`{name}`. A Tool needs a callable `function`; Java Tools also need ordered +`parameter_types`, while Python Tools must not declare them. + +```yaml +tools: + - name: add + type: java + function: com.example.Tools:add + parameter_types: [java.lang.Integer, java.lang.Integer] +``` + +The example shows the YAML shape, not a required business interview. If the user +requested a Tool capability but did not supply its contract, generate a capability- +derived name and a neutral one-string-input/one-string-output function skeleton. +For Java, set `parameter_types: [java.lang.String]`; for Python, omit +`parameter_types`. Put the unknown request fields, platform client, authentication, +query, and response mapping in TODOs. Do not ask the user to choose those details +before generating the project. + +### Descriptor-backed Resources + +Chat-model and embedding connections/setups, vector stores, and MCP servers require +`name` and `clazz`. Their remaining fields are forwarded to the provider. Use only +arguments documented for the selected provider and language. Resolve one Resource +at a time: ask for `clazz` first, inspect that target-version implementation, then +generate its required arguments as `TODO_REQUIRED_*` fields. Do not ask the user for +those values, and do not copy a model, endpoint, or authentication setting from this +or another example. If target-version metadata does not reveal the full constructor +contract, add `TODO_VERIFY_REQUIRED_PROVIDER_ARGUMENTS` and report the gap. + +For chat-model connections/setups, embedding-model connections/setups, and vector +stores, the descriptor's `type` is independent of the custom Action/Tool and Flink +entry-point language. Offer aliases from both Python and Java buckets when the +target-version loader exposes the corresponding wrapper. A Python-loaded YAML may +use `type: java`; a Java-loaded YAML may use `type: python`. Set `type` explicitly +from the selected Resource implementation and add its integration/runtime artifact. +Do not apply this cross-language rule to MCP servers, Skills, Tools, Prompts, or +other Resource types without verified bridge support. + +The required `name` is an internal YAML identifier, not automatically a user +decision. For one Resource of a role, generate the defaults documented in +`application-patterns.md`, including `chat_model_connection` and `chat_model`, and +write the matching Setup `connection` reference directly. Preserve existing names. +Ask for naming input only to resolve multiple-resource ambiguity, satisfy an exact +external reference, or follow a user-requested convention. + +The bundled YAML loader has no general environment-variable interpolation for +descriptor arguments. Do not turn an unknown credential into `${API_KEY}` or claim +that it will be resolved. A literal provider argument in YAML is different from +interpolation: when the user chooses plaintext for local testing, write the literal +value to an untracked local YAML using the provider's documented field, such as +`api_key`, and load that YAML directly. Do not force programmatic `addResource` +registration merely because interpolation is unavailable. + +Keep the local YAML outside tracked application resources where practical, add its +exact path or pattern to `.gitignore`, verify it is ignored, and ensure the local +runner or command loads that file. Warn once that the file and any locally built +artifact containing it hold plaintext. Never echo or reproduce the value in logs, +commands, diffs shown to the user, or the final report. Programmatic registration +and provider-supported secret stores remain valid alternatives, not mandatory +replacements for the user's local-testing choice. + +MCP prompts and Tools are discovered dynamically and referenced by each prompt or +Tool's advertised name, just like local Prompts and Tools. The MCP server Resource +name is separate. Verify discovered names and collisions against the MCP server; +the static schema cannot do so. + +### Runtime Skills + +At least one source list must be non-empty. YAML describes the selected source; it +does not decide how Skills are deployed: + +| YAML field | Runtime scheme | Supported loader | Required deployment condition | +|---|---|---|---| +| `paths` | `local` | Python and Java | Each TaskManager can resolve the directory or ZIP path | +| `urls` | `url` | Python and Java | Each TaskManager can reach the HTTP(S) ZIP URL | +| `classpath` | `classpath` | Java only | Skill resources are present on the runtime classpath, normally in the application JAR | +| `package` | `package` | Python only | Skill resources are package data in an installed Python package/wheel | + +Preserve an existing explicit source unless the user requests a change. For a new +application, do not ask for source paths, URLs, packages, classpath locations, or +distribution. Generate a visibly unresolved declaration that lists the valid forms: + +```yaml +skills: + - name: runtime_skills + # TODO(required): replace this placeholder with the chosen source. + # Supported forms: paths, urls, Python package, or Java classpath. + paths: [TODO_REQUIRED_SKILL_SOURCE] +``` + +`paths` is a schema-shaped placeholder in this scaffold, not a selected deployment +mode. The generated application is not runtime-ready until the user replaces it. +Do not package, mount, download, or validate a source that the user has not supplied. + +Do not choose `package` merely because the implementation is Python or `classpath` +merely because it is Java. Language only constrains the valid bundled scheme. Use +`paths` or `urls` for a supported cross-language source. A relative `paths` entry is +resolved in the TaskManager runtime, not guaranteed to be the submitting client's +working directory; success in a local MiniCluster does not prove cluster-wide path +availability. + +Multiple source fields can coexist when the user explicitly requests composition. +The loaders append sources in `paths`, `urls`, `classpath`, `package` order, and a +later source replaces an earlier registration with the same Skill frontmatter name. +Avoid duplicate names and implicit fallback behavior. An unsupported scheme fails +at load time; a runtime does not skip it as a fallback. Prefer immutable, versioned +URLs; the YAML schema does not provide a checksum field, so do not invent one. + +Declaring a Skills Resource only makes Skills available. To activate one after its +business content and source are filled, add its `SKILL.md` name to a chat model +setup's `skills` list. Add `allowed_commands` only when the user later supplies +Skill behavior requiring shell operations; `load_skill` and `bash` are added +automatically. + +Runtime Skill instructions are business content. Derive minimal metadata from the +stated capability and leave a focused TODO body. Do not ask implementation questions +or invent runbooks, diagnostic rules, tool sequences, shell commands, or safety +policy. Never inspect or reuse a coding-agent host Skill such as a locally installed +`flink-diag`; host Skills and Flink runtime Skills have different owners and +deployment contracts. + +```markdown +--- +name: capability-derived-skill-name +description: TODO: Refine the runtime Skill purpose and trigger. +--- + +# Capability-derived Skill Name + +TODO: Define the domain workflow, evidence rules, and permitted tools. +``` + +## Language Selection + +Missing `type` means `python` for both loaders. Mark each Action, Tool, and +descriptor-backed Resource with its own implementation language rather than copying +one application-wide value. The bundled snapshot supports bidirectional bridging +for chat-model connections/setups, embedding-model connections/setups, and vector +stores. Consult the target version's YAML docs and plan compiler before assuming +that another Resource type bridges languages. + +The schema default is not a product decision. For a new application, first confirm +the Flink Agents/Flink pair and then the YAML API choice. Only then ask the user to +choose Python or Java before generating functions, Resources, dependencies, or the +Flink entry point. Present Python and Java as native single-select options or a +numbered fallback, not an open-ended language question. Treat them as equal peer +options: use parallel descriptions, label neither one `(Recommended)`, and preselect +neither one. Apply the user's choice to custom Action/Tool implementations and the +Flink entry point, and do not mention a Python/JDK version before it. Preserve +explicit language choices in existing YAML. Do not require a separate cross-language +confirmation when the user selects a bridge-supported Resource implementation; that +Resource's selector already records the choice. + +In the current source, Java `AgentPlan` rejects MCP servers added through +`Agent.addResource`; Java `YamlLoader` represents YAML `mcp_servers` through that +path. Therefore, do not claim that a Java-loaded YAML application with +`mcp_servers` is runnable, regardless of `type`. Use the documented Java +`@MCPServer` programmatic definition for that application, or verify that the +target version has removed this restriction. Do not invent a YAML/MCP bridge or +adapter. + +## Name-resolution Pass + +Before loading, build and check this graph: + +- setup `connection` -> declared connection name; +- setup `prompt` -> declared Prompt name; +- setup `tools[]` -> local Tool or MCP-discovered Tool name; +- setup `skills[]` -> individual runtime Skill frontmatter name; +- embedding setup `connection` -> embedding connection name; +- vector-store `embedding_model` -> embedding setup name; +- `ChatRequestEvent.model` in implementation code -> chat setup name; +- Action and Tool `function` -> importable Python callable or Java static method. + +Schema validation cannot prove every dynamic name or provider argument. Run the +loader/build checks from `verification.md` after schema validation. diff --git a/tools/.rat-excludes b/tools/.rat-excludes index dcd7de725..776daadcc 100644 --- a/tools/.rat-excludes +++ b/tools/.rat-excludes @@ -22,7 +22,8 @@ PULL_REQUEST_TEMPLATE.md .*\.egg-info/* licenses/* skills/* +flink-agents-dev .*\.yaml$ AGENTS.md code_review.md -CLAUDE.md \ No newline at end of file +CLAUDE.md