Skip to content

Improve aqa & testgen#89

Draft
sveto wants to merge 6 commits into
mainfrom
improve-aqa-testgen
Draft

Improve aqa & testgen#89
sveto wants to merge 6 commits into
mainfrom
improve-aqa-testgen

Conversation

@sveto
Copy link
Copy Markdown

@sveto sveto commented May 20, 2026

Subtask 1: add a step "add a step if AI can create the test by itself, it should at least ask or have an option to fulfill the task by itself".

Done by editing the general AQA flow, phases 1 and 2 (data collection and requirements clarification) and just briefly mentioning it in phase 6 (test implementation). Tested statically, scenario-wise and manually.

Subtask 2: add different behavior in cases when the project does (not) have an option on using MCPs.

Done by editing AQA Phase 1 (data collection), Phase 6 (test implementation),testgen-flow.md, mcp-capability-interaction.md.

Subtask 3: change hardcoded names, URLs, tools and IDs to placeholders. Done.

Static validation and CI validation done.

@sveto sveto self-assigned this May 20, 2026
@github-actions github-actions Bot added the enhancement New feature or request label May 20, 2026
@github-actions
Copy link
Copy Markdown
Contributor

Rosetta Triage Review

Summary: This PR extends the AQA and TestGen workflows with two complementary capabilities: a new minimal-input / agent-led execution mode for AQA (allowing automation without TestRail/Confluence MCP), and a shared mcp-capability-interaction.md fragment that gates guided vs. questionnaire paths based on agents/mcp-capability.yaml — enabling both workflows to function in environments where external MCP integrations are absent or partially available.

Findings:

  • Draft status: PR is marked as Draft with CI validation in progress; not yet ready for merge.
  • Well-scoped: Three subtasks are cleanly separated — execution mode locking (Phase 1 Task 0 hard gate), MCP capability routing, and placeholder standardization. No mixed concerns.
  • Hard gate correctness: Task 0 explicitly forbids mode inference from clues (missing TestRail, local HTML) and requires explicit user keywords or a mandatory ask-and-wait loop. This is a strong guardrail against agent hallucination.
  • DRY pattern: mcp-capability-interaction.md as a shared fragment acquired by both AQA and TestGen is a correct application of Rosetta's progressive-disclosure model.
  • Plugin mirrors updated: All plugin variants (Cursor, Copilot standalone/plugin, Codex) correctly mirror the core instruction changes. The bulk of the +2426 additions comes from these mirrors — functionally expected.
  • Version not bumped: gain.json still shows "rosetta": "2.0.19". The changes include a new workflow file (mcp-capability-interaction.md), a new template, and significant behavioral additions to Phase 1. Consider whether a version bump to 2.0.20 (or higher) is warranted before merging.
  • New template file: instructions/r2/core/templates/mcp-capability.example.yaml is well-commented and provides a clear starting point for workspace configuration.
  • Web docs aligned: docs/web/docs/aqa-flow.md and docs/web/docs/testgen-flow.md are updated in sync with the instruction changes — good practice.

Suggestions:

  • Consider bumping gain.json versions.rosetta to reflect the new feature additions before merging.
  • The mcp-capability.example.yaml template includes both confluence and atlassian_confluence keys — confirm whether both are needed or if one is an alias, to avoid confusion for users copying the template.
  • Once CI passes and the Draft flag is removed, this is ready for a standard code review.

Automated triage by Rosetta agent

@sveto sveto changed the title Draft: Improve aqa & testgen Improve aqa & testgen May 21, 2026
@sveto sveto changed the title Improve aqa & testgen Draft: Improve aqa & testgen May 21, 2026
@isolomatov-gd isolomatov-gd marked this pull request as draft May 21, 2026 12:35
@sveto sveto marked this pull request as ready for review May 21, 2026 12:39
@sveto sveto changed the title Draft: Improve aqa & testgen Improve aqa & testgen May 21, 2026
@github-actions github-actions Bot added the documentation Improvements or additions to documentation label May 21, 2026
@github-actions
Copy link
Copy Markdown
Contributor

Rosetta Triage Review

Summary: This PR extends the AQA and TestGen workflows with two new capabilities: (1) a "minimal-input / agent-led" execution mode for AQA Phase 1 that allows test creation without TestRail/Confluence when the user supplies sufficient grounding; and (2) a shared MCP capability routing layer (guided vs questionnaire) driven by an optional workspace agents/mcp-capability.yaml file and three new reusable workflow fragments.

Findings:

  • Modular design is solid: the three new fragments (mcp-capability-interaction.md, mcp-tool-resolution.md, agent-action-resolution.md) each own one concern and are cleanly referenced by parent workflows — consistent with existing Rosetta SRP patterns.
  • AQA Task 0 hard gate is well-engineered: it forbids creating plan artifacts or marking Phase 1 complete until an explicit mode keyword is confirmed, preventing the common failure of inferring minimal-input from missing TestRail data.
  • Dual YAML key naming (confluence for TestGen, atlassian_confluence for AQA) referring to the same Confluence MCP is documented in both the template and the fragment, but remains a latent misconfiguration risk for teams using both workflow families simultaneously — the template comment and the Path conventions table in mcp-capability-interaction.md mitigate this adequately.
  • Plugin mirrors are correct: all five plugin targets (core-codex, core-copilot, core-copilot-standalone, core-cursor-standalone, core-copilot) are updated consistently, matching the expected propagation pattern.
  • gain.json rosettaInstructionUpdates is populated correctly, providing the agent-readable changelog for both subtask 1 and subtask 2.
  • Pre-existing typo in gain.json: "e2e_tests": "din-repository" (extra 'd') — not introduced by this PR but visible in the diff; worth a follow-up fix.
  • Test coverage: only test-library/testgen/trigger.txt is touched for automated validation; PR description states static and manual scenario testing was done. For instruction-content changes this is acceptable, though scenario-level regression notes in the PR body would improve traceability.

Suggestions:

  • Consider renaming the confluence / atlassian_confluence dual-key pattern to a single key in a future cleanup, or add a lint check to the template that warns when only one of the two is set.
  • Fix the gain.json e2e_tests typo (din-repositoryin-repository) in a follow-up PR or squash it here before merge.

Automated triage by Rosetta agent

@sveto sveto marked this pull request as draft May 21, 2026 12:49
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

documentation Improvements or additions to documentation enhancement New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant