Skip to content

feat: Unify approvals and verification - #113

Merged
moshloop merged 5 commits into
mainfrom
feat/approval-workflow-verification
Sep 4, 2026
Merged

feat: Unify approvals and verification#113
moshloop merged 5 commits into
mainfrom
feat/approval-workflow-verification

Conversation

@moshloop

@moshloop moshloop commented Sep 4, 2026

Copy link
Copy Markdown
Member

What

  • Add durable approvals for provider and caller-tool runs, including persistence, expiry, cancellation, and resolution.
  • Replace session DoD gates with workflow verification across CLI and agent runs, with typed reports, progress events, and iteration history.
  • Preserve run posture and caller-owned commit semantics while centralizing verification persistence and documenting AI agent workflows.

Notes

  • Breaking change: remove captain dod and hooks; use captain verify.

…ool runs

Introduce a shared durable approval lifecycle for caller-tool and provider executions. Provider approvals can now be resolved without turn or model-call records, while caller-tool approvals retain credential-scoped identity and validation. Add migration rerun safeguards and integration coverage for approval persistence, expiry, cancellation, and resolution.
Unify command, prompt, and fixture checks across CLI and agent runs, with typed reports, progress events, and persisted iteration history. Fail loudly when verification, models, timeouts, or runners are unconfigured instead of passing silently.

BREAKING CHANGE: Remove the captain dod command and hooks; use workflow verification via captain verify.
Pass the resolved run spec to verifier factories so nested graders inherit the run's model and permissions. Allow hosts with their own commit pipeline to suppress duplicate commit hooks while retaining workflow declarations for validation.
…t workflows

Centralize prompt-run iteration persistence so CLI and embedding hosts record generated and verify-only runs consistently, including complete verification rounds. Add comprehensive AI agent, verification, approvals, and embedding documentation, expose batched latest-verification queries, and surface rejected capability changes accessibly. Keep `captain verify` local-only to prevent unauthenticated remote command execution.
@coderabbitai

coderabbitai Bot commented Sep 4, 2026

Copy link
Copy Markdown

Important

Review skipped

Too many files!

This PR contains 148 files, which is 48 over the limit of 100.

To get a review, reduce the PR to 100 files or fewer by splitting it into smaller PRs or changing its base branch.

Upgrade to a paid plan to raise the limit.

This review couldn't start because sufficient usage credits or metered capacity aren't available. Add credits or update usage-based reviews in the billing tab, then retry.

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Team

Run ID: c1cdfafc-4bcf-4e47-be40-42bd595bc72a

📥 Commits

Reviewing files that changed from the base of the PR and between cc9e897 and 5204d4c.

⛔ Files ignored due to path filters (1)
  • docs/pnpm-lock.yaml is excluded by !**/pnpm-lock.yaml
📒 Files selected for processing (148)
  • README.md
  • cmd/captain/help_test.go
  • cmd/captain/local_only_test.go
  • cmd/captain/main.go
  • docs/src/data/navigation.ts
  • docs/src/pages/agents/approvals.mdx
  • docs/src/pages/agents/embedding.mdx
  • docs/src/pages/agents/index.mdx
  • docs/src/pages/agents/verification.mdx
  • docs/src/pages/prompts/sources-api.mdx
  • migrations/32_execution_approvals.pg.hcl
  • migrations/74_turn_request_approval_identity.sql
  • migrations/81_turn_request_provider_approval_identity.sql
  • migrations/approval_identity_upgrade_integration_test.go
  • migrations/migrations.go
  • migrations/schema_ginkgo_test.go
  • pkg/ai/agent/agent_suite_test.go
  • pkg/ai/agent/commit/workflow.go
  • pkg/ai/agent/hook_context.go
  • pkg/ai/agent/hooks.go
  • pkg/ai/agent/runner.go
  • pkg/ai/agent/runner_test.go
  • pkg/ai/agent/runner_verify_only_ginkgo_test.go
  • pkg/ai/agent/scope.go
  • pkg/ai/agent/verify/exec.go
  • pkg/ai/agent/verify/external.go
  • pkg/ai/agent/verify/external_contract_ginkgo_test.go
  • pkg/ai/agent/verify/external_ginkgo_test.go
  • pkg/ai/agent/verify/llmjudge.go
  • pkg/ai/agent/verify/notice_test.go
  • pkg/ai/agent/verify/progress.go
  • pkg/ai/agent/verify/prompt_hooks_test.go
  • pkg/ai/agent/verify/registry.go
  • pkg/ai/agent/verify/registry_ginkgo_test.go
  • pkg/ai/agent/verify/report_ginkgo_test.go
  • pkg/ai/agent/verify/verify.go
  • pkg/ai/agent/verify/verify_suite_test.go
  • pkg/ai/agent/verify/workflow.go
  • pkg/ai/agent/verify/workflow_test.go
  • pkg/ai/agent/verify_order_test.go
  • pkg/ai/approval/approval_suite_test.go
  • pkg/ai/approval/broker.go
  • pkg/ai/approval/broker_ginkgo_test.go
  • pkg/ai/approval/caller_tool_run_ginkgo_test.go
  • pkg/ai/loop.go
  • pkg/ai/types.go
  • pkg/aichat/execution_database.go
  • pkg/aichat/execution_database_correlation.go
  • pkg/aichat/execution_database_state.go
  • pkg/aiflags/defaults.go
  • pkg/aiflags/unconfigured.go
  • pkg/aiflags/unconfigured_test.go
  • pkg/api/runtime_event.go
  • pkg/api/spec.go
  • pkg/api/verify_merge.go
  • pkg/api/verify_merge_ginkgo_test.go
  • pkg/api/verify_report.go
  • pkg/api/verify_report_ginkgo_test.go
  • pkg/api/workflow.go
  • pkg/api/workspace.go
  • pkg/captainconfig/config.go
  • pkg/captainconfig/config_test.go
  • pkg/claude/tools/tool.go
  • pkg/claude/tools/verify.go
  • pkg/cli/ai.go
  • pkg/cli/ai_agent.go
  • pkg/cli/ai_agent_test.go
  • pkg/cli/ai_sandbox_remote.go
  • pkg/cli/attachments_ginkgo_test.go
  • pkg/cli/captain_config_once.go
  • pkg/cli/dod.go
  • pkg/cli/event_renderer.go
  • pkg/cli/event_renderer_ginkgo_test.go
  • pkg/cli/gitagent_e2e_test.go
  • pkg/cli/gitagent_runtask.go
  • pkg/cli/hook.go
  • pkg/cli/prompt_batch_run.go
  • pkg/cli/prompt_chat.go
  • pkg/cli/prompt_chat_http.go
  • pkg/cli/prompt_observe.go
  • pkg/cli/prompt_render.go
  • pkg/cli/prompt_render_test.go
  • pkg/cli/prompt_run.go
  • pkg/cli/prompt_run_events.go
  • pkg/cli/prompt_run_events_test.go
  • pkg/cli/prompt_run_iterations.go
  • pkg/cli/prompt_run_iterations_test.go
  • pkg/cli/prompt_run_live.go
  • pkg/cli/prompt_run_persist.go
  • pkg/cli/prompt_run_persist_test.go
  • pkg/cli/prompt_run_stream.go
  • pkg/cli/prompt_run_stream_verify.go
  • pkg/cli/prompt_run_workflow_test.go
  • pkg/cli/prompt_schema.go
  • pkg/cli/prompt_schema_verifiers.go
  • pkg/cli/prompt_schema_verifiers_ginkgo_test.go
  • pkg/cli/prompt_spec.go
  • pkg/cli/prompt_spec_test.go
  • pkg/cli/prompt_workflow.go
  • pkg/cli/provider_defaults.go
  • pkg/cli/serve_disabled.go
  • pkg/cli/verify.go
  • pkg/cli/verify_ginkgo_test.go
  • pkg/cli/verify_install_ginkgo_test.go
  • pkg/cli/webapp/src/PromptRunStream.test.tsx
  • pkg/cli/webapp/src/PromptRunStream.tsx
  • pkg/cli/webapp/src/StateMessage.tsx
  • pkg/cli/webapp/src/WhoamiPage.test.tsx
  • pkg/cli/webapp/src/WhoamiTopology.tsx
  • pkg/cli/webapp/src/hooks/useEventSource.test.ts
  • pkg/cli/webapp/src/hooks/useEventSource.ts
  • pkg/cli/webapp/src/hooks/usePromptRunStream.test.tsx
  • pkg/cli/webapp/src/hooks/usePromptRunStream.ts
  • pkg/cli/webapp/src/types/verifyReport.test.ts
  • pkg/cli/webapp/src/types/verifyReport.ts
  • pkg/database/caller_tool_store.go
  • pkg/database/prompt_run_iteration_store.go
  • pkg/database/prompt_run_iteration_store_integration_test.go
  • pkg/database/prompt_run_iteration_store_validation_integration_test.go
  • pkg/database/prompt_run_overview.go
  • pkg/database/session_notice_store.go
  • pkg/database/session_notice_store_integration_test.go
  • pkg/database/tool_approval_store.go
  • pkg/dod/cache.go
  • pkg/dod/cache_test.go
  • pkg/dod/run.go
  • pkg/dod/run_test.go
  • pkg/dod/skill.go
  • pkg/dod/skills/dod-clear.md
  • pkg/dod/skills/dod-run.md
  • pkg/dod/skills/dod-status.md
  • pkg/dod/skills/dod.md
  • pkg/gitagent/hookmain.go
  • pkg/gitagent/hookset.go
  • pkg/gitagent/hookset_ginkgo_test.go
  • pkg/promptrun/hooks.go
  • pkg/promptrun/iterations.go
  • pkg/promptrun/iterations_ginkgo_test.go
  • pkg/promptrun/promptrun_ginkgo_test.go
  • pkg/promptrun/promptrun_suite_test.go
  • pkg/promptrun/provider.go
  • pkg/promptrun/provider_ginkgo_test.go
  • pkg/promptrun/result.go
  • pkg/promptrun/result_ginkgo_test.go
  • pkg/promptrun/run.go
  • pkg/session/message.go
  • pkg/session/transcript_pretty.go
  • pkg/session/transcript_verify_ginkgo_test.go

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.


Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@github-actions

github-actions Bot commented Sep 4, 2026

Copy link
Copy Markdown

Gavel summary

Source Pass Fail Skip Duration

Totals: 0 passed · 0 failed · 0 skipped · -

View full results

Comment thread pkg/cli/prompt_run_iterations.go Fixed
@github-actions

github-actions Bot commented Sep 4, 2026

Copy link
Copy Markdown

Gavel summary

Source Pass Fail Skip Duration
agent 5 0 0 278.01µs
agentcreds 11 0 0 738.39µs
ai 90 0 0 107ms
aichat 151 0 2 47.3s
anthropicmock 1 0 0 1ms
api 291 0 0 117ms
approval 16 0 0 2.1s
attachments 5 0 0 22ms
bash 4 0 0 206.094µs
callertools 10 0 0 116ms
captain 31 0 0 29.8s
claude 22 0 0 13ms
claudeagent 13 0 0 422ms
cli 334 0 0 9.1s
cmux 4 0 0 214.831µs
collections 12 0 0 356.247µs
credentials 5 0 0 93ms
credsync 11 0 0 422ms
database 22 0 0 23.4s
deploy 110 0 0 960ms
genkit 25 0 0 221ms
gitagent 123 0 0 36.4s
github.com/flanksource/captain/migrations 11 0 0 3.9s
github.com/flanksource/captain/pkg/ai 245 0 0 610ms
github.com/flanksource/captain/pkg/ai/agent 26 0 0 450ms
github.com/flanksource/captain/pkg/ai/agent/commit 56 0 0 4.1s
github.com/flanksource/captain/pkg/ai/agent/setup 16 0 0 50ms
github.com/flanksource/captain/pkg/ai/agent/verify 32 0 0 390ms
github.com/flanksource/captain/pkg/ai/agent/worktree 6 0 0 -
github.com/flanksource/captain/pkg/ai/assistanttags 15 0 0 -
github.com/flanksource/captain/pkg/ai/fixture 43 0 0 150ms
github.com/flanksource/captain/pkg/ai/fixture/kubeproxy 2 0 0 20ms
github.com/flanksource/captain/pkg/ai/fixture/mcpproxy 6 0 0 -
github.com/flanksource/captain/pkg/ai/history 54 0 0 -
github.com/flanksource/captain/pkg/ai/internal/gen-model-registry 12 0 0 -
github.com/flanksource/captain/pkg/ai/middleware 25 0 0 -
github.com/flanksource/captain/pkg/ai/observation 2 0 0 -
github.com/flanksource/captain/pkg/ai/pricing 9 0 0 10ms
github.com/flanksource/captain/pkg/ai/prompt 24 0 0 -
github.com/flanksource/captain/pkg/ai/provider 177 0 1 20ms
github.com/flanksource/captain/pkg/ai/provider/claudeagent 54 0 0 1.4s
github.com/flanksource/captain/pkg/ai/provider/cmux 128 0 0 880ms
github.com/flanksource/captain/pkg/ai/provider/genkit 39 0 0 -
github.com/flanksource/captain/pkg/ai/provider/jsonrpc 6 0 0 50ms
github.com/flanksource/captain/pkg/ai/provider/openai 8 0 0 20ms
github.com/flanksource/captain/pkg/aichat 9 0 0 100ms
github.com/flanksource/captain/pkg/aiflags 9 0 0 -
github.com/flanksource/captain/pkg/aimock 50 0 7 290ms
github.com/flanksource/captain/pkg/aimock/anthropicmock 14 0 0 10ms
github.com/flanksource/captain/pkg/aimock/openaimock 18 0 0 20ms
github.com/flanksource/captain/pkg/api 138 0 0 350ms
github.com/flanksource/captain/pkg/api/registry 127 0 0 -
github.com/flanksource/captain/pkg/bash 348 0 0 10ms
github.com/flanksource/captain/pkg/captainconfig 29 0 0 -
github.com/flanksource/captain/pkg/captaintoken 23 0 0 1.8s
github.com/flanksource/captain/pkg/claude 138 0 0 -
github.com/flanksource/captain/pkg/claude/tools 18 0 0 -
github.com/flanksource/captain/pkg/cli 818 0 1 10m3s
github.com/flanksource/captain/pkg/cmux 1 0 0 -
github.com/flanksource/captain/pkg/codexconfig 10 0 0 10ms
github.com/flanksource/captain/pkg/container 72 0 1 -
github.com/flanksource/captain/pkg/database 152 0 0 54.7s
github.com/flanksource/captain/pkg/gitagent 31 0 0 420ms
github.com/flanksource/captain/pkg/gitagent/proxy 12 0 0 20ms
github.com/flanksource/captain/pkg/monitor 61 0 0 5.0s
github.com/flanksource/captain/pkg/sandbox 13 0 0 -
github.com/flanksource/captain/pkg/sandbox/adapter 39 0 0 50ms
github.com/flanksource/captain/pkg/sandbox/presets 13 0 0 -
github.com/flanksource/captain/pkg/session 70 0 0 30ms
history 72 0 0 99ms
jsonl 6 0 0 110ms
migrations 17 0 0 28.7s
openaimock 2 0 0 159.408µs
promptrun 40 0 0 62ms
provider 15 0 0 18ms
registry 52 0 0 22ms
runtimeprofiles 27 0 0 8.5s
session 33 0 0 53ms
tools 58 0 0 11ms
verify 28 0 0 537ms

Totals: 4855 passed · 0 failed · 12 skipped · 14m27s

View full results

Validate workflow judge prompts before provider construction so invalid declarations remain attributable even when provider setup fails. Also correct result JSON map capacity estimation.
@moshloop
moshloop merged commit 8c7bc2d into main Sep 4, 2026
12 checks passed
@moshloop
moshloop deleted the feat/approval-workflow-verification branch September 4, 2026 14:26
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants