Reviewotron

An agentic code review bot that uses Claude AI to review GitHub pull requests and push events. It posts inline review comments on PRs, commit comments on pushes to develop, and sends Slack notifications.

Reviewotron includes a multi-agent security analysis pipeline that detects injection, XSS, command injection, authentication, authorization, and SSRF vulnerabilities. Security findings go through adversarial validation before being reported, keeping noise low.

How It Works

Reviewotron runs as an HTTP server that receives GitHub webhook events. It can review on PR open/update, on pushes to develop, or when someone posts a REVIEW comment on a PR. All triggers are off by default — see Defaults below.

For each enabled trigger, the bot:

Receives the webhook at the /github endpoint
Validates the signature using the configured webhook secret (HMAC-SHA256)
Fetches the repo config from .reviewotron.json in the repo (via GitHub API), or uses defaults
Fetches the diff for the PR or push (for REVIEW comments, also fetches the full PR via the API to recover head.sha, since issue_comment webhooks don't carry it)
Filters the diff — removes ignored paths, checks size limits
Runs review plugins concurrently:
- General review — Claude analyzes the diff for bugs, style, logic, performance, etc.
- Security review — A multi-agent pipeline scans for vulnerabilities (see below)
Posts results:
- PR events: a single GitHub PR review with inline comments when findings or errors exist
- Push events: commit comments for critical/warning findings + a Slack message
- REVIEW comments: same as PR events

Event Flow

GitHub Webhook (POST /github)
    │
    ├─ Signature validation (HMAC-SHA256)
    ├─ Event parsing (pull_request, push, or issue_comment)
    ├─ Config fetch from .reviewotron.json
    ├─ Diff fetch + filtering
    │
    ├─ General Review Plugin (Claude Sonnet)
    │     └─ Structured output: summary + findings
    │
    ├─ Security Review Plugin (multi-agent)
    │     ├─ Triage Agent (Haiku) → route signals
    │     ├─ Analysis Agents (Sonnet, parallel) → candidate findings
    │     ├─ Validator Agent (Sonnet) → confirm/reject
    │     └─ Memory Curator (Haiku, async) → update memory
    │
    ├─ Merge + deduplicate findings
    │
    └─ Post results
          ├─ PR → GitHub PR review when there is something to report
          └─ Push → commit comments + Slack notification

Supported GitHub Events

Event	Trigger	Gated by	Output
`pull_request` (opened, reopened, ready_for_review)	PR opened, reopened, or marked ready	`auto_review_pr_open`	GitHub PR review with inline comments when there is something to report
`pull_request` (synchronize)	New commits pushed to a PR	`auto_review_pr_sync`	GitHub PR review with inline comments when there is something to report
`push` (to `refs/heads/develop`)	Code pushed to develop	`review_pushes_to_develop`	Commit comments + Slack message
`issue_comment` (created, on a PR, body is `REVIEW` or `REVIEW <sha>`)	Manual trigger via PR comment	`auto_review_on_comment`	GitHub PR review with inline comments when there is something to report

REVIEW reviews the current PR head. REVIEW <sha> reviews only that commit's patch; 7--40 character hexadecimal short SHAs are accepted when they identify exactly one commit in the PR. The target must belong to the PR. Malformed commands are ignored; unrelated or ambiguous targets receive an explanatory PR comment. Historical reviews can become outdated if later commits modify the same lines. The bot must have the pull_request GitHub App permission and the Issue comment webhook event subscribed.

For PR reviews, Reviewotron adds an eyes reaction while a review is running. On automatic PR events the reaction is attached to the PR; on manual REVIEW comments it is attached to the trigger comment. The eyes reaction is removed before posting a review. If the review runs to completion with no findings, no PR review is posted and Reviewotron posts a PR comment saying LGTM :+1:. If every changed file is excluded before review, Reviewotron posts an explicit skip notice instead; that is not an approval.

Events are processed asynchronously — the webhook returns 200 accepted immediately, and the review runs in the background.

Defaults

All four automatic-review triggers default to false. A repo without a .reviewotron.json (or one that doesn't set the relevant flags) receives no reviews. Opt in via .reviewotron.json:

Flag	Effect when `true`
`auto_review_pr_open`	Review PRs on open / reopen / ready-for-review
`auto_review_pr_sync`	Review PRs when new commits land on them
`review_pushes_to_develop`	Review pushes to the `develop` branch
`auto_review_on_comment`	Review when someone posts a `REVIEW` comment on a PR

Manual REVIEW comments bypass the dedup that protects the automatic flow from re-reviewing the same head SHA — by design, since the manual trigger means the user wants a fresh review.

Agent Helper Mode

Reviewotron ships as a binary built for the supported nspawn userspace and its runtime libraries. Another agent can call it to review code on demand — for example, an app-building agent reviewing the project it just generated before publishing, then re-running after each change.

Key points

No files required. The LLM API key is read from --openrouter-api-key / --anthropic-api-key, else the OPENROUTER_API_KEY / ANTHROPIC_API_KEY environment variables, else a --secrets file if you choose to provide one (in that order). An OpenRouter key is preferred when both are available. A secrets.json is not read unless you pass --secrets explicitly.
Redirectable endpoint. On an OpenRouter key, OPENROUTER_BASE_URL points the provider at an OpenAI-compatible proxy instead of openrouter.ai — see OpenRouter Base URL Override.
Configurable in layers. Local CLI reviews merge built-in defaults, the user-global config, .reviewotron.json, .reviewotron.local.json, and --config in that order. Explicit behavior flags such as --no-security have the final say.
Self-describing config. reviewotron config-help prints the config JSON Schema (field names, types, enum domains, descriptions) so an agent can discover the available knobs before deciding what to pass via --config.
Security on by default. In local mode the multi-agent security pipeline runs by default (it is off by default for webhooks). Disable it with --no-security. The general code review also runs by default.

Three ingestion modes, all printing the same review JSON:

Mode	Command	What it reviews
Single file	`review-path FILE`	One file, as newly-added code
Folder (Git or not)	`review-path DIR`	Every file under a directory, as newly-added code
Diff / delta	`review-diff --diff -`	A unified diff on stdin (or `--diff FILE`, or a generated Git working-tree diff)

The short form reviewotron . is the recommended local entry point. It uses Git delta mode for a directory inside a worktree, path mode for a single file, and path mode outside Git. A clean Git worktree returns no changes to review. The Git delta includes staged and unstaged tracked changes, deletions, renames, and non-ignored untracked files. Untracked files are synthesized as added-file diffs and still pass the existing hidden, generated, binary, and size filters.

Output contract

With --output json:

Success → stdout is { "summary": "...", "findings": [ ... ] }, exit code 0.
Failure (bad path, missing key, invalid config, review error) → stdout is { "error": "<message>" }, non-zero exit code.

A caller can branch on the exit code and parse one JSON object either way. Logs go to stderr; only the JSON object is written to stdout.

Examples

export OPENROUTER_API_KEY=sk-or-v1-...   # or ANTHROPIC_API_KEY=sk-ant-...
reviewotron . --output json

Review a finished app folder (raise the size limits for whole-project reviews):

reviewotron review-path ./my-app \
  --config '{"max_files": 500, "max_diff_lines": 50000}' \
  --output json

Review a single file:

reviewotron review-path ./my-app/src/payments.ts --output json

Review an incremental change passed as a diff:

git -C ./my-app diff | reviewotron review-diff --diff - --root ./my-app --output json

Discover the config knobs, then run without the security pipeline:

reviewotron config-help                                   # JSON Schema of the config
reviewotron review-path ./my-app --no-security --output json

Notes

The security pipeline runs by default in local mode; turn it off with --no-security. The flag owns the on/off decision, while --config still controls the security details (vuln_classes, model tiers, thresholds). Security analysis adds extra model calls (triage + per-class analysis + validation), so expect higher cost and latency than a general-only review.
review-path treats every file as newly added, so the whole file is in scope (not only changed lines). Directory walks skip hidden entries (.git, .env, …), build/dependency directories (node_modules, _build, dist, build, target, vendor, venv, __pycache__, coverage), symlinks, and binary/oversized files.
Whole-folder reviews easily exceed the default max_files (50) and max_diff_lines (2000); raise them via --config (e.g. '{"max_files": 500, "max_diff_lines": 50000}'), otherwise the run returns an error explaining which limit was hit.
Each invocation runs independently. Omit --state (the default) so repeated runs always produce a fresh review instead of skipping as a duplicate.

Setup

Prerequisites

OCaml toolchain with opam
An Anthropic API key
A GitHub personal access token (or GitHub App installation) for each repo
(Optional) A Slack bot token for push notifications

Build

make build        # Build the project
make test         # Run tests
make fmt          # Format code
make clean        # Clean build artifacts

Install the executable without sudo:

make install
PREFIX="$HOME/.local" make install

Users already working in the project opam switch can use opam install . instead.

Install From a Release

Published releases ship a prebuilt x86_64 Linux binary, so there is nothing to compile on the target machine. Each release has two assets: the reviewotron-vX.Y.Z-linux-x86_64.tar.gz archive and a SHA256SUMS file.

Download both, verify the checksum, and extract the binary:

VERSION=X.Y.Z
BASE="https://github.com/ahrefs/reviewotron/releases/download/v$VERSION"

curl -fLO "$BASE/reviewotron-v$VERSION-linux-x86_64.tar.gz"
curl -fLO "$BASE/SHA256SUMS"
sha256sum --check --ignore-missing SHA256SUMS

tar -xzf "reviewotron-v$VERSION-linux-x86_64.tar.gz"

That leaves the executable at reviewotron-v$VERSION-linux-x86_64/reviewotron. Move it wherever you keep binaries and make sure that location is on your PATH — /usr/local/bin for a system-wide install, ~/.local/bin for a single user. Then check it runs:

reviewotron --version

The binary is dynamically linked against the usual system libraries (libcurl, libssl, libcrypto, libevent, libgmp, libpcre2, libstdc++, zlib), which are present on a standard Ubuntu 24.04 host. If one is missing, the binary fails at startup naming the library it could not load; install the matching distribution package.

Secrets File

Create a secrets.json file (see secrets.json.example):

{
  "repos": [
    {
      "url": "https://github.com/org/repo",
      "gh_token": "ghp_xxxxxxxxxxxx",
      "gh_hook_secret": "your-webhook-secret"
    }
  ],
  "openrouter_api_key": "sk-or-v1-xxxxxxxxxxxx",
  "slack_access_token": "xoxb-xxxxxxxxxxxx"
}

Either LLM key works; openrouter_api_key is used when both are present. Swap in "anthropic_api_key": "sk-ant-xxxxxxxxxxxx" to call the Anthropic API directly.

Fields:

Field	Required	Description
`repos`	Yes	List of repositories to monitor
`repos[].url`	Yes	Full GitHub repository URL (e.g. `https://github.com/org/repo`)
`repos[].gh_token`	Yes*	GitHub personal access token with `repo` scope
`repos[].gh_hook_secret`	No	Webhook secret for HMAC signature validation
`repos[].auth`	Yes*	Alternative to `gh_token` — GitHub App installation auth (see below)
`openrouter_api_key`	Yes†	OpenRouter API key; preferred when both keys are set
`anthropic_api_key`	Yes†	Anthropic API key for Claude, used when no OpenRouter key is present
`slack_access_token`	No	Slack bot token for posting messages

*Either gh_token or auth must be set per repo. Using gh_token is the simpler option.

†At least one LLM key must be resolvable, from either of these fields or the OPENROUTER_API_KEY / ANTHROPIC_API_KEY environment variables. Provider selection is credential-driven: an OpenRouter key wins when both are available, otherwise the Anthropic key is used. Blank values count as absent. To route OpenRouter traffic through a proxy, see OpenRouter Base URL Override.

For local-only review-diff usage, repos may be an empty list as long as one LLM key is resolvable. The webhook server still requires at least one configured repo by default.

GitHub App Installation Auth

Instead of a personal access token, you can authenticate as a GitHub App installation:

{
  "repos": [
    {
      "url": "https://github.com/org/repo",
      "auth": [
        "AppInstallation",
        {
          "installation_id": "12345678",
          "client_id": "Iv1.xxxxxxxxxx",
          "pem": "-----BEGIN RSA PRIVATE KEY-----\n...\n-----END RSA PRIVATE KEY-----"
        }
      ],
      "gh_hook_secret": "your-webhook-secret"
    }
  ]
}

App installation tokens are automatically refreshed and cached (55-minute TTL).

OpenRouter Base URL Override

When an OpenRouter API key is in use, the endpoint defaults to https://openrouter.ai/api/v1. Set OPENROUTER_BASE_URL to route OpenRouter traffic through an OpenAI-compatible proxy instead — for example on a host whose egress is firewalled off from openrouter.ai, where a local proxy holds the real key and substitutes it server-side:

export OPENROUTER_API_KEY=sk-or-v1-proxy-placeholder-key-replaced-by-llm-proxy
export OPENROUTER_BASE_URL=http://127.0.0.1:18080/api/v1

Environment variable only. There is deliberately no CLI flag, secrets-file field, or config-schema field: this is deployment configuration, not review configuration.
No trailing slash. Request paths are appended directly to the base URL, so the value should end at the API version (.../api/v1, not .../api/v1/). Trailing slashes are stripped defensively, but do not rely on it.
Blank or whitespace-only is treated exactly as unset, leaving the default endpoint in place.
Plain http:// is supported, for proxies on loopback.
The key is still required and still only checked for non-emptiness, so a placeholder value is fine when the proxy supplies the real key.
Affects the OpenRouter path only. The direct Anthropic path (ANTHROPIC_API_KEY) is unchanged, as is the cross-lab fallback routing, which follows whatever base URL is in effect.

GitHub Webhook

Configure a webhook in your GitHub repository settings:

Setting	Value
Payload URL	`https://your-server:1338/github`
Content type	`application/json`
Secret	Same value as `gh_hook_secret` in secrets.json
Events	Select Pull requests and Pushes

Start the Server

./reviewotron run --port 1338 --secrets secrets.json --state state.json

Verify it's running:

curl http://localhost:1338/ping

Configuration

Each repo can have a .reviewotron.json file in its root. For GitHub webhooks, this is fetched from the repo via the GitHub Contents API on each event. Local CLI reviews additionally load these optional files, from lowest to highest precedence:

~/.config/reviewotron/.reviewotron.json
$XDG_CONFIG_HOME/reviewotron/.reviewotron.json
./.reviewotron.json
./.reviewotron.local.json
--config '{...}'

When both global locations exist, $XDG_CONFIG_HOME wins. Objects merge recursively, while scalar values and arrays are replaced. Missing files are ignored; an existing invalid file fails the local review. The local file is useful for uncommitted developer-specific settings and should not be added to version control. Webhook/server commands do not read the user-global files.

Full Configuration Reference

{
  "max_diff_lines": 2000,
  "max_files": 50,
  "max_tokens_per_review": 100000,
  "model": "claude-sonnet-4-6",
  "ignored_paths": ["*.test.js", "vendor/**"],
  "ignored_file_regexes": ["^snapshots/.*\\.golden$"],
  "ignore_generated_files": true,
  "ignored_authors": ["dependabot[bot]"],
  "auto_review_pr_open": false,
  "auto_review_pr_sync": false,
  "review_pushes_to_develop": false,
  "auto_review_on_comment": false,
  "review_draft_prs": false,
  "system_prompt_override": null,
  "slack_channel": "#code-reviews",
  "show_review_cost": false,
  "debug_artifacts": false,
  "review_plugins": {
    "general": {
      "enabled": true,
      "system_prompt_override": null
    },
    "security": {
      "enabled": false,
      "vuln_classes": ["injection", "xss", "command_injection", "authn", "authz", "ssrf", "path_traversal", "policy_regression"],
      "always_analyze_vuln_classes": [],
      "triage_model_tier": "fast",
      "analysis_model_tier": "standard",
      "analysis_effort": "medium",
      "validator_model_tier": "standard",
      "confidence_threshold": "medium",
      "memory_max_tokens": 5000,
      "metrics_artifacts": false,
      "debug_artifacts": false
    }
  }
}

Config Fields

Field	Default	Description
`max_diff_lines`	`2000`	Maximum total diff lines to review. PRs exceeding this are skipped.
`max_files`	`50`	Maximum files to review after ignored, custom-regex, and generated files are removed.
`max_tokens_per_review`	`100000`	Token budget hint for the review agent.
`model`	`claude-sonnet-4-6`	Model ID for the general review agent.
`ignored_paths`	`[]`	Glob patterns for files to exclude from review. Supports `` and `*` wildcards.
`ignored_file_regexes`	`[]`	Regular expressions matched against repository-relative file paths to exclude from review. Catch-all patterns are rejected; use explicit `ignored_paths` entries when you intentionally want to exclude a complete set of files.
`ignore_generated_files`	`true`	Exclude conservatively detected generated files before `max_files` and `max_diff_lines` are enforced. Set to `false` to review generated artifacts.
`ignored_authors`	`[]`	GitHub usernames whose PRs/pushes should be skipped.
`auto_review_pr_open`	`false`	Review PRs when they are opened, reopened, or marked ready.
`auto_review_pr_sync`	`false`	Review PRs when new commits are pushed to them.
`review_pushes_to_develop`	`false`	Review pushes to the `develop` branch.
`auto_review_on_comment`	`false`	Review when someone posts a top-level PR comment whose body is `REVIEW` or `REVIEW <sha>` (after trimming). Requires the GitHub App to subscribe to Issue comment events.
`review_draft_prs`	`false`	Include draft PRs in automatic reviews. By default drafts are skipped regardless of `auto_review_pr_open` / `auto_review_pr_sync`.
`system_prompt_override`	`null`	Replace the default general review system prompt entirely.
`slack_channel`	`null`	Slack channel for push review notifications. Requires `slack_access_token` in secrets.
`show_review_cost`	`false`	Append a cost summary footer to PR reviews.
`debug_artifacts`	`false`	Write raw agent output dumps when structured output cannot be parsed. Sensitive and opt-in.
`review_plugins`	(see below)	Per-plugin configuration.

Generated-file detection is intentionally conservative. It includes exact __generated__ and gen path components, path components or file stems ending in _gen, file stems starting with generated_, common generated artifact suffixes such as minified assets, .map files, and protobuf outputs, and generated-file header markers. Broad folders such as generated/, dist/, build/, and vendor/ remain reviewable unless excluded with ignored_paths or ignored_file_regexes.

Plugin Configuration

General Plugin

Field	Default	Description
`enabled`	`true`	Enable/disable the general code review.
`system_prompt_override`	`null`	Override the general review prompt (plugin-level).

Security Plugin

Field	Default	Description
`enabled`	`false`	Enable/disable security analysis.
`vuln_classes`	All 8 classes	Which vulnerability types to scan for.
`always_analyze_vuln_classes`	`[]`	Vulnerability classes that bypass `confidence_threshold`. Classes listed here are implicitly enabled even if absent from `vuln_classes`. Use sparingly for high-risk repos or temporarily while tuning recall.
`triage_model_tier`	`"fast"`	Model tier for the triage agent.
`analysis_model_tier`	`"standard"`	Model tier for per-class analysis agents.
`analysis_effort`	`"medium"`	Analysis effort: `"low"`, `"medium"`, `"high"`, or `"xhigh"`. OpenRouter sends it as reasoning effort; direct Anthropic sends native effort with adaptive thinking where the model supports that level. Set to `null` for the provider default.
`validator_model_tier`	`"standard"`	Model tier for the adversarial validator.
`confidence_threshold`	`"medium"`	Minimum triage confidence to trigger analysis for enabled classes. `"high"` = only high-confidence signals. `"medium"` = high + medium. `"low"` = all signals.
`memory_max_tokens`	`5000`	Target size limit for the repo's security memory file.
`metrics_artifacts`	`false`	Write compact security metrics artifacts under the review debug dir's `security/` subdirectory. These omit source code and prompt bodies.
`debug_artifacts`	`false`	Write full redacted per-stage security debug artifacts under the review debug dir's `security/` subdirectory. Sensitive and opt-in.

Model Tiers

Tier	Model	Typical Use
`"fast"`	`claude-haiku-4-5-20251001`	Triage, memory curator
`"standard"`	`claude-sonnet-5`	Analysis agents, validator, general review
`"strong"`	`claude-opus-4-8`	Reserved for complex codebases

Vulnerability Classes

Value	Description
`"injection"`	SQL injection, NoSQL injection, query string construction
`"xss"`	Cross-site scripting (reflected, stored, DOM-based)
`"command_injection"`	OS command injection via exec/system/popen
`"authn"`	Authentication bypass, weak token validation, missing expiry
`"authz"`	Authorization flaws, IDOR, missing permission checks
`"ssrf"`	Server-side request forgery via user-controlled URLs
`"path_traversal"`	Path traversal and file exposure: user-controlled paths reaching file reads, writes, or downloads, plus archive extraction that escapes its root
`"policy_regression"`	Security policy/control regressions such as broad sudo, CI/cloud/RBAC permission expansion, privileged Kubernetes workload settings, and disabled TLS/auth/CSRF controls

Skip Behavior

Reviewotron skips events in these cases:

Bot senders — any login ending in [bot]
Ignored authors — usernames in the ignored_authors list
Non-reviewable actions — PR closed, edited, or other non-code-change actions
Draft PRs — skipped until marked ready
Already reviewed — same PR + head SHA (or same push after SHA) already processed
Empty diff — all files filtered by ignored_paths, ignored_file_regexes, or generated-file detection; Reviewotron posts a skip notice, not LGTM
Diff too large — exceeds max_diff_lines after ignored, custom-regex, and generated files are removed
Non-develop pushes — only refs/heads/develop is reviewed

Security Review Pipeline

When the security plugin is enabled, every diff goes through a multi-agent pipeline:

1. Triage (Haiku, single-shot)

Before triage, Reviewotron runs a deterministic scan over changed paths and added hunk lines for advisory security signals such as dangerous APIs, risky paths, sensitive files, changed security controls, and stateful operations. These signals are hints only: they are summarized by category, vulnerability hint, and affected file for triage, with only the strongest exact hints included. They never become findings and never route directly to analysis.

The triage agent scans the diff for security-relevant patterns and classifies them by vulnerability type. This is intentionally biased toward over-flagging — it's cheap to run an analysis agent that finds nothing, costly to miss a real issue.

The triage agent outputs signals with confidence levels (high, medium, low). The confidence_threshold config controls which signals proceed to analysis for enabled vulnerability classes. always_analyze_vuln_classes is the explicit override that bypasses the threshold; classes listed there are implicitly enabled even if absent from vuln_classes.

2. Analysis (Sonnet, per vulnerability class, parallel)

For each flagged vulnerability class, a specialized agent runs deep analysis:

Source identification — Where does user-controlled input enter?
Sink identification — Where does data reach a dangerous operation?
Data flow tracing — Can the source reach the sink? Traces through variables, function calls, returns.
Sanitization evaluation — Is there adequate, context-correct sanitization on the path?

For policy_regression, the same finding schema is used with a policy proof instead of a runtime user-input flow: source is the changed principal/grant/config entry or removed control, sink is the effective privileged capability or weakened boundary, flow is changed line -> effective policy/control state -> concrete action now possible, and sanitization is the missing or inadequate scoping/mitigation.

Analysis agents can fetch additional files from the repo via the GitHub Contents API when they need to trace a data flow beyond the diff. Each run starts from a focused, class-specific analysis question and the triage evidence. The agent is instructed to inspect changed regions and direct dependencies first, fetch more files only to close a specific evidence gap, and return no finding when a bounded check cannot establish the required source/effect, sink/capability, and missing control.

Analysis depth is budgeted by vulnerability class, triage confidence, and signal count. High-confidence AuthN/AuthZ/SSRF signals still get the most room because those classes often need cross-file context; medium/low-confidence and policy-regression runs are kept tighter to avoid broad repo archaeology.

3. Validation (Sonnet, adversarial)

All candidate findings from all analysis agents pass through a single validator agent. It acts as an adversarial false-positive filter, checking:

The claimed source actually accepts external input
The claimed sink actually performs the dangerous operation
Every step in the flow path is backed by evidence (file + line)
The sanitization assessment is correct
A confirmed result includes a concrete proof-by-construction: reproducible trigger, source-to-sink trace, missing control, expected impact, and explicit assumptions

For policy_regression, validation does not require a user-controlled runtime source, but it does require exact file/line evidence, a concrete effective privilege/control change, a concrete action now possible, no unresolved assumptions, and enough proof to reject vague "security relevant" policy edits.

Findings that fail validation are dropped. Confirmed validator results without concrete proof are downgraded after parsing and are not surfaced. This is by design — a noisy security reviewer that cries wolf loses developer trust. Dropped findings are logged for offline prompt tuning.

4. Memory Curation (Haiku, async)

After the review is posted, a curator agent runs asynchronously to update the repo's security memory with learnings from the review. This is fire-and-forget — it doesn't block the review.

Severity Mapping

Analysis Confidence	Post-Validation Severity
High + Confirmed	Critical
Medium + Confirmed	Warning
Low + Confirmed	Warning

Slack Integration

Push reviews (to develop) optionally send a Slack notification. This requires:

A slack_access_token in secrets.json — a Slack bot token (xoxb-...) with chat:write permission
A slack_channel set in the repo's .reviewotron.json

The message includes:

Pusher name and commit count
Link to the compare view on GitHub
Review summary text
Finding counts (critical, warnings, suggestions)
Color-coded: red if any critical findings, green otherwise

If the security plugin encountered an error, a note is appended to the Slack message.

If slack_access_token is not configured, Slack posting is silently skipped.

State and Persistence

State File

The --state flag enables persistent state tracking. The state file (JSON) records:

PR reviews: repo URL, PR number, head SHA, timestamp, review costs
Push reviews: repo URL, after SHA
Generic change reviews: repo key, change key, timestamp, review costs

For GitHub webhooks, this prevents duplicate reviews — if the same PR at the same commit SHA is already recorded, the review is skipped. Local diff reviews record their repo_key and change_key in the same state file, but currently do not skip duplicates. State is trimmed to the 500 most recent records per repo key.

Without --state, state is in-memory only and lost on restart. This means reviews may be duplicated after a server restart.

Local reviews do not write state, feedback, memory, or debug artifacts into the reviewed working tree by default. --state and --feedback-dir are explicit persistence locations; choose paths outside the repository when using them.

Security Memory Files

The security pipeline maintains per-repo memory files outside the working tree by default: under $XDG_STATE_HOME/reviewotron/memory/{repo-slug}.md, or ~/.local/state/reviewotron/memory/{repo-slug}.md when XDG_STATE_HOME is unset. In persistent server mode with an explicit feedback store, memory lives beside feedback/debug data; for example, if feedback evidence is under ./var/reviewotron-feedback-evidence/, memory files go under ./var/memory/{repo-slug}.md. These are plain-text markdown files (target ~5000 tokens) that accumulate knowledge about the repo:

Architecture notes (frameworks, DB access patterns, auth middleware)
Known safe patterns (parameterized queries, auto-escaping templates)
Known risk areas (shell command construction, raw HTML rendering)
Suppressions (accepted risks with context)

Memory is injected into every security agent's prompt, reducing redundant file fetching and pattern re-discovery across reviews.

The memory curator rewrites the repo brief asynchronously after a review. If two reviews update the same brief concurrently, the last write wins; the brief is intentionally architectural rather than per-finding state.

Debug Dumps

Raw agent debug dumps are disabled by default. Set the top-level debug_artifacts configuration field to true when diagnosing a structured-output failure:

reviewotron . --config '{"debug_artifacts": true}'

When enabled, dumps are written under $XDG_STATE_HOME/reviewotron/debug/{repo-slug}/{sha-prefix}/, or ~/.local/state/reviewotron/debug/{repo-slug}/{sha-prefix}/ when XDG_STATE_HOME is unset. In persistent server mode they use a sibling root next to feedback evidence; for example, if feedback evidence is under ./var/reviewotron-feedback-evidence/, debug dumps go under ./var/debug/{repo-slug}/{sha-prefix}/. These contain raw agent output and should be treated as sensitive.

Security metrics/debug artifacts are separate and opt-in via review_plugins.security.metrics_artifacts and review_plugins.security.debug_artifacts. Metrics write compact JSON files under the review debug dir's security/ subdirectory; full security artifacts additionally write redacted stage inputs and outputs and should be treated as sensitive.

OpenTelemetry Traces

OpenTelemetry tracing is off by default in public/local runs. No trace exporter is started, and no trace network traffic is emitted unless it is explicitly enabled by environment.

Enable local OTLP trace export with:

REVIEWOTRON_OTEL=1 reviewotron run --secrets secrets.json

With no endpoint variable set, traces go to the default http://127.0.0.1:4318. REVIEWOTRON_OTEL=1 on its own is only correct when a node-local OTLP/HTTP collector (agent or sidecar) is listening there — if nothing is listening on that address, tracing looks enabled but every span is silently dropped. Ahrefs/private deployments without a node-local collector must also set an endpoint:

REVIEWOTRON_OTEL=1
OTEL_EXPORTER_OTLP_ENDPOINT=http://127.0.0.1:4318

OTEL_EXPORTER_OTLP_ENDPOINT is a base URL; /v1/traces is appended. OTEL_EXPORTER_OTLP_TRACES_ENDPOINT is used verbatim and must include the /v1/traces path itself:

OTEL_EXPORTER_OTLP_TRACES_ENDPOINT=http://127.0.0.1:5996/v1/traces

Setting either endpoint variable to a non-empty value enables tracing by itself — REVIEWOTRON_OTEL=1 is not required in that case. REVIEWOTRON_OTEL=0 or OTEL_SDK_DISABLED=true still force tracing off even when an endpoint is set.

An endpoint variable that is present but empty or whitespace-only (for example a templated OTEL_EXPORTER_OTLP_ENDPOINT= line with no value substituted) is treated as unset for the enable check, but the OTLP client would otherwise read it verbatim and export to a broken URL. At startup Reviewotron detects this and normalizes the variable to its effective value (the default endpoint, or the resolved base plus /v1/traces), logging a warning when it does. Avoid relying on this: either set a full URL or omit the variable entirely.

Export is OTLP over HTTP/protobuf — point endpoint variables at a collector's HTTP receiver (4318-style port), not a gRPC-only 4317 endpoint. Span batches flush roughly every 2s with retries; export failures are logged to stderr (verbosity controlled by OTEL_LOG_LEVEL). On graceful shutdown, queued spans are flushed, but reviews still in flight when the server stops may produce incomplete or missing traces — expected given the fire-and-forget webhook design.

Other supported standard variables:

OTEL_EXPORTER_OTLP_HEADERS=Authorization=Bearer%20<token>
OTEL_RESOURCE_ATTRIBUTES=service.namespace=devtools,deployment.environment=prod

OTEL_EXPORTER_OTLP_HEADERS is needed for authenticated backends. OTEL_SDK_DISABLED=true disables tracing even when REVIEWOTRON_OTEL=1 is set. OTEL_SERVICE_NAME is respected; otherwise spans use service.name=reviewotron.

The existing /home/user/reviewotron/log/reviewotron.*.json Vector path remains JSON log ingestion only. It is not a trace drain; traces are emitted via OTLP.

Review Feedback

When webhook mode is started with --state, Reviewotron also enables feedback persistence for GitHub PR review bodies and inline PR review comments. Review bodies and inline findings include this prompt:

Was this review helpful? React with :+1: or :-1:.

Each inline finding comment also gets a hidden marker:

<!-- reviewotron-feedback-id: rvf_... -->

GitHub hides the marker in the UI but keeps it in the API body, which lets the collector map a posted review comment back to a local feedback target.

By default, feedback data is stored next to the state file, not inside state.json:

/path/to/state.json
/path/to/reviewotron-feedback-targets.json
/path/to/reviewotron-feedback-events.jsonl
/path/to/reviewotron-feedback-evidence/

Use --feedback-dir /durable/path on both run and collect-feedback to keep targets, events, and evidence bundles somewhere other than the state file's directory. This is recommended when --state points at a temporary or ephemeral location. When reviewotron run has a feedback store, it collects reaction counts in the server process. A target is polled immediately the first time, then waits one hour between polls by default; pass --poll-interval-seconds 60 to run the server poller on a minute cadence.

reviewotron-feedback-targets.json stores polling state for inline PR review comments and top-level PR review bodies. Inline comments are collected through GitHub's REST reactions API; review-body reactions are collected through a minimal GitHub GraphQL query using the review node_id. reviewotron-feedback-events.jsonl stores append-only aggregate events such as reaction count changes, comment ID resolution, and target finalization.

For each successfully posted GitHub PR review with feedback persistence enabled, Reviewotron also writes one immutable evidence bundle under reviewotron-feedback-evidence/<review_batch_id>/. Target records link each feedback ID back to the bundle; inline targets also link to a finding ID. Bundles contain the reviewed filtered diff, the posted review/comment bodies with feedback prompts and inline markers, routed findings with plugin-level provenance, review costs, review config, and fetched file metadata hashes. They are intended as input for a later feedback-review agent/command that correlates GitHub reactions with review context and produces improvement recommendations; humans should not need to inspect the JSON files manually.

Privacy rules:

Only aggregate +1 and -1 counts are stored.
Evidence bundles are written only for successfully posted GitHub PR reviews.
Raw reaction objects are not stored.
Webhook payloads are not stored.
Raw prompts, agent transcripts, tool outputs, and unrelated logs are not stored.
Fetched file contents are not stored by default; only path, byte count, and SHA-256 metadata are written.
GitHub user identity fields such as sender, user, login, name, email, author, committer, pusher, and avatar_url are not written to feedback files.

Polling stops after the earliest of five days from target creation, 24 hours after the first qualifying human PR interaction, or PR close/merge. A closed PR receives one final poll before the target is marked closed.

The collect-feedback command is a one-shot fallback for backfills, debugging, or smoke tests when the server is not running:

reviewotron collect-feedback --secrets secrets.json --state /path/to/state.json --feedback-dir /durable/path

Both the server poller and the one-shot collector are idempotent. They resolve missing GitHub review comment IDs from hidden markers, poll review-body and inline-comment reaction counts, update target state, and append JSONL events only when counts change or targets finalize.

After collection, summarize the local feedback store with:

reviewotron feedback-report --state /path/to/state.json --feedback-dir /durable/path

Use --output json when feeding the parsed feedback into another tool or a future feedback-review agent. The report joins target records, aggregate events, and evidence bundles, grouping feedback by posted PR review and preserving the feedback ID, target kind, inline finding ID/plugin/source when present, reaction counts, reviewed PR metadata, GitHub discussion/review URL, and evidence bundle path. For large feedback sets, keep investigations bounded with filters such as:

reviewotron feedback-report --state /path/to/state.json --feedback-dir /durable/path \
  --sentiment negative --limit 20 --brief

Use review_batch_id to open one evidence bundle, finding_id to select the exact finding in findings.json or posted_review.json, and comment_id or github_comment_url to return to the GitHub discussion.

CLI Usage

`reviewotron [PATH]` — Smart Local Review

reviewotron
reviewotron .
reviewotron ./src/file.ml
reviewotron --mode auto .
reviewotron --mode diff .
reviewotron --mode path .
reviewotron --commit HEAD~2 src/

With no path, PATH defaults to .. In auto mode a directory inside a Git worktree reviews the Git delta; a single file or a path outside Git uses path mode. --mode diff requires Git, while --mode path reviews the selected file or directory as newly added code. A Git directory with no changes exits with no changes to review; if no base ref can be inferred, pass --base or use --mode path.

With --commit COMMIT, the commit-ish is resolved to a full SHA and only that commit's first-parent patch is reviewed. The supplied path scopes the patch; staged, unstaged, and untracked worktree changes are ignored.

The smart command accepts the local review options documented below, including --root, --base, --config, --output, and --no-security. It is also the command used by .claude/skills/reviewotron/SKILL.md: the skill locates an installed binary, falls back to dune exec in this checkout, and summarizes the JSON response without duplicating configuration logic.

`reviewotron run` — Start the Webhook Server

reviewotron run [OPTIONS]

Option	Default	Description
`-p`, `--port`	`1338`	HTTP server port
`--secrets`	`secrets.json`	Path to secrets file
`--config-filename`	`.reviewotron.json`	Config filename to look for in repos
`--state`	(none — in-memory)	Path to state file for persistence
`--feedback-dir`	sibling paths next to `--state`	Directory for feedback targets, events, and evidence bundles
`--poll-interval-seconds`	`3600`	Minimum seconds between feedback polls; use `60` for minute polling
`--logfile`	(stderr)	Log file path
`--loglevel`	(default)	Log level: `debug`, `info`, `warn`, `error`

`reviewotron check` — Parse a Webhook Payload (Dry Run)

reviewotron check --event-type pull_request --payload payload.json [OPTIONS]

Parses and displays a GitHub webhook payload without starting the server or performing any review. Useful for verifying payload parsing.

Option	Required	Description
`--event-type`	Yes	GitHub event type (`pull_request` or `push`)
`--payload`	Yes	Path to JSON payload file
`--secrets`	No	Path to secrets file (defaults to `secrets.json`; must exist for initialization)

`reviewotron review-diff` — Review a Local Unified Diff

reviewotron review-diff [OPTIONS]

Runs the same core review engine against a local unified diff and prints the final review to stdout. Logs go to stderr unless --logfile is set. The diff can be a file (--diff FILE), stdin (--diff -), a selected commit (--commit COMMIT), or — when --diff and --commit are omitted — a Git diff generated from the merge-base of HEAD and the inferred base ref, including staged, unstaged, and non-ignored untracked changes. Commit mode reviews only the selected commit against its first parent and does not inspect the worktree. This path does not fetch or publish through GitHub; local file-content expansion uses --root.

The LLM API key is resolved from --openrouter-api-key / --anthropic-api-key, then the OPENROUTER_API_KEY / ANTHROPIC_API_KEY environment variables, then a --secrets file if one is given — no secrets file is required. An OpenRouter key is preferred when both are available; set OPENROUTER_BASE_URL to route it through a proxy (see OpenRouter Base URL Override). Configuration uses the layered local CLI precedence described in Configuration. See Agent Helper Mode.

Option	Default	Description
`--diff`	Git diff against inferred base	Path to a unified diff file
`--commit`	(none)	Commit-ish to review; defaults the change key to `commit/<full-sha>`
`--base`	inferred from Git	Base ref for generated diffs; tries `origin/HEAD`, `origin/main`, `origin/master`, then the upstream remote
`--root`	Git worktree root, then cwd	Repository root for local file-content lookups
`--repo-key`	`local:<root>`	Stable repository key for config, memory paths, and state
`--change-key`	digest of filtered diff	Stable change key recorded in state
`--title`	inferred from base or diff file	Title passed to review agents
`--description-file`	(none)	Optional file used as the review description
`--config-filename`	`.reviewotron.json`	Config file loaded from `--root`, or absolute config path
`--config`	(none)	Inline config JSON; overrides any config file
`--openrouter-api-key`	(none)	OpenRouter API key (preferred when set); overrides `$OPENROUTER_API_KEY` and any secrets file
`--anthropic-api-key`	(none)	Anthropic API key; overrides `$ANTHROPIC_API_KEY` and any secrets file
`--no-security`	(off)	Disable the security pipeline (on by default in local mode)
`--output`	`markdown`	Output format: `markdown` or `json`
`--secrets`	(none)	Optional secrets file; the API key is taken from `--openrouter-api-key` / `--anthropic-api-key`, then `$OPENROUTER_API_KEY` / `$ANTHROPIC_API_KEY`, then this file
`--state`	(none — in-memory)	Optional state file updated after a successful review

JSON output is an object with a review-level summary and a machine-readable findings list:

{
  "summary": "The review found one startup compatibility issue in session metadata handling.",
  "findings": [
    {
      "file": "backend/safer-claude-code/safer_claude_code.ml",
      "line": 492,
      "level": "warning",
      "category": "bug",
      "summary": "Legacy session-id file from old scc crashes startup because ensure_dir refuses to treat a regular file as a directory",
      "failure_scenario": "Any user who ran a previous scc has a regular file at <scc_metadata>/sessions/<wt_basename> holding their last session UUID. After upgrading, the first scc -f or scc run-on calls prepare_session_id_mount, which calls ensure_dir(Filename.dirname host_path) — i.e. ensure_dir on the legacy file path. ensure_dir sees S_REG and fails. scc aborts on startup until the user manually removes the legacy file."
    }
  ]
}

`reviewotron review-path` — Review a File or Directory

reviewotron review-path PATH [OPTIONS]

Reviews a single file or an entire directory by treating every file as newly added, reusing the same engine, output formats, and JSON contract as review-diff. This is how to review code that has no Git history — a single file, a freshly generated project, or a non-Git working tree.

For a file, the file's parent directory becomes the review root (so context lookups resolve siblings). For a directory, PATH is walked recursively in sorted order; hidden entries, build/dependency directories, symlinks, and binary/oversized files are skipped (see Agent Helper Mode).

Option	Default	Description
`PATH`	(required)	File or directory to review
`--config`	(none)	Inline config JSON; overrides any config file
`--openrouter-api-key`	(none)	OpenRouter API key (preferred when set); overrides `$OPENROUTER_API_KEY` and any secrets file
`--anthropic-api-key`	(none)	Anthropic API key; overrides `$ANTHROPIC_API_KEY` and any secrets file
`--no-security`	(off)	Disable the security pipeline (on by default in local mode)
`--output`	`markdown`	Output format: `markdown` or `json`
`--repo-key`	`local:<root>`	Stable repository key for config, memory, and state
`--change-key`	digest of the synthesized diff	Stable change key recorded in state
`--title`	inferred from the path	Title passed to the review agents
`--config-filename`	`.reviewotron.json`	Config filename loaded from the root, or absolute config path
`--state`	(none — in-memory)	Optional state file updated after a successful review

Whole-folder reviews commonly exceed the default max_files / max_diff_lines limits; raise them with --config (see Agent Helper Mode).

Local Security Timing and Artifact Comparison

To compare local filesystem review behavior before and after enabling the security pipeline, run the same diff twice with stable repo/change keys and measure wall time. Use a throwaway state file so duplicate-review detection does not skip the second run.

General-only baseline:

/usr/bin/time -p \
  dune exec -- src/reviewotron.exe review-diff \
  --root /path/to/repo \
  --diff /tmp/change.diff \
  --repo-key local-security-timing \
  --change-key general-only-1 \
  --no-security \
  --output json \
  --logfile /tmp/reviewotron-general.log \
  --state /tmp/reviewotron-general-state.json \
  > /tmp/reviewotron-general.json

Security-enabled run with metrics and redacted debug artifacts:

SECURITY_CONFIG='{
  "max_files": 500,
  "max_diff_lines": 50000,
  "show_review_cost": true,
  "review_plugins": {
    "security": {
      "metrics_artifacts": true,
      "debug_artifacts": true
    }
  }
}'

/usr/bin/time -p \
  dune exec -- src/reviewotron.exe review-diff \
  --root /path/to/repo \
  --diff /tmp/change.diff \
  --repo-key local-security-timing \
  --change-key security-observed-1 \
  --config "$SECURITY_CONFIG" \
  --output json \
  --logfile /tmp/reviewotron-security.log \
  --state /tmp/reviewotron-security-state.json \
  > /tmp/reviewotron-security.json

Inspect the security metrics line in /tmp/reviewotron-security.log and the artifact directory under /tmp/debug/local-security-timing/<diff-digest-prefix>/security/. metrics.json and fetch_stats.json are compact and omit prompt/source bodies; the full debug files include redacted stage inputs and outputs and should be treated as sensitive.

To isolate just the security pipeline cost, add "general": {"enabled": false} under review_plugins in SECURITY_CONFIG.

`reviewotron config-help` — Print the Config Schema

reviewotron config-help

Prints the review configuration as a JSON Schema — every field with its type, enum domain (for vuln_classes, model tiers, confidence), and a one-line description. An agent can read this to discover which knobs exist and what they accept, then pass chosen values via --config. Takes no options and makes no network calls.

`reviewotron collect-feedback` — Poll Review Feedback Reactions

reviewotron collect-feedback --secrets secrets.json --state state.json [OPTIONS]

Runs one GitHub reaction collection pass for feedback targets stored next to state.json. Server mode normally collects feedback automatically; this command is for backfills, debugging, and smoke tests.

Option	Required	Description
`--secrets`	Yes	Secrets file with repo authentication
`--state`	Yes	State file whose sibling feedback files should be loaded
`--feedback-dir`	No	Directory containing feedback targets, events, and evidence bundles; must match the server's `--feedback-dir` when set
`--poll-interval-seconds`	No	Minimum seconds between polls for the same feedback target; defaults to 3600
`--logfile`	No	Log file path
`--loglevel`	No	Log level: `debug`, `info`, `warn`, `error`

`reviewotron feedback-report` — Summarize Collected Feedback

reviewotron feedback-report --state state.json [OPTIONS]

Reads local feedback files and evidence bundles without contacting GitHub or an LLM. Markdown output is intended for quick inspection; JSON output is intended for downstream tools.

Option	Required	Description
`--state`	Yes	State file whose sibling feedback files should be loaded
`--feedback-dir`	No	Directory containing feedback targets, events, and evidence bundles; must match the server's `--feedback-dir` when set
`--output`	No	`markdown` or `json`; defaults to `markdown`
`--sentiment`	No	Filter targets: `all`, `reacted`, `positive`, `negative`, `mixed`, or `unreacted`; defaults to `all`
`--review-batch-id`	No	Limit output to one evidence bundle/review batch
`--pr`	No	Limit output to one pull request number
`--limit`	No	Limit output to the first N matching feedback targets
`--brief`	No	Omit finding message snippets from markdown output

Endpoints

Path	Description
`/ping`	Health check — returns uptime
`/github`	GitHub webhook receiver

Cost Tracking

Every agent call tracks token usage and estimates cost:

Per agent: input tokens, output tokens, cache read tokens, cache creation tokens, model ID, number of tool-use turns, files fetched, estimated USD cost
Per plugin: aggregated agent costs (general, security)
Per review: total across all plugins

Costs are:

Logged at info level after each review
Stored in state.json alongside the review record (when state persistence is enabled)
Optionally shown in the PR review footer (when show_review_cost: true)

Cost footer example:

Review cost: 5 agents (general: 1 agent, security: 4 agents), ~$0.42

Pricing

Costs are estimated using a built-in pricing table that includes prompt caching rates:

Model Family	Input	Output	Cache Write (5m)	Cache Read
Claude Opus 4.x	$5.00/MTok	$25.00/MTok	$6.25/MTok	$0.50/MTok
Claude Sonnet 4.x	$3.00/MTok	$15.00/MTok	$3.75/MTok	$0.30/MTok
Claude Haiku 4.5	$1.00/MTok	$5.00/MTok	$1.25/MTok	$0.10/MTok

Cache write tokens are charged at 1.25x the base input price (5-minute TTL). Cache read tokens are charged at 0.1x the base input price. Cache token counts are extracted from the Anthropic API response and tracked per-agent.

The pricing table is a single record in the codebase (lib/cost_tracking.ml) — update it when prices change.

Limitations

Diff Size

PRs with more than max_diff_lines (default 2000) total diff lines after ignored/generated files are removed are skipped entirely. There is no partial review — it's all or nothing. For large PRs, consider breaking them into smaller ones.

Push Reviews

Only pushes to refs/heads/develop are reviewed. Other branches, including main/master, are not reviewed on push. PR reviews cover all branches.

File Content Fetching

The general review plugin fetches up to 5 key files for additional context (added or modified files only)
Security analysis agents can fetch any file via get_file_content, bounded by a dynamic step budget derived from vulnerability class, triage confidence, and signal count
All file fetches use the PR head SHA as the git ref, so agents see the PR branch state (not the default branch)

Static Analysis Only

The security pipeline performs static analysis on the diff and referenced files. It cannot:

Execute code or run tests
Detect runtime-only vulnerabilities
Analyze compiled/minified code meaningfully
Fully model infrastructure or policy state outside the reviewed diff and fetched files

Security Scope

8 vulnerability classes are supported. Other classes (e.g., cryptographic weaknesses, deserialization) are not covered.
The triage agent may miss security signals in unusual code patterns. Bumping triage_model_tier to "standard" (Sonnet) can improve recall at higher cost.
AuthN/AuthZ/SSRF analysis from diff context alone is inherently limited. These classes produce the most false negatives.

Webhook Signature Validation

If no gh_hook_secret is configured for a repo, webhook signature validation is skipped — the event is accepted without verification. While the review will fail at the GitHub API step if no auth token is configured, it's best practice to always set a webhook secret.

Duplicate Prevention

Duplicate review prevention relies on the state file. Without --state, or after a server restart with in-memory-only state, the same PR/push may be reviewed again.

Supported Local Runtime

Released binaries target x86_64 Linux and are dynamically linked against the usual system libraries — see Install From a Release. Other architectures and static linking are not part of this phase. Publishing a release is handled by make gh-release (scripts/release.sh), which builds, packages, and uploads from a prepared build environment. It requires a clean working tree and takes the version from dune-project.

Concurrent Reviews

Multiple reviews can run concurrently (events are processed via Lwt.async). The security memory queue handles concurrent appends safely, but there's no global rate limiting on Anthropic API calls.

Troubleshooting

Review not triggering

Check the webhook delivery log in GitHub (Settings > Webhooks > Recent Deliveries)
Verify the server is running: curl http://your-server:1338/ping
Check the server logs for skip reasons:
- "bot sender" — the event was from a bot account
- "ignored author" — the author is in ignored_authors
- "action ... not reviewable" — the PR action doesn't trigger reviews
- "draft PR" — mark the PR as ready for review
- "already reviewed at ..." — duplicate detection fired
Check that the repo URL in secrets.json matches exactly (including https://github.com/...)

Review fails

"no auth configured for repo ..." — the repo URL in the webhook doesn't match any entry in secrets.json
"failed to fetch config" — GitHub API error fetching .reviewotron.json (check token permissions)
"triage agent failed" / "analysis agent failed" — provider API error (check the resolved LLM key, rate limits, and OPENROUTER_BASE_URL if a proxy is configured)
"failed to post review" — GitHub API error posting the review (check token scopes: needs repo or pull_request:write)

Security findings not appearing

Check that review_plugins.security.enabled is true in .reviewotron.json (it is false by default)
Check the confidence_threshold — "high" is very selective. Try "medium" or "low"; for temporary high-recall tuning, add specific enabled classes to always_analyze_vuln_classes
Check the logs for "triage: no actionable signals" (the diff may not contain security-relevant code)
Check for "validator rejected" messages — the finding was detected but rejected as a false positive
Bump analysis_model_tier to "strong" for complex codebases

Debug dumps

When an agent produces output that can't be parsed as structured JSON and debug_artifacts is enabled, look under $XDG_STATE_HOME/reviewotron/debug/{repo-slug}/{sha-prefix}/, or ~/.local/state/reviewotron/debug/{repo-slug}/{sha-prefix}/; in persistent server mode, look under the sibling root next to the feedback evidence root, for example ./var/debug/{repo-slug}/{sha-prefix}/. Without that opt-in, the log reports the failure without writing the transcript.

Known Issues

No rate limiting for Anthropic API calls. Concurrent reviews (e.g., multiple PRs opened at once) will all call the Anthropic API simultaneously. There is no built-in throttling or queue. The SDK handles 429 errors with automatic retry and exponential backoff, so transient rate limits self-heal. At typical usage (a handful of monitored repos), this is unlikely to be an issue.

Architecture (for contributors)

src/
  reviewotron.ml          CLI entrypoint (cmdliner: run + check commands)
  request_handler.ml      HTTP server, webhook routing, signature validation

lib/
  api.ml                  Module type signatures (Github, Agent_runner, Slack)
  api_remote.ml           Production implementations (real HTTP calls)
  api_local.ml            Mock implementations (for testing)

  context.ml              Application context: secrets, config cache, state
  config_types.ml         All configuration types ([@@deriving json])
  github_types.ml         GitHub API request/response types
  slack_types.ml          Slack API types

  github.ml               Event parsing, signature validation
  github_auth.ml          GitHub token/JWT auth (PAT + App Installation)

  reviewer.ml             Plugin orchestrator (Make functor)
  review_plugin.ml        Plugin interface type
  general_review_plugin.ml  General code review + validation
  security_review_plugin.ml Multi-agent security pipeline

  agent_runner.ml         Generic agent execution via ocaml-ai-sdk
  triage_agent.ml         Triage agent config + prompt
  analysis_agent.ml       Per-vuln-class analysis agent framework
  validator_agent.ml      Adversarial validation agent
  memory_curator_agent.ml Memory update curator agent

  security_types.ml       All security pipeline types
  security_tools.ml       get_file_content tool for agents
  security_memory.ml      Memory file + queue I/O

  review_types.ml         Finding, severity, review output types
  review_format.ml        Finding → PR comment / Slack formatting
  review_prompt.ml        General review prompt construction

  cost_tracking.ml        Per-agent + per-review cost estimation
  diff_parser.ml          Unified diff parser + path filtering
  state.ml / state_types.ml  Persistent state (review dedup)
  http_util.ml            HTTP request helper

test/
  test.ml                 Main test suite (golden-file tests)
  test_diff_parser.ml     Diff parser unit tests
  test_security_corpus.ml Security corpus test runner (calls Claude — on-demand)
  test_helpers.ml         Test context setup
  mock_api_responses/     Golden-file fixtures
  mock_payloads/          Sample webhook payloads
  security_corpus/        Synthetic vulnerable/safe diffs per vuln class

The codebase uses OCaml functors for testability - Reviewer.Make takes Github, Agent_runner, and Slack module implementations, so tests can inject mock versions (Api_local) without any HTTP calls.

Adding a Review Plugin

The general plugin is special — its summary becomes the review body. Every other plugin only emits findings, and they share one shape. To add a findings plugin:

Write a Make (AI : Api.Agent_runner) functor with name and a run that takes ~ctx ~repo_url ~config ~diff ~diff_text ~metadata ~debug_dir and returns (Review_types.finding list * Cost_tracking.agent_cost list) Lwt.t (the security_review_plugin.ml shape).
Add a config slice — a field on review_plugins_config in config_types.ml (with [@@deriving json, jsonschema] so it shows up in config-help).
Add one entry to the findings_plugins list in review_engine.ml (fp_name, fp_source, fp_enabled, fp_run).

The engine runs all enabled findings plugins in parallel, tags each plugin's findings with its fp_source for deduplication, and aggregates costs under fp_name. (Dedup currently privileges From_security on line collisions; new plugins use From_general unless they warrant the same treatment.)

Mock Agent Tests

The default test suite does not call external LLM providers. Tests instantiate plugins with Api_local.Agent_runner, which still exercises the production orchestration path but returns deterministic JSON from test/mock_api_responses/ based on the agent config.name.

These mock-agent tests are intended to cover agent plumbing and contracts:

the expected agents are invoked in order
mock JSON parses against the current schemas
filtering, validation, deduplication, error handling, and cost tracking behave deterministically
accepted/rejected findings are mapped into the final review output correctly

They are not evidence that a prompt is high quality, that confidence is calibrated, or that a real model will find the right issues. Prompt quality should be measured separately with an eval corpus that runs real model calls on labeled diffs. The on-demand security corpus runner is the current pattern for that kind of provider-backed check.

Keep file-based mock responses small and purposeful. Prefer adversarial fixtures that lock down one contract edge, such as a validator confirming a finding while echoing a damaged copy, over large "realistic" model transcripts. When a test only needs plugin-local behavior, prefer a small in-memory fake runner instead of adding another broad JSON fixture.

Name		Name	Last commit message	Last commit date
.claude/skills/reviewotron		.claude/skills/reviewotron
.serena		.serena
docs		docs
lib		lib
scripts		scripts
src		src
test		test
tools/replay		tools/replay
.gitignore		.gitignore
.ocamlformat		.ocamlformat
AGENTS.md		AGENTS.md
CLAUDE.md		CLAUDE.md
Makefile		Makefile
dune-project		dune-project
reviewotron.opam		reviewotron.opam
reviewotron.opam.template		reviewotron.opam.template
secrets.json.example		secrets.json.example

Latest commit

History

Folders and files

Repository files navigation