pr-review-agent

A TypeScript CLI agent that reviews a GitHub PR with an LLM and writes real review comments, reviewer assignments, and approve/comment actions back to GitHub.

Live demo PR: yahyo7/react#1 (open this — you will see 3 inline review comments left by the agent, plus an escalation summary)

---

What it does

Given one PR URL and a --mode conservative|aggressive flag:

1. Reads the PR (metadata, full diff, collaborators) via the GitHub API
2. Sends the PR to OpenAI for analysis, getting back a structured verdict (findings + confidence + suggested reviewers)
3. Applies the --mode policy to that verdict to decide: approve vs escalate
4. Writes the result to GitHub:
   • approve → submits an APPROVE review with a summary
   • escalate → posts a COMMENT review with inline comments on the offending lines, requests the suggested reviewers, and posts a per-reviewer comment explaining what to focus on
5. Logs every LLM call (prompt, model, response, tokens, latency) to logs/observability.jsonl

---

Install

Requires Node.js 20+.

git clone <this-repo-url>
cd pr-review-agent
npm install

---

Configure

cp .env.example .env

Edit .env:

GITHUB_TOKEN=github_pat_...
OPENAI_API_KEY=sk-...
OPENAI_MODEL=gpt-4o          # optional, defaults to gpt-4o-2024-08-06

GitHub token scopes (fine-grained recommended): Contents: Read, Metadata: Read, Pull requests: Read and Write.

Production note: the token should belong to a dedicated bot user, not the same user who opens PRs. GitHub blocks self-approval (422), so if the agent runs as the PR author the APPROVE event will fail.

---

Run

# Aggressive mode: low approval bar (bias toward auto-approve)
npm run dev -- https://github.com/<org>/<repo>/pull/<n> --mode aggressive

# Conservative mode: high approval bar (bias toward escalate)
npm run dev -- https://github.com/<org>/<repo>/pull/<n> --mode conservative

# Production-ish: compile and run
npm run build && node dist/index.js https://github.com/<org>/<repo>/pull/<n> --mode conservative

The agent prints the LLM verdict to stdout and progress to stderr. The actual outcome lives on GitHub — open the PR in your browser to see what landed.

---

How --mode actually changes behavior

The LLM's job is analysis only — it produces findings and a confidence score (0..1). The mode flag is a deterministic policy threshold applied to that output:

Mode	Approve if
aggressive	confidence ≥ 0.5 and no critical findings
conservative	confidence ≥ 0.85 and no critical and no major findings
Otherwise	escalate to humans

This means changing review policy is a one-line edit to two named constants in src/decision.ts — no prompt engineering required. Model and policy are decoupled.

---

Observability

Every LLM call appends one line to logs/observability.jsonl:

{"ts":"...","call_id":"a1b2c3d4","stage":"analyze","model":"gpt-4o-2024-08-06","prompt_messages":[...],"response":{...},"usage":{"prompt_tokens":...,"completion_tokens":...,"total_tokens":...},"latency_ms":...}

Query examples (after a few runs):

# Average latency per stage
cat logs/observability.jsonl | jq -s 'group_by(.stage) | map({stage: .[0].stage, avg_ms: (map(.latency_ms) | add / length)})'

# Total tokens spent today
cat logs/observability.jsonl | jq -s 'map(.usage.total_tokens) | add'

# All findings the agent has ever made
cat logs/observability.jsonl | jq '.response.findings[]?'

Plus a one-line stderr summary per call during runs:

[llm] call=a1b2 stage=analyze model=gpt-4o tokens=954→215 latency=3164ms

---

Architecture

CLI (index.ts)
  └→ github.ts        read PR + collaborators
  └→ analyzer.ts      single OpenAI call with structured-output schema
       └→ observability.ts   wraps every LLM call, logs JSONL
  └→ decision.ts      pure function: (verdict, mode) → action
  └→ writer.ts        Octokit writes (createReview / requestReviewers / issues.createComment)

Single-pass pipeline, not an agent loop. The task is read → reason → write. A cyclical agent (LangGraph-style) would add failure modes (reflection drift, infinite loops) for zero benefit on this problem.

Structured outputs over raw JSON. analyzer.ts calls OpenAI with response_format: zodResponseFormat(VerdictSchema, ...), so the response is server-side-validated against the schema. No retry-on-bad-JSON loops needed.

Defense in depth on reviewers. The LLM may hallucinate a username. We pass the collaborator list into the prompt and post-filter in code against the same list — anything the model invents gets dropped before any GitHub write.

Graceful degradation. If inline review comments fail with 422 (a common cause: LLM picked a line outside the diff hunks), the writer retries with the same findings rendered into the review body. The review still lands.

See PLAN.md for the full architecture write-up with mermaid diagrams, time budget, and interview defense notes.

---

Tests

npm test

10 unit tests on src/decision.ts (the highest-ROI test target — pure function, encodes the mode-flag policy). Boundary cases at the threshold values are explicitly covered.

✓ src/decision.test.ts (10 tests) 2ms

---

Known limitations (honest assessment)

• Severity calibration: the LLM occasionally under-rates security issues (e.g., classifying SQL injection as major instead of critical). Mitigation path: few-shot examples in the prompt, or a specialized security-review pass. Not done in v0.1.
• Missed findings: simple bugs like missing zero-division checks may slip through. Same mitigation: targeted few-shot calibration.
• Single LLM call: very large PRs (>100k tokens of input) trigger a soft warning but still go through as one call. A map-reduce chunker would help here; intentionally deferred (YAGNI on the tested PR sizes).
• No retry on transient OpenAI/GitHub failures beyond what the SDKs do by default. Production deployments would add a wrapping retry with jitter.

---

AI tools used while building

• Claude Code (Sonnet 4.6 → Opus 4.7) for code generation, structured planning (PLAN.md), and incremental refactors. All architectural decisions were authored, reviewed, and defended by the developer — Claude was a fast typist and a sounding board, not the architect.

---