Implement DiffKit repo search MVP on Cloudflare + Livegrep

[stylessh]

Summary

Implements the RFC MVP for multi-repo search using Cloudflare as control plane and Livegrep as data plane, including Worker APIs, D1 metadata/job/index tables, queue consumers/producers, cron-driven sync orchestration, and a new self-hosted apps/search service for VPS deployment.

Also adds explicit OSS/self-host deployment docs with:

• concrete Livegrep Docker commands,
• local end-to-end test steps,
• VPS deployment checklist and reverse-proxy recommendations.

Changes

• Added D1 schema + migration for:
  • search_repo_registry
  • search_jobs
  • search_index_builds
• Added Worker search module (apps/dashboard/src/lib/search-worker.ts) with:
  • GET /api/search
  • POST /api/search/repos
  • GET /api/search/repos/:id/status
  • NOT_INDEXED handling + interactive bootstrap enqueue + ETA buckets
  • Livegrep query forwarding + normalized response (results, repo_status, partial, trace_id)
  • Queue processing skeleton for sync / index jobs
  • Cron scheduling for hot/warm/cold sync cadence
  • Index manifest retention cleanup in R2 (30-day policy)
• Wired Worker handlers in entry-worker.ts:
  • fetch routing for /api/search*
  • queue handler for search jobs
  • scheduled handler for periodic sync/cleanup
• Added Cloudflare env/bindings typing in src/env.d.ts for Livegrep/search-control/queues/R2.
• Updated wrangler.jsonc.example with queue producer/consumer bindings, cron trigger, search artifacts R2 bucket, and search vars.

New apps/search service (self-hosted)

Added a new Node/TypeScript app intended for VPS/private deployment:

• apps/search/src/server.ts
  • GET /healthz
  • GET /api/v1/search (proxy to Livegrep upstream)
  • POST /internal/repos/sync (auth-protected)
  • POST /internal/index/build (auth-protected)
• apps/search/src/repo-sync.ts
  • mirror clone/fetch pipeline using git
• apps/search/src/index-build.ts
  • placeholder index build artifact + manifest generation (ready to swap for real livegrep build commands)
• apps/search/src/state.ts
  • persistent local state for repos/build metadata
• apps/search/src/livegrep-client.ts
  • upstream query adapter + response normalization
• apps/search/src/config.ts
  • env-driven runtime config
• apps/search/README.md, .env.example, Dockerfile, .dockerignore

Documentation updates

• Expanded apps/search/README.md with:
  • exact Livegrep Docker commands (indexer/backend/frontend)
  • exact apps/search local run commands
  • exact local API smoke tests (/healthz, /api/v1/search, internal sync/index)
  • VPS deployment checklist and reverse-proxy examples
• Added “Self-hosted Search (Livegrep + apps/search)” section in root README.md for OSS users.

Open-question defaults from RFC were applied:

• User-added repos enqueue interactive jobs immediately.
• Max repo onboarding size cap set to 10,000 MB.
• Default-branch-only indexing retained.
• Manifest retention policy set to 30 days.

Test Plan

• pnpm --filter @diffkit/dashboard check-types
• pnpm --filter @diffkit/dashboard exec biome check src/lib/search-worker.ts src/entry-worker.ts src/db/schema.ts src/env.d.ts
• pnpm --filter @diffkit/search check-types
• pnpm --filter @diffkit/search check

Screenshots

Before	After
N/A	N/A

  

[coderabbitai]

Important

Review skipped

Draft detected.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Repository UI

Review profile: CHILL

Plan: Pro Plus

Run ID: 6ebd48cf-4853-4074-935b-ca62ff25af36

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

• 🔍 Trigger review

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch stylessh/diffkit-search-mvp-20fc

---

_{Comment @coderabbitai help to get the list of available commands and usage tips.}