Rayan9064 · Rayan9064 · Jul 21, 2026 · Jul 21, 2026
diff --git a/README.md b/README.md
@@ -1,53 +1,64 @@
-# DumpIt
+# DumpIt — Your AI Second Brain
 
-DumpIt is an AI knowledge vault for saved links. Save useful resources, organize them into collections, and ask questions across your private dump plus public shared resources with cited source cards.
+DumpIt is an AI-powered knowledge vault for saved links, plain-text notes, and PDF documents. Save useful resources, organize them into collections, and ask questions across your private vault plus public shared resources with cited source cards.
 
-Built with Next.js 14, TypeScript, Tailwind CSS, Firebase Auth, Firestore, Firebase Admin SDK, and Gemini.
+Built with **Next.js 14 (App Router)**, **TypeScript**, **Tailwind CSS**, **Firebase Auth & Firestore**, **Firebase Admin SDK**, **Google Gemini AI (RAG & Embeddings)**, **@sentry/nextjs**, and **@upstash/ratelimit**.
+
+---
 
 ## Features
 
-- Firebase authentication with email/password and Google sign-in.
-- Private resource library with tags, notes, collections, search, and visibility controls.
-- Shared Dump for discovering public resources from other users.
-- URL capture and metadata enrichment.
-- Server-side RAG indexing for saved links:
-  - fetch URL content
-  - extract readable text
-  - chunk text
-  - create Gemini embeddings
-  - store chunks in Firestore `resource_chunks`
-- Ask DumpIt AI search modes:
-  - `My Dump`: your indexed resources
-  - `Shared`: public resources from other users
-  - `All`: your resources plus shared public resources
-- Answers include citations and source cards when matching indexed chunks exist.
+- **Multi-Format Capture:**
+  - **Links:** Auto-enrichment of titles, descriptions, and tags via web scraping.
+  - **Notes:** Plain-text ideas, code snippets, and structured thoughts.
+  - **PDF Documents:** Fast in-memory text extraction for PDF uploads up to 10MB.
+- **AI Search & RAG (Ask DumpIt):**
+  - `My Dump`: Query your private indexed vault.
+  - `Shared`: Discover public resources saved by the community.
+  - `All`: Search across your vault plus community shared resources.
+  - Answers include exact citations and source cards.
+- **Organization & Curation:**
+  - Collections, tags, search filtering, and custom public profiles (`/u/[username]`).
+  - Cursor-based pagination on dashboard for high performance at scale.
+  - Skeleton shimmer card loading states.
+  - Duplicate resource detection.
+- **Enterprise Infrastructure & Performance:**
+  - Sentry exception monitoring across Client, Server, and Edge runtimes.
+  - Upstash Redis API rate limiting (60 req/min auth, 20 req/min public).
+  - PostHog telemetry & product analytics.
+  - Dynamic SEO generation via Next.js `robots.ts` and dynamic `sitemap.ts`.
+  - Browser extension support (Chrome Extension).
+
+---
 
 ## How RAG Works
 
-Saving a link creates a `resources` document, but AI search depends on indexing. The server fetches the saved URL, extracts readable page text, chunks it, embeds each chunk with Gemini, and writes vectorized chunks to Firestore.
+Saving a resource creates a `resources` document in Firestore. AI search relies on server-side background indexing:
+
+1. **Extraction:**
+   - **For Links:** Fetches page content and extracts readable text.
+   - **For PDFs:** Parses PDF binary in memory via `pdf-parse` and extracts plain text into `captured_text`.
+   - **For Notes:** Uses the note content directly.
+2. **Chunking & Embedding:**
+   - Splits text into contextual chunks.
+   - Generates 768-dimensional vector embeddings using Google's Gemini Embedding API.
+3. **Storage & Search:**
+   - Stores vectorized chunks in Firestore `resource_chunks`.
+   - Executes vector similarity searches against user queries.
 
 ```mermaid
 flowchart TD
-    URL["Saved URL"] --> Fetch["Server fetches page"]
-    Fetch --> Extract["Extract readable text"]
-    Extract --> Chunk["Chunk text"]
-    Chunk --> Embed["Gemini embedding"]
+    Input["Link / Note / PDF"] --> Extract["Extract Text (Fetch / pdf-parse)"]
+    Extract --> Chunk["Chunk Text"]
+    Chunk --> Embed["Gemini Embedding (768d)"]
     Embed --> Store["Firestore resource_chunks"]
-    Ask["User question"] --> QueryEmbed["Gemini query embedding"]
-    QueryEmbed --> Vector["Firestore vector search"]
+    Ask["User Question"] --> QueryEmbed["Gemini Query Embedding"]
+    QueryEmbed --> Vector["Firestore Vector Search"]
     Store --> Vector
-    Vector --> Answer["Gemini answer with citations"]
+    Vector --> Answer["Gemini Answer with Citations"]
 ```
 
-A saved resource is useful to Ask DumpIt only after `index_status` becomes `indexed`.
-
-Indexing can fail or be skipped when:
-
-- the page blocks server-side fetches
-- content is behind login
-- content is rendered only by client-side JavaScript
-- the page has little readable text
-- Gemini or Firestore configuration is missing
+---
 
 ## Quick Start
 
@@ -61,13 +72,17 @@ npm run dev
 
 Open [http://localhost:3000](http://localhost:3000).
 
+---
+
 ## Chrome Extension
 
-DumpIt includes a Chrome extension for one-click capturing, side panel search, and text selection clipping. For details on local installation and configuration, see the [Extension README](dumpit-extension/README.md).
+DumpIt includes a Chrome extension for one-click link capturing, side-panel search, and text selection clipping. For setup instructions, see the [Extension README](dumpit-extension/README.md).
+
+---
 
 ## Environment Variables
 
-Use Firebase variables for Firebase only, and Gemini variables for AI only. Do not reuse Firebase keys as Gemini keys.
+Configure your environment variables in `.env.local` (and in Vercel for production deployments):
 
 ```env
 # Firebase Client SDK (browser-safe)
@@ -88,34 +103,34 @@ FIREBASE_PRIVATE_KEY="-----BEGIN PRIVATE KEY-----\n...\n-----END PRIVATE KEY----
 GEMINI_API_KEY=
 GEMINI_MODEL=gemini-2.5-flash
 GEMINI_EMBEDDING_MODEL=gemini-embedding-001
-```
 
-Important:
+# App URL & SEO
+NEXT_PUBLIC_APP_URL=https://dumpit-three.vercel.app
 
-- `GEMINI_API_KEY` must come from Google AI Studio / Gemini API.
-- `NEXT_PUBLIC_FIREBASE_API_KEY` is the Firebase Web SDK key and is not valid for Gemini.
-- Keep `GEMINI_MODEL=gemini-2.5-flash` for v1. `gemini-2.5-pro` may have no free-tier quota and can return 429 errors.
-- Set Gemini variables in Vercel without quotes or `Bearer`.
-- Redeploy Vercel after changing environment variables.
+# Monitoring & Rate Limiting (Optional)
+NEXT_PUBLIC_SENTRY_DSN=
+UPSTASH_REDIS_REST_URL=
+UPSTASH_REDIS_REST_TOKEN=
+NEXT_PUBLIC_POSTHOG_KEY=
+NEXT_PUBLIC_POSTHOG_HOST=
+```
 
-## Firestore Vector Indexes
+---
 
-Ask DumpIt requires Firestore vector indexes for the query shapes used by the app.
+## Firestore Vector Indexes
 
-Create the private search index:
+Ask DumpIt requires Firestore vector indexes for semantic search:
 
 ```bash
+# Private search index
 gcloud firestore indexes composite create \
   --project=YOUR_PROJECT_ID \
   --collection-group=resource_chunks \
   --query-scope=COLLECTION \
   --field-config=order=ASCENDING,field-path=user_id \
   --field-config=vector-config='{"dimension":"768","flat": "{}"}',field-path=embedding
-```
 
-Create the shared/all search index:
-
-```bash
+# Shared / All search index
 gcloud firestore indexes composite create \
   --project=YOUR_PROJECT_ID \
   --collection-group=resource_chunks \
@@ -125,38 +140,18 @@ gcloud firestore indexes composite create \
   --field-config=vector-config='{"dimension":"768","flat": "{}"}',field-path=embedding
 ```
 
-Monitor index creation:
-
-```bash
-gcloud firestore operations list --project=YOUR_PROJECT_ID
-```
-
-Wait for `state: SUCCESSFUL` and `state: READY` before testing AI search.
+---
 
 ## Commands
 
 ```bash
-npm run dev
-npm run typecheck
-npm test -- --run
-npm run build
-npm run secret-scan
+npm run dev        # Run Next.js dev server
+npm run typecheck  # TypeScript type checking
+npm run build      # Build production bundle
+npm test           # Run Vitest unit tests
 ```
 
-## Deployment
-
-See [docs/deployment.md](docs/deployment.md) for the Vercel, Firebase, Gemini, and Firestore index runbook.
-
-Production checklist:
-
-- Firebase Auth enabled.
-- Firestore enabled.
-- Firebase Admin service account variables set in Vercel.
-- Gemini API key from AI Studio set as `GEMINI_API_KEY`.
-- `GEMINI_MODEL=gemini-2.5-flash`.
-- Both Firestore vector indexes are `READY`.
-- Save a resource and confirm it becomes `indexed`.
-- Test Ask DumpIt in `My Dump`, then `Shared`, then `All`.
+---
 
 ## Documentation
 

diff --git a/docs/api-spec.md b/docs/api-spec.md
@@ -6,87 +6,105 @@ Authenticated endpoints require:
 Authorization: Bearer <firebase_id_token>
 ```
 
-Server routes derive `uid` from the verified Firebase token and do not trust client-supplied owner IDs.
+Server routes derive `uid` from the verified Firebase token and do not trust client-supplied owner IDs. All routes enforce rate-limiting via `@upstash/ratelimit` when configured (60 requests/minute for authenticated users, 20 requests/minute for public queries).
+
+---
 
 ## /api/resources
-- GET: fetch authenticated user's resources. Optional query: `collectionId`.
-- POST: create a resource. Body: `title`, `link`, `note`, `tag`, `is_public`, `collection_ids`, `new_collection`.
-- PUT: update an owned resource. Body: `id`, `title`, `link`, `note`, `tag`, `is_public`, `collection_ids`.
-- DELETE: delete an owned resource. Query: `id`.
-- Create/update performs best-effort indexing and returns an `indexing` object.
-- Saved resources are not AI-searchable until indexing writes `resource_chunks` and sets `index_status` to `indexed`.
+- **GET:** Fetch authenticated user's resources or public profile resources.
+  - Query Params:
+    - `collectionId` (optional): Filter resources by collection ID.
+    - `username` + `public=true` (optional): Fetch public resources for user `@username`.
+    - `cursor` (optional): Document ID for cursor-based pagination.
+    - `limit` (optional): Number of items to return (default: 20, max: 100).
+  - Response: `{ success, resources, nextCursor }`.
+- **POST:** Create a link or text note resource.
+  - Body: `{ title, link?, note?, tag?, is_public?, collection_ids?, new_collection?, captured_text? }`.
+  - Duplicate detection: Returns `409 Conflict` if the link URL was already saved by the user.
+- **PUT:** Update an owned resource. Body: `{ id, title, link, note, tag, is_public, collection_ids }`.
+- **DELETE:** Delete an owned resource. Query: `?id=<resource_id>`.
+
+---
+
+## /api/resources/pdf
+- **POST:** Upload and parse a PDF document.
+  - Request format: `multipart/form-data`.
+  - Fields:
+    - `file`: PDF binary file (max size: 10MB).
+    - `title` (optional): Custom document title (defaults to filename).
+    - `note` (optional): Custom description or notes.
+    - `is_public` (optional): `'true'` | `'false'`.
+    - `collection_ids` (optional): JSON array string of collection IDs.
+  - Processing: Parses PDF in memory via `pdf-parse`, extracts text (up to 50,000 characters), creates a resource with `tag: 'PDF'`, and queues background RAG indexing.
+  - Response: `{ success, resource }`.
+
+---
 
 ## /api/collections
-- GET: fetch authenticated user's collections, or public shared collections with `?shared=true`.
-- POST: create a collection for the authenticated user.
-- PUT: update an owned collection. Body: `collectionId`.
-- PATCH: reorder owned collections. Body: `orderedIds`.
-- DELETE: delete an owned collection. Query: `collectionId`.
+- **GET:** Fetch authenticated user's collections, or public shared collections with `?shared=true`.
+- **POST:** Create a collection for the authenticated user.
+- **PUT:** Update an owned collection. Body: `collectionId`.
+- **PATCH:** Reorder owned collections. Body: `orderedIds`.
+- **DELETE:** Delete an owned collection. Query: `collectionId`.
+
+---
 
 ## /api/collections/memberships
-- POST: add an owned resource to an owned collection. Body: `collectionId`, `resourceId`.
-- DELETE: remove an owned resource from an owned collection. Body or query: `collectionId`, `resourceId`.
+- **POST:** Add an owned resource to an owned collection. Body: `{ collectionId, resourceId }`.
+- **DELETE:** Remove an owned resource from an owned collection. Body or query: `{ collectionId, resourceId }`.
+
+---
 
 ## /api/public-resources
-- GET: fetch public resources from other users. Requires auth.
-- POST: copy a public resource to the authenticated user's private dump. Body: `resourceId`.
+- **GET:** Fetch public resources from other users. Requires auth.
+- **POST:** Copy a public resource to the authenticated user's private dump. Body: `{ resourceId }`.
+
+---
 
 ## /api/user-profile
-- POST: create or update authenticated user's profile.
-- GET: get authenticated user's profile, or stats with `?type=stats`.
-- PUT: update authenticated user's profile.
+- **GET:** Get authenticated user's profile, or stats with `?type=stats`.
+- **POST:** Create or update authenticated user's profile.
+- **PUT:** Update authenticated user's profile.
+
+---
 
 ## /api/enrich
-- POST: public metadata extraction. Body: `{ "url": "https://example.com" }`.
-- Response: `{ title, description, suggestedTag, favicon }`.
+- **POST:** Public link metadata extraction. Body: `{ "url": "https://example.com" }`.
+- **Response:** `{ title, description, suggestedTag, favicon }`.
+
+---
 
 ## /api/check-username
-- GET or POST: check username uniqueness.
+- **GET or POST:** Check username availability.
+
+---
 
 ## /api/ai/search
-- POST: semantic search across indexed chunks.
-- Body: `{ "query": "firebase auth", "mode": "mine" | "shared" | "all", "limit": 8 }`.
-- Response: `{ success, results }`.
-- Requires Firestore vector indexes for the selected mode.
+- **POST:** Semantic vector search across indexed chunks.
+- **Body:** `{ "query": "firebase auth", "mode": "mine" | "shared" | "all", "limit": 8 }`.
+- **Response:** `{ success, results }`.
 
-## /api/ai/ask
-- POST: RAG answer generation with citations.
-- Body: `{ "question": "What should I read about Firebase auth?", "mode": "mine" | "shared" | "all", "limit": 8 }`.
-- Response: `{ success, answer, sources }`.
-- Retrieval runs before answer generation. If no matching chunks are found, the API returns an answer explaining that no indexed resources matched.
-- Uses `GEMINI_MODEL`, recommended `gemini-2.5-flash`, for answer generation.
+---
 
-## /api/ai/reindex-resource
-- POST: retry indexing for an owned resource.
-- Body: `{ "resourceId": "..." }`.
-- Response: `{ success, status, chunksIndexed, error? }`.
-- Use this when `index_status` is `failed`, `skipped`, or stale after changing a resource URL.
+## /api/ai/ask
+- **POST:** RAG answer generation with citations.
+- **Body:** `{ "question": "What should I read about Firebase auth?", "mode": "mine" | "shared" | "all", "limit": 8 }`.
+- **Response:** `{ success, answer, sources }`.
 
-## AI Search Modes
-- `mine`: private and public resources owned by the authenticated user.
-- `shared`: public resources owned by other users.
-- `all`: authenticated user's resources plus other users' public resources.
+---
 
-## AI/RAG Operational Requirements
+## /api/ai/reindex-resource
+- **POST:** Retry indexing for an owned resource.
+- **Body:** `{ "resourceId": "..." }`.
+- **Response:** `{ success, status, chunksIndexed, error? }`.
 
-- `GEMINI_API_KEY` must be a valid Google AI Studio / Gemini API key.
-- `GEMINI_MODEL=gemini-2.5-flash` is recommended for production v1.
-- `GEMINI_EMBEDDING_MODEL=gemini-embedding-001`.
-- Firestore vector index for `user_id + embedding` is required for `mine`.
-- Firestore vector index for `is_public + user_id + embedding` is required for `shared` and `all`.
-- Resource chunks use 768-dimensional embeddings.
+---
 
 ## Error Codes
-- 400: Bad Request
-- 401: Unauthorized
-- 404: Not Found
-- 409: Conflict
-- 429: AI provider quota exceeded
-- 500: Internal Server Error
-
-Common AI failure causes:
-
-- Invalid Gemini key: replace `GEMINI_API_KEY` with an AI Studio key and redeploy.
-- Gemini quota exceeded: use `gemini-2.5-flash` or enable billing/quota.
-- Missing Firestore vector index: create the exact index command returned by Firestore.
-- No indexed chunks: save/reindex a resource and wait for `index_status=indexed`.
+- **400:** Bad Request / Missing Fields
+- **401:** Unauthorized
+- **404:** Not Found
+- **409:** Conflict / Duplicate Link
+- **422:** Unprocessable Entity (e.g. Scanned / Image-only PDF)
+- **429:** Rate limit / AI provider quota exceeded
+- **500:** Internal Server Error