Skip to content

overview.md

Latest commit

 

History

History
143 lines (116 loc) · 24.3 KB

overview.md

File metadata and controls

143 lines (116 loc) · 24.3 KB
Raw

Desktop app overview

SurfSense is a desktop app for research over your own documents: add files, ask questions that are answered with citations into them, and turn them into deliverables such as summaries, slides, quizzes and podcasts. It serves one person on one machine and works offline: the parser, the embedding model and the model runtimes run locally, and everything the app knows lives on this machine, in one SQLite file and the files under its data directory. This page is the map of surfsense_local/; each feature has its own page, listed at the end.

Code: surfsense_local/backend/, surfsense_local/frontend/, surfsense_local/electron/ Decisions: ADR 0004, ADR 0005, ADR 0008, ADR 0009, ADR 0011, ADR 0016, ADR 0017

Bird's-eye view

A file is uploaded, parsed to markdown, cut into passages and embedded. A chat turn retrieves the passages nearest the question, and a model answers from them, citing each one. A Studio job hands the selected documents' text to a model and stores the result as a document, with a file for ten of its twelve formats. Every design choice follows from who this is for:

  • One user. There are no accounts, no auth and no users table. The API binds to 127.0.0.1 and accepts any origin, because the packaged renderer loads from file://.
  • Offline. Docling parses, bge-small embeds on onnxruntime, llama-server generates text, sd-server generates images and audio.cpp's server voices podcasts, all on this machine. Remote destinations (Hugging Face for model search and downloads, TDF's archive for the optional Office pack, and each remote host a connection points at) stay off until the user allows them (egress.md). The app carries no telemetry or analytics code.
  • SQLite. surfsense.db holds every table, the search index included: FTS5 for keywords, sqlite-vec for vectors. huey.db holds the job queues. Next to the hosted stack there is no Postgres or Zero, no Celery or Redis, no LangGraph and no Docker.

Processes

Electron main ─┬─ api            FastAPI on 127.0.0.1, free port    ─┐
               ├─ worker-ingest  Huey consumer, "ingest", 1 thread   ├─ surfsense.db, huey.db
               ├─ worker-studio  Huey consumer, "studio", 4 threads ─┘
               ├─ llamacpp       llama-server in router mode
               ├─ sdcpp          sd-server, started once an image model is chosen
               ├─ audiocpp       audiocpp_server, started once the API names an audio model
               ├─ audiocpp-transcription  the same binary, started once the API names a speech model
               └─ opencode       opencode serve, started once the API writes its configuration

BrowserWindow (Vite SPA) ── HTTP ──> api
api ── admission ── HTTP ──> llama-server, remote OpenAI-compatible endpoints
worker-studio, opencode ── POST /internal/models/text/generate, /agent/model/v1 ──> api
worker-studio ── HTTP ──> sd-server, audiocpp, remote image and speech endpoints
worker-ingest, worker-studio ── POST /internal/events ──> api ── SSE /workspaces/{id}/events
  • Electron's main process starts four sidecars at boot and supervises them (sidecars/boot.ts, sidecars/): the API, one Huey worker per queue, and llama-server. Packaged builds run frozen binaries from the app's resources; pnpm dev runs main.py and worker.py <queue> on the backend's .venv interpreter, synced by predev's own uv run scripts. Not through uv run: Windows kills Electron's children with it but not theirs, so after a Ctrl-C the last session's Studio worker, still reading the queue, took the next jobs.
  • llama-server starts whenever its pinned build is staged, in dev too (pnpm build:llamacpp, which predev runs). It reads per-model arguments from a preset file once at startup, so Electron restarts it when the API rewrites that file (local-models/runtime.md).
  • sd-server takes its model as startup arguments, so it cannot start at boot. Whenever its pinned build is staged, in dev too (pnpm build:sdcpp, which predev runs), watchImageModel asks the API every 5 seconds which files to run, and starts, restarts or stops it on a change. The API names a model only while a Studio job needs one and for up to 5 minutes after, so its weights are not held beside the chat model all session (studio.md).
  • audiocpp_server refuses an empty model list, so it starts only once the API has written audio/server.json under the data directory, in dev too (pnpm build:audiocpp, which predev runs). On Windows and Linux that script compiles audio.cpp, and without a C++ toolchain the app runs without local audio (packaging). Electron checks that file every 5 seconds and restarts the server when it changes. Electron sets the machine-wide flags: the CPU backend, half the logical cores up to 8, one loaded model, and an unload after 5 idle minutes. The API writes that file whenever an audio model is installed or deleted, and at startup (local-models/catalog.md). The Studio worker voices podcasts there, at SURFSENSE_LOCAL_AUDIO_BASE_URL, and unloads the model when a podcast ends (studio.md).
  • The same audiocpp_server runs a second time as audiocpp-transcription, for speech recognition, from its own audio/transcription.json and on its own port, so a voice install, a podcast's unload or a busy voice model never touches it (ADR 0056). Electron checks that file every 5 seconds: it starts the server once the file is there, restarts it when the file changes and stops it when the file goes. It takes the voices' flags, with no eSpeak. Unlike the voices' server it is restarted after a crash, at most once every 10 seconds, and left stopped after 3 crashes in 5 minutes until the file changes. The API alone is told its address, SURFSENSE_LOCAL_TRANSCRIPTION_BASE_URL. The API writes that file once a speech recogniser is downloaded, and only when its bytes change (catalog), so the server runs only then. The API's transcription routes are its only caller, one request at a time through one gate (admission).
  • Only the API gates the window. Electron waits up to 60 seconds for /health and gives up at once if the API exits. llama-server is best-effort; its state shows through /llm/providers.
  • opencode, the agent's engine, starts only once the API has written agent/opencode.json under the data directory, in dev too (pnpm build:opencode, which predev runs, stages it unless SURFSENSE_LOCAL_OPENCODE_ENABLED=0: agent). Electron checks for that file every 2 seconds: it starts opencode once the file is there, stops it when the file goes, and restarts it after a crash with a growing delay, stopping after three quick exits until the API writes the file again (agent). A rewrite does not restart it, because opencode reads the file per folder the first time the folder is used; the API makes it read the file again through POST /global/dispose (agent proposal). Electron removes the file at boot, so each run's API writes its own. At boot Electron picks opencode's port and a random password, which it passes to the API alone as SURFSENSE_LOCAL_OPENCODE_URL and SURFSENSE_LOCAL_OPENCODE_PASSWORD. opencode's environment is built from a few system variables rather than inherited, with its home and XDG folders under agent/opencode/, its own fetches switched off and its proxy pointed at the discard port (sidecars/opencode.ts). serve keeps running when its parent is killed, so Electron records the pid, port and password in agent/opencode-process.json, and the next boot stops a recorded process that still accepts that password.
  • Before a quit, whether from the menu or by closing the last window, Electron asks the API how many chat replies are being written; with any, it asks the person, and on Quit has the API store each reply before anything is stopped (chat). Cancel keeps the app open.
  • On macOS and Linux each child runs in its own process group. On quit Electron sends SIGTERM and, after 5 seconds, SIGKILL; the group is also killed as soon as the sidecar itself exits, because a child that ignores SIGTERM would outlive the app. On Windows it kills the process tree. A single-instance lock hands a second launch to the first window, because two sets of sidecars would fight over the SQLite file.
  • The Python sidecars are configured through SURFSENSE_LOCAL_* variables: the API's host and port, the data and models directories, the llama-server and sd-server addresses, the images folder, audio.cpp's address and folder where it is staged, and SURFSENSE_LOCAL_SECRET, the key that encrypts stored API keys (connections.md). No other sidecar receives the secret. The API alone also gets SURFSENSE_LOCAL_SHELL_PID, Electron's own pid, the root of what Settings › Resources counts as the app (resource-usage.md), and the speech recognition server's address.

Layer boundary

Concern Runs in
HTTP routes, OpenAPI, the chat and events streams api
Migrations and the default workspace, at startup api
Upload streaming, dedup and enqueueing api
Query embedding and retrieve() for chat api, in process
Chat runs, the chat stream and thread titles api
Admission to the local runtime, and text generation for every other process api (admission)
Model catalog, downloads and the hardware probe api
Parse, chunk, embed and index a document worker-ingest
Generate and render an artifact, index its body worker-studio
Sidecar lifecycle, the keychain secret, updates, opening files natively Electron main

The rules that keep the processes out of each other's way:

  • The API owns the schema. upgrade_to_head() runs in the API's lifespan before anything else, and the workers only read and write rows (ADR 0005). The API then seeds a workspace named "My Workspace" if none exists.
  • A job is enqueued after the commit that writes its row. The worker is another process and would otherwise look for a row the request had not committed.
  • Workers change status through worker/jobs.py: begin_job marks a row processing unless it was cancelled, raise_if_cancelled runs between steps, and finish_job writes the terminal status only if a cancel has not won the race. A running job is never killed; it notices a cancel at its next step.
  • Every connection runs in WAL mode with a 5-second busy timeout, foreign keys on and sqlite-vec loaded, and opens transactions with BEGIN IMMEDIATE, so a read-then-write waits for the lock instead of failing. The price is that no transaction may stay open across a slow call. The API raises if a request opens one on the event loop, and async handlers run each stretch of session work through transact() (api/dependencies.py).
  • Ingest runs one job at a time because it saturates a CPU and writes to the file the API serves from. Studio mostly waits on a model, so it runs four. Separate queues keep an import from sitting in front of a summary (ADR 0008).

Freshness

  • Workers call POST /internal/events after each status change (worker/notify.py): ingest sends a documents event keyed by document id, Studio an artifacts event keyed by artifact id. The notice is best-effort with a 2-second timeout; losing one costs a live update, never the job.
  • The API announces each reply, chat or agent, starting, changing state and ending on the same stream, as a chat-runs event keyed by thread id with status running, queued, needs-approval or done (chat).
  • The API notifies of its own document changes on the same stream, straight to the broker: a note written, an upload, a rename or edit, a retry, a cancel, and a delete, whose status is deleted (api/notify.py). So a change reaches an open window whoever made it: the window itself, a plugin, or a second window.
  • The API fans each notice out on GET /workspaces/{id}/events (modules/events/) as a named SSE event whose data is {"ids": [...], "status": "..."}, with a transient progress {done_s, total_s} on a recording's event for each transcribed window (documents). The stream opens with a : connected comment and sends : ping after 15 idle seconds. The broker is an in-memory map, which holds because one uvicorn process serves the app.
  • The frontend holds one subscription per workspace, shared by the lists that listen (features/workspaces/workspace-changes.ts), and closes it with the last of them. The sources panel reloads its list on each documents event and the Studio artifact list on each artifacts event; both reload each time a dropped stream is back, for what changed while nothing listened. Because a notice can be lost, each list also refetches every 10 seconds while any of its rows is pending or processing, and stops when none is: a lost notice leaves a row stale for at most that long, and nothing is requested on a timer while nothing is in flight.

Data directory

~/.surfsense/                 ~/.surfsense-dev under `pnpm dev`
├── surfsense.db              every table and the search index
├── huey.db                   the ingest and studio queues
├── models/                   GGUF weights and llama-server's models.ini
├── images/                   sd-server weights (hosts with sd-server staged)
├── electron/                 Electron's userData: secret.bin, updates.json, window and theme prefs
├── agent/                    opencode.json, which the API writes, and opencode's own home (hosts with opencode staged)
└── data/workspaces/<id>/
    ├── documents/<id>/       the upload, under its own name
    ├── chats/<id>/           images a thread's turns carried
    ├── artifacts/<id>/       an artifact's rendered file, named by role
    └── agent/                each agent thread's folder and the text its sources link to
  • The bundled embedding, parser and voice models are read from SURFSENSE_LOCAL_MODELS_DIR: the app's resources when packaged, backend/models under pnpm dev, and <data dir>/models for a bare uv run.
  • Paths under data/ are built from row ids. The only part of a user's filename that reaches the disk is a validated extension.
  • A bare uv run with no SURFSENSE_LOCAL_SECRET writes its key to secret beside the database.
  • SURFSENSE_LOCAL_DATA_DIR, SURFSENSE_LOCAL_HOST and SURFSENSE_LOCAL_PORT override the defaults. Tables and files are detailed in data-model.md.

Frontend

  • A Vite and React SPA with Tailwind and shadcn/ui components in components/ui/, built on Base UI in shadcn's base-nova style; assistant-ui drives the conversation, and still depends on Radix. Packaged, Electron loads frontend/dist/index.html from disk, which is why Vite builds with relative asset paths. In dev it loads the Vite server on port 5173.
  • The API's address comes from the preload: Electron passes it as a command-line argument and preload/index.ts exposes it as window.surfsense.apiUrl. In a bare browser there is no preload, requests stay root-relative, and the Vite dev server forwards /health, /llm, /workspaces, /chat and /artifacts to 127.0.0.1:8000.
  • The preload bridge is the renderer's only other channel: opening or revealing an original file natively, opening an external link (an allowlisted one, one the user confirmed in the link popup, or a listed plugin's page by its id), the platform name, updates, theme, the title bar, this session's log, unexpected sidecar-exit notices, and the operating system's microphone switch for SurfSense. The renderer never sees Node or the sidecars directly. A source with a registered viewer (PDF first) is fetched from the API and previewed in place of the left sidebar; other file types keep the native open action (documents.md).
  • Electron approves every permission a session has no handler for, so the main process gives each session request and check handlers as it is created (permissions/), before the app is ready. media is granted only to the main window's main frame on the app's own page, a file in the SPA's own folder when packaged and Vite's origin in development, and only for audio: a camera, a desktop or tab capture and display-capture are refused. Every other permission keeps Electron's own answer, so clipboard writes keep working; a strict allowlist would need an audit of every permission the app uses. The Word snapshot window's session refuses every permission (agent).
  • window.surfsense.microphone answers the operating system's side (microphone/), for the main window's main frame only: status() is systemPreferences.getMediaAccessStatus("microphone") on Windows and macOS, and granted on Linux, which has no switch for an AppImage or a deb; request() asks macOS with askForMediaAccess before the page calls getUserMedia, and on Windows reports the switch as it is; openSettings() opens ms-settings:privacy-microphone or the macOS Privacy › Microphone pane, which allowedExternalUrl cannot, since it passes only https, and answers false on Linux. The page never names the address.
  • API calls go through request() in lib/api.ts, except the Studio viewers, which fetch an artifact's file bytes directly; an <img>, <audio> or download link takes its absolute address from apiUrl(). request() prefixes the address, turns an error body into an ApiError with its code, and when a user action is refused with 403 egress_disabled it asks for consent and retries once.
  • TanStack Query holds most server state: threads and messages, the citation panel, the model catalog and settings, the Studio formats and artifact list, and an open artifact with what its viewer reads. An artifacts event invalidates the list and every open artifact under one key prefix (features/studio/query-keys.ts), so a regenerated artifact that is open shows its new generation. The sources list is component state, reloaded on the events above.
  • There is no router. app/app-bootstrap.tsx asks GET /llm/onboarding, then renders either onboarding (local-models/selection.md) or the lazily loaded dashboard: the workspace rail, threads and sources on the left, the conversation in the middle, and Studio, artifacts and the citation panel on the right. The left and right columns resize by dragging their inner edge, and the conversation keeps at least 520 px (studio).
  • Code is grouped by feature under src/features/. Each HTTP contract lives in one feature's api.ts, next to its hooks and components, and other features import it from there.

Codemap

  • backend/api/: create_app() and its lifespan (migrations, the default workspace, a background catalog warm-up), the per-request session and transact().
  • backend/modules/: one folder per feature, holding whichever of its models, schemas, router and service it needs: workspaces, documents, chunks, chat, artifacts (Studio), events, llm (runtimes, catalog, connections, selection), egress, agent, plugins, license, migration (import) and health. shared.db.import_models() imports every model at startup, because relationships name their targets as strings.
  • backend/worker/: consumer.py drains one queue per process, jobs.py holds status transitions and cancellation, notify.py the change notice, and ingestion/ and studio/ the two pipelines.
  • backend/shared/: configuration, the engine and its pragmas, upgrade_to_head(), the two Huey queues, retrieve(), and the secret that encrypts stored keys.
  • backend/alembic/: revisions 0001 to 0030, all hand-written. env.py has no target_metadata, so autogenerate cannot run by accident.
  • frontend/src/features/: chat, agent, sources, studio, workspaces, dashboard, models, onboarding, settings, egress, plugins, license, migration, updates and feedback.
  • electron/src/main/: index.ts (the window, IPC and the paths it hands to boot), sidecars/ (boot, the supervisor, one spec per sidecar, and the image-model, preset, audio and agent watchers), session-log/ (this run's output, for issue reports), permissions/ (who may use the microphone), microphone/ (the operating system's switch), secret.ts, updater.ts and document-files.ts.

Where to read next

Known gaps

  • The sources list reloads itself on a workspace event rather than through TanStack Query: it is component state, so nothing else that shows a document is invalidated by the same event.
  • When the API does not start, because it exits or /health does not answer within 60 seconds, Electron writes the reason to the session log and quits: no window opens, nothing on screen says why, and the log, kept only in memory, goes with the app (about). A migration snapshot that cannot be written, such as on a full disk, ends this way (data model).