This document covers how Tenant First Aid is deployed, where it runs, how configuration and secrets are managed, how to debug issues, who has access, and how the service is monitored.
Not a developer? Start with Overview for stakeholders. The rest of the document is aimed at contributors and operators.
Tenant First Aid runs as a public website at tenantfirstaid.com. "Deployment" is the process of taking code changes written by volunteers and making them live for users.
flowchart TD
Dev["👩💻 Developer<br/>writes code"] -->|"opens pull request"| Review["👥 Code review<br/>& automated tests"]
Review -->|"approved and merged"| Build["🔨 GitHub Actions<br/>builds the app"]
Build -->|"copies files to server"| Server["🖥️ Digital Ocean<br/>server"]
Server -->|"serves the website"| User["🏠 Tenant<br/>visits the site"]
Key points for stakeholders:
- Who manages deployments? Project admins at Code for PDX control the server and deployment pipeline. See Permissions to request access.
- Where does it run? A single server ("droplet") hosted on Digital Ocean, a cloud provider. It serves both the website UI and the AI-powered backend.
- How often does it deploy? Every time a change is merged into the
mainbranch, the site updates automatically within a few minutes. - Is there a staging environment? Yes — a separate server mirrors production and is used for testing changes before they reach users. It is triggered manually by a maintainer.
- What AI service powers the chatbot? Google's Gemini 2.5 Pro model via Google Cloud (Vertex AI), not the server itself.
| Environment | URL | Deployment trigger | Purpose |
|---|---|---|---|
| Production | tenantfirstaid.com | Automatic on push to main |
Live site for end users |
| Staging | Internal URL (see GitHub environment settings) | Manual (workflow_dispatch) |
Pre-production validation |
| Local | http://localhost:5173 |
Manual (mise run dev) |
Developer iteration and testing |
Additionally, the agent can be run locally or deployed to LangSmith Cloud for evaluation and interactive testing:
| Environment | URL | Deployment trigger | Purpose |
|---|---|---|---|
| LangGraph dev | http://localhost:2024 |
Manual (langgraph dev in backend/) |
Local Studio testing with full agent (tools, RAG). No LangSmith account required |
| LangSmith Cloud | LangSmith dashboard | Git push (auto-deploy) | Browser-based evaluation and Studio for Plus-tier seat holders |
The langgraph.json manifest in backend/ configures both. See backend/evaluate/EVALUATION.md for setup instructions.
Both production and staging have independent sets of secrets and variables managed in GitHub Actions environment settings. This means staging can be pointed at a different server or datastore without affecting production.
For local development setup, see the Quick Start in README.md. The local environment reads credentials from backend/.env (git-ignored) and connects to the same Google Cloud services as production by default, using developer-scoped GCP credentials.
Notable staging-only setting: SHOW_MODEL_THINKING can be toggled on to surface the model's internal reasoning steps for debugging purposes.
- Reproducibility: Python dependencies are pinned in
backend/uv.lock; Node dependencies are pinned infrontend/package-lock.json. The same commit always produces the same build. - No secrets in code: secrets are never committed to the repository. They are stored encrypted in GitHub Actions environments and written to the server at deploy time (see Secrets and configuration).
- Continuous delivery: every merge to
mainautomatically triggers a production deploy via GitHub Actions with no manual steps. - Single concurrency: the
deploy-to-dropletconcurrency group ensures only one deploy runs at a time; a newer push cancels an in-progress deploy. - Supply-chain security: all third-party GitHub Actions are pinned to commit SHAs rather than floating version tags. See CLAUDE.md for the pinning policy.
- Configuration as code: server configuration files (Nginx, systemd) are version-controlled in
config/. See Server configuration for the caveat on syncing them. - External artifact immutability: production external artifacts must not be modified in-place while in use. Follow a copy → modify → update-deployment pattern: create a new artifact version, validate it in staging, then update the relevant environment variable and redeploy. See External artifact lifecycle for the full policy by artifact type.
"External artifacts" are resources that live outside the application code and are referenced by environment variable at runtime — most importantly the Vertex AI RAG corpus, but also cloud storage objects and any future database.
| Artifact type | Current example | Becomes immutable when… |
|---|---|---|
| Vertex AI RAG corpus (data store) | VERTEX_AI_DATASTORE_LAWS |
Deployed to any live environment (staging or production) |
| Cloud storage objects (GCS files, etc.) | — (not currently used) | Deployed and referenced by a running service |
| Database (relational / document) | — (not currently used) | Tables / collections referenced by a live deployment |
Once immutable, an artifact must never be mutated in-place. The risk is that an in-flight request, a rollback, or a concurrent staging run would then see an inconsistent or corrupted view of the data.
flowchart TD
A["Create new corpus version<br/>(mise run upload-to-gcs → create-datastore-gcs)"] --> B["Point staging to new corpus<br/>(update VERTEX_AI_DATASTORE_LAWS in staging env)"]
B --> C["Deploy to staging and validate<br/>(smoke-test the chatbot)"]
C --> D{"Validation passed?"}
D -->|No| E["Debug / iterate on corpus<br/>(new corpus is still mutable)"]
E --> C
D -->|Yes| F["Point production to new corpus<br/>(update VERTEX_AI_DATASTORE_LAWS in production env)"]
F --> G["Deploy to production<br/>(corpus is now immutable)"]
G --> H["Archive old corpus ID<br/>(keep for rollback window)"]
H --> I{"Rollback needed?"}
I -->|Yes| J["Revert VERTEX_AI_DATASTORE_LAWS<br/>to previous corpus ID, redeploy"]
I -->|No — stable for ≥1 sprint| K["Safe to delete old corpus"]
Rules:
- A corpus is mutable until it is first deployed to any live environment.
- Once deployed, it is immutable — never add, remove, or reindex documents in that corpus.
- To update the knowledge base, always create a new corpus (a fresh datastore) via the RAG ingestion pipeline (
mise run upload-to-gcs→mise run create-datastore-gcs; see corpus-dataset). - The old corpus must be retained for at least one sprint (or until the next successful deployment) to support rollback and bug reproduction.
- A corpus may be deleted only after: (1) it is no longer referenced by any environment, and (2) no open bugs require reproducing behavior against it.
To roll back to a previous corpus:
- In GitHub environment settings, update
VERTEX_AI_DATASTORE_LAWSto reference the previous corpus ID. - Trigger a production deploy (push a revert commit or use
workflow_dispatch). - Verify the site is serving correct answers.
- Post a note in
#tenantfirstaid-generalon Discord describing the rollback and the reason.
Code rollbacks (reverting a bad commit) follow the same deploy-on-merge pattern; no special steps are needed beyond creating and merging a revert PR.
When reproducing a production bug:
- Use the staging environment, pointed at the same corpus ID that was active in production at the time of the bug.
- Never delete a corpus while a bug referencing it is still open.
- If the bug is in the LLM's behavior (not the corpus), use LangSmith traces from the time of the incident.
No database or persistent cloud storage is currently used in production. When introduced, apply the same principle: create a versioned snapshot or migration, validate in staging, promote to production, retain the previous state through the rollback window.
The application runs on a single Digital Ocean Droplet:
| Property | Value |
|---|---|
| Provider | Digital Ocean |
| OS | Ubuntu LTS 24.04 |
| CPUs | 2 |
| RAM | 2 GB |
| Domain registrar / DNS | Porkbun — tenantfirstaid.com |
| TLS certificates | Let's Encrypt via Certbot (auto-renewing) |
graph TB
subgraph "Internet"
Users["🏠 Users"]
Certbot["Let's Encrypt<br/>(TLS certs — auto-renews)"]
Porkbun["Porkbun DNS<br/>(tenantfirstaid.com)"]
end
subgraph "Digital Ocean Droplet · Ubuntu 24.04 LTS"
Nginx["Nginx<br/>Reverse proxy + TLS<br/>Ports 443 / 80"]
Systemd["Systemd<br/>Process manager"]
Gunicorn["Gunicorn WSGI<br/>10 workers · Unix socket"]
Flask["Flask Application"]
Config["/etc/tenantfirstaid/<br/>env + credentials"]
end
subgraph "Google Cloud Platform"
Gemini["Gemini 2.5 Pro<br/>(LLM)"]
VertexRAG["Vertex AI RAG<br/>(document retrieval)"]
end
Users -->|HTTPS| Porkbun
Porkbun -->|"resolves to droplet IP"| Nginx
Certbot -.->|"renews certs"| Nginx
Nginx -->|"static files from disk"| Users
Nginx -->|"/api/ → Unix socket"| Gunicorn
Systemd -->|"manages"| Gunicorn
Gunicorn --> Flask
Flask -->|"service account auth"| Gemini
Gemini --> VertexRAG
Config -.->|"env vars at startup"| Gunicorn
Nginx (config: deployment/config/tenantfirstaid.conf) does two things:
- Serves static files: the built React frontend (
frontend/dist/) is served directly from disk, with a fallback toindex.htmlfor client-side routing. - Proxies API requests: requests to
/api/are forwarded to Gunicorn via a Unix domain socket (no TCP overhead).
All HTTP traffic is redirected to HTTPS. TLS is managed by Certbot.
The Flask backend runs under Gunicorn with 10 worker processes and a 300-second timeout (config: deployment/config/tenantfirstaid-backend.service). Systemd restarts the process on failure and ensures it starts on server reboot.
flowchart TD
Trigger["Push to main (production)<br/>or manual trigger (staging)"] --> Checkout["Checkout source code"]
Checkout --> NodeSetup["Set up Node 20<br/>(cached)"]
NodeSetup --> BuildUI["Build React frontend<br/>npm ci && npm run build"]
BuildUI --> SCPBackend["Upload backend/ to droplet via SCP<br/>(replaces existing directory)"]
SCPBackend --> SCPFrontend["Upload frontend/dist to droplet via SCP<br/>(appends — does not replace backend)"]
SCPFrontend --> UVInstall["SSH: install uv on droplet<br/>(skip if already present)"]
UVInstall --> UVSync["SSH: uv sync<br/>(install / update Python deps from lockfile)"]
UVSync --> WriteEnv["SSH: write /etc/tenantfirstaid/env<br/>(inject GitHub secrets as env vars)"]
WriteEnv --> WriteCreds["SSH: write /etc/tenantfirstaid/google-service-account.json<br/>(GCP service account)"]
WriteCreds --> Restart["SSH: systemctl restart tenantfirstaid-backend<br/>+ systemctl reload nginx"]
Workflow files:
- Production:
.github/workflows/deploy.production.yml— triggers on every push tomain. - Staging:
.github/workflows/deploy.staging.yml— triggered manually viaworkflow_dispatchin the GitHub Actions UI.
The two workflows are structurally identical; the only difference is which GitHub Actions environment (production vs staging) they read credentials from.
Nginx config and systemd service are not auto-deployed. The files in
config/are version-controlled here for reference, but the CI pipeline does not copy them to the server. Changes to Nginx or systemd configuration require a manual step by a server admin (see Manual server configuration changes).
flowchart TD
GH["GitHub Actions<br/>Secrets + Variables<br/>(production / staging environments)"]
EnvFile["/etc/tenantfirstaid/env<br/>(chmod 640, root:root)"]
CredsFile["/etc/tenantfirstaid/<br/>google-service-account.json<br/>(chmod 640, root:root)"]
Gunicorn["Gunicorn / Flask<br/>(runtime)"]
GH -->|"deploy workflow writes at each deploy"| EnvFile
GH -->|"deploy workflow writes at each deploy"| CredsFile
EnvFile -->|"EnvironmentFile= in systemd unit"| Gunicorn
CredsFile -->|"GOOGLE_APPLICATION_CREDENTIALS path"| Gunicorn
Secrets exist in two places:
- GitHub Actions environments — managed in repository settings → Environments. Requires maintainer access to view or edit. Each deploy rewrites the on-server files from these values.
- On the server —
/etc/tenantfirstaid/holds the env file and GCP credentials JSON. The directory is restricted to root (750). The files within are readable by root and the group (640). Between deploys these files persist on disk; a new deploy always overwrites them.
⚠️ Potentially obsolete items are flagged — they appear in the deploy workflow env file but have no correspondingos.getenv()call in the application source. They may be vestigial from a previous design; a maintainer should confirm before removing.
| Secret | Purpose | Referenced in |
|---|---|---|
SSH_KEY |
Private key for SSH / SCP access to the droplet | deploy.production.yml, deploy.staging.yml |
FLASK_SECRET_KEY |
deploy.production.yml (env file only) | |
GOOGLE_SERVICE_ACCOUNT_CREDENTIALS |
GCP service account JSON; grants access to Vertex AI (Gemini + RAG). Also used in PR checks for non-fork PRs | deploy.production.yml, deploy.staging.yml, pr-check.yml |
DB_HOST |
deploy.production.yml (env file only) | |
DB_PASSWORD |
deploy.production.yml (env file only) | |
APP_PASSWORD |
SMTP app-specific password for sending feedback emails | deploy.production.yml, backend/tenantfirstaid/app.py |
SSH_USER |
SSH username on the droplet (staging environment only — stored as a secret there) | deploy.staging.yml |
LANGSMITH_API_KEY |
LangSmith API key for LLM trace collection (quality-control review of live conversations) | deploy.production.yml, deploy.staging.yml |
| Variable | Purpose | Referenced in |
|---|---|---|
URL |
Droplet hostname or IP address | deploy.production.yml, deploy.staging.yml |
SSH_USER |
SSH username on the droplet (production — stored as a plain variable) | deploy.production.yml |
FRONTEND_DIR |
Path to frontend/ within the repository |
deploy.production.yml, deploy.staging.yml |
BACKEND_DIR |
Path to backend/ within the repository |
deploy.production.yml, deploy.staging.yml |
REMOTE_APP_DIR |
Deployment root on the server (e.g. /var/www/tenantfirstaid) |
deploy.production.yml, deploy.staging.yml |
SERVICE_NAME |
Systemd service name (e.g. tenantfirstaid-backend) |
deploy.production.yml, deploy.staging.yml |
ENV |
Runtime environment label (prod / staging) |
deploy.production.yml, backend/tenantfirstaid/app.py |
DB_PORT |
deploy.production.yml (env file only) | |
MAIL_PORT |
SMTP port | deploy.production.yml, backend/tenantfirstaid/app.py |
MAIL_SERVER |
SMTP server hostname | deploy.production.yml, backend/tenantfirstaid/app.py |
SENDER_EMAIL |
Sender address for feedback emails | deploy.production.yml, backend/tenantfirstaid/app.py, backend/tenantfirstaid/feedback.py |
RECIPIENT_EMAIL |
Recipient address for feedback emails | deploy.production.yml, backend/tenantfirstaid/feedback.py |
MODEL_NAME |
Gemini model identifier (e.g. gemini-2.5-pro) |
deploy.production.yml, backend/tenantfirstaid/google_models.py |
GOOGLE_CLOUD_PROJECT |
GCP project ID | deploy.production.yml, backend/tenantfirstaid/google_auth.py, pr-check.yml |
GOOGLE_CLOUD_LOCATION |
GCP region (e.g. global) |
deploy.production.yml, backend/tenantfirstaid/google_auth.py |
VERTEX_AI_DATASTORE_LAWS |
Vertex AI Search datastore ID for the Oregon laws corpus. Can be a bare datastore ID or a full resource URI. Additional datastores follow the same VERTEX_AI_DATASTORE_<NAME> pattern and are picked up automatically. |
deploy.production.yml, backend/tenantfirstaid/datastores.py |
SHOW_MODEL_THINKING |
Toggle Gemini reasoning display (staging only; hardcoded false in production) |
deploy.staging.yml, backend/tenantfirstaid/google_models.py |
LANGSMITH_PROJECT |
LangSmith project name that groups traces in the UI (e.g. tenantfirstaid-prod, tenantfirstaid-staging) |
deploy.production.yml, deploy.staging.yml |
Copy backend/.env.example to backend/.env and fill in the required values. See README.md for step-by-step instructions. The .env file is git-ignored and never committed. The variables in .env.example mirror the production configuration but use developer-specific credentials.
The deployment/config/ directory contains reference copies of the two server configuration files:
| File | Deployed to | Managed by |
|---|---|---|
deployment/config/tenantfirstaid.conf |
/etc/nginx/sites-available/tenantfirstaid |
Manual — server admin |
deployment/config/tenantfirstaid-backend.service |
/etc/systemd/system/tenantfirstaid-backend.service |
Manual — server admin |
These files are not auto-deployed. The CI pipeline only deploys application code. If you change a config file in this repository, a server admin must manually copy it to the server and reload the relevant service.
For Nginx config changes:
sudo cp /path/to/tenantfirstaid.conf /etc/nginx/sites-available/tenantfirstaid
sudo nginx -t # Validate config before reloading.
sudo systemctl reload nginxFor systemd service changes:
sudo cp /path/to/tenantfirstaid-backend.service /etc/systemd/system/
sudo systemctl daemon-reload
sudo systemctl restart tenantfirstaid-backendServer admins access the droplet via the Digital Ocean web console — no local SSH setup required:
- Log in to the Digital Ocean control panel.
- Navigate to Droplets and select the
tenantfirstaiddroplet. - Click Console (top right of the droplet detail page) to open a browser-based terminal session.
If you need server access for the first time, see Permissions to request a Digital Ocean team invite.
Before running commands on the server, use available signals to identify the affected layer:
-
Check the GitHub Actions deploy log first. Open the Actions tab and look at the most recent deploy run. A failed step there means the issue is in the deploy pipeline, not the running application.
-
Check DataDog (if you have access). The production backend ships structured logs with trace injection enabled (
DD_LOGS_INJECTION=true). Filter byservice:tenant-first-aidandenv:prod. Look for:- HTTP 5xx errors → likely Gunicorn / Flask issue
- Timeout patterns → check Gunicorn worker count or GCP API latency
- Crash loops (repeated startup logs) → missing or malformed env vars
-
Check the Nginx access log for HTTP status codes if the DataDog agent is not forwarding logs (see View logs below). A 502 means Gunicorn is not running; a 504 means it is running but timing out.
-
Check application logs via journalctl (requires droplet console access) to see Python tracebacks and startup errors.
Once you have a hypothesis, use the commands in the sections below to confirm and fix it.
sudo systemctl status tenantfirstaid-backend# Live-tail application logs.
sudo journalctl -u tenantfirstaid-backend -f
# Last 200 lines.
sudo journalctl -u tenantfirstaid-backend -n 200
# Nginx access and error logs.
sudo tail -f /var/log/nginx/access.log
sudo tail -f /var/log/nginx/error.logsudo systemctl restart tenantfirstaid-backend
sudo systemctl reload nginxsudo cat /etc/tenantfirstaid/envOpen the GitHub Actions tab and select the failing workflow run. Each step's output is visible in the run log.
Each runbook follows Detect → Diagnose → Resolve → Notify steps.
Discord: post all incident notifications and resolution confirmations in
#tenantfirstaid-generalon the Code for PDX Discord.
Symptoms: users see a 502 error; Nginx access log shows 502 responses on /api/ routes.
Detect:
- Check tenantfirstaid.com — if the frontend loads but the chatbot fails, the backend is down.
- Check the most recent GitHub Actions deploy run for errors.
Diagnose:
sudo systemctl status tenantfirstaid-backend
sudo journalctl -u tenantfirstaid-backend -n 50Look for: crash on startup (missing env var), OOM kill, or permission error on the socket.
Resolve:
# Restart the backend service.
sudo systemctl restart tenantfirstaid-backend
sudo systemctl status tenantfirstaid-backend # Confirm it is active (running).If it won't start, check /etc/tenantfirstaid/env for missing or malformed variables. Trigger a fresh deploy to rewrite the env file if needed.
Notify: Post in #tenantfirstaid-general:
⚠️ Incident: backend down (502). Restarted tenantfirstaid-backend at [time]. Monitoring for stability. Root cause: [brief description].
Symptoms: chatbot returns an error message or hangs; journalctl shows google.api_core.exceptions tracebacks.
Common error classes:
| Error | Likely cause | Action |
|---|---|---|
ResourceExhausted / 429 |
Quota exceeded | Wait for quota reset (check GCP console quotas); request quota increase if recurring |
PermissionDenied / 403 |
Service account lacks Vertex AI role, or credentials file is stale | Re-deploy (rewrites credentials file); verify service account roles in GCP IAM |
Unauthenticated / 401 |
Service account key expired or malformed | Re-deploy; if still failing, rotate the GOOGLE_SERVICE_ACCOUNT_CREDENTIALS secret and re-deploy |
ServiceUnavailable / 503 |
GCP regional outage | Check Google Cloud Status; no action until outage clears |
InvalidArgument / 400 |
Model name changed or unsupported | Check MODEL_NAME variable in GitHub environment settings against the Gemini model list |
Diagnose:
sudo journalctl -u tenantfirstaid-backend -n 100 | grep -i "exception\|error\|google"
sudo cat /etc/tenantfirstaid/env | grep -E "MODEL_NAME|GOOGLE_CLOUD|VERTEX"Resolve: apply the action from the table above. For quota issues, consider temporarily disabling the chatbot route in Nginx while waiting.
Notify: Post in #tenantfirstaid-general:
⚠️ Incident: Gemini API errors ([error class]). Status: [investigating / resolved]. Impact: chatbot unavailable / degraded. ETA: [if known].
Symptoms: chatbot responds without legal citations, or journalctl shows VertexAI / datastore errors.
Diagnose:
sudo journalctl -u tenantfirstaid-backend -n 100 | grep -i "retriev\|datastore\|rag\|vertex"
sudo cat /etc/tenantfirstaid/env | grep VERTEX_AI_DATASTOREVerify the corpus ID in the env file matches an existing corpus in GCP Vertex AI Search.
Common causes:
| Symptom | Cause | Fix |
|---|---|---|
NOT_FOUND on datastore ID |
Corpus deleted or wrong ID | Update VERTEX_AI_DATASTORE_LAWS in GitHub environment settings and redeploy |
| Empty retrieval results | Corpus not indexed / wrong metadata filters | Re-ingest documents via the RAG ingestion pipeline (mise run upload-to-gcs → mise run create-datastore-gcs); follow External artifact lifecycle |
| Auth errors on RAG calls | Same as Gemini auth issues | See Gemini runbook above |
Resolve: update VERTEX_AI_DATASTORE_LAWS to a valid corpus ID and trigger a redeploy.
Notify: Post in #tenantfirstaid-general:
⚠️ Incident: RAG retrieval errors — chatbot may respond without citations. Status: [investigating / resolved]. Corpus ID in use: [value from env].
Symptoms: browsers show a certificate error; curl -I https://tenantfirstaid.com returns an SSL handshake error.
Diagnose:
sudo certbot certificates # Show expiry dates.
sudo systemctl status certbot.timer # Check the auto-renewal timer is active.
sudo journalctl -u certbot -n 50 # Look for renewal errors.Resolve:
sudo certbot renew --dry-run # Test renewal without committing.
sudo certbot renew # Renew all certificates.
sudo systemctl reload nginx # Reload Nginx to pick up new certs.If the timer is disabled: sudo systemctl enable --now certbot.timer.
Notify: Post in #tenantfirstaid-general:
⚠️ Incident: TLS certificate expired / near expiry. Renewed at [time]. Next expiry: [date from certbot certificates].
Symptoms: GitHub Actions deploy run shows a red ✗; the site may be on an older version or partially broken.
Detect: check the Actions tab for the failing run and identify the failing step.
Common failure points:
| Failing step | Likely cause | Fix |
|---|---|---|
| Build UI | npm dependency or build error | Fix in code; re-merge or trigger workflow_dispatch |
| Upload via SCP | Droplet unreachable or SSH key invalid | Check droplet status in Digital Ocean console; verify SSH_KEY secret is current |
| uv sync | Lockfile conflict or network error | Re-run the workflow; if persistent, check uv.lock in the repo |
| Write env file | Malformed secret value (newline in secret) | Edit the offending secret in GitHub environment settings |
| systemctl restart | Service unit file missing or broken | Manually apply deployment/config/tenantfirstaid-backend.service to the server |
Resolve: fix the root cause and re-trigger the deploy (push a fix commit, or use workflow_dispatch on the staging workflow).
Notify: Post in #tenantfirstaid-general:
⚠️ Incident: deploy failed at step "[step name]" — production may be on previous version. Fix in progress. PR/commit: [link].
| Symptom | Likely cause | Fix |
|---|---|---|
| 502 Bad Gateway | Gunicorn not running | See Runbook: Site down / 502 |
| App starts then crashes | Missing or invalid env var | Check /etc/tenantfirstaid/env; look for the specific error in journalctl |
| GCP API errors | Bad or expired service account credentials | Re-deploy (the workflow rewrites the credentials file) |
| TLS certificate expiry | Certbot renewal failed | See Runbook: TLS certificate expiry |
| Deploy stuck / hanging | Previous deploy still running | Cancel it in the GitHub Actions UI; the concurrency group will allow the next one to proceed |
| Chatbot returns wrong answers | Stale or incorrect RAG corpus | Check VERTEX_AI_DATASTORE_LAWS in the env file; see External artifact lifecycle |
| Role | Access |
|---|---|
| Server admin | Digital Ocean team membership (web console access to the droplet); can restart services, view logs, edit config files, and manage GitHub Actions environment secrets |
| GitHub maintainer | Can trigger manual (staging) deploys and read the deploy workflow output; cannot view GitHub Actions secrets |
| Contributor | No direct server access; contributes via pull requests which deploy automatically on merge |
Access is managed through Code for PDX:
- Complete the Code for PDX onboarding and join the Discord server.
- Post in
#tenantfirstaid-generalon Discord describing the access you need and why. - An existing admin will review the request and grant the appropriate level of access.
For Google Cloud (Vertex AI) access needed for local development, see README.md.
The production Gunicorn service is instrumented with DataDog for log collection and correlation. The following variables are set in the systemd service unit (deployment/config/tenantfirstaid-backend.service):
| Variable | Value | Purpose |
|---|---|---|
DD_SERVICE |
tenant-first-aid |
Service identifier in DataDog |
DD_ENV |
prod |
Environment tag |
DD_LOGS_INJECTION |
true |
Injects trace IDs into log lines |
DD_LOGS_ENABLED |
true |
Enables log forwarding to DataDog |
The DataDog agent and its API key are configured directly on the server by a server admin and are not stored in this repository or the CI pipeline.
LangSmith traces every LLM call made by the live chatbot for ongoing quality-control review of user conversations. Tracing is enabled in production and staging as well as local development; the on/off switch is the presence of a LANGSMITH_API_KEY env var plus the LANGSMITH_TRACING=true / LANGCHAIN_TRACING_V2=true flags (see backend/.env.example).
Traces are grouped in the LangSmith UI by the LANGSMITH_PROJECT variable — typically tenantfirstaid-prod vs tenantfirstaid-staging — so production and staging traffic are viewable separately.
Privacy note: trace content includes user-submitted chat messages, which may contain sensitive housing/legal details. Access to the LangSmith project is therefore limited to maintainers with a documented quality-control need. See Permissions for how to request access.
For running offline evaluations against a dataset, see backend/evaluate/EVALUATION.md.
See issues tagged observability for planned monitoring improvements.
- Architecture.md — code organization and system design
- README.md — local development setup
deployment/config/— server configuration files (Nginx, systemd).github/workflows/deploy.production.yml— production CI/CD workflow.github/workflows/deploy.staging.yml— staging CI/CD workflowbackend/evaluate/EVALUATION.md— LLM evaluation with LangSmith