Nova Retrieve — Enterprise Agentic RAG

A production-ready Agentic RAG framework built on LangChain + LangGraph + Qdrant + BGE-M3.

中文版：README.zh.md

Features

• LangGraph state machine — query rewriting → routing → retrieval → document grading → generation → hallucination check → answer-usefulness check, every edge has a fallback.
• CRAG / Self-RAG inspired — auto-rewrites the query when retrieval is thin, auto-regenerates when the answer is hallucinated, and falls back to web search after exhausting retries.
• Local embeddings (BGE-M3) — loads the model from a local directory via sentence-transformers; your data never leaves your network. Just set EMBEDDING_LOCAL_PATH.
• Qdrant vector store — single-command Docker deployment, collection is auto-created on first run.
• OpenAI-compatible LLM — plug in DeepSeek / Qwen / Zhipu / any compatible endpoint.
• Tavily web fallback — switches to live web search when the local index can't answer.
• FastAPI + SSE streaming — per-node event stream that lets the frontend render the agent's reasoning trace live.
• Built-in Web UI — zero-build single-page frontend served at /ui/, with live agent step / timing / citation rendering.
• MCP server — also exposes the stack as MCP tools (rag_search / rag_answer / rag_collections) over stdio or streamable-http, so agents like Hermes can query your knowledge base natively.

Architecture

                ┌──────────────┐
                │ rewrite_query│
                └──────┬───────┘
                       ▼
                ┌──────────────┐
                │route_question│
                └──┬────────┬──┘
       vectorstore│        │web_search
                  ▼        ▼
              ┌──────┐  ┌─────────┐
              │retrieve  │web_search│
              └──┬───┘  └────┬────┘
                 ▼           │
        ┌──────────────┐    │
        │grade_documents│    │
        └──┬───────┬────┘    │
   relevant│  none │         │
           ▼       ▼         │
       ┌────────┐ transform  │
       │generate│◄──query──┐ │
       └───┬────┘          │ │
           ▼          retry│ │
   ┌────────────────┐      │ │
   │hallucination_  │──no──┘ │
   │grader (CRAG)   │        │
   └───┬────────────┘        │
       ▼ yes                 │
   ┌────────────┐            │
   │answer_grader│──no→transform_query
   └───┬────────┘
       ▼ useful
      END

Quick start

1. Start Qdrant

docker compose up -d qdrant

2. Install dependencies

python -m venv .venv && source .venv/bin/activate
pip install -e .

Point EMBEDDING_LOCAL_PATH in .env at your local BGE-M3 directory (it should contain config.json, tokenizer.json, model.safetensors, etc.). Leave it empty to pull the model from HuggingFace on first run.

3. Configure

cp .env.example .env
# edit LLM_BASE_URL / LLM_API_KEY / TAVILY_API_KEY

4. Ingest documents

python -m scripts.ingest_docs ./data/docs

5. Run the server

uvicorn app.main:app --host 0.0.0.0 --port 8000

Open http://localhost:8000/ in a browser — you'll be redirected to the Web UI at /ui/.

Or use the interactive CLI:

python -m scripts.chat_cli

API

Endpoint	Method	Description
/health	GET	Health check
/ingest	POST	Ingest files or directories
/chat	POST	Blocking endpoint — returns the full answer with citations
/chat/stream	POST	SSE stream — step events per node, final answer event with the result

Examples

curl -X POST http://localhost:8000/chat \
  -H "Content-Type: application/json" \
  -d '{"question":"What is our refund policy?"}'

SSE stream:

curl -N -X POST http://localhost:8000/chat/stream \
  -H "Content-Type: application/json" \
  -d '{"question":"Has GPT-5 been released?"}'

MCP server (Hermes / external agents)

Besides the HTTP API, the RAG stack is exposed as an MCP server (app/mcp_server.py) so an MCP-capable agent — e.g. Hermes — can call it as a native tool. It's a thin adapter: it reuses the same retriever and LangGraph pipeline, no logic is duplicated.

Tools

Tool	Cost	Description
rag_search	cheap, no LLM	Semantic search; returns raw source chunks with provenance + similarity scores for the agent to reason over and cite itself. Fully offline (local BGE-M3 + Qdrant).
rag_answer	heavier, own LLM calls	Runs the full agentic-RAG pipeline and returns a synthesized, hallucination-graded answer with a Sources list.
rag_collections	cheap	Lists Qdrant collections (knowledge bases) with point counts.

Transports

RAG_MCP_TRANSPORT selects the transport:

• stdio (default) — the MCP client spawns this process and talks over stdin/stdout. Use for same-host integration.
• http — long-lived networked service (streamable-http) at http://<host>:<port>/mcp. Use for server-to-server / offline-box integration.

Under stdio, stdout is the JSON-RPC channel: the server pins all logging to stderr and never calls setup_logging(). Don't print to stdout from tools.

Run

pip install -e .          # installs the `mcp` dep + the `nova-mcp` console script

# A) HTTP service — recommended when Hermes runs on another host
RAG_MCP_TRANSPORT=http RAG_MCP_WARMUP=1 nova-mcp
#   → serving streamable-http on http://0.0.0.0:8765/mcp

# B) stdio — same host, client spawns the process
RAG_MCP_TRANSPORT=stdio nova-mcp        # or: python -m app.mcp_server

RAG_MCP_WARMUP=1 eager-loads the embedding model + Qdrant collection at startup so the first request doesn't pay that cost — recommended for the http service.

Wire it into Hermes

HTTP transport (RAG reachable over the network):

# Hermes cli-config.yaml
mcp_servers:
  knowledge_base:
    url: http://<myrag-host-ip>:8765/mcp

stdio transport (RAG on the same host as Hermes):

mcp_servers:
  knowledge_base:
    command: nova-mcp
    args: []
    env:
      RAG_MCP_TRANSPORT: stdio
      EMBEDDING_LOCAL_PATH: /abs/path/to/bge-m3
      QDRANT_URL: http://localhost:6333
      LLM_BASE_URL: http://<llm-endpoint>/v1
      LLM_API_KEY: <key>

Under stdio Hermes spawns the process fresh — it does not inherit your shell or .env. Pass everything the RAG needs (embedding path, Qdrant URL, LLM endpoint) explicitly via env:. Note rag_search needs no LLM; only rag_answer calls LLM_BASE_URL.

Config

Env	Default	Meaning
RAG_MCP_TRANSPORT	stdio	stdio or http
RAG_MCP_HOST	0.0.0.0	http bind host
RAG_MCP_PORT	8765	http bind port
RAG_MCP_MAX_CHARS	1500	per-chunk body cap in rag_search (protects the agent's context window)
RAG_MCP_WARMUP	0	1 = eager-load embedding/collection at startup

Troubleshooting — a weak local model never retrieves

Symptom: the MCP server connects (hermes mcp test <server> lists the tools) but the agent answers from its own knowledge and never calls rag_search / rag_answer.

With a strong model this is rare. With a small local model (e.g. llama-3.1-8B via vLLM) it is almost guaranteed — and the cause is the tool surface, not the model. Diagnose in this order:

1. Can the model emit tool calls at all? curl the LLM endpoint directly with a single fake tool and "tool_choice":"auto"; confirm choices[0].message.tool_calls is populated. vLLM for Llama 3.1 needs --enable-auto-tool-choice --tool-call-parser llama3_json, else it returns HTTP 400 on any request carrying tools.
2. Are the tools actually registered? hermes mcp test <server>, the /tools command, or grep ~/.hermes/logs/agent.log for registered N tool(s).
3. How many tools is the request carrying? The verbose log's Tools: N / prompt_tokens line — 25 tools (~12K tokens of schemas) drowns an 8B.
4. Which tool did it actually call? A wrong tool means a distractor is winning.

The four root causes we hit with llama-3.1-8B, and the fixes (all in ~/.hermes/config.yaml unless noted):

#	Root cause	Fix
1	25 tools (default hermes-cli preset) overload the 8B	platform_toolsets: { cli: [file, mcp-<server>] } — the whitelist must include mcp-<server> or it silently drops the RAG tools
2	The generic clarify tool hijacks the turn	agent: { disabled_toolsets: [clarify] }
3	No standing steering — Hermes drops the MCP server's instructions; only each tool's description reaches the model	Add an agent.personalities entry that names mcp_<server>_rag_search and activate it via /personality. The strengthened tool docstrings in mcp_server.py also help — they are the only per-tool channel Hermes forwards.
4	The 8B hallucinates the optional collection arg → queries a non-existent Qdrant collection → "no results"	Server-side, rag_search / rag_answer take a single required arg (done in this repo). Also drop the MCP prompt/resource meta-tools: mcp_servers.<server>.tools: { prompts: false, resources: false, include: [rag_search, rag_answer, rag_collections] }

After each change: validate the YAML (python3 -c "import yaml; yaml.safe_load(open('/root/.hermes/config.yaml'))"), restart Hermes, re-check with /verbose. Healthy state: Tools: ≈ 7, the first Tool call: is mcp_<server>_rag_search / rag_answer with only a query / question arg, and the RAG log queries the default collection (QDRANT_COLLECTION).

Minimize the tool surface, kill distractor/meta tools, put steering where the host actually forwards it (tool descriptions + personality, not the MCP instructions field), and give tools a minimal signature. Strong models tolerate a messy tool surface; weak local ones do not.

Keeping Hermes' other tools without breaking retrieval

The trim above (cli: [file, mcp-<server>]) is a debugging step — it strips Hermes down to file + RAG to isolate the overload. That is not the goal state if you want to keep Hermes' normal capabilities. Two mechanics matter (model_tools.py):

• platform_toolsets.<platform> is a whitelist of toolset names. Once set, only those toolsets load — MCP tools are an ordinary mcp-<server> toolset and are not auto-attached; omit it and the RAG tools silently vanish.
• agent.disabled_toolsets is subtracted last, even from composite presets like hermes-cli — the surgical way to drop only clarify while keeping everything else.
• Names compose: [hermes-cli, mcp-<server>] = the full preset ∪ the RAG tools.

The catch is a real trade-off for a weak model only: full preset + RAG ≈ 28 tools puts an 8B straight back into overload (it stops calling RAG). This is the 8B's ceiling, not a config bug — a stronger orchestration model removes the trade-off entirely. Under an 8B, pick per priority:

Option	platform_toolsets.cli	Outcome
A — full capability	[hermes-cli, mcp-<server>]	Hermes intact; RAG retrieval unreliable (8B drowns), only partly rescued by the personality
B — pragmatic (recommended)	[file, web, terminal, todo, mcp-<server>]	Keeps the day-to-day tools + RAG; ~12 tools, 8B still retrieves fairly reliably
C — RAG-first	[file, mcp-<server>]	Most reliable RAG; Hermes reduced to file I/O

All three also keep agent.disabled_toolsets: [clarify] and the steering personality. Recommended path: start from B and add back the specific toolsets you actually use (web/terminal/browser/skills/todo/tts/cronjob — see Available toolsets in cli-config.yaml.example), checking Tools: N in /verbose after each — empirically an 8B stays reliable up to ≈12 tools and starts dropping RAG past ≈20.

Project layout

app/
├── config.py           # pydantic-settings config
├── main.py             # FastAPI app + /ui static mount
├── mcp_server.py       # MCP adapter (stdio / streamable-http) for Hermes
├── api/                # routes and schemas
├── core/               # llm / embeddings / vectorstore / logging
├── ingest/             # loader / chunker / pipeline
├── retrieval/          # retriever
└── agent/              # state / nodes / edges / prompts / graph / tools
web/                    # single-page frontend (no build step)
├── index.html
├── styles.css
└── app.js
scripts/
├── ingest_docs.py
└── chat_cli.py

Extension points

• Reranker — drop a BGE-Reranker into retrieval/ for a two-stage rerank.
• Multi-tenancy — ChatRequest.collection is already wired for per-tenant collection isolation.
• Caching — adding Redis caching on route_question / grade_documents cuts cost significantly.
• Observability — set LANGSMITH_API_KEY for end-to-end tracing.