A Semantic Operating Layer for Local Knowledge
LLMWiki is not a Retrieval-Augmented Generation framework. It is a semantic operating layer that continuously transforms heterogeneous digital artifacts into an evidence-aware knowledge space.
Repository scope: this repository is the architecture/specification, evaluation protocol, LaTeX source, and small reference integrations. It is not the complete integrated desktop application. The runnable integrated implementation is maintained separately in Folding-JimiJimi.
- Developer Handoff Note — What changed in the latest build, verified build state, rebuild instructions
- Project Vision — Mission, goals, non-goals, success metrics
- System Philosophy — 8 foundational architectural principles
- Architecture Overview — Three-layer decomposition, data flow, module contracts
- Evidence Wire Contract — WC/1 token-lean, lossless evidence transport
- Evidence Packing Research — Budgeted selection, limits, acceptance experiment
- Discord-Native LLMWiki — Event journal, Gateway/REST sync, Markdown projection, commands
- Discord Integration Quickstart — Runnable reference implementation
- Roadmap — Milestones, timeline, resource estimates
- Design Decisions — 15 recorded decisions with rationale
- Technical Specification — 12 Volumes + Appendices (LaTeX source)
- Contributing — Development setup, standards, workflow
- Changelog — Release history
| Aspect | Traditional RAG | LLMWiki |
|---|---|---|
| Primary Store | Vector DB | Property Graph (RocksDB) |
| Retrieval | ANN only | Hybrid: HNSW + BM25 + Graph Adjacency |
| Verification | Reranker (optional) | Mandatory Evidence Gate (cross-encoder + calibration) |
| Deployment | Cloud API | Local binary / Air-gapped / Container |
| Updates | Re-embed all | Incremental (Tree-sitter diff) |
| Attribution | Post-hoc | Built-in (content-addressed units) |
| Lineage | None | Full (WAL replay) |
- Event Log is Durable Truth — Graph/vector/keyword indices are rebuildable projections of the append-only log, not independent sources of truth
- Graph is the Canonical Query-Time Representation — among the projections, the property graph is what the planner queries first
- SHA-256 is Content Identity, Not Location Identity — content identity and occurrence (file+span) identity are separate; see Appendix B
- Evidence Gate is Mandatory (Four Stages) — Relevance, Structural/Source-Validity, Claim Support, and Evidence Coverage; empty set = "insufficient evidence"
- Local-First, LLM-Optional — Critical path has zero external calls
- DAG Planner — Parallel, explainable, budgeted
- Incremental Consistency — Per-projection, per-sequence: once a projection's last_applied_sequence reaches n, it matches log replay 1..n
- Deterministic Replay — Same input → same output; crash recovery via a sequence-numbered, idempotent projection scanner
- Mutation Requires Approval — No write to a source artifact bypasses the approval pipeline (Volume 12) and produces a rollback record
- Evidence Text Survives Context Shaping — Budget selection and WC/1 may remove envelope overhead, never rewrite cited evidence
- Conversation Sources are Replayable — Discord edits/deletes append events; Markdown, search, and summaries remain rebuildable projections
The complete technical specification is organized as 12 volumes:
| Volume | Title | Focus |
|---|---|---|
| 01 | Architecture | Three-layer design, invariants, module contracts, crash consistency |
| 02 | Semantic Parsing | Tree-sitter, SCM, chunking, embedding, extraction |
| 03 | Knowledge Graph | Property graph, RocksDB, entity resolution, hybrid index |
| 04 | Retrieval | Hybrid search, Evidence Gate, budgeted context selection, WC/1 transport, verified synthesis |
| 05 | Agent Runtime | DAG planner, executor, deterministic context assembly, SSE streaming, budget allocation |
| 06 | Evaluation | Evidence quality, context efficiency, quality-at-budget A/B, CI gates |
| 07 | Deployment | Local, container, K8s, air-gapped, monitoring |
| 08 | API Reference | REST/gRPC/WS, auth, rate limits, SDKs |
| 09 | Developer Guide | Extension points (parsers, chunkers, embedders, tools, auth) |
| 10 | Patent | Draft claims (unfiled; pending counsel review and prior-art search) |
| 11 | Research Paper | Architecture and evaluation protocol; benchmark results are preliminary targets pending public release of LLMWikiBench |
| 12 | Mutation and Approval Engine | Document write-back, format-aware patching, human approval, rollback, resync |
Appendices: Mathematical Notation, Data Models (content identity vs. occurrence identity), Config Schema, Grammar Specs, Evaluation Protocol, Related Work, Design Decisions
make pdfRequires: TeX Live / MacTeX with latexmk, lualatex (via fontspec), bibtex
Windows/MiKTeX note: if
bibtexreports "I couldn't open database file" despitebibliography/LLMWiki.bibexisting, setBIBINPUTSto a Windows-style path before building:export BIBINPUTS="<repo-root-absolute-path>;". This is akpsewhichpath-resolution quirk, not a missing file.
This repository was iteratively developed with AI assistance. Treat the LaTeX
specification and benchmark tables as design material unless they point to a
present, reproducible script and test result; the executable eval harness under
eval/scripts/ is the authority for the currently implemented reference behavior.
Apache License 2.0 (see LICENSE)
LLMWiki: Transforming heterogeneous digital artifacts into an evidence-aware knowledge space.