knowledge-llm

Objective

Ask grounded questions over a UK environmental regulation corpus.

The current scope is environmental permitting, waste, water discharge, habitats, and related regulator guidance.

how

import legal documents scripts/uk_env_law_importer.py

documents/chunks SQLite ↓ FTS5 lexical retrieval ↓ embedding retrieval ↓ legal concept extraction ↓ rules table ↓ question planner

Today the implemented parts are:

• corpus import into SQLite documents/chunks
• FTS5 lexical retrieval
• embedding retrieval
• retrieval comparison in the Flask UI
• corpus browse/search pages

The rule extraction and planner stages are not built yet.

Structure

.
|-- app/
|   |-- __init__.py
|   |-- db.py
|   |-- routes.py
|   |-- services/
|   |   |-- embeddings.py
|   |   `-- retrieval.py
|   |-- static/
|   |   `-- css/app.css
|   `-- templates/
|       |-- base.html
|       |-- corpus.html
|       |-- document_detail.html
|       `-- index.html
|-- data/
|   |-- raw/
|   `-- uk_env_law.db
|-- instance/
|   `-- .gitkeep
|-- scripts/
|   |-- build_chunk_embeddings.py
|   |-- build_search_suggestions.py
|   |-- dev.sh
|   |-- sample_job.py
|   `-- uk_env_law_importer.py
|-- tests/
|   `-- test_app.py
|-- .env.example
|-- .gitignore
|-- config.py
|-- requirements.txt
`-- run.py

Local setup

source /Users/navinkumar/work/vindaloo/.venv/bin/activate
python -m pip install -r requirements.txt
cp .env.example .env
python run.py

The Flask app currently exposes:

• / for grounded question answering over the corpus
• /corpus for browsing and FTS5-searching imported source documents
• /corpus/<doc_id> for viewing extracted text and chunked passages

By default the app is configured to use Ollama on banana with gemma4:e4b.

Processes

The repo currently has a few separate background or maintenance processes.

1. Import the corpus

python scripts/uk_env_law_importer.py

This fetches source material, writes raw files under data/raw/, and rebuilds the documents and chunks tables in data/uk_env_law.db.

2. Build chunk embeddings

python scripts/build_chunk_embeddings.py

This fills chunk_embeddings for the configured embedding model so the embedding and hybrid retrieval modes work.

3. Build search suggestions

python scripts/build_search_suggestions.py

This rebuilds a derived search_suggestions table from titles, headings, section refs, authorities, and topic tags. It is intended for future autocomplete/typeahead support and is kept separate from the source corpus tables on purpose.

Search And Indexing

• chunk_search is an FTS5 virtual table derived from documents + chunks
• it is rebuilt automatically by the app when needed
• chunk_embeddings is derived data and is populated by scripts/build_chunk_embeddings.py
• search_suggestions is derived data and is populated by scripts/build_search_suggestions.py

To compare embedding-based retrieval, build chunk embeddings first:

python scripts/build_chunk_embeddings.py

basics

ruff check
ruff check --fix