Graph-based retrieval for RAG, built on spreading activation — like a spider web.
Instead of cutting retrieval off at top-k, the query is injected into a
vector graph as an energy seed and spreads outward with decay. Strongly
related nodes light up first; weakly related but genuinely connected nodes
light up later, through multiple hops. The answer is built from the whole web,
not from one cluster of near-duplicates.
Status: Phase 1 — measured on three benchmarks, and the gate is not passed yet. The web beats both baselines on MuSiQue and 2WikiMultihopQA and loses to iterative retrieval on HotpotQA. The numbers are below, including the one that does not flatter the project. The design has been public from day one so the idea can be judged — and challenged — early.
Python 3.11+ on Windows, macOS or Linux. One install, one word to start.
Windows (PowerShell)
pip install spiyweb # or: uv tool install spiyweb
cd C:\path\to\your-project
spiywebmacOS / Linux
pip3 install spiyweb # or: uv tool install spiyweb (pipx works too)
cd ~/path/to/your-project
spiywebspiyweb takes the window over: a welcome box with the spider, a
transcript, a prompt, a status bar. Everything happens from there - type a
question as a plain sentence, or a / command. The input box says what to
type first until you have typed anything. Only closing the terminal ends it.
A first session, without leaving the screen:
/demo three recorded questions spread on a sample corpus
/doctor what is installed, what is missing, and the fix
/install index the index-time extras and the spaCy model, in the background
/index pick a folder of .txt/.md, pick the options, build
who built the tower? a plain sentence is a question for the active index
/show 2 the second passage in full: text, source, how the energy got there
/compare who built it? the same question as plain top-k, and what only the web found
/index runs in the background with its stage and elapsed time in the
status line (and the terminal bell when a long build ends), so the screen
never freezes; /kill stops it. The index it builds becomes the active one
- the status bar names it,
/indexesswitches between the ones nearby, and it is remembered next time./tune damping 0.8 seed 6changes the spread for the session and asks the last question again;/configholds the lasting knobs (profile, warm follow-ups, ring map, speed, glyphs, colour, mouse)./savewrites the query on screen to a Markdown file./helplists every command,?the shortcuts, tab completes commands and folders, and up/down remembers earlier sessions.
Your own application can be watched too: start it in a second terminal -
or from the prompt with ! python app.py - and every query it makes
through spiyweb.open_index(...) plays on screen as it happens.
pip install spiyweb on its own pulls in nothing - the core is pure
Python. Building an index needs the index-time extras once (/install index
in the monitor, or pip install "spiyweb[index]" plus
python -m spacy download xx_ent_wiki_sm).
If the console cannot draw braille or box lines (an old conhost, a plain
TERM=dumb), SPIYWEB_ASCII=1 spiyweb uses the plain glyph set; a pipe or
a CI job gets the usage text and exit 2 instead of a prompt.
Classic RAG retrieval has two structural weaknesses:
- A hard cutoff has no notion of indirect relevance. When an answer is
distributed across several documents, no single one of them may be among
the most similar chunks — so
top-knever sees it. - Repetition is treated as content. Ten near-identical chunks fill ten context slots and tell the model one thing, badly.
Formally: spreading activation over a sparse multi-layer graph, mathematically equivalent to Personalized PageRank with a damping factor. It reduces to repeated sparse matrix–vector products — cheap and numerically stable.
HOP 0 Q = 10.0 -> first contact (cosine): A = 5.60 C = 4.40
HOP 1 A -> B 2.24, D 1.12 C -> D 1.76, E 0.88 (dies)
HOP 2 D = 1.12 + 1.76 = 2.88 <- converging evidence
HOP 3 energy below threshold, the web stops on its own
RESULT A 5.60 | C 4.40 | D 2.88 | B 2.24 | F 1.73
D is never the single most similar node to the query, yet it ranks third —
because two independent weak paths converged on it. That promotion is the
entire value proposition.
What the monitor shows: the ranking bars growing in hop by hop on the
left, the ring map on the right, the ledger line under both, the numbered
top passages under that - and, only when the run found them, the things a
ranking hides: contradictions kept on both sides, claims the corpus
disputes, themes with no bridge between them. No socket, no server, no code
in your application: the monitor leaves a marker in .spiyweb/, the
library appends its records there while the marker is fresh, the monitor
tails them. /find locates the file that imports spiyweb and the index it
opens; /lint reads the corpus's shape; /replay plays a recorded query
again; examples/hello_web.py is a three-question app to watch first.
The same verbs work without the screen, for scripts and pipes:
pip install "spiyweb[index]"
python -m spacy download xx_ent_wiki_sm
spiyweb index docs/ my-index # a directory of .txt/.md -> an index
spiyweb query my-index "what happened afterwards"
spiyweb lint my-index # what is wrong with the CORPUSspiyweb index docs/ my-index --propositions adds the second node layer:
a local LLM (Ollama, llama3.1:8b by default) splits every passage into
short, self-contained facts, and those join the graph beside the passages.
It costs one LLM call per passage, and whether it pays depends on the
corpus. On MuSiQue it was the difference that mattered: the web over a
passage-only graph beat plain top-k by +.005 (inside the noise), and
over the same corpus with propositions by +.047, CI [+.029, +.067]. On
a real documentation corpus (below) it was not: the web beat top-k
without propositions, and adding them changed nothing measurable while the
build took 33 minutes instead of 2. Try the passage-only index first. An
interrupted run resumes where it stopped; --llm-model,
--llm-url and --llm-key-env point it at another model or any
OpenAI-compatible API, the last one naming the environment variable that
holds the key rather than the key.
lint is the diagnostic that needs no query: it reads the graph's shape and
reports islands nothing bridges, hubs that grind arriving energy into dust,
near-identical passages that will compete for one seed slot, and the sources
that contradict the rest of the corpus.
spiyweb version says which extras are installed and prints the pip install
line for the ones that are not — it works on a bare install, which is exactly
when you need it to.
The core itself is pure Python with no dependencies at all. Similarities come
from the caller; core/ never computes them.
git clone https://github.com/Yigtwxx/spiyweb && cd spiyweb
uv sync --group dev && uv run pytestfrom spiyweb import Graph, propagate
graph = Graph.from_edges(
[
("A", "A_dup", 0.0), # near duplicate, edge suppressed by dedup
("A", "B", 0.8),
("A", "D", 0.4),
("C", "D", 0.6),
("C", "E", 0.3),
("D", "F", 0.5),
]
)
result = propagate(graph, seeds={"A": 0.9, "C": 0.7})
[(node, round(energy, 3)) for node, energy in result.ranked()]
# [('A', 5.625), ('C', 4.375), ('D', 2.875), ('B', 2.25), ('F', 1.725)]
result.activations["D"].contributors # ('A', 'C') — converging evidence
result.stop_reason # 'threshold' — the web stopped itselfE is missing because 0.875 of energy reached it against a floor of 1.5, and
A_dup is missing because its edge was suppressed and its share redistributed.
Neither outcome came from a result-count parameter.
pip install "spiyweb[index]"import spiyweb
from spiyweb.indexing import DocumentInput, TextUnit, build_index
from spiyweb.indexing import SentenceTransformerEmbedder, load_spacy_pipeline
docs = [
DocumentInput(source_id="handbook", units=(TextUnit(text=part) for part in parts))
for parts in my_documents
]
build_index(
docs,
"data/mydocs",
embedder=SentenceTransformerEmbedder(),
entity_pipeline=load_spacy_pipeline(),
)
with spiyweb.open_index("data/mydocs") as index:
answer = index.retrieve("who signed off on the change?", profile="precise")
for passage in answer.passages:
print(f"{passage.energy:5.2f} {passage.votes} votes {passage.text[:70]}")
print(answer.profile) # which profile ran; "explore" when you named none
print(answer.confidence) # total energy, node count, hop depth
print(answer.dedup_mode) # which duplicate rules actually ran
for path in answer.paths(): # how the energy reached each node
print(path)There is no k. The web stops when its energy falls below the threshold -
that self-termination is the argument against top-k, so a k= parameter
here would quietly reintroduce the thing being argued against. Slice
answer.passages if you want fewer.
profile is optional. Name none and pass no config, and the library runs
explore (spiyweb.DEFAULT_PROFILE) - the bare RetrievalConfig() carries
the canonical worked example above and cannot spread past five seeds, so it
is what you get only when you build it yourself, and retrieve() warns if
what you built cannot leave hop 0.
open_index wires duplicate suppression correctly, which is not a detail:
the mechanism needs a config AND a similarity backend, and this project's own
measurement campaign ran with it silently off for want of the second half.
answer.dedup_mode is the receipt.
When the corpus changes, sync instead of rebuilding. Pass the whole current
corpus; only new or edited chunks are embedded and extracted again, removed
ones are dropped, and the result is artifact-for-artifact the index a fresh
build_index would write:
from spiyweb.indexing import sync_index
report = sync_index(
docs,
"data/mydocs",
embedder=SentenceTransformerEmbedder(),
entity_pipeline=load_spacy_pipeline(),
)
print(report.reused, report.added, report.changed, report.removed)spiyweb index <docs> <out> does the same when <out> already holds an
index; --force rebuilds from scratch. A different embedding model is
refused rather than mixed into the old vectors.
pip install "spiyweb[langchain]" # or "spiyweb[llamaindex]"from spiyweb.integrations.langchain import SpiywebRetriever
retriever = SpiywebRetriever(index=spiyweb.open_index("data/mydocs"))
docs = retriever.invoke("who signed off on the change?")
# docs[0].metadata -> node_id, source_id, energy, votes, hop, ...spiyweb.integrations.llamaindex.SpiywebLlamaRetriever returns
NodeWithScores scored by energy. Both keep the web's own order and have no
k by default (max_documents / max_nodes cap the list after the
ranking); import spiyweb never imports either framework.
import spiyweb is the query-time contract: everything in spiyweb.__all__
and nothing else. spiyweb.indexing is the index-time contract, on the same
terms; it imports with nothing installed, and only the FAISS-bound names ask
for pip install "spiyweb[store]". Anything reached through a submodule path
(spiyweb.core.*, spiyweb.evaluation.*) is internal and may change without
notice. Both surfaces are snapshot-tested, and changes to them are written
down in CHANGELOG.md.
Spreading activation over graphs is not new (HippoRAG, GraphRAG, LightRAG, RAPTOR). Spiyweb's claimed differentiators are elsewhere:
- Redundancy becomes a vote, not noise. When a near-duplicate is found during propagation, its edge is severed and its energy share redistributed — and the surviving idea's vote count goes up. Repetition turns into corpus-support evidence instead of burning context slots. To our knowledge this dynamic dedup-to-vote conversion has no published equivalent.
- Honesty outputs. Every retrieval returns a confidence score (total energy, node count, hop depth), corpus-gap warnings (two dense clusters with no bridge), contradiction records with a ready-made, LLM-free question for the user, and activation paths as explanations the LLM can cite. The retriever can say "I don't know, and here is why." One honest limit: the contradiction mechanism works on the edges it is given, but the bundled detector that finds them is weak - it caught 31.6% of WikiContradict's 253 annotated pairs, 9.5% end to end. Treat contradiction records as a bonus, not a guarantee.
- Coloured multi-seed bridging. A decomposed query injects differently coloured seeds; a node where two colours meet is a bridge — exactly where a multi-hop answer lives.
- The web stops itself. Termination is a relative energy threshold, not a "return N results" parameter.
Every query an open index answers is recorded as a self-contained trace: the activated subgraph, the passages' text, the energy ledger and the settings the call ran with. Nothing else is needed to read it back - no index, no store, no dependency.
Traces are held in memory (the last 200) and cost no disk unless asked:
index = spiyweb.open_index("my-index", trace=spiyweb.TraceConfig(directory="traces"))That writes JSONL, and a machine that holds no index can read it back:
from spiyweb import load_traces
for record in load_traces("traces/traces.jsonl"):
print(record.query, record.profile, record.ledger)The layout of a recorded call - hop rings, layers, a side-by-side against
plain top-k - lives in spiyweb.scene (numpy only, spiyweb[view]), and
the hop-ring rule itself in spiyweb.rings (nothing at all), so one query
produces one picture no matter what draws it. The terminal monitor is what
draws it now; the browser face that used to was removed after 0.1.2.
| Item | Decision |
|---|---|
| Benchmark | MuSiQue (multi-hop) |
| Gate | Beat both baselines — plain top-k and iterative retrieval — by a meaningful margin |
| Reference | HippoRAG results reported alongside |
| Metrics | 65% multi-hop accuracy + 35% Novelty@k, plus bridge-node recall |
| Embedding | multilingual-e5-large |
| Store | numpy + FAISS, single file |
| LLM (index-time only) | Local-first (Ollama); free APIs optional |
| Environment | Python 3.11 + uv · macOS / Windows / Linux |
Phase 1 also succeeds if it fails clearly: a reliable negative number is a better outcome than a polished library built on an unmeasured assumption.
S@5 = 0.65 · support recall + 0.35 · Novelty@5. 1000 questions per run, paired bootstrap intervals, winning configuration applied unchanged to every dataset after the first.
| dataset | SPIYWEB | iterative (IRCoT-style) | plain top-k | verdict |
|---|---|---|---|---|
| MuSiQue (tuning, seed 42) | .5094 | .4631 | .3090 | passes, +.046 CI [+.030, +.062] |
| MuSiQue (confirmation, seed 123) | .5073 | .4420 | .3046 | passes, +.065 CI [+.048, +.082] |
| 2WikiMultihopQA | .7130 | .687 | .468 | passes, +.026 CI [+.016, +.037] |
| HotpotQA | .6228 | .6428 | .5623 | fails: −.020 CI [−.032, −.009] |
RESULTS.md is the complete record — one table with every
number Phase 1 produced: the gate, the metric decomposition, the per-hop
breakdown, all five failed rescue rounds, every mechanism ablation, the
contradiction-detection measurements, and the seven limits that go with them.
The gate asks for both baselines, so HotpotQA is a failure, not a footnote. The diagnosis is that the advantage is depth-dependent: 74.5% of HotpotQA questions already have all their gold in the dense top-5, where Novelty@5 is 0 by construction and spreading can only displace.
Five pre-registered attempts to close that gap were all recorded as negative, and Phase 1 closed on that. Four worked in the ranking layer (a confidence gate, blending, novelty-free slots); the fifth moved inside propagation, letting the decomposition's colour count pick the query profile. It won +.0025 CI [+.0005,+.0050] on the tuning set, was not confirmed on the seed-123 set, and changed HotpotQA by exactly zero — not one question's window moved. One earlier attempt helped HotpotQA and hurt the deeper sets, which is exactly the trade the gate is meant to refuse.
So the honest headline is a result with a condition attached: the advantage over both baselines is real on the deeper sets and does not transfer to a benchmark that is entirely 2-hop.
One fixed reader (llama3.1:8b, temperature 0, the top 5 passages of each
system, at most 12 words) answered all 4000 questions. Token F1, paired
intervals:
| dataset | top-k | iterative | SPIYWEB | vs top-k | vs iterative |
|---|---|---|---|---|---|
| MuSiQue (seed 42) | .186 | .270 | .294 | +.108 [+.082, +.133] | +.024 [−.001, +.048] |
| MuSiQue (seed 123) | .200 | .292 | .293 | +.094 [+.070, +.118] | +.001 [−.024, +.026] |
| 2WikiMultihopQA | .437 | .525 | .561 | +.123 [+.095, +.152] | +.035 [+.008, +.062] |
| HotpotQA | .620 | .689 | .666 | +.046 [+.021, +.069] | −.023 [−.045, −.002] |
The retrieval advantage reaches the answer: far ahead of top-k on every
set, ahead of or level with the iterative baseline on the deeper sets, and
behind it on HotpotQA - the same shape as the retrieval table.
The 15 file-and-archive pages of the official Python 3.11 library docs
(os.path, pathlib, shutil, zipfile, json, subprocess ...), 3005
blocks indexed exactly as spiyweb index reads them, and 30 questions
written before the index existed - 15 answered by one page, 15 that need
two ("how do I convert rows of a CSV file into a JSON array?"). Recall@5
of the gold pages:
| top-k | SPIYWEB | |
|---|---|---|
| all 30 | .867 | .983 (+.117 [+.050, +.200]) |
| one-page questions | 1.000 | 1.000 |
| two-page questions | .733 | .967 |
Build: 2 minutes, 13 MB, ~85 ms per query on a laptop CPU. With
--propositions: 33 minutes, 2655 LLM calls, 36 MB, and no measurable
gain (web −.017, inside the noise). n = 30 is small - this is a case
study, not a benchmark.
The coloured web decomposes the question with an LLM and extracts intermediate answers; propagation itself is milliseconds, the calls are not. Fifty MuSiQue questions, every call real, laptop RTX 4070:
| p50 | p95 | LLM calls | |
|---|---|---|---|
| top-k | 0.08 s | 0.09 s | 0 |
| iterative | 3.8 s | 5.4 s | 4.0 |
| SPIYWEB, default models | 9.1 s | 9.7 s | 2.2 |
| SPIYWEB, one model for both roles | 1.3 s | 1.9 s | 3.2 |
The default splits the work between two models - qwen3.5:9b decomposes,
llama3.1:8b extracts intermediate answers - and on an 8 GB card they do
not fit together, so the server reloads one on every query. Giving
decomposition to llama3.1:8b as well removes the reload: the web drops to
1.3 s, three times faster than the iterative baseline. Its retrieval
quality is not measured - every quality number on this page uses the
default pair, and the one-model run splits questions into more parts (3.2
calls instead of 2.2). The library never calls an LLM itself - you
decompose and pass the parts to retrieve_colored() - so the lesson for
your own pipeline is: on an 8 GB card, use one model for every LLM step.
In the harness that is --decomp-model llama3.1:8b.
HippoRAG 2's released question sets match our corpora passage for passage, so recall@5 is directly comparable (their numbers are published, ours are measured; their reader and ours differ, so answers are not compared):
| MuSiQue | 2Wiki | HotpotQA | |
|---|---|---|---|
| HippoRAG 2 (published) | 74.7 | 90.4 | 96.3 |
| SPIYWEB | 67.0 | 95.8 | 91.7 |
Three newer embedders replaced multilingual-e5-large in a pre-registered
screen. All three lifted dense recall@5 on MuSiQue (+.05 to +.12) and all
three lost on HotpotQA (−.03 to −.08), so none passed. The strongest,
multilingual-e5-large-instruct, was then run through the whole pipeline
anyway: the web's recall rose (+.024 [+.008, +.040]), but the iterative
baseline rose more, and overtook it (S@5 −.025 [−.041, −.011]). The
default stays multilingual-e5-large; EmbeddingConfig(model=..., query_prefix=..., passage_prefix=...) takes another one.
CLAUDE.md— condensed engineering ground truth: the settled invariants, the architecture boundaries, and the rules a change must not break. The full design specification and the decision log (every choice with its rationale and its rejected alternatives) are kept privately.
site/ holds the project's landing page: a static Astro site, no UI framework,
deployed to Vercel with site as the project root. The engravings are
public-domain spider plates, halftoned at build time from
site/images.manifest.json; the outputs in site/public/images are committed,
so a build never fetches anything.
cd site
npm install
npm run dev # local preview
npm run build # static output in site/dist
npm run images # re-halftone after editing images.manifest.json
npm run favicon # re-render the favicon set from scripts/favicon.source.svgDesign feedback is welcome right now — especially prior art for the dedup-to-vote mechanism. See CONTRIBUTING.md.
Apache-2.0 © 2026 Yigit Erdogan