Skip to content

feat(sandbox): conversation memory poisoning sandbox and exploit (#22) - #56

Open
opitaru-sys wants to merge 2 commits into
GenAI-Security-Project:mainfrom
opitaru-sys:feat/conversation-memory-poisoning-sandbox
Open

feat(sandbox): conversation memory poisoning sandbox and exploit (#22)#56
opitaru-sys wants to merge 2 commits into
GenAI-Security-Project:mainfrom
opitaru-sys:feat/conversation-memory-poisoning-sandbox

Conversation

@opitaru-sys

@opitaru-sys opitaru-sys commented Aug 24, 2026

Copy link
Copy Markdown

Summary

Relates to #22.

Adds a persistent-memory sandbox and a paired exploit that demonstrate Conversation Memory Poisoning / Context Injection (GenAI Red Teaming Manual 4.2.1.3).

  • sandboxes/llm_memory_local — a persistent-memory variant of llm_local. It keeps the same FastAPI mock + Ollama backend and adds a SQLite memory layer around the chat endpoint, using the stack suggested in the issue (ollama, mirascope, gradio, sqlite).
  • exploitation/conversation_memory_poisoning — an end-to-end attack against that sandbox.

The vulnerability

Memory is keyed by a scope string, not by an individual conversation, and the default scope (global) is shared by every session. The chat endpoint:

  1. Recall (sink): before calling the model, loads every fact stored for the scope and injects it as a leading system message labelled as trusted context.
  2. Write (source): after replying, scans the latest user message for directive phrases (remember that ..., from now on ...) and persists what follows.

So a single message ending in remember that ... plants an instruction that is silently injected into the prompt of every later session in the same scope, including a victim's otherwise-clean session. This maps to OWASP LLM01 (Prompt Injection), persisted / cross-session variant.

What the exploit does

attack.py runs two phases against a shared scope:

  1. Poison — one session plants a malicious download link via a remember that ... message.
  2. Persistence check — reads GET /memory to confirm the payload was stored (model-independent proof the write path is exploitable).
  3. Probe — a brand-new session asks an innocent question and is steered by the injected memory.

Payloads use fictional, non-resolving .test domains only; nothing leaves the sandbox.

Reproducing

# Offline, no container or model needed: proves the poisoning logic deterministically
cd sandboxes/llm_memory_local && make unit

# Full stack (requires Ollama + Podman)
cd exploitation/conversation_memory_poisoning && make all

The sandbox README also includes a curl walkthrough of the poison → persist → probe sequence.

Notes for reviewers

  • SQL is fully parameterized; input is sanitized and length-bounded; a carriage-return guard prevents directive smuggling.
  • Deterministic, model-free unit tests cover the write path, scope isolation, the injection preamble, and the input guards (make unit).
  • A threat-model note (threat_model/MEMORY_TM_report.md) documents the trust boundaries and the mitigations a production memory design should add.
  • Directory name follows the name suggested in [Backlog]: Conversation Memory Poisoning #22 (sandboxes/llm_memory_local).
  • Structure, Makefile targets, and container setup mirror sandboxes/llm_local; formatted with black and isort.

Happy to split this into two PRs (sandbox first, exploit second) if that reviews more easily.

Closes GenAI-Security-Project#22.

Add sandboxes/llm_memory_local, a persistent-memory variant of llm_local that
demonstrates Conversation Memory Poisoning / Context Injection (GenAI Red Teaming
Manual 4.2.1.3), plus exploitation/conversation_memory_poisoning, which plants a
durable instruction in one session and shows a separate, fresh session being steered
by it.

Sandbox:
- SQLite-backed, scope-keyed memory (ollama, mirascope, gradio, sqlite per the issue).
- Recall injects stored memory as a leading system message; the "remember that ..."
  write path persists user text across sessions, so a fact planted in one session is
  injected into every later session that shares the scope.
- Parameterized SQL throughout, input sanitization, a carriage-return smuggling guard,
  and bounded entry length.
- Deterministic, offline unit tests for the store (make unit).
- GET/DELETE /memory endpoints for model-independent persistence verification.
- Threat model note with defensive guidance.

Exploit:
- Two-phase attack: poison via one session, confirm persistence through /memory
  (model-independent), then probe from a fresh session.
- Fictional, non-resolving .test payloads only; nothing leaves the sandbox.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The client mirrors sandboxes/llm_local: gradio_app.py uses the mirascope v0 class
interface (mirascope.v0) and the gradio 5.x ChatInterface(theme=...) signature. An
unpinned resolution floats mirascope to 2.x (which drops mirascope.v0) and gradio to
6.x (which drops the theme kwarg), breaking the Gradio UI and the container build. Cap
both to their current major lines so the lock is reproducible and the UI works.

Verified end-to-end with the server + Gradio + Ollama (llama3.1:8b): the poisoning
reproduces (cross-session persistence, behavioural steering, and scope isolation all
confirmed).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant