feat(sandbox): conversation memory poisoning sandbox and exploit (#22) - #56
Open
opitaru-sys wants to merge 2 commits into
Open
Conversation
Closes GenAI-Security-Project#22. Add sandboxes/llm_memory_local, a persistent-memory variant of llm_local that demonstrates Conversation Memory Poisoning / Context Injection (GenAI Red Teaming Manual 4.2.1.3), plus exploitation/conversation_memory_poisoning, which plants a durable instruction in one session and shows a separate, fresh session being steered by it. Sandbox: - SQLite-backed, scope-keyed memory (ollama, mirascope, gradio, sqlite per the issue). - Recall injects stored memory as a leading system message; the "remember that ..." write path persists user text across sessions, so a fact planted in one session is injected into every later session that shares the scope. - Parameterized SQL throughout, input sanitization, a carriage-return smuggling guard, and bounded entry length. - Deterministic, offline unit tests for the store (make unit). - GET/DELETE /memory endpoints for model-independent persistence verification. - Threat model note with defensive guidance. Exploit: - Two-phase attack: poison via one session, confirm persistence through /memory (model-independent), then probe from a fresh session. - Fictional, non-resolving .test payloads only; nothing leaves the sandbox. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The client mirrors sandboxes/llm_local: gradio_app.py uses the mirascope v0 class interface (mirascope.v0) and the gradio 5.x ChatInterface(theme=...) signature. An unpinned resolution floats mirascope to 2.x (which drops mirascope.v0) and gradio to 6.x (which drops the theme kwarg), breaking the Gradio UI and the container build. Cap both to their current major lines so the lock is reproducible and the UI works. Verified end-to-end with the server + Gradio + Ollama (llama3.1:8b): the poisoning reproduces (cross-session persistence, behavioural steering, and scope isolation all confirmed). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Relates to #22.
Adds a persistent-memory sandbox and a paired exploit that demonstrate Conversation Memory Poisoning / Context Injection (GenAI Red Teaming Manual
4.2.1.3).sandboxes/llm_memory_local— a persistent-memory variant ofllm_local. It keeps the same FastAPI mock + Ollama backend and adds a SQLite memory layer around the chat endpoint, using the stack suggested in the issue (ollama,mirascope,gradio,sqlite).exploitation/conversation_memory_poisoning— an end-to-end attack against that sandbox.The vulnerability
Memory is keyed by a scope string, not by an individual conversation, and the default scope (
global) is shared by every session. The chat endpoint:systemmessage labelled as trusted context.remember that ...,from now on ...) and persists what follows.So a single message ending in
remember that ...plants an instruction that is silently injected into the prompt of every later session in the same scope, including a victim's otherwise-clean session. This maps to OWASP LLM01 (Prompt Injection), persisted / cross-session variant.What the exploit does
attack.pyruns two phases against a shared scope:remember that ...message.GET /memoryto confirm the payload was stored (model-independent proof the write path is exploitable).Payloads use fictional, non-resolving
.testdomains only; nothing leaves the sandbox.Reproducing
The sandbox README also includes a
curlwalkthrough of the poison → persist → probe sequence.Notes for reviewers
make unit).threat_model/MEMORY_TM_report.md) documents the trust boundaries and the mitigations a production memory design should add.sandboxes/llm_memory_local).sandboxes/llm_local; formatted withblackandisort.Happy to split this into two PRs (sandbox first, exploit second) if that reviews more easily.