MeClear

MeClear: Cooperative Game-Theoretic Attribution and Risk-Aware Memory Clearance for Long-Horizon LLM Agents

Paper · Project Page · Website Source

This repository contains the reproducible experiment pipeline for MeClear. It builds conversation-scoped Mem0 memory pools from LoCoMo, constructs controlled harmful-memory cases, compares attribution methods, verifies deletion-based clearance, and exports the semantic data used by the paper figures.

The repository is self-contained at runtime: source data, memory indexes, provider traces, stage artifacts, baseline outputs, analysis exports, and figures all live below this project directory. LoCoMo data and API credentials are intentionally not committed.

Each LoCoMo conversation is isolated as its own memory universe. The main experiment runs five durable stages:

1. Memory pool: update Mem0 sequentially over turns within a conversation. Different conversations may be built concurrently.
2. Original QA: retrieve the clean top-5 memories and retain questions that are correct in both repeated evaluations.
3. Error injection: generate controlled memories conditioned on the question and clean context. Cases cover precise factual errors, precise temporal errors, and explicit conflicts under direct, redundant-conflict, and joint-interaction structures. A case is retained only when the injected context changes a previously correct answer.
4. Attribution: evaluate LOO-only, sampled-Shapley-only, and full attribution. Sampled Shapley uses checkpoints at 4, 8, and 16 permutations; full attribution combines the LOO screen with sampled Shapley.
5. Delete check: remove each method's estimated harmful set and evaluate the resulting answer twice using the same binary correctness exit.

The primary semantic outputs include Recall@|M|, harmful-set recall and precision, exact harmful-set recovery (H = M), and binary recovery after clearance. Here M is the injected harmful set and H is the method's estimated harmful set. For redundant conflicts, each injected memory remains a distinct member of M.

Two independent baseline runners consume a completed main run without modifying it:

• Classical baselines: ContextCite and ProxySPEX, using 32 coalition-mask evaluations per case.
• LLM baseline: few-shot direct harmful-memory attribution using the configured diagnosis role (qwen3.5-plus in the released configuration).

MeClear/
|-- configs/                  # Frozen experiment and provider configuration
|-- data/                     # LoCoMo download target; dataset is gitignored
|-- prompts/                  # Memory, answer, judge, fault, and LLM-baseline prompts
|-- resources/                # Pinned local BM25 tokenizer resource
|-- scripts/                  # Dataset acquisition utilities
|-- website/                  # Astro project page and GitHub Pages source
|-- src/meclear/
|   |-- runtime/              # Provider, LoCoMo, and Mem0 integration
|   |-- core/                 # Shared records, ports, QA, and BM25 functions
|   |-- domain/               # Experiment records and stage contracts
|   |-- functions/            # Attribution, injection, QA, metrics, concurrency
|   |-- pipeline/             # Durable main experiment orchestration
|   |-- baselines/            # Classical and LLM baseline packages
|   |-- analysis/             # Read-only semantic figure-data interface
|   `-- visual/               # Five publication-figure renderers
|-- artifacts/                # Published stage payloads, exports, and figures
|-- runs/                     # Run graphs, attempts, audit records, native indexes
`-- tests/                    # Unit and pipeline tests

Run all commands in this README from the repository root. The frozen development environment uses Python 3.13. Create it with Conda:

conda env create -f environment.yml
conda activate meclear
python -m pip install "https://github.com/explosion/spacy-models/releases/download/en_core_web_sm-3.8.0/en_core_web_sm-3.8.0-py3-none-any.whl"

Alternatively, use an existing Python 3.13 environment:

python -m pip install -e ".[test]"
python -m pip install "https://github.com/explosion/spacy-models/releases/download/en_core_web_sm-3.8.0/en_core_web_sm-3.8.0-py3-none-any.whl"

The major pinned dependencies are Mem0 2.0.12, spaCy 3.8.14, FastEmbed 0.8.0, Qdrant Client 1.18.0, OpenAI 2.48.0, shapiq 1.6.0, XGBoost 3.3.0, Matplotlib 3.11.1, and en_core_web_sm 3.8.0. Exact Python package versions are in pyproject.toml. The spaCy model is pinned separately because it is distributed as a model wheel.

Mem0 uses the spaCy model for memory/query lemmatization and entity extraction. A missing model can leave some calls apparently functional while changing retrieval behavior, so MeClear treats it as a failed protocol precondition.

If a Windows certificate chain blocks only the GitHub model wheel, retry that one command with --trusted-host github.com --trusted-host objects.githubusercontent.com. This is a per-command download exception; do not disable SSL verification globally.

Download the public LoCoMo dataset and verify its frozen SHA-256 digest:

python scripts/download_locomo.py
python scripts/download_locomo.py --check

The expected file is data/locomo10.json. The loader accepts LoCoMo categories 1-4 with reference answers and isolates all questions, turns, memories, and vector indexes by conversation.

Create .env from .env.example and configure:

KIMI_API_KEY=...
ZHIZENGZENG_API_KEY=...
EMBEDDING_BASE_URL=...
EMBEDDING_API_KEY=...
EMBEDDING_MODEL=...

Model roles and access boundaries are frozen in configs/runtime.yaml:

The Answer and Judge use different model families. No role receives LoCoMo gold evidence. Provider endpoints and model names can be adapted for another deployment, but doing so creates a different experimental configuration and therefore a different run hash.

Validate the complete local setup, check the dataset plan, then probe the configured Answer, Judge, and Fault roles:

meclear preflight
meclear plan-locomo
meclear capability

preflight checks Python and protocol-critical package versions, repository resources, required environment-variable names, the pinned spaCy model, LoCoMo hash and shape, offline BM25 loading, and resolved Mem0 source hashes. It makes zero API calls and never prints credential values. capability is the separate provider-backed check.

The release includes a bounded provider-backed smoke profile. It uses the first internal session of conv-30 and three evidence-supported questions, keeps at most nine causal cases, runs sampled Shapley at L=4, and runs the classical baselines at B=8. ProxySPEX records HPO as requested but disables it at this smoke budget because five-fold R2 tuning requires at least ten unique masks; the formal B=32 profile keeps HPO enabled. These reductions validate engineering connectivity; smoke metrics are not paper results.

The experiment-conv47.yaml, baseline-conv47.yaml, and llm-baseline-conv47.yaml files provide a second, independent maintainer-validation profile. They exercise a different memory pool and include a known Redundant-compatible QA set; they remain smoke configurations and are not part of the paper denominator.

meclear --config configs/smoke/experiment.yaml plan-locomo
meclear --config configs/smoke/experiment.yaml capability
meclear --config configs/smoke/experiment.yaml run-locomo \
  --run-id meclear-smoke-r1

meclear-baseline --config configs/smoke/baseline.yaml plan-locomo \
  --source-run-id meclear-smoke-r1 --run-id meclear-classical-smoke-r1
meclear-baseline --config configs/smoke/baseline.yaml run-locomo \
  --source-run-id meclear-smoke-r1 --run-id meclear-classical-smoke-r1 \
  --allow-provider-calls

meclear-llm-baseline --config configs/smoke/llm-baseline.yaml plan-locomo \
  --source-run-id meclear-smoke-r1 --run-id meclear-llm-smoke-r1
meclear-llm-baseline --config configs/smoke/llm-baseline.yaml run-locomo \
  --source-run-id meclear-smoke-r1 --run-id meclear-llm-smoke-r1 \
  --allow-provider-calls

Smoke outputs are isolated below runs/smoke/ and artifacts/smoke/. Analysis must use main checkpoint 4 and baseline checkpoint 8; meclear figure-data infers these from the smoke profile, while meclear-visualize exposes --main-final-checkpoint 4, --baseline-final-checkpoint 8, and --allow-partial. Figure 5 case IDs are run outputs and should be supplied with --figure5-cases.

Export the normalized read-only dataset after all three runs complete:

meclear --config configs/smoke/experiment.yaml figure-data \
  --main-run-id meclear-smoke-r1 \
  --baseline-run-id meclear-classical-smoke-r1 \
  --llm-baseline-run-id meclear-llm-smoke-r1 \
  --baseline-run-root runs/smoke/baselines/classical/dataset/meclear-classical-smoke-r1 \
  --llm-baseline-run-root runs/smoke/baselines/llm/dataset/meclear-llm-smoke-r1 \
  --all-completed --output artifacts/smoke/analysis/locomo-smoke-r1

Choose two accepted case IDs from case_registry.jsonl, then render the five-figure smoke preview:

meclear-visualize --workspace-root . \
  --main-run-root runs/smoke/main/dataset/meclear-smoke-r1 \
  --baseline-run-root runs/smoke/baselines/classical/dataset/meclear-classical-smoke-r1 \
  --llm-baseline-run-root runs/smoke/baselines/llm/dataset/meclear-llm-smoke-r1 \
  --main-final-checkpoint 4 --baseline-final-checkpoint 8 \
  --figure5-cases <case-id-1> <case-id-2> \
  --formats png --output artifacts/smoke/figures/r1 --allow-partial

Use a stable run ID because baselines refer to it explicitly:

meclear run-locomo \
  --run-id meclear-main \
  --pool-build-workers 3 \
  --session-run-workers 1 \
  --max-workers 20

pool-build-workers controls parallel conversation pool construction; turns within each conversation remain serial. session-run-workers controls how many conversation pipelines run concurrently after pools are ready. max-workers is the global provider-task bound and cannot exceed the frozen runtime limit.

Inspect the parent graph or a stage payload:

meclear inspect-locomo --run-id meclear-main
meclear inspect-locomo-session \
  --run-id meclear-main \
  --conversation-id conv-26 \
  --stage attribution

Evaluator-only payloads, which may include reference-bearing records, are hidden by default. Add --include-evaluator-only only for authorized audit work.

Resume a failed or interrupted run:

meclear run-locomo --run-id meclear-main --resume

To invalidate one conversation from a known stage and recompute its downstream stages:

meclear run-locomo \
  --run-id meclear-main \
  --resume \
  --conversation-id conv-43 \
  --restart-from-stage error-injection

Every stage publishes an immutable manifest containing input artifact IDs, configuration and implementation hashes, counts, timing, and payload hashes. Attempts and provider calls remain under the corresponding run directory, so interruption recovery does not require editing published artifacts.

Plan and run the classical baselines after the selected main conversations have completed:

meclear-baseline plan-locomo \
  --source-run-id meclear-main \
  --run-id meclear-classical

meclear-baseline run-locomo \
  --source-run-id meclear-main \
  --run-id meclear-classical \
  --conversation-workers 2 \
  --allow-provider-calls

Run the direct LLM attribution baseline independently:

meclear-llm-baseline plan-locomo \
  --source-run-id meclear-main \
  --run-id meclear-llm

meclear-llm-baseline run-locomo \
  --source-run-id meclear-main \
  --run-id meclear-llm \
  --conversation-workers 2 \
  --allow-provider-calls

Both runners are resumable and write only to runs/baselines/* and artifacts/baselines/*. Use inspect-locomo with the corresponding executable to inspect progress. Classical deletion verification can be selectively recomputed with rerun-delete-locomo.

The meclear.analysis package is read-only with respect to experiment runs. It validates manifests, intersects jointly usable conversations and cases, and converts raw artifacts into typed semantic tables.

Export the full paired dataset used by plotting:

meclear figure-data \
  --main-run-id meclear-main \
  --baseline-run-id meclear-classical \
  --llm-baseline-run-id meclear-llm \
  --all-completed \
  --require-full \
  --output artifacts/analysis/locomo-full

Render all five figures from the completed runs:

meclear-visualize \
  --workspace-root . \
  --main-run-root runs/main/dataset/meclear-main \
  --baseline-run-root runs/baselines/classical/dataset/meclear-classical \
  --llm-baseline-run-root runs/baselines/llm/dataset/meclear-llm \
  --output artifacts/figures/locomo-current

Formal rendering requires the complete paired dataset. --allow-partial exists only for development previews. PDF, SVG, and PNG are produced by default.

The paper website is maintained in website/ and deploys through the repository-level GitHub Pages workflow.

cd website
npm ci
npm run dev

The production build automatically uses /MeClear/ as its GitHub Pages base path.

python -m pytest

Tests use fake providers and temporary run directories. The real LoCoMo loader integration test runs when data/locomo10.json is present; it is skipped otherwise. Provider-backed experiments are never started by the test suite.

API-backed construction, fault generation, QA, coalition evaluation, and deletion verification can be expensive. Run plan-locomo, verify model identifiers and concurrency limits, and use fixed run IDs before starting. Do not edit a frozen configuration during a run: resumption rejects configuration or dataset hash changes.

The repository does not redistribute LoCoMo or model outputs. Consult the upstream LoCoMo and provider licenses before publishing derived artifacts.

The MIT License in the repository root applies only to code authored for MeClear. See LICENSE.

Third-party resources retain their original licenses and are not relicensed under MIT. The vendored Qdrant BM25 English stopword resource is distributed under Apache License 2.0; see THIRD_PARTY_NOTICES.md and licenses/Apache-2.0.txt.