LycheeMem/LycheeMem

LycheeMemory is a compact memory framework for LLM agents. It starts from efficient conversational memory—through structured organization, lightweight consolidation, and adaptive retrieval—and gradually extends toward action-aware, usage-aware memory for more capable agentic systems.

---

---

🔥 News

• [07/07/2026] OpenAI-compatible Chat Completions endpoints are now available, with request-level consolidation control via consolidate or store.
• [05/08/2026] Transformer memory reranker v0 improves evidence selection in semantic memory search, with positive hit@10 gains on LoCoMo and zero-shot LongMemEval-S / MSC-MemFuse / HotpotQA fixtures. See Transformer Reranker v0.
• [04/29/2026] Hermes and Claude Code plugin integrations are now available, bringing LycheeMemory's automatic recall, turn mirroring, and consolidation workflow to more agent runtimes. Setup guides: Hermes · Claude Code
• [04/26/2026] Visual (Multimodal) Memory module added! See Visual Memory.
• [04/13/2026] LycheeMem is now LycheeMemory.
• [04/03/2026] The project now supports installation via pip install lycheemem. You can easily start the service from anywhere using lycheemem-cli!
• [03/30/2026] We evaluated LycheeMemory on PinchBench with the OpenClaw plugin: compared to OpenClaw's native memory, it achieved an ~6% score improvement, while reducing token consumption by ~71% and cost by ~55%!
• [03/28/2026] Semantic memory has been upgraded to Compact Semantic Memory (SQLite + LanceDB), no Neo4j required. See /quick-start for details.
• [03/27/2026] OpenClaw Plugin is now available at /openclaw-plugin ! Setup guide →
• [03/26/2026] MCP support is available at /mcp !
• [03/23/2026] LycheeMemory is now open source: GitHub Repository →

---

🔗 Related Projects

LycheeMemory is part of the 3rd-generation Lychee (立知) large model series, which focuses on memory intelligence, continual learning, and long-context reasoning.

We welcome you to explore our related works:

• LycheeMemory (ACL 2026, CCF-A): a unified framework for implicit long-term memory and explicit working memory collaboration in large language models

• LycheeMem (this project): long-term memory infrastructure for LLM-based agents

• LycheeDecode (ICLR 2026, CCF-A): selective recall from massive KV-cache context memory

• LycheeCluster (ACL 2026, CCF-A): structured organization and hierarchical indexing for context memory

---

⚡ Quick Start

Prerequisites

• Python 3.9+
• An LLM API key (OpenAI, Gemini, or any litellm-compatible provider)

Installation

Install the core package:

pip install lycheemem

Recommended install with the default transformer memory reranker:

pip install "lycheemem[rerank]"

The rerank extra adds PyTorch / Transformers runtime dependencies. With it installed, LycheeMemory enables the hosted LycheeMem/reranker checkpoint by default. Without the extra, the core memory system still works and reranking falls back safely.

Once installed, you can start the backend server instantly using the CLI:

lycheemem-cli

For development or if you prefer to run from source:

git clone https://github.com/LycheeMem/LycheeMem.git
cd LycheeMem
pip install -e .

Configuration

Create a .env file in your working directory and fill in your values. The full template in .env.example also includes session/user DB paths, JWT settings, and working-memory thresholds; the snippet below shows the most important ones:

# LLM — litellm format: provider/model
LLM_MODEL=openai/gpt-4o-mini
LLM_API_KEY=sk-...
LLM_API_BASE=                     # optional
Embedder
EMBEDDING_MODEL=openai/text-embedding-3-small
EMBEDDING_DIM=1536
EMBEDDING_API_KEY=                # optional
EMBEDDING_API_BASE=               # optional

Supported LLM providers (via litellm):

openai/gpt-4o-mini · gemini/gemini-2.0-flash · ollama_chat/qwen2.5 · any OpenAI-compatible endpoint

Transformer Reranker

LycheeMemory includes a transformer reranker for semantic memory search. It can improve evidence selection when the correct memory is already in the wider candidate pool.

For the smoothest experience, install LycheeMemory with the rerank extra:

pip install "lycheemem[rerank]"

After that, no extra model command is required. The reranker is enabled by default and loads the current v0 checkpoint from Hugging Face on first use:

EXPERIMENTAL_TRANSFORMER_RERANK=true
TRANSFORMER_RERANK_MODEL_PATH=LycheeMem/reranker

To disable it explicitly:

EXPERIMENTAL_TRANSFORMER_RERANK=false

If you prefer to pin the model to a local directory, download it once and point the same variable at that path:

mkdir -p ~/.cache/lycheemem/models
huggingface-cli download LycheeMem/reranker \
  --local-dir ~/.cache/lycheemem/models/reranker-v0
export TRANSFORMER_RERANK_MODEL_PATH=~/.cache/lycheemem/models/reranker-v0

The base install still works without PyTorch or Transformers. If rerank dependencies or the checkpoint are unavailable, LycheeMemory logs a warning, disables reranking for that process, and continues with baseline memory search. See Transformer Reranker v0 for metrics, limitations, and diagnostics.

Start the Server

If you installed via pip, you can start the LycheeMemory background service from anywhere using:

lycheemem-cli

(If running from source, you can also use python main.py to start the server.)

The API is served at http://localhost:8000. Interactive docs at /docs.

main.py currently starts Uvicorn without enabling live reload. For development reload, run Uvicorn directly, for example:

>

> uvicorn src.api.server:create_app --factory --reload

---

🎨 Web Demo

A frontend demo is included under web-demo/. It provides a chat interface alongside live views of the semantic memory tree, skill library, and working memory state.

cd web-demo
npm install
npm run dev      # served at http://localhost:5173

Make sure the backend is running on port 8000 (or update proxy settings in web-demo/vite.config.ts) before starting the frontend.

---

🦞 OpenClaw Plugin

LycheeMemory ships a native OpenClaw plugin that gives any OpenClaw session persistent long-term memory with zero manual wiring.

What the plugin provides:

• lychee_memory_smart_search — default long-term memory retrieval entry point
• Automatic turn mirroring via hooks — the model does not need to call append_turn manually
• User messages are appended automatically
• Assistant messages are appended automatically
• /new, /reset, /stop, and session_end automatically trigger boundary consolidation
• Proactive consolidation on strong long-term knowledge signals

Under normal operation:

• The model only calls lychee_memory_smart_search when recalling long-term context
• The model may call lychee_memory_consolidate manually when an immediate persist is warranted
• The model does not need to call lychee_memory_append_turn at all

Quick Install

openclaw plugins install "/path/to/LycheeMem/openclaw-plugin"
openclaw gateway restart

See the full setup guide: openclaw-plugin/INSTALL_OPENCLAW.md

---

🔧 MCP

LycheeMemory also exposes an HTTP MCP endpoint at http://localhost:8000/mcp.

• Available tools: lychee_memory_smart_search, lychee_memory_search, lychee_memory_append_turn, lychee_memory_consolidate
• lychee_memory_consolidate works for sessions that already contain mirrored turns from /chat, /memory/reason, or lychee_memory_append_turn

MCP Transport

• POST /mcp handles JSON-RPC requests
• GET /mcp exposes the SSE stream used by some MCP clients
• The server returns Mcp-Session-Id during initialize; reuse that header on later requests

Client Configuration

For any MCP client that supports remote HTTP servers, configure the MCP URL as:

http://localhost:8000/mcp

Generic config example:

{
  "mcpServers": {
    "lycheemem": {
      "url": "http://localhost:8000/mcp"
    }
  }
}

Manual JSON-RPC Flow

1. Call initialize 2. Reuse the returned Mcp-Session-Id 3. Send initialized 4. Call tools/list 5. Call tools/call

Initialize example:

curl -i -X POST http://localhost:8000/mcp \
  -H "Content-Type: application/json" \
  -d '{
    "jsonrpc": "2.0",
    "id": 1,
    "method": "initialize",
    "params": {
      "protocolVersion": "2025-03-26",
      "capabilities": {},
      "clientInfo": {
        "name": "debug-client",
        "version": "0.1.0"
      }
    }
  }'

Tool call example:

curl -X POST http://localhost:8000/mcp \
  -H "Content-Type: application/json" \
  -H "Mcp-Session-Id: " \
  -d '{
    "jsonrpc": "2.0",
    "id": 2,
    "method": "tools/call",
    "params": {
      "name": "lychee_memory_smart_search",
      "arguments": {
        "query": "what tools do I use for database backups",
        "top_k": 5,
        "mode": "compact",
        "include_graph": true,
        "include_skills": true
      }
    }
  }'

Recommended MCP Usage Pattern

1. Use /chat or /memory/reason with a stable session_id to write conversation turns, or mirror external host turns with lychee_memory_append_turn. 2. Use lychee_memory_smart_search in compact mode for the default one-shot recall path. 3. Use lychee_memory_search only when you explicitly want raw retrieval results for debugging or custom host-side processing. 4. After the conversation ends, call lychee_memory_consolidate with the same session_id.

---

📚 Memory Architecture

LycheeMemory organizes memory into three complementary stores:

Working Memory	Semantic Memory	Procedural Memory	Visual Memory
(Episodic) Session turns Summaries Token budget management	(Typed Action Store) 7 MemoryRecord types Conflict-aware Record Fusion Hierarchical memory tree Action-aware hierarchical retrieval Usage feedback loop + RL-ready statistics	(Skills) Skill entries HyDE retrieval	(Multimodal) VLM-driven image understanding Dual embedding (caption + CLIP visual) Text ↔ image cross-modal retrieval Ebbinghaus forgetting curve Three-layer storage (SQLite + LanceDB + filesystem)

💾 Working Memory

The working memory window holds the active conversation context for a session. It operates under a dual-threshold token budget:

• Warn threshold (70%) — triggers asynchronous background pre-compression; the current request is not blocked.
• Block threshold (90%) — the pipeline pauses and flushes older turns to a compressed summary before proceeding.

Compression produces summary anchors (past context, distilled) + raw recent turns (last N turns, verbatim). Both are passed downstream as the conversation history.

🗺️ Semantic Memory

Semantic memory is organised around typed MemoryRecords plus action-grounded retrieval state. The storage layer is SQLite (FTS5 full-text search) + LanceDB (vector index), while retrieval is conditioned on recent context, tentative action, constraints, and missing slots.

Memory Record Types

Each memory entry is stored as a MemoryRecord. The memory_type field distinguishes seven semantic categories:

| Type | Description | |------|-------------| | fact | Objective facts about the user, environment, or world | | preference | User preferences (style, habits, likes/dislikes) | | event | Specific events that have occurred | | constraint | Conditions that must be respected | | procedure | Reusable step-by-step procedures / methods | | failure_pattern | Previously failed action paths and their causes | | tool_affordance | Capabilities and applicable scenarios of tools/APIs |

Beyond text, every MemoryRecord carries action-facing metadata (tool_tags, constraint_tags, failure_tags, affordance_tags) and usage statistics (retrieval_count, action_success_count, etc.) to seed future reinforcement-learning signals. Retrieval logs also persist retrieval_plan, action_state, response excerpts, and later user feedback so the system can close a lightweight action-outcome loop without training.

Related MemoryRecords can be fused online by the Record Fusion Engine into denser CompositeRecords. Composite entries persist direct child_composite_ids, so long-term semantic memory is organised as a hierarchical memory tree instead of a flat bag of summaries.

Four-Module Pipeline

Module 1: Compact Semantic Encoding

A single-pass pipeline that converts conversation turns into a list of MemoryRecords:

1. Typed extraction — LLM extracts self-contained facts and assigns a semantic category to each record. 2. Decontextualization — Pronouns and context-dependent phrases are expanded into full expressions, so each record is understandable without the original dialogue. 3. Action metadata annotation — LLM annotates each record with memory_type, tool_tags, constraint_tags, failure_tags, affordance_tags, and other structured labels.

record_id = SHA256(normalized_text) — naturally idempotent; duplicate content is deduplicated automatically.

Module 2: Record Fusion, Conflict Update, and Hierarchical Consolidation

Triggered online after each consolidation. No LLM calls — pure embedding cosine similarity math:

1. Deduplication — For each new record, ANN search finds existing records of the same memory_type with cosine similarity > 0.85. Near-duplicates are soft-expired; composites covering affected source records are invalidated. 2. Clustering — ANN search builds a similarity graph (cosine > 0.75) over surviving records. Union-Find finds connected components; each component containing at least one new record becomes a candidate cluster. 3. Composite construction — The representative record (highest confidence / most recent) provides semantic_text; entities, tags, and temporal fields are merged from all cluster members. A new CompositeRecord is written to SQLite + LanceDB. 4. Hierarchy rounds — The same clustering pass runs over CompositeRecords, producing composite → composite abstractions and persisting child_composite_ids so the memory tree can keep growing upward.

Module 3: Action-Aware Hierarchical Retrieval

Retrieval is organised around the hierarchical memory tree, using CompositeRecords as the primary retrieval unit. The current query, recent context, and ActionState jointly condition holistic relevance judgement at the composite level; matched composites are expanded down the memory tree to atomic MemoryRecords on demand; and a reflection loop driven by adequacy assessment covers any residual information gaps.

Composite-Level Relevance Judgement

Retrieval first operates at the CompositeRecord level. An ANN vector search pre-filters to the top-20 semantically nearest CompositeRecords, then a single LLM call performs holistic relevance judgement over those candidates: each composite is either selected as relevant or excluded; among those selected, the LLM additionally flags entries whose summary is too abstract to fully answer the query and therefore warrant expansion to their underlying atomic records. The ANN pre-filter keeps the LLM judgement bounded to one call regardless of how many CompositeRecords exist in the database.

Memory Tree Expansion

For composites flagged as requiring expansion, the retrieval engine recursively traverses source_record_ids and child_composite_ids down the memory tree to retrieve the corresponding atomic MemoryRecords. This preserves the broad semantic overview provided by high-level composites while enabling precise access to fine-grained evidence when the query demands it, balancing retrieval efficiency with detail coverage.

Reflection-Based Supplementary Recall

After the initial candidate set is formed, the engine assesses the adequacy of the current context. When a coverage gap is detected, multi-channel supplementary recall is activated: FTS full-text and vector channels (both semantic_text and normalized_text paths) extend coverage at the MemoryRecord level, and a direct vector recall over the episode turns index recovers dialogue content not yet distilled into MemoryRecords. The reflection loop runs for a bounded number of rounds, continuing only while information gaps remain.

Module 4: Candidate Aggregation and Context Enrichment

After all phases complete, candidates are aggregated and ranked by source tier for top-k selection: composites selected by the composite-level relevance judgement receive the highest priority, followed by atomic MemoryRecords from tree expansion, with supplementary recall results ranked last. All candidates are then enriched with episodic context — original dialogue excerpts from the session store are retrieved and appended to each candidate's display text, providing the downstream SynthesizerAgent with fully sourced, contextualised background.

🛠️ Procedural Memory — Skill Store

The skill store preserves reusable how-to knowledge as structured skill entries, each carrying:

• Intent — a short description of what the skill does.
• doc_markdown — a full Markdown document describing the procedure, commands, parameters, and caveats.
• Embedding — a dense vector of the intent text, used for similarity search.
• Metadata — usage counters, last-used timestamp, preconditions.

Skill retrieval uses HyDE (Hypothetical Document Embeddings): the query is first expanded into a hypothetical ideal answer by the LLM, then that draft text is embedded to produce a query vector that matches well against stored procedure descriptions, even when the user's original phrasing is vague.

---

🖼️ Visual Memory

Visual Memory stores image-grounded knowledge through a three-layer architecture: SQLite (metadata + FTS5), LanceDB (dual vector index), and local filesystem (raw ima