Team-Level Memory Hub Stack for Long-Horizon AI Agents

Long-horizon agents fail when they run out of context, not when they run out of cleverness. A coding agent that has already read your repo three times this week will happily read it again. A support agent will ask the same onboarding questions in every thread. A research agent will re-derive the same constraints because the last session was flushed. The symptom looks like a context-window problem, but the real issue is architectural: there is no durable, queryable team memory between runs.

This guide is for the senior AI or platform engineer who has to fix that. The reader is building a multi-agent workflow for a team that already ships real work: product specs, support playbooks, incident runbooks, and production code. The team cannot afford to re-explain the world to every agent, but it also cannot dump every document into the prompt and hope. The solution is a narrow, permission-scoped memory hub that turns conversations, documents, and code into reusable assets, then binds only the relevant assets to each agent at query time.

The current discovery signals point in the same direction. TencentDB Agent Memory has been trending as a team-level memory hub that stores assets locally and integrates with OpenClaw and Hermes. Hacker News discussions around agent persistence keep returning to the same complaint: agents do not inherit the previous run. DEV guides are starting to treat memory as a distinct layer, not an afterthought. The takeaway is that memory is becoming a stack component, not a prompt trick.

If you want the broader taxonomy first, the companion post AI Agent Stacks: Skills, Plugins, MCP, ACP, Memory and Workflows maps the whole layer cake. This article is the micro-niche version: one memory hub, four asset types, a layered retrieval pipeline, and a concrete operating workflow.

Recommended stack

Recommended stack

Reader: senior AI/full-stack or platform engineer running long-horizon agents for a small-to-medium team.

Job: give every agent persistent, permission-scoped memory without leaking private context or bloating prompts.

Risk and budget: medium-to-high privacy risk; local-first storage is required; model spend is bounded by retrieval caps and tiered routing.

Model roles: a small model for asset classification and routing; a long-context model for distilling conversations into L1–L3 memory; an embedding model for local vector search (default is sqlite-vec); the agent's own reasoning model stays unchanged.

Agent/interface: OpenClaw or Hermes Agent as the primary client. A custom SDK client is fine if it speaks the Memory Hub OpenAPI and respects /v3/tools/list + /v3/tools/call.

Skills: openclaw-memory plugin for chat memory capture; hermes-skill-sync for skill asset exchange; repo ingest for CodeGraph; document ingest for Wiki.

Plugins: Memory Proxy as a chat-completion shim so existing agents do not need to be rewritten.

MCP/tool categories: read-only code and document indexers, an append-only conversation capture, an asset-management hub, and fine-grained ACL enforcement.

Memory pattern: store four asset types (Chat Memory, Skill, Wiki, CodeGraph) with fixed binding + ACL; retrieve by layer (L3/L2 bootstrap, L1/L0 fallback) with caps.

Approval rule: new chat memory and skills are private by default; team sharing requires explicit review; restricted visibility needs ACL sign-off.

What problem this stack solves

The standard RAG answer to memory is "put everything in a vector store and retrieve the top-k chunks." That works for ad-hoc search, but it does not work for agents that need continuity:

• Context bloat: every run starts by re-reading the same docs.
• Recall failure: the right chunk exists, but the retrieval query misses it.
• Privacy leakage: one agent sees another agent's private notes.
• Loss of provenance: a fact is retrieved, but its source, version, and owner are gone.
• No skill reuse: a workflow that one agent learned last week is forgotten next week.

A memory hub solves these by keeping four distinct assets, each with its own schema, lifecycle, and access rules. The hub does not run the agent loop; it equips the loop with the right context at the right time.

Stack selection: models, agents, tools, and memory

Model roles

The architecture keeps the agent's reasoning model separate from the memory system. The reasoning model can be any current coding or chat model. The memory system adds three supporting roles:

1. Classification/routing model. A small, cheap model decides whether a new turn should be captured, which asset type it maps to, and which team or agent owns it. This keeps the capture pipeline fast and prevents every typo from becoming a permanent memory.
2. Distillation model. A long-context model runs asynchronously to compress L0 conversations into L1 atoms, L2 scenarios, and L3 persona or core memories. Because this is offline, you can use a stronger model without blocking the agent loop.
3. Embedding/retrieval model. The default backend is local SQLite with the sqlite-vec extension. You can point the indexer at a local sentence-transformer model or a small cloud embedding API, but the repository runs zero external dependencies by default.

You do not need frontier models for any of these. The capture layer should be cheap; the distillation layer should be reliable; the retrieval layer should be local.

Agent and interface choice

The cleanest integration path is through an existing agent framework that already has a memory plugin:

• OpenClaw: the @tencentdb-agent-memory/memory-tencentdb plugin exposes chat memory as a tool. The agent calls /v3/tools/list to see what memory assets it has, then /v3/tools/call to read the right ones.
• Hermes Agent: supports Skill asset sync, so a proven workflow discovered by one agent can be imported as a governed skill for another.
• Custom client: the Memory Hub serves an OpenAPI spec. Any client that can call POST /v3/tools/call and respect the JSON schema can participate. This is useful if you already have an internal agent harness.

The Memory Proxy is the bridge for teams that do not want to rewrite agents. It sits between the agent and the LLM API, captures turns as L0, and routes them to the Memory Core. The agent continues to use its normal chat-completion endpoint; the proxy adds memory without touching the agent's control flow.

Skills, plugins, and MCP/tool categories

Think of the tool surface in four categories:

• Capture tools: Memory Proxy (conversation), repo importer (CodeGraph), document importer (Wiki).
• Distillation tools: the Memory Core pipeline, which converts raw captures into L1–L3 layers.
• Management tools: Memory Hub for asset review, versioning, visibility, and agent loadout.
• Runtime tools: /v3/tools/list and /v3/tools/call so agents query memory on demand instead of carrying it in every prompt.

You do not need MCP for this stack. The hub exposes its own REST surface. If you already use MCP, wrap the hub endpoints as a memory server; if not, use the OpenAPI adapter directly.

Memory pattern and privacy boundaries

The golden rule is: retrieve less, but retrieve the right things.

By default, every captured memory is private to the owner. Sharing is explicit and logged. The hub supports four visibility levels:

This pattern protects against the most common memory failure: one agent accidentally including another user's personal facts or another team's confidential notes. It also makes compliance review easier because every asset has an owner, a version, and a visibility label.

Budget and token controls

Memory can become expensive if the agent retrieves too much. The hub enforces three caps:

1. Item count cap: limit how many memory items can be returned per tool call.
2. Character budget: cap the total characters injected into context.
3. Timeout limit: bound how long retrieval can block the agent loop.

These caps apply at every layer. L3 and L2 memories are small and fast. L1 and L0 are only fetched when the higher layers do not satisfy the query. This tiered fallback is cheaper than dumping a full conversation history into every prompt.

Ecosystem integration: how the pieces connect

The published deployment pattern runs three services together from deploy/global-images:

1git clone https://github.com/Tencent/TencentDB-Agent-Memory.git
2cd TencentDB-Agent-Memory/deploy/global-images
3cp .env.example .env
4#Edit .env: two LLM groups (memory group + proxy group)
5./start-all.sh

After the script finishes, the Memory Hub panel is available at http://localhost:8125, and the proxy prints a one-line configuration that you can paste into Claude Code or another compatible client.

Service boundaries

• Memory Core owns the L0–L3 pipeline and the asset schema. It writes to local SQLite + sqlite-vec by default and runs the async distillation jobs.
• Memory Hub is the control plane: teams, agents, asset library, loadout bindings, and access control.
• Memory Proxy is the capture shim for existing chat clients. It is optional if your agent uses the plugin directly.

All three can run from a single Docker Compose file, but you can scale the Core independently if you have heavy ingestion traffic.

The local-first default

The default storage is SQLite with the sqlite-vec extension. That means the entire memory layer can run on a laptop or inside your existing VPC without calling an external vector database. For teams with strict data-residency requirements, this is the strongest privacy boundary. If you later need horizontal scaling, swap the storage backend through the adapter interface; the asset and API layers stay the same.

Importing existing knowledge

The hub is cold-start friendly. You can seed it before any agent asks a question:

• Codebases: import a repository and the CodeGraph asset indexes files, symbols, callers, and callees. The current implementation prioritizes public HTTPS repositories; private SSH support is on the roadmap.
• Documents: import product docs, design specs, and runbooks to build Wiki pages with a link graph.
• Past sessions: import old agent conversation logs and the system extracts Chat Memory and candidate Skills.

This turns the learning cost the team has already paid into an asset library new agents can load on day one.

Context engineering and agent steering

The L0–L3 layer model

The memory pipeline is the heart of the system. Raw conversation is not thrown into a vector store. It is distilled into four layers:

Generation and retrieval are both layered. In normal operation, L2/L3 memories bootstrap the agent quickly. If the agent needs a specific fact, the system falls back to BM25 + vector retrieval + RRF over L1/L0. The agent prompt should instruct the model to trust L3/L2 for context and to request L1/L0 only when it needs evidence.

Fixed binding and agent loadout

Every memory asset is registered uniformly. The hub uses fixed binding + ACL to decide what an agent can see:

1. Narrow the permission scope by team, user, agent, and visibility.
2. Retrieve assets relevant to the current query.
3. Inject only the selected assets into the agent context, respecting the character and item caps.

This means you can give the Release Agent the release checklist skill, the Reviewer Agent the incident Chat Memory, and the Builder Agent the project CodeGraph, without each agent seeing the other's private notes. Switching agents or frameworks only requires re-equipping; you do not retrain.

Steering the agent with tool discovery

Agents should not assume memory exists. They should discover it:

1Step 1: Call /v3/tools/list to see available memory assets.
2Step 2: Decide which asset answers the current user request.
3Step 3: Call /v3/tools/call to fetch the asset.
4Step 4: Summarize what was loaded before answering.

This four-step discipline prevents the agent from hallucinating memories that were not retrieved. It also makes the retrieval trace explicit for debugging and evaluation.

Operating workflow

Day 0: stand up the hub

1. Clone the deployment repository and copy .env.example to .env.
2. Fill in two LLM groups: one for memory distillation, one for the proxy chat shim.
3. Run ./start-all.sh and confirm the panel at http://localhost:8125 loads.
4. Create a team and define the first set of agents (Builder, Reviewer, Scout, etc.).
5. Set default visibility to private for all new captures.

Day 1: seed the library

1. Import the main codebase and let CodeGraph finish indexing.
2. Import the product wiki / runbooks and let the Wiki asset build its link graph.
3. Import any useful past agent sessions; review the extracted Skills before sharing them.
4. Assign base loadouts: every coding agent gets CodeGraph; every research agent gets Wiki; every support agent gets the public FAQ Wiki.

Day 2: run with capture enabled

1. Route one agent through the Memory Proxy or the OpenClaw memory plugin.
2. Confirm L0 conversations appear in the hub.
3. Wait for the async pipeline to produce L1 atoms and L2 scenarios.
4. Review a sample of L1/L2 outputs for accuracy before they are promoted to team visibility.

Day 3: refine retrieval

1. Define per-agent retrieval caps: max items, max characters, max latency.
2. Tune the L3/L2 bootstrap prompt so the agent trusts high-level context but verifies facts.
3. Add restricted ACLs for sensitive assets (security runbooks, customer data policies, HR docs).
4. Run an eval set: can the agent answer a factual question that requires L1 fallback?

Ongoing: govern and prune

1. Review new private memories weekly and promote the reusable ones to team or agent visibility.
2. Delete or archive stale L0 conversations after L2/L3 distillation is verified.
3. Version Skills that change often; deprecate outdated Wiki pages.
4. Audit ACL changes in the hub log.

Safety, source validation, and human approval

Memory is a high-risk component because it accumulates facts, opinions, and mistakes over time. Treat it like a database that agents can read:

• Source validation: every L1 atom must carry a pointer to its L0 source. Do not trust distilled facts unless the source conversation can be inspected.
• Human review for promotion: private-to-team sharing should require at least one human review. Sensitive assets should use restricted ACLs.
• No customer payloads in memory: capture interaction patterns and constraints, not PII, credentials, or raw support tickets.
• Sanitize before ingest: run documents and code through the same checks you use for any production asset: no secrets, no malicious files, no oversized blobs.
• Version and rollback: because assets are versioned, you can roll back a bad skill or a contaminated persona without rebuilding the whole system.
• Rate limits and quotas: the proxy and plugin should have per-user/per-agent rate limits. The distillation queue should be throttled so one noisy channel does not starve the pipeline.

Alternatives and when to use them

If your agents are short-lived and stateless, a full memory hub is overkill. If your agents are part of a team that ships real work, the hub is usually worth the operational cost.

Evaluation criteria and failure modes

Evaluation criteria

1. PersonaMem-style recall: can the agent apply user preferences and facts after an extended interaction? The published benchmark reports a 48% → 76% improvement with memory enabled. Reproduce this on your own conversation logs before trusting it.
2. Retrieval precision: for a set of known questions, the right asset should be in the top-3 results at least 85% of the time.
3. Context window savings: measure prompt token count with and without the hub. You should see a measurable drop as L3/L2 replace repeated full-document reads.
4. Privacy leakage: run adversarial queries where an agent should not see another user's private memory. The failure rate must be zero.

Failure modes to watch

• Over-distillation: L3 persona becomes stale because user preferences changed. Mitigation: version the persona and let users revoke outdated facts.
• False recall: an L1 atom is wrong but gets retrieved as fact. Mitigation: require source pointers and let agents call back to L0.
• Asset sprawl: too many low-quality skills and wiki pages. Mitigation: mandatory review before team sharing and periodic archival.
• Tool hallucination: agent calls a memory asset that does not exist. Mitigation: always start from /v3/tools/list and summarize loaded assets.
• Storage bloat: raw L0 conversations pile up. Mitigation: archive L0 after distillation is verified, with a retention policy.

FAQ

Does the memory hub replace my vector database? Not necessarily. It uses sqlite-vec by default for local vector search, but the asset and ACL layers are the real value. You can swap in another vector backend if you need horizontal scale.

Can I use this without OpenClaw or Hermes? Yes. The Memory Hub exposes an OpenAPI spec. Any client that can call /v3/tools/list and /v3/tools/call can use it. The Memory Proxy even lets existing chat-completion clients participate without code changes.

How is this different from standard RAG? RAG answers "what can be found?" The memory hub also answers "who can use it, which version is valid, which agent should receive it, and how it was distilled." It adds ownership, versioning, skills, and tiered retrieval.

What should not be stored in memory? PII, credentials, raw customer tickets, and anything that cannot survive a team review. Capture patterns, constraints, and decisions instead.

How do I prevent one agent from seeing another agent's work? Default new captures to private, use team visibility only after review, and bind sensitive assets to restricted ACLs. The loadout system means each agent gets only the assets it needs.

Is the hub production-ready? The project labels its Team Memory as Beta. Start with a single team, monitor distillation quality, and keep L0 sources available for verification before promoting memories to shared visibility.

Recommended stack card

For a team that needs persistent, permission-scoped agent memory:

• Capture: Memory Proxy for existing clients, OpenClaw/Hermes plugins for new agents.
• Distill: Memory Core with local SQLite + sqlite-vec, async L0→L1→L2→L3 pipeline.
• Manage: Memory Hub for teams, agents, asset library, loadouts, and ACLs.
• Retrieve: tiered fallback from L3/L2 to L1/L0, with item, character, and timeout caps.
• Govern: private-by-default, explicit sharing, restricted ACLs, source pointers, and versioned assets.

If you want to see how this same disciplined stack mindset applies to other parts of the AI layer, read the RAG eval gate for TypeScript support agents, the MCP security threat modeling guide, and the guide on structured outputs for reliable AI APIs. For observability, the OpenTelemetry GenAI production guide covers tracing the exact retrieval calls this hub will generate.

Sources

• TencentDB Agent Memory repository, README and technical overview. https://github.com/TencentCloud/TencentDB-Agent-Memory/blob/feat/server_team/README.md
• TencentDB Agent Memory npm package metadata. https://www.npmjs.com/package/@tencentdb-agent-memory/memory-tencentdb
• TencentDB Agent Memory installation guide. https://github.com/TencentCloud/TencentDB-Agent-Memory/blob/feat/server_team/INSTALL.md
• CodeGraph project by Colby McHenry, used by TencentDB Agent Memory. https://github.com/colbymchenry/codegraph
• Andrej Karpathy, "LLM Wiki" gist on treating documentation as an LLM-maintained knowledge artifact. https://gist.github.com/karpathy/442a6bf555914893e9891c11519de94f
• Hermes Agent repository by Nous Research. https://github.com/nousresearch/hermes-agent
• OWASP Top 10 for Large Language Language Model Applications, memory and privacy considerations. https://genai.owasp.org/llm-top-10/