AgentCon Perth 202601 / 22

Your Agent's Memory
Shouldn't Be a
Markdown File

Give it a database brain instead — with chDB.

Auxten Wang · Technical Director @ ClickHouse · Creator of chDB

memory.md
↓
SQL
a real local database
About me02 / 22
Auxten Wang

Auxten Wang

@auxten · auxten.com

Technical Director @ ClickHouse
Creator of chDB — acquired by ClickHouse in 2024
Building ClickMem — agent memory on chDB
Ex-Shopee · CovenantSQL · Baidu · 360 · 4Paradigm
The status quo03 / 22

How agents "remember" today

Append lines to a text file — memory.md, CLAUDE.md, AGENTS.md — then paste the whole thing back into the prompt.

memory.mdCLAUDE.mdAGENTS.md

It works on day one. It does not survive contact with a real session.

 memory.md
# Project notes
- user prefers TypeScript
- API base = api.acme.dev
- use Poetry for this repo
- deploy on Fridays (?)
- actually, repo moved to uv
- prod base = api.acme.com
- ... 2,000 more lines ...
The pain04 / 22

A familiar agent failure

Yesterday

It hit a constraint and found the fix that worked.

Today

Same task. It reads the prompt — not yesterday's fix.

Result

Repeats the failed plan. Burns tool calls on retries.

The model isn't dumb. The right context just wasn't queryable in time.
Why it breaks05 / 22

A markdown file is write-only memory

✗No query. You load all of it (context bloat) or grep it (no ranking by relevance).
✗No recency, no dedup. Stale notes sit next to fresh ones; nothing decays.
✗No similarity. It can't answer "what's related to what I'm doing now?"
✗No revision, no audit. When did the belief change? From what, to what?
✗It only grows. Straight into the context window: more tokens, slower, needle lost.
A bigger prompt is not a memory. It's a bill.
The reframe06 / 22

Your agent's context is really data

Four kinds of it — and recall is choosing the few that matter now.

Context is four kinds of data; recall is a query
Recall = find the few facts that matter now. A query, not a bigger prompt.
Why a database07 / 22

"Can't I just use what I have?"

SQLite

Great for tiny state. No columnar scan, no vectors, no remote files.

Pandas

Great in memory. Dies past RAM. Can't query files you never loaded.

DuckDB

Closest match — but fewer formats and no mature cloud path.

Vector DB

Similarity only. Can't filter + join + aggregate + audit in one query.

Vector search is just an index. Agent memory is the whole database around it.
chDB08 / 22

chDB: a rocket engine
on a bicycle

The ClickHouse engine, embedded in your Python process — SQLite-simple, built to scan millions of rows.

# pip install chdb
from chdb import session as chs
sess = chs.Session("./agent")
sess.query("SELECT count() FROM file('log.pq')")
Same engine: ClickHouse SQL, JOINs, vectors.
Persistent: local tables on disk for memory.
A rocket engine on a bicycle
Local like SQLite. Analytical like ClickHouse.
chDB API09 / 22

It drops into your Python data stack

70+ formats

Parquet, CSV, JSON, Arrow, ORC — read & write, no converters.

Zero-copy Pandas / Arrow

Query a DataFrame or Arrow table in place — no copy, no load.

Pandas-style API

Prefer DataFrames to raw SQL? chDB v4 speaks that too.

It fits your existing Python code — and LLMs already write SQL & pandas extremely well.
Agent memory10 / 22

What good agent memory needs

Memory the agent can filter, rank, and grow — not a prompt blob.

Memories are rows

Filter, rank, and audit them with plain SQL.

Recall = similarity + filter

Meaning and business rules in one query.

Local now, cloud later

Same SQL from laptop to ClickHouse Cloud.

Recall11 / 22

Recall is one SQL query

Rank by meaning, keep the active ones, pinned first.

cosineDistance

how close two meanings are (0 = identical)

Need project / tags / task history? Add a WHERE or JOIN — same query.

SELECT content,
       1 - cosineDistance(embedding, :q) AS meaning
FROM memory
WHERE is_deleted = 0            -- rules
ORDER BY pinned DESC, meaning DESC
LIMIT 8
Try doing that to a markdown file.
Lifecycle12 / 22

Memory has a lifecycle

Beliefs change. "Use Poetry" → "this repo standardizes on uv."

remember to recall to conflict to revise; append-only history
Append-only13 / 22

Don't update it.
Append to it.

Every revision is a new immutable row with a version. Deletes are a soft flag, never a real delete.

Full revision history — time travel for beliefs.
append-only = ClickHouse's fastest, zero-write-amplification path.

(For history, skip ReplacingMergeTree — it converges to one state and folds the past away.)

CREATE TABLE memory
(
    memory_id   UUID,
    content     String,
    embedding   Array(Float32),
    version     UInt64,
    is_deleted  UInt8 DEFAULT 0,
    created_at  DateTime64(3) DEFAULT now64()
)
ENGINE = MergeTree
ORDER BY (memory_id, version);
One table, three answers14 / 22

One append-only table answers all three

-- NOW: latest version per memory, minus soft-deletes
SELECT * FROM (
  SELECT * FROM memory ORDER BY memory_id, version DESC LIMIT 1 BY memory_id
) WHERE is_deleted = 0;

-- HISTORY: how this belief evolved
SELECT * FROM memory WHERE memory_id = :id ORDER BY version;

-- POINT-IN-TIME: what did the agent believe as of version t?
SELECT * FROM (
  SELECT * FROM memory WHERE version <= :t
  ORDER BY memory_id, version DESC LIMIT 1 BY memory_id
) WHERE is_deleted = 0;
The scoreboard15 / 22

So… should it be a markdown file?

Capability
memory.md
chDB
Human-readable
✓
query / export
Git-reviewable
✓
snapshot
Zero setup
✓
embedded
Filter + rank by relevance
✗
✓
Semantic recall (vectors)
✗
✓
Dedup / conflict resolution
✗
✓
Queryable history + point-in-time
✗
✓
Scales to GBs, archives cheaply
✗
✓
Markdown is a great human interface. It is a weak memory engine.
One surface16 / 22

Memory ≈ Observability ≈ History

All three are append-only by nature — the same shape as the memory model. One local engine serves them as a single query surface.

Agent memory

decisions, facts, prior solutions.

Observability

traces of what the agent did.

History

the full conversation log.

Langfuse (LLM observability, now ClickHouse) builds on exactly this — columnar + append-only.
Local hot path17 / 22

Recall sits inside the agent loop

5–20 tool calls / turn. Put recall over the wire and jitter becomes think-time jitter — and tokens.

agent loop with data calls crossing the network
In-process recall: no RTT, no retries, no detours. Stability is a token problem.
Scale path18 / 22

Start local, graduate to ClickHouse

Day 1

chDB session on a laptop or agent runtime

→
Day 100

ClickHouse Cloud when memory is shared or large

# Day 1 — query local memory
sess.query("SELECT content FROM memory WHERE project = 'checkout'")

# Day 100 — repoint the same name at the cluster (SELECT unchanged)
sess.query("""CREATE OR REPLACE VIEW memory AS
              SELECT * FROM remote('cloud:9440', db, memory)""")
ClickMem19 / 22

Meet ClickMem

The whole pattern — lifecycle, recall, local-first — packaged open-source on chDB.

create→ insert→ recall→ evaluate

Embeddings: Qwen/Qwen3-Embedding-0.6B — runs locally, nothing leaves the box.

github.com/auxten/clickmem

ClickMem architecture
Where it runs20 / 22

Where the brain runs: AWS Lambda MicroVMs

chDB is the only data-infrastructure launch partner for Lambda MicroVMs. Every isolated session carries its own private chDB — Firecracker isolation, suspend / resume, zero network round-trips.

Each isolated Lambda MicroVM carries its own private chDB and federates out
Close21 / 22

Three takeaways

01

Context is data.

Filter it, rank it, join it, version it, evaluate it.

02

Recall is a query.

Semantic meaning + business rules, in one SQL.

03

Don't store memory in a markdown file.

chDB now; ClickHouse when it's shared or large.

Thank you22 / 22

Give your agent a database brain

pip install chdb

Build it

github.com/chdb-io/chdb
github.com/auxten/clickmem

Read it

Docs: clickhouse.com/docs/chdb
Blog & slides: auxten.com

ClickHouse Cloud is where your data lives.
chDB is what your agent thinks with.
Lambda MicroVMs is where it thinks, in private.
Q&A