From Vibe Coding
to Verifiable Agents
Memory, traces, and feedback loops for agents you can actually trust at work.
Auxten Wang · Technical Director @ ClickHouse · 16 Sep 2026
I build databases.
I also run work through agents.
Auxten Wang
@auxten · auxten.com
I spent it on coding-agent tokens.
What I bought was not code. It was six expensive lessons.
It worked.
I couldn't tell you why.
My agent once wrote a social reply that earned 50+ likes. I understood every word.
I had no idea why people loved it.
Make 600+ behaviors match.
Change one import. Existing pandas code should behave the same—while ClickHouse does the heavy work underneath.
# before
import pandas as pd
# after
import chdb.datastore as pd
result = (
pd.read_parquet("sales.parquet")
.query("amount > 1000")
.groupby("region")
.sum()
)Verifiable means:
something else says you're right.
Build a system that can tell.
Agents cheat.
A test fails because row order differs. The fastest way to green?
actual.sort_values(...)
expected.sort_values(...)
assert_equal(actual, expected)Test passes
The incompatible behavior is now hidden.
Our rule
Never normalize away a difference. Fix it—or mark an explicit, reviewed XFAIL.
Put rules in the repo.
Then keep deleting them.
Bring in someone with zero memory.
Daily agent
Knows the history. Knows every workaround. Eventually learns to step around every rough edge.
Fresh reviewer
No project context. Asked to be critical, rational, and evidence-driven. Sees what insiders stopped seeing.
Creators and reviewers need different context.
Files carry context.
JSON carries decisions.
Agents write artifacts to disk. The next agent reads only what it needs.
APPROVE · REJECT · ESCALATE are structured—not guessed from prose.
Task: real Prophet notebook
Generated: 18 mirror tests
✓ 7569 passed
Review: IMPROVE
{
"decision": "IMPROVE",
"reason": "weak assertion",
"rollback_on_failure": true
}
We were building three systems.
Memory
Keep the right knowledge across runs.
Traces
Record what happened and why.
Feedback loops
Turn failures into better behavior.
Every lesson survived a switch of agent framework.
Every handoff loses context.
Days of architecture decisions, rejected ideas, naming rules, and weird edge cases…
Then the next agent starts from a README.
A Markdown file is write-only memory.
# Things to remember
- User likes vegetarian food
- Use Firebase for new apps
- Never use approach B
- Prefer short reports
- Temporary workaround...
- Another exception...
- Another rule...Bad memory is worse than no memory.
One vegetarian week
becomes a permanent identity: “the user never eats meat.”
One discarded Firebase idea
keeps returning in every architecture review.
Working memory · current task state
Episodic memory · what happened and when
Semantic memory · reviewed facts and decisions
Recall is one SQL query.
ClickHouse, inside your process.
Think of it as SQLite for analytical data: one package, no database server.
pip install chdb
memory = recall(task, project, user)Give the digital teammate a name, boundary, and history.
I had no idea what was running.
Bugs, deployments, scheduled jobs, reports—everything landed in one chat.
No run record. No owner. No final state.
“The agent is working on it.”
First, acknowledge.
Then, always return.
1 · Accepted
Before any model call, persist the task and return an acknowledgement.
2 · Terminal state
Every run ends as success, failure, timeout, or cancelled.
Do not mix the loops.
Developer loop
Microsoft Foundry tracing, Application Insights, or Langfuse. Debug one run.
Organization loop
Agent 365 + Entra Agent ID. Inventory, access, activity, lifecycle.
Analysis loop
Your own SQL and evaluations. Find patterns across thousands of runs.
Instrument once.
Choose the views you need.
A trace is just a wide table.
I opened one bad run and saw the missing context, failed tool call, retries, latency, and token cost.
Then I stopped debugging one run at a time.
SELECT tool, countIf(status='error')
FROM spans GROUP BY tool;
SELECT skill, sum(tokens)
FROM spans GROUP BY skill;
SELECT quantile(0.95)(latency_ms)
FROM spans WHERE name='memory.recall';Full visibility creates a lot of data.
Compress
Columnar storage makes repetitive span attributes cheap.
Tier
Keep hot detail. Aggregate older runs. Apply explicit TTLs.
Redact
Remove secrets, credentials, and personal data before export.
The agent reviews the agent.
1 · Observe
Collect traces and outcomes.
2 · Find bad cases
Hallucinations, loops, weak decisions.
3 · Improve
Rules, skills, tests, prompts.
Periodic, evidence-driven, reviewable.
The supervisor catches laziness, hallucinations, and fake tests—but never approves irreversible actions for itself.
The 8 a.m. patrol.
One read-only agent checks ads, analytics, subscriptions, releases, and model errors.
Its real job is to tell me whether to panic.
Nothing critical.
3 medium anomalies.
engagedSessions ↓72%
but sessions, users, and events are normal.
Likely a tracking artifact—not an outage.
Every bad trace is the start of an eval set.
Groundedness
Did the answer follow evidence?
Safety
Did policy hold under pressure?
Business outcome
Did the actual task succeed?
My agent found a bug in its own runtime.
fix: avoid restart loop on plugin metadata updates
One rule classified install timestamps as a no-op before the catch-all restart rule.
Closed loop does not mean closed oversight.
No silent self-editing
Rules and skills change only through reviewable diffs.
No shared superuser
Each agent gets its own identity and least privilege.
No universal architecture
A solo setup is not a 5,000-agent tenant design.
Identity around the agent.
Evidence around every run.
/ Foundryagent + tools
Three changes. No grand platform rewrite.
Do this before adding another agent.
Do not hire a teammate who…
forgets everything
Memory needs retrieval, lifecycle, and provenance.
leaves no record
Every run needs identity, spans, and a final state.
never learns
Failures should become evaluations and reviewed changes.
Prompt engineering is the entry ticket.
System engineering is the moat.
Questions?
@auxten · auxten.com