PyCon JP 2026 · 広島 Hiroshima

Rediscovering
DataFrames

100× Analytics Without Leaving Pandas

再発見

Auxten Wang · ClickHouse · Creator of chDB@auxten

Auxten Wang

@auxten · auxten.com

Technical Director @ ClickHouse

Creator of chDB — acquired by ClickHouse in 2024

Builds embedded databases & data tooling for Python

Ex-Shopee · CovenantSQL · Baidu · Qihoo 360

Everyone's first data tool

import pandas as pd
df = pd.read_csv("sales.csv")
df[df.amount > 100] \
.groupby("region").amount.sum()

In every tutorial, every notebook, every team

Muscle memory for millions of engineers

The DataFrame dialect LLMs write best

This API took 15 years
to learn everywhere.

The pandas API is an asset. The engine underneath is the problem.

限界

Then your data grew.

Memory
Error

a 10 GB DataFrame chokes a 16 GB laptop

1 / 16

cores doing the work while fifteen watch

> query

reading from S3 takes longer than the analysis itself

Your code didn't hit the ceiling. The execution model did.

The ceiling has three walls

EAGER

every step in the chain materializes a full intermediate copy

SINGLE-CORE

the GIL-era design: one thread computes, the rest idle

ALL-IN-RAM

the whole table decompresses into memory before row one is read

33 MB

Parquet on disk

→

1.5 GB

as a pandas DataFrame

×45 blow-up,
measured on my laptop.

Not fixable with another .apply() trick. It's the engine.

The usual exits all cost the same thing

Spark

a cluster to babysit, for one notebook

Polars

fast — but a new API for the whole team

DuckDB

great engine — but now you write SQL

Rewrite

months of migration, retraining, re-testing

非常口 — emergency exit

Every exit asks you to leave the API. What if you didn't have to?

Change one line

- import pandas as pd + import chdb.datastore as pd
df  = pd.read_parquet("events.parquet")
top = (df[df.status == 200]
        .groupby("user_id").amount.sum()
        .sort_values(ascending=False).head(10))

同じコード。
違うエンジン。
Same code.
Different engine.

The rest of the file doesn't know. 一行だけ — one line.

chDB: a rocket engine on a bicycle

# pip install chdb
import chdb
chdb.query("SELECT count()
FROM file('orders.parquet')")
# schema inferred · nothing loaded

The ClickHouse engine, in-process — like SQLite

No server, no config, no connection string

Columnar · vectorized · every core

Same engine that runs analytics at Anthropic, Cursor, Vercel, Cloudflare.

Layered, not mixed — 広島風

ソース sauce — your pandas calls

df[df.x > 1].groupby(…).sum() — untouched

麺 noodles — the lazy op chain

every call recorded, nothing executed

キャベツ cabbage — one SQL plan

the whole chain compiles to a single query

生地 batter — the chDB engine

columnar · vectorized · all cores

鉄板 teppan — your Python process

everything cooks in-process, no server

len() · print() · for — the コテ flip: the whole stack executes at once.

Where the 100× lives

chDB

×1.35

DuckDB

×1.40

Polars

×9.99

pandas

×71.18

Worst single query: pandas 94.3 s vs chDB 0.037 s — ×2,550

各駅停車 → のぞみ

ClickBench · DataFrame category · 43.77 GiB · same c6a.metal machine · relative runtime, lower is better

Same DataFrame workload. ×71 less waiting — before any tuning.

10M rows, 20 everyday ops — wins 14 / 20

pandas keeps 6 —
and we'll tell you
which ones.

Complex chains win big. Tiny cached ops stay with pandas — honesty slide ahead.

Your data lives on many islands

pd.read_parquet("https://…/house_0.parquet") # the lake
pd.read_csv("crm_exports.tar :: users_*.csv") # archives
pd.read_sql("SELECT …", "postgresql://prod")  # live DBs
    

measured this week, this laptop

2,772,030 rows on S3 — counted in 3.3 s, no download
parquet ⋈ CSVs-inside-a-tar ⋈ groupby — one chain

実測

70 table functions · 72 input formats — counted from the engine

Like the Miyajima torii: a gateway standing in the water. The data never moves.

It reads what the query needs — not the file

house_0.parquet · 33 MB · 2.77M rows · 14 columns ▼ HTTP Range reads

type

price

date

town

street

…9 more columns

footer

■ fetched for avg(price) by type ■ skipped ■ footer = schema + stats, read first

3.3 s

len() on the remote file — footer + counts only, no download

2 / 14

columns fetched for the aggregation — ranged reads, no full scan

344 vs 975 MB

peak RAM, chDB streaming vs pandas on the same file

Ranged reads: you pay for the columns you touch — not the file.

Live: a 10M-row access log, migrated

import pandas as pd # ← the only line we'll touch
logs = pd.read_parquet("access_log.parquet") # 10M rows
slow = logs[logs.url.str.contains("/search/")]
p99 = slow.latency_ms.quantile(0.99)

0.66 s

pandas, every run — filter is the hot path

367.8

the p99 latency it computes — remember this number

String filter +
quantile — pandas'
least favorite food.

Ordinary pipeline, ordinary laptop. Now change one line — live.

Same file. Same answer. One line later.

- import pandas as pd + import chdb.datastore as pd # …nothing else changes…

0.66 s

pandas

0.16 s

chdb.datastore — ×4.1

367.8

identical answer, checked every run

実測

×4.1 on a laptop, file in page cache. The ×71 arrives as the data grows — same code.

Same numbers out, less waiting. That's the entire migration.

Vector search is a column, not a service

from datastore import F
docs = pd.read_parquet("docs.parquet") # emb: Array(Float32)
docs["dist"] = F.cosine_distance(
docs.emb, F.array(*query_vec))
docs.sort_values("dist").head(3)

top-3, measured

api rate limits  0.000 ← the query itself
refund policy    0.946
shipping delay   1.119

実測

No second database to run

Embeddings live next to the rows they describe

Schema gotcha: needs non-nullable Float32 arrays

No vector DB. An Array column and a sort.

Time series, the calls you already know

d = pd.read_parquet("sensor.parquet")
daily = d.resample("D", on="ts").value.mean()
d["ma24"] = d.value.rolling(24).mean()

daily means, measured

2026-01-01   99.81
2026-01-02  101.78
2026-01-03   99.08  …

実測

resample · rolling · shift · diff

Backed by ClickHouse's date & window functions

Windows run columnar, on all cores

The ops that hurt most
at 3am on-call — faster.

Same idioms. The clock runs columnar now.

金継ぎ — show the cracks in gold

Small cached data: pandas wins

value_counts on a cached file: pandas 0.13 s, DataStore 1.61 s. Below ~1M rows, don't bother switching.

API coverage is not 100%

during prep, str.extract inside a groupby came back empty — mirror-test your pipeline, don't assume.

DuckDB is also excellent

×1.40 vs our ×1.35 — a coin flip. chDB is the ClickHouse flavor: more formats, and a straight path to a shared cluster.

Tools you can trust tell you when not to use them.

Migrate like a daruma: one eye first

Swap at the I/O boundary. read_parquet / read_csv first — the cheapest, biggest wins.

Move the hot paths. The slow groupbys, the string filters — one column, one op at a time.

Mirror-test everything. Same input, both engines, assert equal — never trust, always verify.

Keep the escape hatch. .to_pandas() drops any result back — any op, any time.

Paint the second eye when the pipeline runs green.

Three takeaways

The pandas API is the asset. Fifteen years of muscle memory — keep it, swap the engine.

The ceiling is the execution model. Eager, single-core, all-in-RAM — not your code.

Migrate daruma-style. One line, one column at a time — mirror-tested, escape hatch open.

pandas is the language. chDB is the engine. You never had to leave.

ありがとうございました！

Thank you.

pip install chdb

github.com/chdb-io/chdb

clickhouse.com/docs/chdb

auxten.com — slides & verified demos

Q&A

Auxten Wang · ClickHouse · Creator of chDB@auxten

RediscoveringDataFrames

Auxten Wang

Everyone's first data tool

Then your data grew.

The ceiling has three walls

EAGER

SINGLE-CORE

ALL-IN-RAM

The usual exits all cost the same thing

Spark

Polars

DuckDB

Rewrite

Change one line

chDB: a rocket engine on a bicycle

Layered, not mixed — 広島風

ソース sauce — your pandas calls

麺 noodles — the lazy op chain

キャベツ cabbage — one SQL plan

生地 batter — the chDB engine

鉄板 teppan — your Python process

Where the 100× lives

10M rows, 20 everyday ops — wins 14 / 20

Your data lives on many islands

measured this week, this laptop

It reads what the query needs — not the file

Live: a 10M-row access log, migrated

Same file. Same answer. One line later.

Vector search is a column, not a service

top-3, measured

Time series, the calls you already know

daily means, measured

金継ぎ — show the cracks in gold

Small cached data: pandas wins

API coverage is not 100%

DuckDB is also excellent

Migrate like a daruma: one eye first

Three takeaways

Thank you.

Rediscovering
DataFrames