What is chDB?

import chdb
import pandas as pd

df = pd.DataFrame({'user_id': range(1_000_000),
                    'category': ['A', 'B', 'C'] * 333334})

result = chdb.query("""
    SELECT category, COUNT(*) as cnt
    FROM Python(df)
    GROUP BY category
""", "DataFrame")
  

Same ClickHouse engine you already know — compiled as a shared library, embedded in your Python process. No infrastructure.

The Problem: Pandas at Scale

"I have a DataFrame. I want to run SQL on it. Give me back a DataFrame. Fast."

On an 8 GB laptop, that JOIN will OOM and kill your process.

What if you could keep writing Pandas code, but run it up to 247x faster with 23x less memory?

The Journey to Zero-Copy

Three generations of engineering to eliminate every serialization bottleneck

Is chDB Fast?

chDB is ClickHouse — same codebase, same vectorized engine, same SIMD pipelines.

So how do we make it the fastest SQL engine on Pandas DataFrame?

The secret: Don't let Python slow down ClickHouse.

• Zero serialization — read/write DataFrame memory directly
• Minimize GIL — batch all Python calls, then release
• Bypass CPython — rewrite hot paths (string encoding) in C++

The next 3 slides show exactly how we did each of these.

How Zero-Copy Works

For numeric-heavy workloads, the boundary crossing cost is effectively constant, regardless of data size

Breakthrough 1: GIL Two-Phase Design

// Phase 1: prepareColumnCache() — GIL held
{
    py::gil_scoped_acquire acquire;
    for (auto & col : column_cache) {
        // Pandas: df[col].to_numpy() → array
        py::array arr = column.attr("to_numpy")();
        col.buf    = arr.data();   // raw pointer
        col.stride = arr.strides(0);
        col.row_count = arr.size();
    }
}  // GIL released — shared across all streams

// Phase 2: N parallel PythonSource streams
// each reads a row slice via raw pointer math
void insert_from_ptr(const void* ptr,
    MutableColumnPtr& col, size_t offset,
    size_t count, size_t stride) {
    // Direct memcpy — no GIL, no Python API
    const char* start = (char*)ptr
                        + offset * sizeof(T);
    col->appendRawData(start, count);
}
      

Breakthrough 2: C++ String Encoding

// Step 1: Get raw pointer + kind without
// Python C API (via pybind11 non_limited_api)
getPyUnicodeUtf8(obj, data, length,
                 kind, codepoint_cnt,
                 direct_insert);

// Step 2: Convert in C++ — no GIL needed
static size_t ConvertPyUnicodeToUtf8(
    const void* input, int kind,
    size_t codepoint_cnt,
    Offsets& offsets, Chars& chars) {
  switch (kind) {
  case 1: // Latin-1 (1 byte/char)
    for (size_t i = 0; i < codepoint_cnt; ++i)
      offset += utf8proc_encode_char(
        ((uint8_t*)input)[i], &chars[offset]);
    break;
  case 2: // UCS-2 (2 bytes/char)
    for (size_t i = 0; i < codepoint_cnt; ++i)
      offset += utf8proc_encode_char(
        ((uint16_t*)input)[i], &chars[offset]);
    break;
  case 4: // UCS-4 (4 bytes/char)
    /* same pattern with uint32_t */
  }
}
      

Breakthrough 3: Native JSON on DataFrame

import pandas as pd, chdb

data = pd.DataFrame({
    'event': [
        {'type': 'click',
         'meta': {'x': 100, 'y': 200}},
        {'type': 'scroll',
         'meta': {'x': 150, 'y': 300}},
        {'type': 'click',
         'meta': {'x': 200, 'y': 400}}
    ]
})

# Dot notation on nested JSON!
result = chdb.query("""
    SELECT
        event.type,
        event.meta.x AS x_coord,
        event.meta.y AS y_coord
    FROM Python(data)
    WHERE event.type = 'click'
""", "DataFrame")

#   event.type  x_coord  y_coord
# 0      click      100      200
# 1      click      200      400
      

Perfect for log analytics, event data, API responses — no pre-processing, no schema definition

DataStore: Four-Layer Architecture

Layer 1 — Pandas API
ds['col'] > 25, ds.groupby(), ds.merge()
Every call returns a lazy object

Layer 2 — LazyOp Chain
Records operations without touching data
filter, select, assign, apply, groupby...

Layer 3 — Query Planner
Triggered by print(), len(), .columns
Splits pipeline into segments, routes to engine

Layer 4 — Execution
chDB segments ↔ Pandas segments
DataFrames flow via memoryview

Lazy Execution & explain()

from chdb import DataStore

ds = DataStore.from_file("users.parquet")
ds = ds[ds["age"] > 30]
ds = ds.sort_values("salary",
                     ascending=False)
ds = ds.head(100)

# Nothing has executed yet!
print(ds.explain())
      

Execution Plan
════════════════════════════════
 [1] File: users.parquet

Operations:
────────────────────────────────
 [2] [chDB] WHERE: age > 30
 [3] [chDB] ORDER BY: salary DESC
 [4] [chDB] LIMIT: 100
      

Segment Execution: Best of Both Worlds

ds = DataStore.from_file("data.parquet")
ds = ds[ds["age"] > 25]                    # → chDB segment
ds["name_upper"] = ds["name"].str.title()   # → Pandas segment (str.title)
ds = ds.sort_values("age")                  # → chDB segment
  

• Auto-splits the pipeline at SQL/Pandas boundaries
• ClickHouse handles what it can; Pandas handles the rest (300+ CH functions + full Pandas ecosystem)
• Seamless handoff via Python memoryview — for numeric data, effectively zero-copy
• Tip: keep Pandas-only ops toward the end for best optimization

The query planner sees through the entire pipeline. You just write Pandas.

Unified Data Sources

DataStore inherits all of ClickHouse's data source support:

# Local files — Parquet, CSV, JSON, ORC...
ds = DataStore.from_file("events.parquet")

# Remote S3
ds = DataStore.uri("s3://bucket/path/to/data.parquet")

# Query a PostgreSQL table
ds = DataStore.from_sql("SELECT * FROM users", engine="postgresql://...")

# Mix sources in a single query — ClickHouse optimizes the execution
ds = DataStore.from_sql("""
    SELECT u.name, e.event_type, e.timestamp
    FROM file('users.parquet') u
    JOIN postgresql('host:5432', 'db', 'events', 'user', 'pass') e
    ON u.id = e.user_id
""")

ds = ds[ds["event_type"] == "purchase"].sort_values("timestamp")
print(ds)  # Lazy — executes here
  

80+ formats. Local, remote, in-memory, cross-database JOINs. Same Pandas-style pipeline. No ETL.

Benchmark: chDB vs Pandas — Speed & Memory (50M Rows)

chDB wins 10/10 on speed (geomean 4.2x) and 10/10 on memory (geomean 2.9x). T04 JOIN: Pandas peaks at 10.5 GB → OOM on 8 GB laptop; chDB uses 1.8 GB → OK.

Output Zero-Copy: chDB vs DuckDB

Query ClickBench hits (1M rows) → export to Pandas DataFrame

AI Writes Pandas. chDB Runs It Fast.

Summary: What chDB v4 Delivers

Not Just Python

chDB runs as a native library in 9 languages:

Python

pip install chdb

Go

github.com/chdb-io/chdb-go

Rust

cargo add chdb-rust

Ruby

gem install chdb-ruby

Node.js

npm i chdb

Bun

chdb-io/chdb-bun

.NET

dotnet add package chdb

Java

chdb-io/chdb-java

Zig

chdb-io/chdb-zig

Runs everywhere — no server, no infra:

Getting Started — 30 Seconds

pip install "chdb>=4.0.0"
  

import chdb.datastore as pd  # swap one import, done

df = pd.read_csv("your_data.csv")

result = (
    df[df['amount'] > 100]
    .groupby('category')
    .agg({'amount': 'sum', 'order_id': 'count'})
    .sort_values('amount', ascending=False)
    .head(10)
)

result.plot(kind='bar')  # Still a real DataFrame
  

No server. No config. No new API to learn.

The Road Ahead

• Native Python UDF — in-process, no subprocess
  Register Python functions as ClickHouse UDFs via @chdb.func decorator, executed inside the vectorized pipeline with full type mapping (PR #9)
• Free-threading build (no-GIL) — Python 3.14+
  chDB compiled with free-threading support, enabling true multi-threaded Python UDF execution within ClickHouse's parallel pipeline (PR #12)
• PyTorch DataLoader integration
  Efficient batching, shuffling, and streaming for ML training
• Hybrid execution
  Seamlessly scale from laptop to ClickHouse Cloud

Thank You!

Q&A

@auxten · auxten.com