CROWDB Blog

Load TPC-H and TPC-DS tables into CROWDB Iceberg

2026-09-30T15:20:00+00:00

I wanted a repeatable way to put recognizable data into the CROWDB Iceberg container before trying SQL engines. Hand-writing a sample table proves little about multi-table imports. crowdb-tpc-loader now generates TPC-H and TPC-DS Parquet, checks every table, uploads the files, and registers them through the Iceberg REST Catalog. This guide uses scale factor 0.01 to exercise that path. It does not run the benchmark queries or claim TPC certification.

Start a local container

These commands use a Linux amd64 development checkout of CROWDB at version 0.2.0. The image below is built locally from source; the test does not establish that a 0.2.0 Docker Hub tag has been published. The single-node profile has one copy and no node-failure protection, so use disposable data.

cd /path/to/crowdb
pixi run build-single-node-container
docker run -d --name crowdb-tpc-demo \
  -p 127.0.0.1:80:80 \
  -v crowdb-tpc-demo-data:/opt/crowdb/data \
  crowdb-iceberg-single-node:dev

Wait until docker inspect crowdb-tpc-demo --format '' says healthy. Port 80 carries the REST Catalog and native Iceberg file endpoint. Get the local client credentials without copying the token into a command line:

umask 077
docker exec crowdb-tpc-demo crowdb-monitor credentials show --format env > /tmp/crowdb-tpc-demo.env
set -a
. /tmp/crowdb-tpc-demo.env
set +a

Keep that file private and remove it when finished. Use a different host port or adjust ICEBERG_URI if port 80 is occupied.

Load both datasets

From a checkout of crowdb-tpc-loader:

cd /path/to/crowdb-tpc-loader
python3 -m venv .venv
. .venv/bin/activate
python -m pip install --only-binary=:all: -e .

crowdb-tpc-loader load --benchmark tpch --sf 0.01 \
  --namespace tpch_demo --report-file ./tpch-demo.json
crowdb-tpc-loader load --benchmark tpcds --sf 0.01 \
  --namespace tpcds_demo --report-file ./tpcds-demo.json

The commands need a fresh namespace. TPC-H uses the tpchgen-cli 3.0.0 binary; TPC-DS uses DuckDB 1.5’s official tpcds extension. The first run may download these generator components. The loader validates the full generated dataset before creating a benchmark table, then uploads each Parquet file and registers it with Iceberg. CROWDB checks the upload’s signed payload or supplied checksum before accepting it; the loader does not download the object for another checksum pass. The loader uploads up to 24 files concurrently by default, then commits one snapshot per table in order. It deletes local staging on success; the JSON reports retain remote file locations, counts, and snapshot IDs. An existing table stops a default load, so choose a new namespace when repeating the experiment.

In my local SF 0.01 run, TPC-H produced 8 tables, 86,805 rows, and 3.23 MB across 8 Parquet files. TPC-DS produced 24 tables, 277,976 rows, and 3.86 MB across 24 Parquet files. There was one file per table in this small run, but the loader accepts multiple shards. TPC-H region was 1,227 bytes while lineitem was 1,924,571 bytes; TPC-DS call_center was 4,917 bytes while store_sales was 985,882 bytes. These are compressed Parquet sizes, not in-memory table sizes.

Select rows in Python

Run this in a fresh process after the loader has removed its local files. The file adapter handles exact-object requests on CROWDB’s native Iceberg endpoint:

import os
from pyiceberg.catalog import load_catalog

catalog = load_catalog(
    "crowdb", type="rest",
    uri=os.environ["ICEBERG_URI"], token=os.environ["ICEBERG_TOKEN"],
    **{"py-io-impl": "crowdb_tpc_loader.crowdb_fileio.CrowdbFileIO"},
)

region = catalog.load_table("tpch_demo.region")
print(region.scan(
    row_filter="r_regionkey == 1",
    selected_fields=("r_regionkey", "r_name"),
).to_arrow().to_pylist())

item = catalog.load_table("tpcds_demo.item")
print(item.scan(
    selected_fields=("i_item_sk", "i_item_id"), limit=5,
).to_arrow().to_pylist())

The first selection returned {'r_regionkey': 1, 'r_name': 'AMERICA'} in my run. The second returned five item rows. For a read-only check of every imported table, including remote Parquet footers and an Iceberg sample scan, run python scripts/verify_crowdb.py ./tpch-demo.json --require-complete --iceberg-scan and the same command with tpcds-demo.json.

Query from DuckDB

I also tested the locally built DuckDB 1.5.6 CLI with its iceberg extension against the CROWDB REST Catalog. With the same credential environment loaded, this selected AMERICA from the imported region table:

/path/to/duckdb/build/release/duckdb <<SQL
INSTALL iceberg;
LOAD iceberg;
INSTALL httpfs;
LOAD httpfs;
CREATE SECRET crowdb_catalog (TYPE ICEBERG, TOKEN '$ICEBERG_TOKEN');
ATTACH '' AS crowdb (TYPE ICEBERG, SECRET crowdb_catalog, ENDPOINT '$ICEBERG_URI');
SELECT r_regionkey, r_name FROM crowdb.tpch_demo.region WHERE r_regionkey = 1;
SQL

The empty ATTACH warehouse selector matters: this CROWDB profile has no named warehouse. DuckDB must use the object credentials supplied by the catalog for this Iceberg table. A separate fixed S3 credential from the monitor could attach the catalog but got HTTP 403 when reading the table’s manifest. I also queried the larger TPC-H SF 10 lineitem file and the newly loaded TPC-DS SF 10 item table; the latter returned (1, 'AAAAAAAABAAAAAAA') for i_item_sk, i_item_id. These are spot queries through the REST Catalog and file endpoint, not TPC query coverage or performance results.

CROWDB’s Iceberg file paths look like S3 paths, but they are native Iceberg objects with table-scoped authority. PyIceberg’s default PyArrow FileIO calls get_file_info for an exact file; in this run that led to ListObjectsV2 and failed. The loader’s small FileIO adapter uses an exact-object request for existence and size. Iceberg’s FileIO guide describes read, write, and seek as essential file operations. General prefix listing and the meaning of the S3 bucket field remain separate design work; this import does not require either.

What the small run measured

I ran one single-node container on a Linux x86_64 host with an Intel Core i9-7960X, 32 logical CPUs, and 62 GiB RAM. The generator used two threads and a 1 GB DuckDB memory limit; Python 3.12.3, PyArrow 23.0.1, and PyIceberg 0.10.0 were installed. There was no performance baseline or concurrent client load. The clean TPC-H import took 29.81 seconds wall time; TPC-DS took 75.83 seconds. A separate client then verified all 32 tables after local staging was gone, including full reads and SHA-256 checks.

The files are tiny, yet each table’s create, upload, and register step took a median 3.33 seconds for TPC-H and 2.85 seconds for TPC-DS. Those steps account for most elapsed time. This shows a significant per-table fixed cost in this setup; the run does not identify which internal metadata call dominates.

I then ran the concurrent loader on larger datasets. TPC-H SF 1 loaded 8.66 million rows in 50 seconds; TPC-H SF 10 loaded 86.59 million rows and 3.64 GB of compressed Parquet in 232 seconds. TPC-DS SF 1 loaded 19.56 million rows in 157 seconds; TPC-DS SF 10 loaded 191.50 million rows and 2.77 GB of compressed Parquet across 24 tables in 453 seconds. Each successful run passed independent manifest, footer, and sample Iceberg scan verification. These are single-node wall times, including generation and table registration, not a benchmark result. TPC-H SF 10’s upload phase took about 176 seconds for 3.64 GB, roughly 20.7 MB/s; its streaming/write path needs profiling. A distributed deployment and standard SQL benchmark queries still need separate tests.

The local Docker volume keeps the Iceberg data when the container stops. To end the example, run docker rm -f crowdb-tpc-demo and remove /tmp/crowdb-tpc-demo.env. Remove the crowdb-tpc-demo-data volume only when you deliberately want to discard the imported tables.

Use crowdb-iceberg container with pandas

2026-09-28T02:30:00+00:00

CROWDB’s crowdb-iceberg container runs an Iceberg REST catalog and its file storage together. This example creates a table of six sample orders, writes the rows through PyIceberg, then reads the stored table into pandas.

Use disposable data with this 0.1.0 evaluation release. The commands below assume a Linux amd64 host and a free local port 80.

1. Start the container

docker run -d --name crowdb-iceberg -p 127.0.0.1:80:80 crowdb/crowdb-iceberg:0.1.0

The port mapping lets Python on your machine reach the catalog and file service.

2. Set up the connection

Install the Python clients and print the connection values from your container:

python3 -m venv .venv
. .venv/bin/activate
pip install 'pyiceberg[pyarrow]==0.11.1' pandas
docker exec crowdb-iceberg crowdb-monitor credentials show --format env

Export the two Iceberg values in the same shell, replacing the examples below with the values printed by your container. Keep the token private.

export ICEBERG_URI='http://localhost'
export ICEBERG_TOKEN='<your container token>'

Start an orders.py file with the connection:

import os

import pandas as pd
import pyarrow as pa
from pyiceberg.catalog import load_catalog

catalog = load_catalog(
    "crowdb",
    type="rest",
    uri=os.environ["ICEBERG_URI"],
    token=os.environ["ICEBERG_TOKEN"],
)

3. Write a table and its data

Add this block to orders.py. The sample rows are created locally; table.append writes them to the container’s Iceberg storage.

orders = pd.DataFrame(
    [
        (1, "Beijing", "paid", 120),
        (2, "Shanghai", "paid", 80),
        (3, "Beijing", "cancelled", 200),
        (4, "Shanghai", "paid", 60),
        (5, "Shenzhen", "paid", 50),
        (6, "Beijing", "paid", 30),
    ],
    columns=["order_id", "city", "status", "amount_usd"],
)
arrow_orders = pa.Table.from_pandas(orders, preserve_index=False)

catalog.create_namespace_if_not_exists("pandas_demo")
table = catalog.create_table("pandas_demo.orders", schema=arrow_orders.schema)
table.append(arrow_orders)

This creates a new pandas_demo.orders table. Use a different table name if you run the script again.

4. Query the stored table with pandas

Add the final block to orders.py, then run python orders.py:

saved_orders = catalog.load_table("pandas_demo.orders").scan().to_pandas()
result = (
    saved_orders[saved_orders["status"] == "paid"]
    .groupby("city", as_index=False)
    .agg(orders=("order_id", "count"), revenue_usd=("amount_usd", "sum"))
    .sort_values("revenue_usd", ascending=False)
)
print(result.to_string(index=False))

python orders.py

The query counts paid orders and sums their revenue for each city:

    city  orders  revenue_usd
 Beijing       2          150
Shanghai       2          140
Shenzhen       1           50

The cancelled order is excluded. saved_orders comes from a fresh Iceberg table scan, so this query uses the data written to the container.

For container configuration and other client operations, see the quick start and Iceberg manual.

Why we’re building CROWDB

2026-09-21T02:30:00+00:00

For more than a decade, I have designed, built, and debugged storage systems as their bottlenecks moved from disks to CPUs, networks, and data movement. The most useful lessons came from the work in between: revisiting an assumption, tracing a slow request, and seeing what a real workload exposed that a clean diagram did not.

That experience is why I’m building CROWDB. I want the storage system to own enough of the data path that we can change it when the workload changes—not just tune the parts we happen to control.

Storage has a new job

Consider a training job that reads data from an Iceberg table. The table engine resolves metadata, reads files, and hands data to a library that prepares batches for the GPU. Somewhere underneath, storage places and protects the bytes. Several components may cache, retry, or copy the same data along the way.

None of those steps is automatically wrong. Separate systems give us useful interfaces and independent ways to evolve. The difficult part is deciding who owns a problem that crosses those interfaces. A table engine cannot change how storage places a file. A storage service usually cannot see the batch the application is trying to assemble.

When I follow a slow request, those boundaries are often more interesting than the individual algorithms. Which component owns this buffer? Why is this metadata fetched again? What happens if the writer retries after only part of the operation became durable? A fast device does not answer those questions.

S3 remains a useful interface. I do not think every workload should be forced to use it as an intermediate representation, though. Tables have commits and snapshots. Datasets have samples and batches. These concepts deserve a place in the storage design, not just conventions layered over object names.

What CROWDB is

CROWDB is a distributed storage platform. S3 and Iceberg have their own access models over a shared storage core. Dataset is a third model we are designing; it is not implemented yet.

An S3 client works with objects. An Iceberg client works with a catalog, tables, snapshots, and immutable files. The two models reuse the same underlying machinery for distributed state and protected chunks. Iceberg is not implemented by asking users to assemble a catalog on top of a separate S3 service.

This distinction is easy to lose in a diagram. “Shared storage” does not mean that uploading an S3 object registers an Iceberg table. The access models still have separate semantics. What they share is the infrastructure underneath.

The first useful question is therefore practical: can a table client talk to the catalog and store its data without another storage service to configure? The native Iceberg path is designed to make that possible. The answer still needs to be tested against each client workflow.

Why build the whole path?

There are good storage engines, object stores, and table formats already. Reusing them is often the right decision. Building another core means taking responsibility for failures that someone else has already spent years finding.

I am accepting that cost because the boundaries are part of what I want to change. If a request crosses an independently owned metadata service, an object gateway, and a separate storage engine, a change to placement or buffer ownership can become a negotiation between systems. We want those contracts to be explicit within CROWDB.

Multi-Paxos, write-ahead logs, B+trees, and erasure coding are not the novelty. Putting them in one repository is not an advantage by itself. The work is making their contracts agree: when a write is durable, who may change its placement, how a buffer stays bounded, and what recovery must finish after a process dies.

Owning those decisions should give us room to improve the path. It does not prove that CROWDB is faster. That needs measurements with a stated workload, hardware, concurrency, and baseline. I would rather publish those results separately than borrow credibility from an architecture drawing.

Own the path that determines the system’s limits.

Three layers, one system

The architecture has three layers. Reading from the application downward helps explain their jobs, but this is a dependency map—not a claim that every byte travels through every box.

FIG. 01 / THREE LAYERSimplementedplanned

S3HTTP objectsIMPLEMENTED

IcebergCatalog + FileIOIMPLEMENTED

DatasetHTTP + native clientIN DESIGN

02Chunk

One protected storage layer

Chunk Stream · Chunk-KV
Placement, bounded streaming, protection and repair

chunk client → Chunk I/O → ChunkDB → DiskIO → DiskDB

01Reusable KV

Replicated metadata and durable state

crowdb-kv · Multi-Paxos · WAL · crowdb-tree · RPC

PLANNED PATH DiskIO buffers → RDMA / GDS → GPU memory. This is a design direction, not a capability of the preview.

The KV layer supplies distributed state; the diagram does not imply that every payload is stored in KV. S3 and Iceberg use the shared core without translating through one another.

Access models keep their meaning

The Access layer implements the interfaces applications use. S3 handles HTTP object operations. Iceberg supplies a native catalog and FileIO path. Both have working implementations in the current repository. Dataset remains in design, including the possibility of native access that does not pass through an HTTP server.

The Iceberg catalog and FileIO belong to one access model. General S3 object access is a separate model. Keeping those roles clear is more useful than calling every path “compatible.”

Chunks own the storage work

The Chunk layer is responsible for placement, protection, streaming, and repair. Chunk Stream provides durable ordered append; Chunk-KV provides a range-partitioned structure. The underlying path continues through Chunk I/O, ChunkDB, DiskIO, and DiskDB.

These names will need their own articles. For this introduction, the important point is ownership: access models can reuse the same storage mechanisms rather than grow their own placement and recovery systems.

Distributed state is a reusable foundation

Underneath, crowdb-kv supplies replicated state using parallel Multi-Paxos slots and write-ahead logging. It is a reusable distributed layer, not the product-level data model.

This separation matters to me. Owning the stack should not require turning it into a single inseparable component. The layers need narrow enough contracts that we can reason about them, test them, and use them independently where that makes sense.

Built for the next data path

The GPU path is an example of why I want this control. Today, CROWDB uses ordinary CPU streaming. Dataset, topology-aware native access, and direct GPU delivery remain planned work.

RDMA or GPUDirect Storage would not become useful merely because we added another endpoint. We would need to reason about buffer ownership, placement, protection, and the lifetime of the data being transferred. Those decisions cross several layers of the system.

The aim is to leave room for that work without changing what an object or a table means. There is no direct-GPU benchmark to show here, and no such capability to enable in the preview. That boundary belongs next to the design, not in a footnote.

Where the project stands

Development status · updated September 29, 2026

S3 and native Iceberg have working implementations. Dataset and direct GPU delivery are still in design. Production use and on-disk upgrade compatibility are not supported.

The repository includes code, tests, and design documents; the product site hosts the user manual. That is where I want this argument to be judged. A useful next step is to inspect the Iceberg guide and tell us which assumption breaks under your workload.

Start with the Iceberg evaluation guide →

Technical references: project README and status, design documents, and Iceberg quick start. This article was first published on September 21 and revised on September 29, 2026 to reflect the current Iceberg implementation and development limits.