Skip to content

Register large DataFrames with DuckDB's Arrow appender - #617

Open
samclark2015 wants to merge 2 commits into
mainfrom
sclark/duckdb-arrow-appender
Open

samclark2015 wants to merge 2 commits into
mainfrom
sclark/duckdb-arrow-appender

Conversation

@samclark2015

@samclark2015 samclark2015 commented Nov 19, 2026 •

Copy link
Copy Markdown
Contributor

Split out of #595 at review, since it affects every DuckDB user rather than only the Data Explorer.

What it does

DuckDBReader::register creates a table from a DataFrame's first 2,048-row chunk and then ran one INSERT … FROM arrow(?, ?) per further chunk. That chunking works around duckdb-rs's ArrowVTab, which can't take a batch larger than one DuckDB vector. Everything after the first chunk now goes through DuckDB's Arrow appender (the appender-arrow feature of the duckdb crate), which chunks to the vector size itself.

  • Speed: registering 10M rows (617 MB of Arrow data) takes about 1.5 s instead of 3.4 s (release build, Apple Silicon).
  • Fallback: a frame the appender can't convert falls back to the old INSERT loop, recreating the table first so a partial append never leaves it half-filled. If the fallback fails too, its error includes the appender's, so the appender error is never silently lost.

Testing

  • cargo test -p ggsql --lib reader::duckdb: 21 pass, including a new test registering 5,000 rows (past one chunk) with nulls, which checks every row arrives in order.
  • CI's build job fails with the test-binary segfault that main has had since the runners moved to Ubuntu 26.04 (see DO NOT MERGE: CI debug for segfault #612); it is not specific to this change.

🤖 Generated with Claude Code

samclark2015 and others added 2 commits November 19, 2026 10:55
`DuckDBReader::register` created the table from the first 2,048-row chunk
and then ran one `INSERT ... FROM arrow(?, ?)` per further chunk, because
duckdb-rs's ArrowVTab cannot take a batch larger than a DuckDB vector. The
rest now goes through DuckDB's Arrow appender, which chunks to the vector
size itself and is about twice as fast: 10M rows (617 MB of Arrow data)
register in 1.5 s instead of 3.4 s. A frame the appender cannot convert
falls back to the INSERT loop, starting over, and if that fails too its
error carries the appender's.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant