Repository navigation
Register large DataFrames with DuckDB's Arrow appender - #617
Open
samclark2015 wants to merge 2 commits into
Open
samclark2015 wants to merge 2 commits into
samclark2015 wants to merge 2 commits into
Conversation
`DuckDBReader::register` created the table from the first 2,048-row chunk and then ran one `INSERT ... FROM arrow(?, ?)` per further chunk, because duckdb-rs's ArrowVTab cannot take a batch larger than a DuckDB vector. The rest now goes through DuckDB's Arrow appender, which chunks to the vector size itself and is about twice as fast: 10M rows (617 MB of Arrow data) register in 1.5 s instead of 3.4 s. A frame the appender cannot convert falls back to the INSERT loop, starting over, and if that fails too its error carries the appender's. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Split out of #595 at review, since it affects every DuckDB user rather than only the Data Explorer.
What it does
DuckDBReader::registercreates a table from a DataFrame's first 2,048-row chunk and then ran oneINSERT … FROM arrow(?, ?)per further chunk. That chunking works around duckdb-rs'sArrowVTab, which can't take a batch larger than one DuckDB vector. Everything after the first chunk now goes through DuckDB's Arrow appender (theappender-arrowfeature of theduckdbcrate), which chunks to the vector size itself.INSERTloop, recreating the table first so a partial append never leaves it half-filled. If the fallback fails too, its error includes the appender's, so the appender error is never silently lost.Testing
cargo test -p ggsql --lib reader::duckdb: 21 pass, including a new test registering 5,000 rows (past one chunk) with nulls, which checks every row arrives in order.buildjob fails with the test-binary segfault thatmainhas had since the runners moved to Ubuntu 26.04 (see DO NOT MERGE: CI debug for segfault #612); it is not specific to this change.🤖 Generated with Claude Code