Repository navigation
Open console query results in Positron's Data Explorer - #595
samclark2015 wants to merge 2 commits into
Conversation
|
Most of the logic of this PR is implemented in the Jupyter kernel and I'm not super fond of that if it can be avoided at all. For the kernel to contain a module so closely tied to positron feels like a code smell. Can you take another round at this with the goal of keeping it as confined as possible to the VS code extension? Also, we plan to spin out the VS Code extension to its own repo that you can have more control over once this is in (just an FYI :-) ) |
|
Sweet! Thanks for this feedback. I churned on it with the 🤖 and came up with this proposal, which moves Positron-specific code into the extension. Kernel additions are minimal and client-agnostic. Let me know how this sounds, and I'll spin up a revision. Proposal: the kernel offers a general results API, and the extension serves the Data ExplorerKernel: a general results API, with no Positron logic.
Extension: everything specific to Positron's Data Explorer.
Why this shape
Tradeoffs
ScopeBoth directives go. The extension calls |
|
I think this is a better shape in general. If possible, it would be nice to reuse the cache mechanism already in place in ggsql, however, care should be taken to not enforce a cached reader on the user (because that change the dialect for them). So, the cache reader should only be used for the results API |
Console query results get a link that opens them in Positron's Data Explorer. The kernel keeps each result in a dedicated in-memory DuckDB results cache (sharing `CacheTables` with `CachingReader`, with pinning so open results are not evicted; the user's reader is never wrapped) and exposes a frontend-agnostic results API over `call_method`. The VS Code extension registers the Data Explorer RPC handler, adds the console link, and pages, sorts, filters and profiles results through that API. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
307d520 to
6711156
Compare
|
The shape of this refactor is great - basically how I envisioned it. Below are some correctness issues found and summarised together with Kimi for you to look through: High
Medium
Low
Test gaps
|
- A failed sort or filter leaves its view as it was: the new order table is built under a new name and only then replaces the old one. - Pages are bounded (MAX_PAGE_ROWS), scattered indices fetch just those rows, and an index past the end is an error rather than a shifted column. - One byte budget (--results-max-bytes) covers unopened results, cached ones including pinned, and order tables. Keeping results is opt-in (--keep-results / GGSQL_KEEP_RESULTS), which the extension sets for console sessions. - result_sql quotes for the session dialect, escapes LIKE with '!', keeps a cell's earlier statements, and leaves out repeated column names. Regex support comes from SqlDialect::sql_regex_match. - Views close with the UI comm. The extension releases a view when the Data Explorer fails to open, ignores requests after close, and never calls a session that is down. - CacheTables takes a byte budget, keeps pins in memory, and has a narrower API; the kernel builds its own DuckDB backend for it. - kind_of matches DuckDB type names exactly. - The DuckDB Arrow appender change moves to its own PR. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
@thomasp85 Thanks for this!! Everything is addressed, and the appender is split out into #617. Some differences from your suggestions:
|
Addresses posit-dev/positron#16137 ("ggsql: View result of SQL query in data explorer"). No Positron changes are needed.
Following review, the Positron-specific code lives in the extension. The kernel gains a small results API that any frontend can use, and it keeps results in a cache of its own, so the session's reader and dialect are never changed.
How it works
ggsql-jupyter/src/results/). Results are kept only when the frontend asks, with--keep-results/GGSQL_KEEP_RESULTS; the extension sets it for console sessions. A kept result's key goes in theexecute_resultmetadata as{"ggsql": {"result": key}}. The calls are documented inmethods.rs; nothing in the module knows about Positron. The only Positron-specific part of the kernel is the route: acall_methodrequest on the UI comm whose method starts withresult_. Closing the UI comm closes every view, so a frontend that reloads or crashes doesn't leave results pinned.CachingReader's table bookkeeping is pulled out intoCacheTables(src/reader/cache.rs), which takes a byte budget and gains pinning (kept in memory, not in__ggsql_cache_meta__). The results API uses one on its own in-memory DuckDB, so the user's reader is never wrapped. A result is copied in on first open and stays pinned while a view of it is open.--results-max-bytes/GGSQL_RESULTS_MAX_BYTES(default 1 GB) covers unopened results, cached results (pinned ones included) and order tables. Closed results go first, then the oldest unopened ones. The cache never spills to disk, and running out of room is an error rather than a crash.ggsql-vscode/src/dataExplorer/). Registers withpositron.dataExplorer.registerRpcHandlerand translates Positron's protocol to the results API: paging, sorting, row filters, column search, profiles (summary stats, histograms, frequency tables), export (CSV/TSV/HTML) and Convert to Code. Convert to Code writes SQL for the session's own database, not the cache: identifiers are quoted for that dialect, a multi-statement cell keeps its earlier statements, and a filter the database can't run is left out with a comment. A session that has ended or restarted isn't called, so an open tab shows as disconnected instead of hanging, and a kernel without the results API is reported as too old.ggsql-vscode/src/resultLinks.ts). Each ggsql session is wrapped. After a marked result in a console session, the extension adds an OSC 8 "Open in Data Explorer" hyperlink tox-ggsql-result:<session>/<key>, and the wrapper'sopenResourcehandles the click. Clicking a link again focuses the open tab, or reopens one that was closed.ggsql.consoleRowssetting changes the number; it reaches the kernel asGGSQL_CONSOLE_ROWS. Notebooks still show up to 100 rows.Why DuckDB for the results cache
ggsql-jupyter/examples/results_bench.rs, release build, Apple Silicon (14 cores), 10M rows (617 MB of Arrow data). The alternative was to sort and filter the Arrow data directly with row indices.DuckDB uses about twice the memory but sorts 4–8× faster, and a multi-second wait on every sort click isn't acceptable for large results. Memory levels off near 2 GB across 30 open/close cycles rather than growing. #617 speeds up the copy with DuckDB's Arrow appender; it's a separate PR because it affects every DuckDB user.
Not in this PR
format_options, Positron's number-formatting settings.Testing
cargo test -p ggsql-jupyter: 121 unit tests pass, including the results store, views, filters, profiles, eviction and pinning, the memory budget, failed sorts and filters, page bounds, and Convert to Code SQL for MySQL, BigQuery, ClickHouse and Databricks.cargo test -p ggsql --lib: passes except two parser comment tests (test_extract_sql_from_first_skips_line_comment,test_validate_visualise_in_comment_is_valid) that also fail onmain.ggsql-jupyter/tests/test_integration.py:call_methodround trips for open, filter, sort, page, profile, export, SQL and close; an unknown key or method is a JSON-RPC error; results are kept only when asked for; closing the UI comm closes its views; and the session's database gains no tables.ggsql-vscode: extension tests pass in CI on macOS, Linux and Windows, including the Data Explorer backend against a recordedcallMethodstand-in, the session wrapper, opening a result, session end and restart, and concurrent requests sharing one view.Demo
ggsql-data-explorer.mp4
🤖 Generated with Claude Code