Skip to content

Open console query results in Positron's Data Explorer - #595

Open
samclark2015 wants to merge 2 commits into
mainfrom
sclark/ggsql-data-explorer
Open

samclark2015 wants to merge 2 commits into
mainfrom
sclark/ggsql-data-explorer

Conversation

@samclark2015

@samclark2015 samclark2015 commented Oct 20, 2026 •

Copy link
Copy Markdown
Contributor

Addresses posit-dev/positron#16137 ("ggsql: View result of SQL query in data explorer"). No Positron changes are needed.

Following review, the Positron-specific code lives in the extension. The kernel gains a small results API that any frontend can use, and it keeps results in a cache of its own, so the session's reader and dialect are never changed.

How it works

Positron Data Explorer
      │  Data Explorer protocol (get_state, get_data_values, set_sort_columns, …)
      ▼
ggsql-vscode   src/dataExplorer/   registerRpcHandler('ggsql', …)
      │  session.callMethod('result_page', …)  →  call_method on the positron.ui comm
      ▼
ggsql-jupyter  src/results/        result_open / _page / _set_sort / _set_filters / _profile / _export / _sql / _close
      │
      ▼
in-memory DuckDB results cache     (CacheTables from ggsql::reader::cache, with pinning)
  • Kernel: results API (ggsql-jupyter/src/results/). Results are kept only when the frontend asks, with --keep-results / GGSQL_KEEP_RESULTS; the extension sets it for console sessions. A kept result's key goes in the execute_result metadata as {"ggsql": {"result": key}}. The calls are documented in methods.rs; nothing in the module knows about Positron. The only Positron-specific part of the kernel is the route: a call_method request on the UI comm whose method starts with result_. Closing the UI comm closes every view, so a frontend that reloads or crashes doesn't leave results pinned.
  • Cache reuse. CachingReader's table bookkeeping is pulled out into CacheTables (src/reader/cache.rs), which takes a byte budget and gains pinning (kept in memory, not in __ggsql_cache_meta__). The results API uses one on its own in-memory DuckDB, so the user's reader is never wrapped. A result is copied in on first open and stays pinned while a view of it is open.
  • Sorting, filtering and paging. Sorting and filtering build an order table of row ids, so scrolling never sorts again. A new order table replaces the old one only once it's built, so a sort or filter that fails leaves the view as it was. A page is at most 10,000 rows, and a row index outside the view is an error rather than a shifted column.
  • One memory budget. --results-max-bytes / GGSQL_RESULTS_MAX_BYTES (default 1 GB) covers unopened results, cached results (pinned ones included) and order tables. Closed results go first, then the oldest unopened ones. The cache never spills to disk, and running out of room is an error rather than a crash.
  • Extension: Data Explorer backend (ggsql-vscode/src/dataExplorer/). Registers with positron.dataExplorer.registerRpcHandler and translates Positron's protocol to the results API: paging, sorting, row filters, column search, profiles (summary stats, histograms, frequency tables), export (CSV/TSV/HTML) and Convert to Code. Convert to Code writes SQL for the session's own database, not the cache: identifiers are quoted for that dialect, a multi-statement cell keeps its earlier statements, and a filter the database can't run is left out with a comment. A session that has ended or restarted isn't called, so an open tab shows as disconnected instead of hanging, and a kernel without the results API is reported as too old.
  • Extension: the link (ggsql-vscode/src/resultLinks.ts). Each ggsql session is wrapped. After a marked result in a console session, the extension adds an OSC 8 "Open in Data Explorer" hyperlink to x-ggsql-result:<session>/<key>, and the wrapper's openResource handles the click. Clicking a link again focuses the open tab, or reopens one that was closed.
  • Shorter console tables. Like R's tibble, a result of up to 20 rows shows in full and a longer one shows its first 10, so the link stays in view. The ggsql.consoleRows setting changes the number; it reaches the kernel as GGSQL_CONSOLE_ROWS. Notebooks still show up to 100 rows.

Why DuckDB for the results cache

ggsql-jupyter/examples/results_bench.rs, release build, Apple Silicon (14 cores), 10M rows (617 MB of Arrow data). The alternative was to sort and filter the Arrow data directly with row indices.

Step Arrow row indices DuckDB cache
Open (copy in) none 3.4 s (1.5 s with #617)
Sort, 1 key (stable) 2.3 s 0.41 s
Sort, 2 keys 4.3–5.0 s 0.51–0.61 s
Filter, contains 116 ms 33 ms
Filter, numeric range 10 ms 92 ms
Filter range, then sort 475 ms 142 ms
Page 100 rows from a sorted view 0.2 ms 18–21 ms
Memory, one result open ~0.8 GB peak ~1.5 GB peak

DuckDB uses about twice the memory but sorts 4–8× faster, and a multi-second wait on every sort click isn't acceptable for large results. Memory levels off near 2 GB across 30 open/close cycles rather than growing. #617 speeds up the copy with DuckDB's Arrow appender; it's a separate PR because it affects every DuckDB user.

Not in this PR

  • Notebooks still get the static HTML table, not Positron's inline data grid.
  • The explorer ignores format_options, Positron's number-formatting settings.
  • Column filters, row labels and filter conditions (OR) are reported as unsupported.
  • DuckDB's own working memory during a sort isn't counted in the budget.

Testing

  • cargo test -p ggsql-jupyter: 121 unit tests pass, including the results store, views, filters, profiles, eviction and pinning, the memory budget, failed sorts and filters, page bounds, and Convert to Code SQL for MySQL, BigQuery, ClickHouse and Databricks.
  • cargo test -p ggsql --lib: passes except two parser comment tests (test_extract_sql_from_first_skips_line_comment, test_validate_visualise_in_comment_is_valid) that also fail on main.
  • ggsql-jupyter/tests/test_integration.py: call_method round trips for open, filter, sort, page, profile, export, SQL and close; an unknown key or method is a JSON-RPC error; results are kept only when asked for; closing the UI comm closes its views; and the session's database gains no tables.
  • ggsql-vscode: extension tests pass in CI on macOS, Linux and Windows, including the Data Explorer backend against a recorded callMethod stand-in, the session wrapper, opening a result, session end and restart, and concurrent requests sharing one view.
  • Two exploratory passes in a Positron dev build found eight bugs, all fixed with tests: the link opened an empty tab, dates showed as day counts, sorting failed after a console restart, standard deviation lost precision on large values and broke on NaN, duplicate column names stopped a result opening, Convert to Code kept filters the database couldn't run, and nested-type profiles never finished loading.

Demo

ggsql-data-explorer.mp4

🤖 Generated with Claude Code

@thomasp85

Copy link
Copy Markdown
Collaborator

Most of the logic of this PR is implemented in the Jupyter kernel and I'm not super fond of that if it can be avoided at all. For the kernel to contain a module so closely tied to positron feels like a code smell. Can you take another round at this with the goal of keeping it as confined as possible to the VS code extension?

Also, we plan to spin out the VS Code extension to its own repo that you can have more control over once this is in (just an FYI :-) )

@samclark2015

Copy link
Copy Markdown
Contributor Author

Sweet! Thanks for this feedback. I churned on it with the 🤖 and came up with this proposal, which moves Positron-specific code into the extension. Kernel additions are minimal and client-agnostic. Let me know how this sounds, and I'll spin up a revision.

Proposal: the kernel offers a general results API, and the extension serves the Data Explorer

Positron Data Explorer
      │  get_data_values, set_sort_columns, …     (Positron's protocol)
      ▼
ggsql-vscode   positron.dataExplorer.registerRpcHandler('ggsql', …)
      │  translates each request into a general ggsql call
      │  session.callMethod('result_page', view, rows, cols)
      ▼
ggsql-jupyter  results API
      answers with only the requested page; the data stays in the kernel

Kernel: a general results API, with no Positron logic.

  • The kernel keeps recent query results as the Arrow batches it already has, up to a limit.
  • It offers a small set of neutral operations on them:
    • result_open(key) returns a view id, the schema and the shape
    • result_set_sort and result_set_filters
    • result_page(view, rows, cols) returns plain values
    • result_profile, result_export and result_close
  • Sorting and filtering produce a list of row indices over the batch, as R's and Python's Data Explorer backends do. The result is never copied, and scrolling only reads the rows on screen.
  • These operations reach the extension through call_method on the positron.ui comm, which the kernel already serves. It's Positron's existing route for an extension to call into a kernel; session.callMethod, which positron-r uses to call into the R kernel, goes through it. That one dispatch point is the only Positron-specific code involved. The operations themselves don't mention Positron, so other frontends could use them later, for example JupyterLab through a comm of its own, or the CLI.

Extension: everything specific to Positron's Data Explorer.

  • It registers as a Data Explorer backend through Positron's extension API (positron.dataExplorer.registerRpcHandler), the same way Positron's Postgres and SQLite drivers do.
  • It translates each request into the operations above. It also handles everything that follows Positron's protocol: mapping column types, formatting values, the shape of column profiles, Convert to Code, and focusing a tab that's already open.
  • It handles the "Open in Data Explorer" link in the console. The kernel prints that link only when the extension asks for it.
  • All of this moves to the new repo with the extension.

Why this shape

  • Performance on large results. Only the visible page crosses between processes, and the result is never copied. The alternative was to send the whole result to the extension and keep a second copy there. Positron's existing backends avoid that: R and Python keep the data in the kernel, and the database drivers query the database directly. Here the kernel plays the role the database plays for the drivers.
  • Less in the kernel. This PR's kernel-side Data Explorer work goes away: the filter and export modules, the private DuckDB copy of results, the @view-result and @view directives, and the comm and tab-focus handling. The kernel's existing Data Explorer for the Connections pane isn't touched.

Tradeoffs

  • Data Explorer requests reach the kernel through the same request channel as code execution, so they wait while a long query runs. The comm this PR adds has the same limitation, and so does R's Data Explorer.
  • Each request takes one more hop (kernel to extension to Positron) than a comm served by the kernel. The database drivers have the same shape, and I don't expect it to be noticeable.

Scope

Both directives go. The extension calls result_open in place of -- @view-result, and drops -- @view. A result is opened from its console link.

@thomasp85

Copy link
Copy Markdown
Collaborator

I think this is a better shape in general. If possible, it would be nice to reuse the cache mechanism already in place in ggsql, however, care should be taken to not enforce a cached reader on the user (because that change the dialect for them). So, the cache reader should only be used for the results API

Console query results get a link that opens them in Positron's Data
Explorer. The kernel keeps each result in a dedicated in-memory DuckDB
results cache (sharing `CacheTables` with `CachingReader`, with pinning so
open results are not evicted; the user's reader is never wrapped) and
exposes a frontend-agnostic results API over `call_method`. The VS Code
extension registers the Data Explorer RPC handler, adds the console link,
and pages, sorts, filters and profiles results through that API.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@samclark2015
samclark2015 force-pushed the sclark/ggsql-data-explorer branch from 307d520 to 6711156 Compare November 17, 2026 20:21
@thomasp85

Copy link
Copy Markdown
Collaborator

The shape of this refactor is great - basically how I envisioned it.

Below are some correctness issues found and summarised together with Kimi for you to look through:

High

  1. Failed sort/filter leaves a view broken (results/view.rs:162-206). New filters/sort are saved before rebuild, which drops the order table and sets ordered=false. If the rebuild fails (OOM sort, a predicate invalid only in combination), pages come back unsorted/unfiltered, selected_rows is stale, and result_sql exports the failed sort. The old state should be kept or restored on error.
  2. One result_page call can abort the kernel (view.rs:30-35, 275-288). Rows::Range materializes every index from first..last — a huge last allocates gigabytes. Scattered indices fetch everything between min and max. And out-of-range indices are silently dropped, so the returned column is shorter than requested and later values land in the wrong rows.
  3. The memory bound doesn't hold.
    • Unopened results are capped by count only (--max-results 16); sixteen large SELECT * results sit in RAM where before they were freed after display. Retention is also gated on SessionKind::PositronConsole (kernel.rs:429) rather than the frontend opting in — per intent 1/3, the extension should turn it on (like GGSQL_CONSOLE_ROWS).
    • Pinned entries are exempt from the budget (cache.rs:332-360), and per-view sort tables (view.rs:200-203) live outside CacheTables entirely, so neither limit counts them.

Medium

  1. result_sql emits wrong SQL for several databases (filter.rs:136, 247, 263, 307). It uses naming::quote_ident (double quotes) instead of the session dialect's quoting — wrong on MySQL, ClickHouse, BigQuery, Databricks, Drill — and ESCAPE '\' is an unterminated string on MySQL/Snowflake. Multi-statement cells get wrapped as one subquery (invalid), and deduplicated column names (a_1) don't exist in the original query. (The PR body's "Not in this PR" section acknowledges the multi-statement case, but the quoting bugs are new and silent.) Also decide whether result_sql belongs in the "neutral" API at all — it exists only for Convert to Code*
  2. A view that is never closed pins its result forever (mod.rs:174, 241-251). Extension reload/crash or a missed result_close leaks the pin; no fallback like releasing views when the UI comm closes. result_open twice on one key also makes two views. Related extension-side leaks: openResult opens the view then loses it if dataExplorer.open rejects (resultLinks.ts:73-85), and requests after closeDataset reopen a view nobody closes (handler.ts:221-227).
  3. Explorer hangs after kernel exit (not restart) — extension side. callMethod requests made while no UI comm is open queue forever; on exit, forgetViews (resultLinks.ts:116) triggers schema_update → get_state → call (handler.ts:229-235) which never returns. An open tab spins instead of showing the disconnected state. Fix by marking the session unavailable on onDidEndSession (unless Restart) / re-available on Ready, or a callMethod timeout.
  4. CacheTables API is too wide and under-checked (cache.rs). ensure_meta_table, touch, drop_entry, evict_over_budget, clear are all pub, but only ensure_meta_table has an external caller (a kernel test); touch/drop_entry/evict_over_budget don't call it, so they error on a fresh instance. It also takes the whole CacheConfig while only using max_bytes, forcing the kernel to pass dummy enabled: true, ttl_secs: u64::MAX (results/mod.rs:101-105) — pass a byte budget instead. And in_memory silently falls back to SQLite without the duckdb feature, while the kernel then sends DuckDB-only SQL (SET temp_directory, DESCRIBE).
  5. kind_of column typing in the kernel is fragile (mod.rs:305-339). Any type name containing "INT" is an integer, so POINT… / ENUM('PRINT',…) are treated as numbers and their filters rejected. (Arguably this mapping belongs in the extension per your intent, but at minimum it needs real type matching.)

Low

  • Link scheme mismatch: the code uses x-ggsql-result:, not x-ggsql-view: as the PR body says (test even rejects x-ggsql-view as "the old one"). Body should be updated.
  • consoleRowsEnv has no upper bound (manager.ts:546-550): values above u32::MAX (or 1e21 → "1e+21") pass validation and the kernel then refuses to start — the exact failure the env-var indirection exists to avoid. Add maximum: 4294967295 in package.json.
  • DuckDB appender rewrite is unrelated (duckdb.rs:167-181, 305-327 + appender-arrow feature): a performance change affecting all DuckDB users, with the appender error swallowed by .is_ok() and a stale comment at duckdb.rs:272. I'd split it into its own PR. (odbc/mod.rs:167 is a harmless drive-by.)
  • Unknown call_method names now error instead of result: null (kernel.rs:762) — contradicts ggsql-jupyter/AGENTS.md:188.
  • Capability check by URI prefix: reader_uri().starts_with("duckdb://") (kernel.rs:757) should come from the dialect.
  • Old-kernel interop: pre-PR kernels answer call_method with null → "Cannot read properties of null" (handler.ts:214). A shape check with a "kernel too old" message would be kinder.
  • Small stuff: unused regex dep in ggsql-jupyter/Cargo.toml:42; pinned column now visible in user-queryable __ggsql_cache_meta__; plain-text console output ignores console_rows (display.rs:351-366); opened map bookkeeping never cleaned up; misplaced comment at kernel.rs:424-426; CHANGELOG cites positron#16137 instead of Open console query results in Positron's Data Explorer #595 and buries the 10-row console truncation behavior change in an "Added" bullet.

Test gaps

  • Nothing asserts the session DB is untouched after result_open (no __ggsql_result_* tables in SHOW TABLES) — that's your dialect-isolation concern, tested.
  • No comm-level tests for result_profile/result_export; result_sql only tested with ANSI/DuckDB quoting.
  • Extension: the openResult success path, onDidEndSession lifecycle, exited-but-not-disposed session, and concurrent-open dedup are untested.

- A failed sort or filter leaves its view as it was: the new order table is
  built under a new name and only then replaces the old one.
- Pages are bounded (MAX_PAGE_ROWS), scattered indices fetch just those rows,
  and an index past the end is an error rather than a shifted column.
- One byte budget (--results-max-bytes) covers unopened results, cached
  ones including pinned, and order tables. Keeping results is opt-in
  (--keep-results / GGSQL_KEEP_RESULTS), which the extension sets for
  console sessions.
- result_sql quotes for the session dialect, escapes LIKE with '!', keeps a
  cell's earlier statements, and leaves out repeated column names. Regex
  support comes from SqlDialect::sql_regex_match.
- Views close with the UI comm. The extension releases a view when the Data
  Explorer fails to open, ignores requests after close, and never calls a
  session that is down.
- CacheTables takes a byte budget, keeps pins in memory, and has a narrower
  API; the kernel builds its own DuckDB backend for it.
- kind_of matches DuckDB type names exactly.
- The DuckDB Arrow appender change moves to its own PR.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@samclark2015

Copy link
Copy Markdown
Contributor Author

@thomasp85 Thanks for this!! Everything is addressed, and the appender is split out into #617.

Some differences from your suggestions:

  • result_sql stays in the results API. "The SQL that reproduces this view against your own database" isn't specific to Positron: a CLI export or a notebook's copy-as-SQL could use it too, and Convert to Code is just the first caller. It also has to stay in the kernel, since writing it needs the session's dialect, the filter-to-SQL code the cache already uses, and the parser to split multi-statement cells.
  • Column kinds (kind_of) stay in the kernel. It now matches type names exactly, but the neutral kinds are part of what lets any frontend use the API.
  • result_open twice on one result still gives two views, each with its own sort and filters. The extension shares one view per Data Explorer tab, so it doesn't leak.
  • An unknown call_method is still an error, like the plot comm's unknown methods. I changed AGENTS.md to match instead.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants