Skip to content

feat(server): rows of a source for a signed-in person (records) - #635

Open
eizedev wants to merge 8 commits into
mainfrom
feat/records-route
Open

eizedev wants to merge 8 commits into
mainfrom
feat/records-route

Conversation

@eizedev

@eizedev eizedev commented Nov 19, 2026

Copy link
Copy Markdown
Owner

Fixes #615

This adds the first piece of shared live reports: POST /api/v1/sources/records hands the rows of one source to a person who signed in, so a host application's backend can show a live report over a source only pivotkit-server reaches. It stores nothing and it is off by default. Record 0015 is set to accepted.

What changes

  • A profile grants the right per source: "records": ["sales"]. The server's policy, a source's policy and files.policy refuse the key.
  • Only a person gets rows: an OAuth token of the configured issuer with a subject, or an external issuer's token with a subject. Static and signed tokens, an OAuth token without a subject and a server without sign-in get 403 NOT_ALLOWED. A source the profile does not name answers the same, whether it exists or not.
  • Only fields at show leave. A field at values or summary answers 403 NOT_ALLOWED with notShown. A hidden field answers 400 INVALID_INPUT exactly as a field that does not exist, and no answer names it. A view only says which fields are needed; its filters are not applied.
  • The answer is { columns, types, rowCount, dataAsOf, withheld, requestId, data }, the columnar shape encode reads. Dates are ISO text without an offset (wall clock).
  • The policy decides in the main thread. The compute worker that holds the data posts a head (columns, types, row count, a view's fields), then writes the chosen columns one piece per read of the answer (records-writer.js). The main thread holds at most one piece.
  • Limits: rest.records.maxRows (50,000), .maxCells (1,000,000), .concurrency (1, one per person), .idleMs (30 s) and sources.<id>.records.maxRows, which cannot be higher than the server's. 413 LIMIT_EXCEEDED comes before the first byte, 503 BUSY with Retry-After beyond the places. The defaults are a first guess, as the issue says.
  • GET /sources adds "records": true per source the caller may read. MCP gets no tool.
  • check-config gets a Rows (REST) section: per profile the sources and the fields that leave or stay; a problem for an unknown source id and for a static or signed token whose profile reads rows; warnings for a source without a show field, a profile whose default is show, and rest.records.concurrency not below compute.workers.
  • A records log line per answer: source, rows, columns, profile, the person as audit.subject says. A cut answer is a warning with reason (idle, client, timeout, error).

Measurement

npm run bench:records -w pivotkit-server, Node 26, Apple M4, 10 cores. One process per row, the REST handler in process, each answer read to its end. Cold: a fresh server reads the CSV. Warm: the worker kept the data.

Rows Answer CPU rows, cold / warm CPU render, cold / warm RSS rows, cold / warm RSS render, cold / warm Main heap rows, warm
10,000 0.8 MiB 197 ms / 18 ms 205 ms / 7 ms 43 MiB / 11 MiB 45 MiB / 0 MiB 0 MiB
50,000 4.2 MiB 427 ms / 61 ms 403 ms / 7 ms 101 MiB / 9 MiB 98 MiB / 1 MiB 0 MiB
100,000 8.3 MiB 694 ms / 118 ms 617 ms / 25 ms 134 MiB / 5 MiB 142 MiB / 2 MiB 0 MiB

The issue expected a rows request to cost less CPU than a render. Cold, both cost about the same, since reading the source dominates. Warm, a rows request costs more CPU in pivotkit-server than a render of the same source (61 ms against 7 ms at 50,000 rows). The pivoting then moves to the browser, which this does not measure. 100,000 rows needed raised limits.

Tests

  • test/records.test.js (26 tests), each item of "Done when" on its own: show fields only and the encode round trip; values/summary/hide never in an answer or error, also via fields and a view; NOT_ALLOWED for every caller without a person and without the right; a claim-bound parameter; the three LIMIT_EXCEEDED cases with their messages; a source limit above the server's; BUSY beyond the cap and per person; a stalled reader losing its place; pieces; one piece per read; two people with different rights; the base headers on the answer and every error, and Vary: Origin; the GET /sources flag; the log line; the four check-config items; the writer against pivotkit-core's data export; spellings of a hidden field, a view as JSON text, a reader that leaves, and a column without a header in check-config.
  • The tests failed before the code (20 of 21 at the time; the options fragment came first).
  • npm test -w pivotkit-server, npm test -w pivotkit-mcp, the repository tests, npm run lint, npm run types, npm run pack:check and zensical build --strict pass locally on macOS. Linux, Windows and the container image only run in CI.

Open points

  • With the default concurrency of 1, one person who loads or reads slowly keeps others at 503 BUSY for up to compute.timeMs. The docs say so. A shorter write limit or a short wait for a place would help; I left that for a follow-up.
  • auth.toolScopes cannot name the route, since it is no operation. rest.requiredScopes and the source's scopes apply.

Docs checked

REST reference (route table, profile keys, #records with setup, who, fields, limits, the host recipe, log and sizing; the new #headers; limits and errors), OpenAPI, sources, security, privacy (overview row and the "leaves or stays" table), upgrading, features, what reaches an AI model, embed recipes, the package README, CHANGELOG, the package AGENTS.md, record 0015 and its index. I searched docs/ for rest.concurrency, results: true and the REST route list.

The upgrading section and the CHANGELOG entry sit where the release PR #634 also writes, so one of the two needs a merge of main.

🤖 Generated with Claude Code

eizedev and others added 7 commits November 19, 2026 02:46
POST /api/v1/sources/records answers the rows of one source to a person
whose profile names the source in records: only fields at show, in the
columnar shape encode() reads, written in pieces by the compute worker
that holds the data. Off by default.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… and a measurement

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… answer; accept record 0015

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ot send

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…read

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…onfig, privacy and host docs

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… route does; docs and tests from the reviews

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

server: rows of a source for a signed-in person (records), off by default

1 participant