Skip to content

feat(server): read an HTTP source's JSON answer while it streams - #647

Open
eizedev wants to merge 4 commits into
mainfrom
feat/http-source-stream
Open

eizedev wants to merge 4 commits into
mainfrom
feat/http-source-stream

Conversation

@eizedev

@eizedev eizedev commented Nov 19, 2026 •

Copy link
Copy Markdown
Owner

An HTTP source now reads a JSON answer while it arrives. Each row becomes a table row at once. The server no longer holds a page's whole body, its parsed objects and the table at the same time.

What changed

  • sources/json-rows.js scans the body as bytes. It parses each row on its own with JSON.parse. It keeps the body around the rows and parses it at the end, so asOf, next and check-source read it as before.
  • guardedFetch gets an opt-in consume option. It hands the decoded body to the caller in pieces. The byte count, the time limit, cancelling, redirects and the address checks stay as they are. Other callers do not use it.
  • tableBuilder in sources/http.js turns each row into a table row when it arrives. tableOf uses it too.
  • Hidden columns are dropped in place before the rows are kept, not in a copy.
  • Where asOf or next points into the rows, the page is read whole, as before.
  • A body that repeats a key on the way to the rows after them is refused. JSON.parse would take the later value, which is only known after the rows were read.
  • CSV is not streamed. The CSV reader needs the whole text for the encoding, the separator and the header. I tried reading a CSV body into one buffer of its stated length. It saved about 8 MB at 34 MB, which is within the noise, so I left it out.

Measurements

bench/server/http-source.js runs a fake backend in its own process and one describe_source call per new server process. Peak RSS is the growth of the server process during the call, the workers included. Peak heap is the main thread's, seen at garbage collections. Node 26 on one macOS laptop, seven columns with one hidden, median of three runs.

Answer Rows Size Peak RSS before after Main heap before after
JSON 10,000 1 MB 80 MB 69 MB 32 MB 23 MB
JSON 100,000 13 MB 302 MB 247 MB 76 MB 66 MB
JSON 151,550 20 MiB 387 MB 299 MB 97 MB 77 MB
JSON 500,000 66 MB 913 MB 591 MB 256 MB 161 MB
CSV 100,000 7 MB 239 MB 236 MB 20 MB 20 MB
CSV 293,222 20 MiB 458 MB 441 MB 20 MB 20 MB
CSV 500,000 34 MB 637 MB 636 MB 20 MB 20 MB

Of the 322 MB saved at 500,000 JSON rows, about 260 MB come from streaming and about 60 MB from dropping hidden columns in place. The rest of the peak is the kept table and the worker that computes the report. The time of the call stayed about the same (1174 ms before, 1106 ms after at 500,000 rows). CSV is unchanged within the noise.

Tests

  • test/json-rows.test.js reads about 50 answers in pieces from one byte up and compares each with JSON.parse of the whole body and the pointer. A seeded test changes those answers at random 3000 times and compares again, in pieces of 1, 2 and 3 bytes and whole. It also covers consume: pieces, gzip, maxBytes mid-body, a consumer's error, a redirect, a cancelled signal and timeMs.
  • test/http-sources.test.js runs the limit tests in both readings, streamed and whole: maxBytes and timeMs over all pages, limits.rows, limits.columns, limits.importCells, depth and a body that is no JSON. A new test fails an answer mid-body (past maxBytes, past timeMs, a closed connection, broken gzip) and checks that nothing is kept and the next call loads every row.
  • The new reader tests and the repeated-rows test failed before the change. The limit tests passed before and after.
  • An independent review and a follow-up review ran before and after the fixes. They found a byte order mark at a row's start and a repeated key after the rows, both accepted where JSON.parse refuses them or finds no rows. Both are fixed with tests that failed before.
  • npm test -w pivotkit-server (781 pass), node --test tests/*.test.js (110 pass), lint, types and npm run pack:check. Only CI runs Linux and Windows.

Docs checked: docs/reference/sources.md (reading the answer, a new section on memory), docs/getting-started/kubernetes.md (sizing, with the numbers and the bench command), the server AGENTS.md, the root AGENTS.md (the bench folder), the server CHANGELOG.md, and the header comments of http.js and fetch.js. The upgrading page needs no entry. Only a backend that repeats a key on the way to the rows would notice.

Fixes #620

🤖 Generated with Claude Code

eizedev and others added 3 commits November 19, 2026 05:15
A JSON answer's rows now become table rows as they arrive (rowsReader in
sources/json-rows.js, fed by guardedFetch's new consume option), so the
server never holds a page's whole body, its parsed objects and the table
at once. The body around the rows is parsed at the end, so asOf, next and
check-source read it as before. Where asOf or next points into the rows,
the page is read whole as before. Hidden columns are dropped in place.

All limits keep their messages: maxBytes and timeMs over all pages,
limits.rows (counted, not kept, past the limit), columns and importCells.
A body with its rows' key twice in one object is refused.

bench/server/http-source.js measures peak RSS and heap against a fake
backend. With 500,000 JSON rows the peak RSS drops from 913 to 591 MB.
CSV answers are unchanged: the reader needs the whole text.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A row that started with a byte order mark was read, while JSON.parse of the
whole body refuses it. The reader now skips the mark at the body's start
only. Adds a seeded mutation test against JSON.parse, consume tests for a
redirect, a cancelled signal and timeMs, and before/after numbers in the
changelog.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…nswer

A repeated key after the rows ("items":[...],"items":5) kept the
streamed rows, while JSON.parse of the whole body has none. Any value on
the pointer's path after the rows is now refused. The mutation test uses
xorshift32 and pieces of 1, 2 and 3 bytes. The sizing docs show 10,000
rows and the main thread's heap, and say how the benchmark measures.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

server: read HTTP sources as a stream instead of whole pages

1 participant