Repository navigation
Conversation
A JSON answer's rows now become table rows as they arrive (rowsReader in sources/json-rows.js, fed by guardedFetch's new consume option), so the server never holds a page's whole body, its parsed objects and the table at once. The body around the rows is parsed at the end, so asOf, next and check-source read it as before. Where asOf or next points into the rows, the page is read whole as before. Hidden columns are dropped in place. All limits keep their messages: maxBytes and timeMs over all pages, limits.rows (counted, not kept, past the limit), columns and importCells. A body with its rows' key twice in one object is refused. bench/server/http-source.js measures peak RSS and heap against a fake backend. With 500,000 JSON rows the peak RSS drops from 913 to 591 MB. CSV answers are unchanged: the reader needs the whole text. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A row that started with a byte order mark was read, while JSON.parse of the whole body refuses it. The reader now skips the mark at the body's start only. Adds a seeded mutation test against JSON.parse, consume tests for a redirect, a cancelled signal and timeMs, and before/after numbers in the changelog. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…nswer
A repeated key after the rows ("items":[...],"items":5) kept the
streamed rows, while JSON.parse of the whole body has none. Any value on
the pointer's path after the rows is now refused. The mutation test uses
xorshift32 and pieces of 1, 2 and 3 bytes. The sizing docs show 10,000
rows and the main thread's heap, and say how the benchmark measures.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
An HTTP source now reads a JSON answer while it arrives. Each row becomes a table row at once. The server no longer holds a page's whole body, its parsed objects and the table at the same time.
What changed
sources/json-rows.jsscans the body as bytes. It parses each row on its own withJSON.parse. It keeps the body around the rows and parses it at the end, soasOf,nextandcheck-sourceread it as before.guardedFetchgets an opt-inconsumeoption. It hands the decoded body to the caller in pieces. The byte count, the time limit, cancelling, redirects and the address checks stay as they are. Other callers do not use it.tableBuilderinsources/http.jsturns each row into a table row when it arrives.tableOfuses it too.asOfornextpoints into the rows, the page is read whole, as before.JSON.parsewould take the later value, which is only known after the rows were read.Measurements
bench/server/http-source.jsruns a fake backend in its own process and onedescribe_sourcecall per new server process. Peak RSS is the growth of the server process during the call, the workers included. Peak heap is the main thread's, seen at garbage collections. Node 26 on one macOS laptop, seven columns with one hidden, median of three runs.Of the 322 MB saved at 500,000 JSON rows, about 260 MB come from streaming and about 60 MB from dropping hidden columns in place. The rest of the peak is the kept table and the worker that computes the report. The time of the call stayed about the same (1174 ms before, 1106 ms after at 500,000 rows). CSV is unchanged within the noise.
Tests
test/json-rows.test.jsreads about 50 answers in pieces from one byte up and compares each withJSON.parseof the whole body and the pointer. A seeded test changes those answers at random 3000 times and compares again, in pieces of 1, 2 and 3 bytes and whole. It also coversconsume: pieces, gzip,maxBytesmid-body, a consumer's error, a redirect, a cancelled signal andtimeMs.test/http-sources.test.jsruns the limit tests in both readings, streamed and whole:maxBytesandtimeMsover all pages,limits.rows,limits.columns,limits.importCells, depth and a body that is no JSON. A new test fails an answer mid-body (pastmaxBytes, pasttimeMs, a closed connection, broken gzip) and checks that nothing is kept and the next call loads every row.JSON.parserefuses them or finds no rows. Both are fixed with tests that failed before.npm test -w pivotkit-server(781 pass),node --test tests/*.test.js(110 pass), lint, types andnpm run pack:check. Only CI runs Linux and Windows.Docs checked:
docs/reference/sources.md(reading the answer, a new section on memory),docs/getting-started/kubernetes.md(sizing, with the numbers and the bench command), the serverAGENTS.md, the rootAGENTS.md(the bench folder), the serverCHANGELOG.md, and the header comments ofhttp.jsandfetch.js. The upgrading page needs no entry. Only a backend that repeats a key on the way to the rows would notice.Fixes #620
🤖 Generated with Claude Code