Repository navigation
Proxy: a conversation reserves a document's bytes before it reads it - #918
Merged
Merged
Conversation
One upstream conversation keeps at most 4 x proxy.max_metadata_bytes of documents (#230), but a document was counted once it had been downloaded, read into memory, base64-encoded for the local cache, serialized, parsed back and decoded. The four fetches of a round each held their document and those copies beside what the conversation kept already: 19 x max_metadata_bytes measured (28 x with the local cache's copies), where the docs said 4 x. A fetch now reserves its document's bytes in the conversation's budget as soon as it knows how many they are and before it has them in memory: the upstream's answer is on disk then, the stored copy is a row's size, the cached copy is the cache's own. A document that does not fit is not read, and a download stops at what the budget has left. Nothing waits for bytes of that budget, so a conversation never holds a part of it while waiting for the rest. The copies made on the way are down to one, the cache's encoding (4/3 of the document): the body is encoded straight into the cache's JSON, which is allocated at its size, and decoded straight from it. A conversation encodes at most one max_metadata_bytes of documents at a time, so it holds at most 5 1/3 x max_metadata_bytes, which the docs now say. Fixes #601 Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012e6DJae3KEQWi1VkmDNZVw
github-merge-queue
Bot
removed this pull request from the merge queue due to failed status checks
Nov 17, 2026
Member
Author
|
The merge queue removed this PR because I do not think this failure is this PR's:
Nothing is changed on the branch. It needs to be queued again, which I cannot do. Generated by Claude Code |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #601
The problem
One upstream conversation keeps at most 4 ×
proxy.max_metadata_bytesof documents (#230), butconversecounted a document only afterfetchreturned it. I traced one fetched document of size S onmain, from the upstream read to the kept value:proxy.spool_memory_bytesSpooledBody::into_bytesVecS (grown by doubling, so up to 2 S of capacity)CachedFetch::foundString, 4/3 Sserde_json::to_vecVec, 4/3 S (up to 8/3 S of capacity)from_sliceString4/3 SVecS, the value keptSo the issue's "3–4 copies" holds: about 11/3 S at the decode, more while the JSON grows. Four fetches run at once, each doing this beside what the conversation already kept, and a document past the budget was still read, stored and cached before it was dropped.
The change
ConversationBudget(crates/core/src/repo/proxy.rs) replaces the counter inconverse. A fetch reserves its document's bytes as soon as it knows how many they are and before it has them in memory. There are three places: the upstream's answer (spooled to disk, its length known), the stored copy (proxy_document.size), and the local cache's copy (its size is read from the cached JSON without decoding it). A document that does not fit is not read into memory.min(max_metadata_bytes, what the budget has left), so a response with aContent-Lengthover it is not downloaded at all.Stringeither way), and the cache allocates that JSON at its exact size. The only copy beside a document being read is the cache's encoding, 4/3 S.max_metadata_bytesof documents is encoded at a time per conversation. A second large document waits, with its bytes reserved and its body still on disk.docs/deployment.md(Memory),docs/format-plugins.md,forge.toml.exampleand the config's doc comment state the bound below.Deadlock and starvation
The bound, before and after
In multiples of
proxy.max_metadata_bytes(32 MiB by default):Measured with
crates/core/tests/proxy_memory.rs: a test binary with a counting allocator, an in-process upstream serving 8 documents of just undermax_metadata_bytes(4 MiB in the test), and a format asking for all of them in one round.mainmainat 28.5 ×.What remains outside the bound
proxy.spool_memory_bytes(2 MiB) of each stays in memory before it spills to disk, 4 at a time. The docs say so.cache.l1_max_capacity× 16 KiB), as before.proxy.max_metadata_memory_bytes, Proxies: one memory budget for the upstream documents conversations hold #673) is still taken after a document is read:fetch_heldis unchanged apart from carrying the reservation. Across conversations, each can be one document and its encoding over that budget while it finds out the budget is spent.docs/deployment.mdalready says.crates/server/images/proxy.rs) are not touched.Behaviour changes to know about
forge-cachegainsget_or_try_insert_local_encoded_with_tags_expiringand serializes every value twice (once to count its bytes).BlobStore::read_allandSpooledBody::into_bytesallocate the known size at once.#770
#770 is the same code and is not fixed here. This change does not add to it: the conversation's reservation is given back together with the document while
fetch_heldwaits for the instance's budget, and nothing here waits on another conversation. If #770 is fixed by giving back the earlier rounds' holds and reading those documents again, the re-read goes throughfetchand reserves again.No admin API changes, so no admin screen.
Verification
Run locally, with Postgres 18 and versitygw from testcontainers (pulled through
mirror.gcr.io):cargo test -p forge-core -p forge-cacheandcargo test -p forge-storage: all pass, including the newproxy_memorytest, the existing Plugin upstream conversations keep every fetched body for all rounds (up to 256 ×max_metadata_bytes), and a WebAssembly plugin gets all of them copied in each round #230 test (an_upstream_conversation_keeps_at_most_its_byte_budget) and the Proxies: nothing bounds the memory of concurrent upstream conversations, so a cold restore can OOMKill the pod #668 instance-budget test, unchanged.proxy_memoryrun againstmain's sources: fails at 28.5 ×.cargo fmt --all --check,cargo clippy --workspace --all-targets -- -D warnings,cargo xtask changelog --check,cargo xtask docs --check,cargo xtask unused,cargo xtask doc-check.Not run locally: the
forge-serverandforge-e2esuites (the session's disk did not hold the server's test build; CI'stestand e2e jobs run them) and the web checks (nothing underweb/changes). The retry between two conversations reading one document has no test of its own.🤖 Generated with Claude Code
https://claude.ai/code/session_012e6DJae3KEQWi1VkmDNZVw