Repository navigation
Conversation
Each probe merge-patched its node's whole window of samples into the shared fleet-role-probe ConfigMap every sample period. The datastore stores the whole object on every patch, so each write was the size of every node's window together. The probe now keeps its window in its data directory on the node (window.json, replaced by a rename once a sample's summary is published) and publishes only window_summary, the tails, counts and last readings the controller judges by, from the module both share. An entry is held to probe_entry_limit_bytes (4 KiB, above a summary for etcd's largest recommended cluster of seven voters with the longest Node names); a summary over it is logged and not published. The controller judges the summary as it judged the window. Because it can no longer drop the samples from before a membership change, a window that began before the change is unknown until it has rolled past it, which a full window does once 100 samples have been taken since, as before. The probe now needs the regrade limit to count samples over it, so the DaemonSet passes PRESSURE_THRESHOLD and AVAILABLE_FLOOR_PERCENT from the same role variables as the controller. Refs #266
The job-memory and job-peak records were each capped at 2048 entries, which the comment sized at half the ConfigMap limit, so the two together could reach the whole limit, and an entry's size depends on the names in it. Each record is now held to JOB_RECORDS_MAX_BYTES, a thirty-second of CONFIGMAP_LIMIT_BYTES (Kubernetes' 1 MiB), counted in UTF-8 bytes and keeping the newest entries; the first entry that does not fit ends the record. Every patch rewrites the whole ConfigMap in the datastore, so this bounds each write of the object as well as its size. Refs #266
The controller size test filled the probe ConfigMap with windows and the autoscaler test expected a record over ARG_MAX to be kept whole. Both records are now bounded, so the tests assert the bounds: every probe entry within probe_entry_limit_bytes, and an oversized stored job record read whole through a file and written back within JOB_RECORDS_MAX_BYTES. Refs #266
`kubectl patch` reads the object before it patches it, so the probe Role without get had every patch refused and no probe ever published, which the mesh role controller scenarios showed as voters with no probe report. Refs #266
A probe from before summaries holds raw samples in its key until its new pod publishes. The controller now drops any entry whose samples field is not a count, so such a host is unknown and a voter's gates do not hold, never unfit on figures it cannot read. Refs #266
The stored-record test dated its entries in the future, so the newest-first bound rightly kept them over the job seen now; it dates them a minute ago. The probe test measuring itself passed a node name the window reader no longer takes, and the oversized-window test used more voters than etcd recommends, whose summary is rightly refused; it uses the most voters with the longest names. Refs #266
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Refs #266
The role controller's probe merge-patched its node's whole 100-sample window into the shared
fleet-role-probeConfigMap every 15 seconds, and the datastore stores the whole object on every patch, so each node's write was the size of every node's window together. The autoscaler'sjob-memoryandjob-peakrecords were capped by entry count, which let the two together reach the ConfigMap limit.Whether these writes cause the control-plane restarts in #256 is not proven. The restarts follow lease renewals timing out against a slow datastore, and the write size is one candidate contributor. This change removes the large writes; only observation after deployment can show whether the restarts stop, which is why this refers to #266 and does not close it.
Reading taken on 20 November 2026, around 08:50 UTC, in the platform namespace (
kubectl get cm -o jsonbyte counts):fleet-role-probe258,561 bytes,autoscaler-status315,256 bytes.Probe. The probe keeps its window in its data directory on the node (
window.json, replaced by a rename once a sample's summary is published, so a replaced pod loses nothing) and publishes onlywindow_summary(inscripts/fleet-role-stats.jq, shared with the controller): the TCP tail and reading count per peer, etcd's round-trip tail per member id, the WAL, commit and fdatasync tails, the memory and restart counts, and the last etcd gauges. An entry is held toprobe_entry_limit_bytes(4 KiB, above the summary for etcd's largest recommended cluster of seven voters, each with a 253-character Node name); a summary over it is logged and not published. The probe needs the regrade limit to count samples over it, so the DaemonSet now passesPRESSURE_THRESHOLDandAVAILABLE_FLOOR_PERCENTfrom the same role variables the controller gets. The probe no longer reads its entry back from the ConfigMap; its Role keepsgetbecausekubectl patchreads the object before patching it.Controller.
scripts/fleet-role.jqjudges the summary with the same thresholds. An entry that is not a summary (raw samples left by a probe from before this change, until its new pod publishes) is read as no report, so the host isunknownand a voter's gates do not hold, neverunfit. One behaviour changes: it can no longer drop the samples from before a membership change, so a window that began before the change isunknownuntil it has rolled past it. A full window rolls past once 100 samples have been taken since, which is when the old filter would also have had a full window, so the time to a verdict after a change is the same.Autoscaler. Each per-job record is held to
JOB_RECORDS_MAX_BYTES, a thirty-second ofCONFIGMAP_LIMIT_BYTES(1 MiB), counted in UTF-8 bytes, newest entries first, and the first entry that does not fit ends the record.Regression checks.
PublishedEntryTestintests/unit/test_fleet_role_probe.pypublishes a full window to seven long-named voters and asserts the entry is withinprobe_entry_limit_bytesand equals the summary of the kept window; it asserts the entry does not grow with the window, and that an entry over the limit is refused and not published.test_each_job_record_is_held_to_its_share_of_the_configmap_newest_firstintests/unit/test_autoscaler.pyfeeds both records several times their budget and asserts each stays inside it, keeps the newest entries and drops the oldest, and that the whole stored object stays inside an eighth of the limit. All the bounds come from named constants, not from today's sizes. The probe's growth test failed before the change (the entry grew by 2,003 bytes over a 99-sample window). The autoscaler test was written before its implementation but not run against the old code.Design check. Premises: no new tooling, the probe and controller already share one jq module. Strongest form: the bounds are executable tests and a run-time refusal in the probe, not prose. Storage form: I kept one key per node in the existing ConfigMap. A per-node Lease or a Node annotation is the native home for per-node heartbeat status, but the probe pods share one ServiceAccount, and RBAC
resourceNamescannot name "this pod's node", so either would needcreateon Leases in the namespace orpatchon every Node, both wider than today'spatchon one named ConfigMap. Per-node ConfigMaps have the same problem. With small summaries, the shared object's whole size is bounded byprobe_entry_limit_bytestimes the measured nodes. Trust boundary: any probe's token can still patch any key of the shared object, as before, so a hostile or faulty node could write an oversized or false entry for itself or another node. Closing that needs an admission check (a ValidatingAdmissionPolicy that matches the key against the token's node-name claim and bounds its size), which this PR does not include.