Skip to content

fix: publish the probe window's summary and bound the autoscaler's per-job records - #267

Draft
Mearman wants to merge 6 commits into
mainfrom
fix/probe-summary-configmap
Draft

Mearman wants to merge 6 commits into
mainfrom
fix/probe-summary-configmap

Conversation

@Mearman

@Mearman Mearman commented Nov 20, 2026 •

Copy link
Copy Markdown
Member

Refs #266

The role controller's probe merge-patched its node's whole 100-sample window into the shared fleet-role-probe ConfigMap every 15 seconds, and the datastore stores the whole object on every patch, so each node's write was the size of every node's window together. The autoscaler's job-memory and job-peak records were capped by entry count, which let the two together reach the ConfigMap limit.

Whether these writes cause the control-plane restarts in #256 is not proven. The restarts follow lease renewals timing out against a slow datastore, and the write size is one candidate contributor. This change removes the large writes; only observation after deployment can show whether the restarts stop, which is why this refers to #266 and does not close it.

Reading taken on 20 November 2026, around 08:50 UTC, in the platform namespace (kubectl get cm -o json byte counts): fleet-role-probe 258,561 bytes, autoscaler-status 315,256 bytes.

Probe. The probe keeps its window in its data directory on the node (window.json, replaced by a rename once a sample's summary is published, so a replaced pod loses nothing) and publishes only window_summary (in scripts/fleet-role-stats.jq, shared with the controller): the TCP tail and reading count per peer, etcd's round-trip tail per member id, the WAL, commit and fdatasync tails, the memory and restart counts, and the last etcd gauges. An entry is held to probe_entry_limit_bytes (4 KiB, above the summary for etcd's largest recommended cluster of seven voters, each with a 253-character Node name); a summary over it is logged and not published. The probe needs the regrade limit to count samples over it, so the DaemonSet now passes PRESSURE_THRESHOLD and AVAILABLE_FLOOR_PERCENT from the same role variables the controller gets. The probe no longer reads its entry back from the ConfigMap; its Role keeps get because kubectl patch reads the object before patching it.

Controller. scripts/fleet-role.jq judges the summary with the same thresholds. An entry that is not a summary (raw samples left by a probe from before this change, until its new pod publishes) is read as no report, so the host is unknown and a voter's gates do not hold, never unfit. One behaviour changes: it can no longer drop the samples from before a membership change, so a window that began before the change is unknown until it has rolled past it. A full window rolls past once 100 samples have been taken since, which is when the old filter would also have had a full window, so the time to a verdict after a change is the same.

Autoscaler. Each per-job record is held to JOB_RECORDS_MAX_BYTES, a thirty-second of CONFIGMAP_LIMIT_BYTES (1 MiB), counted in UTF-8 bytes, newest entries first, and the first entry that does not fit ends the record.

Regression checks. PublishedEntryTest in tests/unit/test_fleet_role_probe.py publishes a full window to seven long-named voters and asserts the entry is within probe_entry_limit_bytes and equals the summary of the kept window; it asserts the entry does not grow with the window, and that an entry over the limit is refused and not published. test_each_job_record_is_held_to_its_share_of_the_configmap_newest_first in tests/unit/test_autoscaler.py feeds both records several times their budget and asserts each stays inside it, keeps the newest entries and drops the oldest, and that the whole stored object stays inside an eighth of the limit. All the bounds come from named constants, not from today's sizes. The probe's growth test failed before the change (the entry grew by 2,003 bytes over a 99-sample window). The autoscaler test was written before its implementation but not run against the old code.

Design check. Premises: no new tooling, the probe and controller already share one jq module. Strongest form: the bounds are executable tests and a run-time refusal in the probe, not prose. Storage form: I kept one key per node in the existing ConfigMap. A per-node Lease or a Node annotation is the native home for per-node heartbeat status, but the probe pods share one ServiceAccount, and RBAC resourceNames cannot name "this pod's node", so either would need create on Leases in the namespace or patch on every Node, both wider than today's patch on one named ConfigMap. Per-node ConfigMaps have the same problem. With small summaries, the shared object's whole size is bounded by probe_entry_limit_bytes times the measured nodes. Trust boundary: any probe's token can still patch any key of the shared object, as before, so a hostile or faulty node could write an oversized or false entry for itself or another node. Closing that needs an admission check (a ValidatingAdmissionPolicy that matches the key against the token's node-name claim and bounds its size), which this PR does not include.

Each probe merge-patched its node's whole window of samples into the shared
fleet-role-probe ConfigMap every sample period. The datastore stores the
whole object on every patch, so each write was the size of every node's
window together.

The probe now keeps its window in its data directory on the node
(window.json, replaced by a rename once a sample's summary is published) and
publishes only window_summary, the tails, counts and last readings the
controller judges by, from the module both share. An entry is held to
probe_entry_limit_bytes (4 KiB, above a summary for etcd's largest
recommended cluster of seven voters with the longest Node names); a summary
over it is logged and not published.

The controller judges the summary as it judged the window. Because it can
no longer drop the samples from before a membership change, a window that
began before the change is unknown until it has rolled past it, which a
full window does once 100 samples have been taken since, as before.

The probe now needs the regrade limit to count samples over it, so the
DaemonSet passes PRESSURE_THRESHOLD and AVAILABLE_FLOOR_PERCENT from the
same role variables as the controller.

Refs #266
The job-memory and job-peak records were each capped at 2048 entries, which
the comment sized at half the ConfigMap limit, so the two together could
reach the whole limit, and an entry's size depends on the names in it.

Each record is now held to JOB_RECORDS_MAX_BYTES, a thirty-second of
CONFIGMAP_LIMIT_BYTES (Kubernetes' 1 MiB), counted in UTF-8 bytes and
keeping the newest entries; the first entry that does not fit ends the
record. Every patch rewrites the whole ConfigMap in the datastore, so this
bounds each write of the object as well as its size.

Refs #266
The controller size test filled the probe ConfigMap with windows and the
autoscaler test expected a record over ARG_MAX to be kept whole. Both records
are now bounded, so the tests assert the bounds: every probe entry within
probe_entry_limit_bytes, and an oversized stored job record read whole through
a file and written back within JOB_RECORDS_MAX_BYTES.

Refs #266
`kubectl patch` reads the object before it patches it, so the probe Role
without get had every patch refused and no probe ever published, which the
mesh role controller scenarios showed as voters with no probe report.

Refs #266
A probe from before summaries holds raw samples in its key until its new pod
publishes. The controller now drops any entry whose samples field is not a
count, so such a host is unknown and a voter's gates do not hold, never
unfit on figures it cannot read.

Refs #266
The stored-record test dated its entries in the future, so the newest-first
bound rightly kept them over the job seen now; it dates them a minute ago.
The probe test measuring itself passed a node name the window reader no
longer takes, and the oversized-window test used more voters than etcd
recommends, whose summary is rightly refused; it uses the most voters with
the longest names.

Refs #266
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant