Repository navigation
test(readme): derive the spec-FILE counts from BASELINE, not by hand - #955
Merged
Merged
Conversation
The README states how many ruby/spec files the sweep covers five times:
| spec files | 2,202 of 2,206 produced a result; 4 produced none |
… agreed on every one of the 2,206 per-file records except …
| spec files in the corpus | 3,814 — of which 2,206 are swept … |
count of each of the 2,206 spec files, and the run fails if any file …
of the 2,206 files is compared with its own record, which needs no …
Nothing derived any of them. TestReadmeConformanceCountMatchesTheBaseline
guards the passing EXAMPLE total — the number that moves most often — and
PRECISELY BECAUSE that one is guarded, the file counts sitting beside it
read as equally measured. They were not.
⛔ MEASURED, not assumed: setting the table to "2,203 of 2,206" leaves
TestReadmeConformanceCountMatchesTheBaseline GREEN. It skips FILEFAIL
rows on purpose ("a FILEFAIL row carries no count"), so it sums examples
and can say nothing about how many FILES there are. Two numbers on one
line, one guarded and one not, with nothing to tell a reader which.
The counts are correct today — OK=2202, FILEFAIL=4, total=2206 — so this
changes no prose. A corpus sweep that adds or retires spec files moves
2,206 in five places and 2,202 in one, by hand, and the only reader who
could catch a missed edit is one who happens to count BASELINE.
⛔ THE HELPER KEYS ON THE RECORD SHAPE, NOT ON THE WORD. `grep -c
FILEFAIL` over BASELINE answers 5 where there are 4 records: the fifth
hit is the header line that DOCUMENTS the format —
# One record per spec file: OK<TAB>path<TAB>passing, or FILEFAIL<TAB>path.
A comment that names a value is the oldest way a count goes wrong by one.
Both denominators are asserted, not just the figures: 1 triple statement
and at least 4 single ones must MATCH, because a rewording that moves a
claim out of the pattern's reach takes it out of this test without taking
it out of the README — and a test that then matched nothing would pass.
⛔ What this gate does NOT judge, stated so the silence is not read as
coverage: 3,814 and 1,608 describe ruby/spec's whole tree, which is not
in this repository. BASELINE cannot answer for them. The test checks only
that the SWEPT figure inside that sentence is the same 2,206 as
everywhere else. A gate that quietly judged part of a sentence would be
worse than none.
Verified by mutation, not by reading: 2,202 -> 2,203 turns it red naming
README.md:145 and the right figure; reverted, green, logging "BASELINE:
OK=2202 FILEFAIL=4 total=2206; README statements checked: 1 triple, 4
single". gofmt clean, `go test .` green, buildscan --vet under all 19.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Five hand-written file counts, none of them derived
The README states how many ruby/spec files the sweep covers five times:
Nothing derived any of them.
⛔ And the guard next door makes it worse, measured
TestReadmeConformanceCountMatchesTheBaselineguards the passing example total — the number that moves most often. Precisely because that one is guarded, the file counts sitting beside it read as equally measured.Measured, not assumed: setting the table row to
2,203 of 2,206leaves that test green. It skips FILEFAIL rows on purpose — "a FILEFAIL row carries no count" — so it sums examples and can say nothing about how many files there are. Two numbers on one line, one guarded and one not, with nothing to tell a reader which is which.The counts are correct today —
OK=2202 FILEFAIL=4 total=2206— so this changes no prose. What it changes is that a corpus sweep which adds or retires spec files can no longer move2,206in five places and2,202in one by hand and have the only possible reviewer be someone who happens to count BASELINE.⛔ The helper keys on the record shape, not on the word
grep -c FILEFAILover BASELINE answers 5 where there are 4 records. The fifth hit is the header line that documents the format:A comment that names a value is the oldest way a count goes wrong by one. So
baselineRecordCountsswitches on the tab-separated first field of a non-comment line, fails on any record it cannot classify rather than dropping it from a denominator the README quotes, and fails on zero records rather than calling a format change an empty corpus.Both denominators are asserted, not only the figures
One triple statement and at least four single ones must match. A rewording that moves a claim out of the pattern's reach takes it out of this test without taking it out of the README — and a test that then matched nothing would pass. The thresholds are what the README held when this was written; a drop means a claim went unread, not that the README got shorter.
⛔ What this does NOT judge
Stated here so the silence is not mistaken for coverage: 3,814 and 1,608 describe ruby/spec's whole tree, which is not in this repository, so BASELINE cannot answer for them. The test checks only that the swept figure inside that sentence is the same
2,206as everywhere else. A gate that quietly judged part of a sentence would be worse than none.Verified by mutation, not by reading
2,202→2,203: red, namingREADME.md:145and the right figure — whileTestReadmeConformanceCountMatchesTheBaselinestayed green on the same file, which is the measurement that justifies this gate existing.BASELINE: OK=2202 FILEFAIL=4 total=2206; README statements checked: 1 triple, 4 single.gofmt -l .prints nothing;go test .green;buildscan --dir . --vet→ "go vet passed under all 19".🤖 Generated with Claude Code