Skip to content

test(readme): derive the spec-FILE counts from BASELINE, not by hand - #955

Merged
tannevaled merged 1 commit into
mainfrom
readme-specfile-counts
Nov 16, 2026
Merged

tannevaled merged 1 commit into
mainfrom
readme-specfile-counts

Conversation

@tannevaled

Copy link
Copy Markdown
Contributor

Five hand-written file counts, none of them derived

The README states how many ruby/spec files the sweep covers five times:

| spec files | 2,202 of 2,206 produced a result; 4 produced none |
… agreed on every one of the 2,206 per-file records except …
| spec files in the corpus | 3,814 — of which 2,206 are swept and 1,608 are not |
count of each of the 2,206 spec files, and the run fails if any file passes
of the 2,206 files is compared with its own record, which needs no margin and

Nothing derived any of them.

⛔ And the guard next door makes it worse, measured

TestReadmeConformanceCountMatchesTheBaseline guards the passing example total — the number that moves most often. Precisely because that one is guarded, the file counts sitting beside it read as equally measured.

Measured, not assumed: setting the table row to 2,203 of 2,206 leaves that test green. It skips FILEFAIL rows on purpose — "a FILEFAIL row carries no count" — so it sums examples and can say nothing about how many files there are. Two numbers on one line, one guarded and one not, with nothing to tell a reader which is which.

The counts are correct today — OK=2202 FILEFAIL=4 total=2206 — so this changes no prose. What it changes is that a corpus sweep which adds or retires spec files can no longer move 2,206 in five places and 2,202 in one by hand and have the only possible reviewer be someone who happens to count BASELINE.

⛔ The helper keys on the record shape, not on the word

grep -c FILEFAIL over BASELINE answers 5 where there are 4 records. The fifth hit is the header line that documents the format:

# One record per spec file: OK<TAB>path<TAB>passing, or FILEFAIL<TAB>path.

A comment that names a value is the oldest way a count goes wrong by one. So baselineRecordCounts switches on the tab-separated first field of a non-comment line, fails on any record it cannot classify rather than dropping it from a denominator the README quotes, and fails on zero records rather than calling a format change an empty corpus.

Both denominators are asserted, not only the figures

One triple statement and at least four single ones must match. A rewording that moves a claim out of the pattern's reach takes it out of this test without taking it out of the README — and a test that then matched nothing would pass. The thresholds are what the README held when this was written; a drop means a claim went unread, not that the README got shorter.

⛔ What this does NOT judge

Stated here so the silence is not mistaken for coverage: 3,814 and 1,608 describe ruby/spec's whole tree, which is not in this repository, so BASELINE cannot answer for them. The test checks only that the swept figure inside that sentence is the same 2,206 as everywhere else. A gate that quietly judged part of a sentence would be worse than none.

Verified by mutation, not by reading

  • 2,202 → 2,203: red, naming README.md:145 and the right figure — while TestReadmeConformanceCountMatchesTheBaseline stayed green on the same file, which is the measurement that justifies this gate existing.
  • Reverted: green, logging BASELINE: OK=2202 FILEFAIL=4 total=2206; README statements checked: 1 triple, 4 single.
  • gofmt -l . prints nothing; go test . green; buildscan --dir . --vet → "go vet passed under all 19".

🤖 Generated with Claude Code

The README states how many ruby/spec files the sweep covers five times:

    | spec files | 2,202 of 2,206 produced a result; 4 produced none |
    … agreed on every one of the 2,206 per-file records except …
    | spec files in the corpus | 3,814 — of which 2,206 are swept … |
    count of each of the 2,206 spec files, and the run fails if any file …
    of the 2,206 files is compared with its own record, which needs no …

Nothing derived any of them. TestReadmeConformanceCountMatchesTheBaseline
guards the passing EXAMPLE total — the number that moves most often — and
PRECISELY BECAUSE that one is guarded, the file counts sitting beside it
read as equally measured. They were not.

⛔ MEASURED, not assumed: setting the table to "2,203 of 2,206" leaves
TestReadmeConformanceCountMatchesTheBaseline GREEN. It skips FILEFAIL
rows on purpose ("a FILEFAIL row carries no count"), so it sums examples
and can say nothing about how many FILES there are. Two numbers on one
line, one guarded and one not, with nothing to tell a reader which.

The counts are correct today — OK=2202, FILEFAIL=4, total=2206 — so this
changes no prose. A corpus sweep that adds or retires spec files moves
2,206 in five places and 2,202 in one, by hand, and the only reader who
could catch a missed edit is one who happens to count BASELINE.

⛔ THE HELPER KEYS ON THE RECORD SHAPE, NOT ON THE WORD. `grep -c
FILEFAIL` over BASELINE answers 5 where there are 4 records: the fifth
hit is the header line that DOCUMENTS the format —

    # One record per spec file: OK<TAB>path<TAB>passing, or FILEFAIL<TAB>path.

A comment that names a value is the oldest way a count goes wrong by one.

Both denominators are asserted, not just the figures: 1 triple statement
and at least 4 single ones must MATCH, because a rewording that moves a
claim out of the pattern's reach takes it out of this test without taking
it out of the README — and a test that then matched nothing would pass.

⛔ What this gate does NOT judge, stated so the silence is not read as
coverage: 3,814 and 1,608 describe ruby/spec's whole tree, which is not
in this repository. BASELINE cannot answer for them. The test checks only
that the SWEPT figure inside that sentence is the same 2,206 as
everywhere else. A gate that quietly judged part of a sentence would be
worse than none.

Verified by mutation, not by reading: 2,202 -> 2,203 turns it red naming
README.md:145 and the right figure; reverted, green, logging "BASELINE:
OK=2202 FILEFAIL=4 total=2206; README statements checked: 1 triple, 4
single". gofmt clean, `go test .` green, buildscan --vet under all 19.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@tannevaled
tannevaled merged commit e1f7f29 into main Nov 16, 2026
26 checks passed
@tannevaled
tannevaled deleted the readme-specfile-counts branch November 16, 2026 04:02
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant