Skip to content
Merged
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 3 additions & 1 deletion layouts/_default/list.html
Original file line number Diff line number Diff line change
Expand Up @@ -72,7 +72,9 @@ <h3>Runtime conformance — ruby/spec</h3>
<ul>
<li>CI gates <b>file by file</b>, not on a total: no spec file may pass fewer examples than its recorded row, and none may stop loading. The sum of those rows is <b>23,672</b>, re-recorded at Ruby 4.0.9 (2026-11-03) on the pinned runner that enforces it — it is what CI guarantees, while the measured total above is what one dated run did. One file (<code>core/random/bytes_spec.rb</code>) is unstable between 8 and 9 passing examples; since 2026-11-04 its row holds <b>8</b>, its floor, because a row recorded at the high draw makes the next ordinary run a red build. This replaced a single shrink-only floor of 22,475, which needed a margin wider than the largest spec file and so could not see a regression smaller than that.</li>
<li><b>Only <code>language/</code> and <code>core/</code> are swept.</b> ruby/spec's <code>library/</code>, <code>optional/</code>, <code>command_line/</code> and <code>security/</code> trees are not run at all, so this figure is <b>not</b> a percentage of Ruby, and most of the standard library sits outside it.</li>
<li>Those unswept trees were measured <b>once</b>, on 2026-11-10, against MRI 4.0.7 through the same shim — a dated run, not a gate. Across the 970 files where the two engines differ the gap is <b>2,341</b> examples, and the shape matters more than the size: <b>78 files cannot be measured at all</b> (every <code>command_line/</code> spec is built on <code>ruby_exe</code>, and <code>optional/capi</code> needs a compiled C extension — MRI itself scores zero in them), the <b>largest single debt is a hang rather than a missing feature</b> (<code>Net::FTP</code> is 442 examples behind, with 39 of its 56 files killed by the clock on rbgo and none on MRI), and the outright absences are comparatively small (<code>Coverage</code> 53 examples, <code>WeakRef</code> 9).</li>
<li>Those unswept trees were measured <b>once</b>, on 2026-11-10, against MRI 4.0.7 through the same shim — a dated run, not a gate. Across the 970 files where the two engines differ the gap is <b>2,341</b> examples — a <b>floor</b> rather than a total, for the reason two points below — and the shape matters more than the size: <b>78 files cannot be measured at all</b> (every <code>command_line/</code> spec is built on <code>ruby_exe</code>, and <code>optional/capi</code> needs a compiled C extension — MRI itself scores zero in them), and the outright absences are comparatively small (<code>Coverage</code> 53 examples, <code>WeakRef</code> 9).</li>
<li><b>The hangs that used to be the largest debt are mostly gone.</b> Where this page reported <code>Net::FTP</code> 442 examples behind with 39 of its 56 files killed by the clock, <code>Net::FTP</code> now has <b>0 files killed and is 128 behind</b> (re-measured 2026-11-13, MRI 4.0.6), <code>Net::HTTP</code> is at 0 from 27, and across the corpus <b>75 killed files are down to 8</b>. One symptom, three unrelated causes: a blocking socket call holding rbgo's emulated GVL, a <code>TCPSocket.open { }</code> that never closed the socket so the peer waited for an EOF that never came, and a <code>Thread#kill</code> that could not interrupt a blocking native call.</li>
<li><b>The largest debt is now unimplemented surface — and it was invisible while its own control scored zero.</b> <code>library/socket</code>'s <code>spec_helper.rb</code> calls <code>MSpec.enable_feature</code>, which the shim did not define, so all 190 of those files scored <b>0 under MRI as well</b>, and a control of zero is indistinguishable from a module with no gap. Repaired, the module measures <b>MRI 1790 / rbgo 224</b> — about <b>−1560</b>, more than <code>matrix</code>, <code>bigdecimal</code>, <code>stringscanner</code>, <code>date</code>, <code>zlib</code>, <code>stringio</code> and <code>openssl</code> put together — and it is <code>Addrinfo</code>, <code>recvmsg</code> and the socket option accessors rather than anything that hangs. The two figures are deliberately not added together: they come from different runs against different MRI patch releases.</li>
<li>Of the examples that ran, <b>93.3%</b> pass (22,488 of 24,116); 477 more are skipped because the shim stubs <code>ruby_exe</code>, <code>ruby_cmd</code> and <code>output_to_fd</code> — three helpers that would need a subprocess or real file-descriptor redirection — and 10 of 2,206 spec files produce no result. Skips and unreadable files sit outside the ratio entirely.</li>
<li>The guarantee has climbed from a floor of <b>6,000</b> on 2026-08-03 to a per-file baseline summing <b>23,672</b>. Two earlier jumps were fixes to the <b>measurement</b> — the mspec shim, and a parser upgrade that made 13 previously-unparseable spec files visible — which are gains in what the measurement can <i>see</i>, not in what the VM can <i>do</i>.</li>
</ul>
Expand Down
Loading