Skip to content

fix(pstack): carry TLC warnings into the statechart report - #1254

Open
dev-dami wants to merge 1 commit into
cursor:mainfrom
dev-dami:fix/statechart-report-tlc-warnings
Open

dev-dami wants to merge 1 commit into
cursor:mainfrom
dev-dami:fix/statechart-report-tlc-warnings

Conversation

@dev-dami

Copy link
Copy Markdown

Refs #1235 (level 1: carry TLC's warnings into report.md)

Why

statechart could list a rule under "Rules that hold" when TLC had warned that the model cannot break it. The common cause is an action that sets a variable and also lists it as UNCHANGED, so the action can never fire. TLC prints a warning (code 2143), but it reached tlc.out at most, and report.md gave a clean pass. With the spec from #1235 on main:

## Rules that hold
NeverPaidTwice, NoDeadlock

The chart has no BankAccepts arrow, and nothing else says so.

Root cause

report() never read TLC's warning messages. The first fix I tried read them from the first verdict run only (the run that tlc.out prints). Review found that this was not enough. When a deadlock or an invariant breaks first, TLC stops before it evaluates the bad action, so only the bounded rerun that judges the remaining rules prints the warning. judge() in run.ts threw that output away; it keeps later runs' messages only for broken rules. So the false pass came back as soon as the config had one more rule that breaks early.

What changed

  • frontends/verdict.ts: readVerdict now also collects the run's level-3 (warning) messages in the pass it already makes. It leaves out two of them:
    • 2401 (TLC_ENVIRONMENT_JVM_GC), TLC's advice to run Java with -XX:+UseParallelGC. Every run prints it, and it says nothing about the model.
    • 2284 (TLC_FEATURE_LIVENESS_CONSTRAINTS), in bounded reruns only. Statechart adds the CONSTRAINT ChartBound that causes it there; an unbounded run or the user's own CONSTRAINT keeps it.
  • run.ts: judge() gathers the warnings of every verdict run, once each, in first-seen order, and returns them on CheckResult.warnings. The drawing run (ChartRun) is not read.
  • backends/report.ts: when there are warnings, the report gets a ## TLC warnings section above ## Rules that hold, with each warning in a fenced block. The exit code does not change; a warning is not a failure.
  • SKILL.md: one bullet under the report notes saying where the section comes from and what it leaves out.
  • Regenerated harness/pstack/ with scripts/generate-pstack-harness-trees.mjs.

Level 2 of the issue, listing the actions of Next that made no step, is not in this PR.

Tests

  • New corpus case UnchangedWarning: the spec from pstack statechart: a rule "holds" when the action that could break it can never fire, and the report drops TLC's warning #1235, recorded with TLC 1.7.4. The warning is in the first run.
  • New corpus case UnchangedAfterDeadlock: one initial state deadlocks, so the warning appears only in verdict-2.out (a bounded rerun).
  • Two CLI tests on those recordings. Each checks that the warning appears exactly once, above ## Rules that hold, and that UseParallelGC does not appear. The second also checks that the deadlock break is still reported. Both failed before the change: the first on main, the second on the first-run-only version.
  • PickWorker/report.md now includes TLC's 2279 warning ("Declaring symmetry during liveness checking is dangerous …"), which comes from that case's own SYMMETRY + PROPERTY config. MediaPlayer's report is unchanged: its only extra warning is the 2284 from statechart's bounded rerun. All other corpus recordings and reports are byte-identical.

Verification

Run on this branch, Linux, Node 24.22:

  • node --disable-warning=ExperimentalWarning --test pstack/skills/*/scripts/test/*.test.ts without TLA2TOOLS: 350 tests, 278 pass, 0 fail, 72 skipped (main: 340 / 270 / 0 / 70).
  • The statechart tests with TLA2TOOLS set to tla2tools.jar 1.7.4 and a JDK 21: 385 / 385 pass (main: 373 / 373). These include the live TLC runs of every corpus case.
  • node scripts/generate-pstack-harness-trees.mjs --check: up to date.
  • node scripts/validate-plugins.mjs: all plugins validated.
  • git diff --check: clean.
  • Extra scenarios run with real TLC against main and this branch. Exit codes and chart.mmd were identical in every case.
    • Two different warnings.
    • A warning with a break, before and after the bad action.
    • A model above the 500-state chart budget, both when its rules hold and when one breaks (the paths branch).
    • A run that hits the 100,000-state rerun bound ("Rules not checked").
    • SYMMETRY or CONSTRAINT with a PROPERTY in the user's config.
    • Twelve warnings.
    • A spec with no rules.
    • Specs with no warning except the GC advice. Their report.md is byte-identical to main.
  • A warning text can't close the fence early: SANY rejects a backtick in a TLA+ string, and TLC crashes on one in a cfg string. A warning whose text contains ## Rules that hold stays inside its fence.

Known limits

  • tlc.out is still the first run only. A warning that only a later run printed is in report.md but not in tlc.out. The SKILL.md bullet says why.
  • Only top-level TLC messages are read. TLC prints warnings at top level; nesting happens for messages embedded in exceptions.
  • Not checked on macOS or Windows.

A rule could be listed under "Rules that hold" while TLC warned that
the model cannot break it, for example when an action sets a variable
and also lists it as UNCHANGED. The warning reached tlc.out only.

report.md now gives TLC's warnings under "TLC warnings", above "Rules
that hold", once each. They come from every verdict run, not only the
first: a run that stops at a deadlock or a broken invariant can stop
before TLC evaluates the bad action, so only a later run prints the
warning. The report leaves out TLC's advice on Java's garbage collector
(code 2401), and in a bounded rerun the warning about constraints during
liveness checking (code 2284), because statechart adds that constraint
itself. The exit code does not change.

Refs cursor#1235
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant