1.5.0: tests by the mcpp test spec, test sessions, tests beside each other

.agents/docs/2026-11-10-tests-from-mcpp.md

@@ -1,6 +1,6 @@
 # Tests as mcpp states them: programs, parts, cases, live results, metrics and history
 
-Status, 2026-11-10: implemented in mcpp-ide 1.3.0 (mcpp-ide#8), with the client side of mcpp#917; the backend's own tests are mtest's.
+Status, 2026-11-10: implemented in mcpp-ide 1.3.0 (mcpp-ide#8), with the client side of mcpp#917; the backend's own tests are mtest's. mcpp-ide 1.5.0 reads tests by the mcpp test spec (last section): the static reading of cases with mtest's manifest is withdrawn.
 
 ## Scope and principle
 
@@ -183,3 +183,32 @@ insights are empty and say so.
 - Neovim: the smoke test covers live results, the virtual text and `:Mcpp test.insights`.
 - CLion: the comparison failure and the nested suites, in the plugin's unit tests.
 - The insights page is rendered by a Chromium test, as the graph page is.
+
+## 1.5.0: the mcpp test spec
+
+mcpp 2026.11.10.1 states what an editor may rely on of any test framework as the mcpp test spec
+(mcpp docs/specs/test.md, SPEC-017 v2.0): a base every framework implements (B1–B6) and optional
+capabilities a framework states of itself (O1–O9). From 1.5.0 the backend reads tests by that spec and
+knows no framework by name.
+
+- **Cases.** A program's cases are its listing (B3, `mcpp test --cases`), one listing per project. The
+  last listing is kept: `test/cases` with `list` lists again when the project changed since, and
+  without it answers the kept listing. Writing a file builds nothing; the cases shown stay until a
+  listing after a build or a test replaces them. No case is read from a source.
+- **Selection.** Cases run by identifier (P4, `--case`). A configuration names a test and its cases;
+  no framework is named.
+- **Outcomes.** A case carries its checks (O3: held, failed; each failure's statement and whether the
+  case went on) and its own output (O8).
+- **Frameworks.** What a framework offers beyond the base is what `mcpp test --guide` states of it
+  (B6): its name, shown beside a program, and its capabilities.
+- **Sessions.** A framework that states a test session (`test-session`, O9), with an mcpp that gives
+  launches (`test-launch-records`, B5), runs a selection of cases without a new `mcpp test`: the backend
+  starts the session once per project and speaks JSON-RPC 2.0 to it on its standard streams. A run is
+  answered at once; its build, its cases and its end arrive as notifications and become the job's
+  `test/caseEvent` and `job/finished`. A run is registered when its answer is read, before any
+  notification of it. The session ends at the end of its input, hence with the backend; a lifetime
+  bound by the system is not used, since on Linux it follows the thread that started the process.
+  `mcpp.test.session = "off"` runs every case through `mcpp test`.
+- **Concurrency.** A test runs in a lane of its own; two tests, or a test and a build, do not wait for
+  each other. `mcpp.runtime.concurrency` holds only clean and pack.
+

CHANGELOG.md

@@ -2,6 +2,39 @@
 
 The changelog is kept by series (`X.Y`); each section lists its components' releases.
 
+## 1.5
+
+### 1.5.0
+
+Every component is 1.5.0: the backend, mcpp-vscode, mcpp-clion and mcpp-nvim (from 1.4.0).
+
+- **Tests by the mcpp test spec** (mcpp docs/specs/test.md, SPEC-017 v2.0). The backend knows no test
+  framework by name. A program's cases are its listing (`mcpp test --cases`, B3), one listing per
+  project, kept: a listing is asked when cases are first shown and after a build or a test, and writing
+  a file shows the last listing and builds nothing. Cases run by identifier (`--case`, P4), whatever
+  framework or language wrote them. A framework's name and capabilities are what `mcpp test --guide`
+  states of it (B6). The framework table (`spec/test-frameworks.json`), the static reading of cases with
+  mtest's manifest and the setting `mcpp.test.macros` are removed.
+- **Checks and output.** A case carries the checks it ran and those that failed, each failure's
+  statement and whether the case went on (O3), and its own output (O8). VS Code shows one message per
+  failed check, a diff where both sides are stated, and the failed checks on the case's line.
+- **Test sessions** (O9). Where a project's framework states a test session and its mcpp gives launches
+  (`test-launch-records`, mcpp 2026.11.10.1), a run of some cases goes through the session, started once
+  per project: its cases arrive as they begin and end, and a run after the first does not start
+  `mcpp test` again. `mcpp.test.session` (`auto`, `off`) chooses.
+- **Tests run beside each other.** A test runs at once in a lane of its own: two tests of a project, or
+  a test beside a build, do not wait for each other, and cancelling one leaves the other (mcpp
+  2026.11.10.1 releases its build directory before tests run). `mcpp.runtime.concurrency` holds only
+  clean and pack.
+- **One configuration for every test command.** A listing, a run and a debug launch take the profile,
+  the target and `mcpp.task.testArgs` alike. Discovery after events is debounced. Debugging some cases of
+  a protocol program states the protocol's variables and the selection.
+- **CLion.** A run configuration names a test and its cases; its framework field is removed. The gutter
+  places a case's run on the first element of the line its program lists it at.
+- **Neovim.** Signs list the cases when a file is read, when the session starts and after a run.
+- **Protocol.** `protocol.schema.json` states `test/cases`, `test/runCases`, `test/caseResult` and
+  `test/caseEvent` by the spec; `framework` of `test/runCases` is accepted and not read.
+
 ## 1.4
 
 ### 1.4.0

build.mcpp

@@ -45,7 +45,7 @@ int main() {
     const std::filesystem::path root = env_or_empty("MCPP_MANIFEST_DIR");
     const std::filesystem::path out = env_or_empty("MCPP_OUT_DIR");
     const std::filesystem::path spec = root / "spec";
-    const std::array<std::pair<const char*, const char*>, 9> files{{
+    const std::array<std::pair<const char*, const char*>, 8> files{{
         {"kSettingsJson", "settings.json"},
         {"kPresetsJson", "presets.json"},
         {"kCommandsJson", "commands.json"},
@@ -54,7 +54,6 @@ int main() {
         {"kManifestZhCnJson", "i18n/manifest.zh-cn.json"},
         {"kTomlSchemaJson", "toml-schema.json"},
         {"kBuildScriptApiJson", "buildscript-api.json"},
-        {"kTestFrameworksJson", "test-frameworks.json"},
     }};
     std::string module =
         "// Generated by mcpp-ide's build.mcpp from spec/ (not part of the repository).\n"

docs/architecture.md

@@ -75,29 +75,37 @@ mcppls what a compiler answers better.
 
 ## A test run, end to end
 
-The editors show tests as `mcpp test` states them (design 2026-11-10). mcpp builds and runs the
-programs; mtest's protocol is how a program's cases report to mcpp; mtest's store (spec P12) is the form
-in which mcpp keeps a run (`target/mtest`, or `[test] store`). The backend reads those three; it runs no
-other test runner.
+The editors show tests as `mcpp test` states them, by the mcpp test spec (mcpp docs/specs/test.md). mcpp
+builds and runs the programs and states their cases and results; a framework is whatever library a
+program is written with, and the backend knows none by name. What a framework offers beyond the base,
+it states of itself (`mcpp test --guide`, B6).
 
 1. `test/list` runs `mcpp test --list` and answers each program with its files, its `main` and its parts.
    A protocol program of several files (`[test] programs = "directory"`) runs as one process per file,
-   `<program>/<stem>`. mcpp states the parts with `test-parts` (mcpp#917); before it, the backend derives
-   them as mcpp 2026.11.9.1 forms them. The programs are kept per project.
-2. `test/cases` reads the cases of every file of a program with mtest's static manifest (their kinds,
-   tags and the bounds on their metrics), and with `list` the program's own listing (`--cases`).
-3. A test job runs `mcpp test --message-format json`. Each record becomes `test/result`, with the
-   program it belongs to, the part it is and its cases, each status mapped (`xfail` passes, `xpass`
-   fails, `invalid` is errored). With `test-case-stream` (mcpp#917) each case is sent as it begins and
-   ends (`test/caseEvent`).
-4. VS Code marks the Testing view's items, a failure with both sides as a diff, and ends a case's line
-   with its duration and metrics. CLion fills an ID-based test tree: a suite per program and per file, a
-   comparison failure when both sides are stated. Neovim writes `mcpp://tests`, the quickfix list and
-   the signs.
-5. `test/insights` reads the store's last runs (20 by default) and answers the last run, its failures
-   and since when, the slowest and the flaky cases, each metric against its bound, and the schedule
-   with its critical path. VS Code and CLion draw it with one page (`webui/tests.html`); Neovim prints
-   it.
+   `<program>/<stem>`. mcpp states the parts with `test-parts`; before it, the backend derives them as
+   mcpp 2026.11.9.1 forms them. The programs are kept per project.
+2. `test/cases` answers a program's cases as the program lists them (`mcpp test --cases`, B3): one
+   listing of the project, kept until the project changes.
+3. A test job runs `mcpp test --message-format json` with one set of arguments for every test command
+   (the profile, the target, `mcpp.task.testArgs`). Each record becomes `test/result`, with the program
+   it belongs to, the part it is and its cases, each status mapped (`xfail` passes, `xpass` fails,
+   `invalid` is errored); a case carries its checks (O3) and its own output (O8). With
+   `test-case-stream` each case is sent as it begins and ends (`test/caseEvent`). A test job runs in a
+   lane of its own: two tests, or a test and a build, do not wait for each other.
+4. Where a framework states a test session (`test-session`, O9) and mcpp gives launches
+   (`test-launch-records`, B5), `test/runCases` runs the cases through it: the session is started once
+   per project, speaks JSON-RPC on its standard streams, answers a run at once and states its build,
+   its cases and its end as notifications, which become the job's `test/caseEvent` and `job/finished`.
+   The session ends with its input, so it ends with the backend. `mcpp.test.session = "off"` runs
+   every case through `mcpp test`.
+5. VS Code marks the Testing view's items, one message per failed check (a diff where both sides are
+   stated), and ends a case's line with its duration, its failed checks and its metrics. CLion fills an
+   ID-based test tree: a suite per program and per file, a comparison failure when both sides are
+   stated. Neovim writes `mcpp://tests`, the quickfix list and the signs.
+6. `test/insights` reads the last runs mcpp keeps (`target/mtest`, or `[test] store`; 20 by default)
+   and answers the last run, its failures and since when, the slowest and the flaky cases, each
+   metric against its bound, and the schedule with its critical path. VS Code and CLion draw it with
+   one page (`webui/tests.html`); Neovim prints it.
 
 Every step reads what the project's mcpp states and nothing more: an older mcpp gives programs and
 their results, and a view that needs a later feature is left out (`capabilities/explain`).

mcpp.lock

@@ -89,13 +89,6 @@ source  = "index+mcpplibs@0.4.2"
 hash    = "fnv1a:c06e4ca4f73f228d"
 targets = ["aarch64-openkal", "x86_64-linux-gnu", "x86_64-openkal"]
 
-[package."mcpplibs.mtest.manifest"]
-namespace = "mcpplibs.mtest"
-version = "0.5.2"
-source  = "index+mcpplibs.mtest@0.5.2"
-hash    = "fnv1a:0caef37dd958cfef"
-targets = ["aarch64-openkal", "x86_64-linux-gnu", "x86_64-openkal"]
-
 [package."nlohmann.json"]
 namespace = "nlohmann"
 version = "3.12.0"

mcpp.toml

@@ -9,7 +9,7 @@
 # M2/M3), where the system it runs upon is answered when the image is loaded.
 [package]
 name        = "mcpp-ide"
-version     = "1.4.0"
+version     = "1.5.0"
 description = "The IDE backend for mcpp: sessions, jobs, records and view models over mcpp's control contract"
 license     = "Apache-2.0"
 authors     = ["mcpp-community"]
@@ -80,8 +80,6 @@ main = "src/bin/dev.cpp"
 # runtime and loader remain the platform's.
 mcpplibs.os   = "0.4.2"
 nlohmann.json = "3.12.0"
-# mtest's static case manifest: the cases a test file declares, read as mtest reads them.
-mcpplibs.mtest.manifest = "0.5.2"
 
 # The tests are mtest's (mtest docs/layout.md): tests/unit, tests/e2e, tests/cts and tests/bench, each
 # directory one program whose files declare their cases; what every test shares is testsupport/, a

mide-plugins/clion/README.md

@@ -99,7 +99,7 @@ An individual packaged plugin can be checked without rebuilding the backend:
 
 ```bash
 xvfb-run -a gradle --no-daemon -p integration-tests test \
-  -PpluginZip=/absolute/path/mcpp-clion-1.4.0.zip \
+  -PpluginZip=/absolute/path/mcpp-clion-1.5.0.zip \
   -PmcpplsZip=/absolute/path/mcppls-clion-0.21.0.zip \
   -PideVersion=2026.1.5.2
 ```

mide-plugins/clion/gradle.properties

@@ -2,7 +2,7 @@
 # where it installs (2026.1 and later); the verifier checks both ends.
 platformType = CL
 platformVersion = 2026.2.3
-pluginVersion = 1.4.0
+pluginVersion = 1.5.0
 # The mcppls CLion plugin this one depends on (<depends>io.github.sunrisepeak.mcppls</depends>):
 # its release's one package for every system, fetched for the sandbox, the tests and the verifier.
 mcpplsPluginVersion = 0.21.0

mide-plugins/clion/src/main/kotlin/io/github/mcppcommunity/mcppclion/McppSession.kt

@@ -63,14 +63,14 @@ class McppSession(private val project: Project) : Disposable {
             if (key == "mcpp.ide.backendPath") restart() else pushSettings()
         })
         on("manifest/changed") { forgetSources() }
-        // A build ended: a test program can list the cases its macros generate (§7.3).
+        // A build or a test ended: the programs are listed again (mcpp test spec B3).
         on("job/finished") { p ->
             val o = p.asJsonObject
             if (o["status"]?.asString == "ok" && o["kind"]?.asString in setOf("build", "test")) {
                 cases.keys.toList().forEach { key -> key.split('\u0000').takeIf { it.size == 2 }?.let { refreshCases(it[0], it[1], list = true) } }
             }
         }
-        // A C++ source saved: what was read from it (main, a framework's cases) is read again.
+        // A C++ source saved: what was read from it (main) is read again.
         project.messageBus.connect(this).subscribe(com.intellij.openapi.vfs.VirtualFileManager.VFS_CHANGES,
             object : com.intellij.openapi.vfs.newvfs.BulkFileListener {
                 override fun after(events: MutableList<out com.intellij.openapi.vfs.newvfs.events.VFileEvent>) {
@@ -297,6 +297,8 @@ class McppSession(private val project: Project) : Disposable {
     // when unknown, and the editor's markers recomputed when the answer arrives.
     private val sources = java.util.concurrent.ConcurrentHashMap<String, JsonObject>()
     private val cases = java.util.concurrent.ConcurrentHashMap<String, JsonObject>()
+    // The programs whose cases are being asked for: one listing at a time each.
+    private val asking = java.util.concurrent.ConcurrentHashMap.newKeySet<String>()
 
     fun entriesFor(path: String): JsonObject? {
         val known = sources[path]
@@ -322,30 +324,33 @@ class McppSession(private val project: Project) : Disposable {
 
     fun casesFor(root: String, test: String): JsonObject? {
         val known = cases["$root\u0000$test"]
-        if (known == null) refreshCases(root, test, list = false)
+        if (known == null) refreshCases(root, test, list = true)
         return known
     }
 
     fun refreshCases(root: String, test: String, list: Boolean) {
+        val key = "$root\u0000$test"
+        if (!asking.add(key)) return
         request("test/cases", JsonObject().apply {
             addProperty("root", root)
             addProperty("test", test)
             addProperty("list", list && TrustedProjects.isProjectTrusted(project))
-        }).thenAccept { r ->
-            cases["$root\u0000$test"] = r.asJsonObject
+        }).whenComplete { r, error ->
+            asking.remove(key)
+            if (error != null) return@whenComplete
+            cases[key] = r.asJsonObject
             restartMarkers()
         }
     }
 
-    /** A source changed: what was read from it is read again on its next use. */
+    /** A source changed: what was read from it is read again on its next use. Its program's cases
+     *  stay as they were listed until a build or a test lists them again. */
     fun forgetSource(path: String) {
         sources.remove(path)
-        cases.keys.removeIf { it.split('\u0000').getOrNull(1)?.let { test -> path.replace('\\', '/').endsWith("/tests/$test.cpp") } == true }
     }
 
     private fun forgetSources() {
         sources.clear()
-        cases.clear()
     }
 
     private fun restartMarkers() = com.intellij.openapi.application.ApplicationManager.getApplication().invokeLater {