diff --git a/.agents/docs/2026-11-10-tests-from-mcpp.md b/.agents/docs/2026-11-10-tests-from-mcpp.md index 3782f61..86e4250 100644 --- a/.agents/docs/2026-11-10-tests-from-mcpp.md +++ b/.agents/docs/2026-11-10-tests-from-mcpp.md @@ -1,6 +1,6 @@ # Tests as mcpp states them: programs, parts, cases, live results, metrics and history -Status, 2026-11-10: implemented in mcpp-ide 1.3.0 (mcpp-ide#8), with the client side of mcpp#917; the backend's own tests are mtest's. +Status, 2026-11-10: implemented in mcpp-ide 1.3.0 (mcpp-ide#8), with the client side of mcpp#917; the backend's own tests are mtest's. mcpp-ide 1.5.0 reads tests by the mcpp test spec (last section): the static reading of cases with mtest's manifest is withdrawn. ## Scope and principle @@ -183,3 +183,32 @@ insights are empty and say so. - Neovim: the smoke test covers live results, the virtual text and `:Mcpp test.insights`. - CLion: the comparison failure and the nested suites, in the plugin's unit tests. - The insights page is rendered by a Chromium test, as the graph page is. + +## 1.5.0: the mcpp test spec + +mcpp 2026.11.10.1 states what an editor may rely on of any test framework as the mcpp test spec +(mcpp docs/specs/test.md, SPEC-017 v2.0): a base every framework implements (B1–B6) and optional +capabilities a framework states of itself (O1–O9). From 1.5.0 the backend reads tests by that spec and +knows no framework by name. + +- **Cases.** A program's cases are its listing (B3, `mcpp test --cases`), one listing per project. The + last listing is kept: `test/cases` with `list` lists again when the project changed since, and + without it answers the kept listing. Writing a file builds nothing; the cases shown stay until a + listing after a build or a test replaces them. No case is read from a source. +- **Selection.** Cases run by identifier (P4, `--case`). A configuration names a test and its cases; + no framework is named. +- **Outcomes.** A case carries its checks (O3: held, failed; each failure's statement and whether the + case went on) and its own output (O8). +- **Frameworks.** What a framework offers beyond the base is what `mcpp test --guide` states of it + (B6): its name, shown beside a program, and its capabilities. +- **Sessions.** A framework that states a test session (`test-session`, O9), with an mcpp that gives + launches (`test-launch-records`, B5), runs a selection of cases without a new `mcpp test`: the backend + starts the session once per project and speaks JSON-RPC 2.0 to it on its standard streams. A run is + answered at once; its build, its cases and its end arrive as notifications and become the job's + `test/caseEvent` and `job/finished`. A run is registered when its answer is read, before any + notification of it. The session ends at the end of its input, hence with the backend; a lifetime + bound by the system is not used, since on Linux it follows the thread that started the process. + `mcpp.test.session = "off"` runs every case through `mcpp test`. +- **Concurrency.** A test runs in a lane of its own; two tests, or a test and a build, do not wait for + each other. `mcpp.runtime.concurrency` holds only clean and pack. + diff --git a/CHANGELOG.md b/CHANGELOG.md index 2c2b194..cd9c600 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,39 @@ The changelog is kept by series (`X.Y`); each section lists its components' releases. +## 1.5 + +### 1.5.0 + +Every component is 1.5.0: the backend, mcpp-vscode, mcpp-clion and mcpp-nvim (from 1.4.0). + +- **Tests by the mcpp test spec** (mcpp docs/specs/test.md, SPEC-017 v2.0). The backend knows no test + framework by name. A program's cases are its listing (`mcpp test --cases`, B3), one listing per + project, kept: a listing is asked when cases are first shown and after a build or a test, and writing + a file shows the last listing and builds nothing. Cases run by identifier (`--case`, P4), whatever + framework or language wrote them. A framework's name and capabilities are what `mcpp test --guide` + states of it (B6). The framework table (`spec/test-frameworks.json`), the static reading of cases with + mtest's manifest and the setting `mcpp.test.macros` are removed. +- **Checks and output.** A case carries the checks it ran and those that failed, each failure's + statement and whether the case went on (O3), and its own output (O8). VS Code shows one message per + failed check, a diff where both sides are stated, and the failed checks on the case's line. +- **Test sessions** (O9). Where a project's framework states a test session and its mcpp gives launches + (`test-launch-records`, mcpp 2026.11.10.1), a run of some cases goes through the session, started once + per project: its cases arrive as they begin and end, and a run after the first does not start + `mcpp test` again. `mcpp.test.session` (`auto`, `off`) chooses. +- **Tests run beside each other.** A test runs at once in a lane of its own: two tests of a project, or + a test beside a build, do not wait for each other, and cancelling one leaves the other (mcpp + 2026.11.10.1 releases its build directory before tests run). `mcpp.runtime.concurrency` holds only + clean and pack. +- **One configuration for every test command.** A listing, a run and a debug launch take the profile, + the target and `mcpp.task.testArgs` alike. Discovery after events is debounced. Debugging some cases of + a protocol program states the protocol's variables and the selection. +- **CLion.** A run configuration names a test and its cases; its framework field is removed. The gutter + places a case's run on the first element of the line its program lists it at. +- **Neovim.** Signs list the cases when a file is read, when the session starts and after a run. +- **Protocol.** `protocol.schema.json` states `test/cases`, `test/runCases`, `test/caseResult` and + `test/caseEvent` by the spec; `framework` of `test/runCases` is accepted and not read. + ## 1.4 ### 1.4.0 diff --git a/build.mcpp b/build.mcpp index 37a1522..8c6cebd 100644 --- a/build.mcpp +++ b/build.mcpp @@ -45,7 +45,7 @@ int main() { const std::filesystem::path root = env_or_empty("MCPP_MANIFEST_DIR"); const std::filesystem::path out = env_or_empty("MCPP_OUT_DIR"); const std::filesystem::path spec = root / "spec"; - const std::array, 9> files{{ + const std::array, 8> files{{ {"kSettingsJson", "settings.json"}, {"kPresetsJson", "presets.json"}, {"kCommandsJson", "commands.json"}, @@ -54,7 +54,6 @@ int main() { {"kManifestZhCnJson", "i18n/manifest.zh-cn.json"}, {"kTomlSchemaJson", "toml-schema.json"}, {"kBuildScriptApiJson", "buildscript-api.json"}, - {"kTestFrameworksJson", "test-frameworks.json"}, }}; std::string module = "// Generated by mcpp-ide's build.mcpp from spec/ (not part of the repository).\n" diff --git a/docs/architecture.md b/docs/architecture.md index b02eac0..c7a9e30 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -75,29 +75,37 @@ mcppls what a compiler answers better. ## A test run, end to end -The editors show tests as `mcpp test` states them (design 2026-11-10). mcpp builds and runs the -programs; mtest's protocol is how a program's cases report to mcpp; mtest's store (spec P12) is the form -in which mcpp keeps a run (`target/mtest`, or `[test] store`). The backend reads those three; it runs no -other test runner. +The editors show tests as `mcpp test` states them, by the mcpp test spec (mcpp docs/specs/test.md). mcpp +builds and runs the programs and states their cases and results; a framework is whatever library a +program is written with, and the backend knows none by name. What a framework offers beyond the base, +it states of itself (`mcpp test --guide`, B6). 1. `test/list` runs `mcpp test --list` and answers each program with its files, its `main` and its parts. A protocol program of several files (`[test] programs = "directory"`) runs as one process per file, - `/`. mcpp states the parts with `test-parts` (mcpp#917); before it, the backend derives - them as mcpp 2026.11.9.1 forms them. The programs are kept per project. -2. `test/cases` reads the cases of every file of a program with mtest's static manifest (their kinds, - tags and the bounds on their metrics), and with `list` the program's own listing (`--cases`). -3. A test job runs `mcpp test --message-format json`. Each record becomes `test/result`, with the - program it belongs to, the part it is and its cases, each status mapped (`xfail` passes, `xpass` - fails, `invalid` is errored). With `test-case-stream` (mcpp#917) each case is sent as it begins and - ends (`test/caseEvent`). -4. VS Code marks the Testing view's items, a failure with both sides as a diff, and ends a case's line - with its duration and metrics. CLion fills an ID-based test tree: a suite per program and per file, a - comparison failure when both sides are stated. Neovim writes `mcpp://tests`, the quickfix list and - the signs. -5. `test/insights` reads the store's last runs (20 by default) and answers the last run, its failures - and since when, the slowest and the flaky cases, each metric against its bound, and the schedule - with its critical path. VS Code and CLion draw it with one page (`webui/tests.html`); Neovim prints - it. + `/`. mcpp states the parts with `test-parts`; before it, the backend derives them as + mcpp 2026.11.9.1 forms them. The programs are kept per project. +2. `test/cases` answers a program's cases as the program lists them (`mcpp test --cases`, B3): one + listing of the project, kept until the project changes. +3. A test job runs `mcpp test --message-format json` with one set of arguments for every test command + (the profile, the target, `mcpp.task.testArgs`). Each record becomes `test/result`, with the program + it belongs to, the part it is and its cases, each status mapped (`xfail` passes, `xpass` fails, + `invalid` is errored); a case carries its checks (O3) and its own output (O8). With + `test-case-stream` each case is sent as it begins and ends (`test/caseEvent`). A test job runs in a + lane of its own: two tests, or a test and a build, do not wait for each other. +4. Where a framework states a test session (`test-session`, O9) and mcpp gives launches + (`test-launch-records`, B5), `test/runCases` runs the cases through it: the session is started once + per project, speaks JSON-RPC on its standard streams, answers a run at once and states its build, + its cases and its end as notifications, which become the job's `test/caseEvent` and `job/finished`. + The session ends with its input, so it ends with the backend. `mcpp.test.session = "off"` runs + every case through `mcpp test`. +5. VS Code marks the Testing view's items, one message per failed check (a diff where both sides are + stated), and ends a case's line with its duration, its failed checks and its metrics. CLion fills an + ID-based test tree: a suite per program and per file, a comparison failure when both sides are + stated. Neovim writes `mcpp://tests`, the quickfix list and the signs. +6. `test/insights` reads the last runs mcpp keeps (`target/mtest`, or `[test] store`; 20 by default) + and answers the last run, its failures and since when, the slowest and the flaky cases, each + metric against its bound, and the schedule with its critical path. VS Code and CLion draw it with + one page (`webui/tests.html`); Neovim prints it. Every step reads what the project's mcpp states and nothing more: an older mcpp gives programs and their results, and a view that needs a later feature is left out (`capabilities/explain`). diff --git a/mcpp.lock b/mcpp.lock index 477ed06..0c5ad56 100644 --- a/mcpp.lock +++ b/mcpp.lock @@ -89,13 +89,6 @@ source = "index+mcpplibs@0.4.2" hash = "fnv1a:c06e4ca4f73f228d" targets = ["aarch64-openkal", "x86_64-linux-gnu", "x86_64-openkal"] -[package."mcpplibs.mtest.manifest"] -namespace = "mcpplibs.mtest" -version = "0.5.2" -source = "index+mcpplibs.mtest@0.5.2" -hash = "fnv1a:0caef37dd958cfef" -targets = ["aarch64-openkal", "x86_64-linux-gnu", "x86_64-openkal"] - [package."nlohmann.json"] namespace = "nlohmann" version = "3.12.0" diff --git a/mcpp.toml b/mcpp.toml index e2a6337..db96ded 100644 --- a/mcpp.toml +++ b/mcpp.toml @@ -9,7 +9,7 @@ # M2/M3), where the system it runs upon is answered when the image is loaded. [package] name = "mcpp-ide" -version = "1.4.0" +version = "1.5.0" description = "The IDE backend for mcpp: sessions, jobs, records and view models over mcpp's control contract" license = "Apache-2.0" authors = ["mcpp-community"] @@ -80,8 +80,6 @@ main = "src/bin/dev.cpp" # runtime and loader remain the platform's. mcpplibs.os = "0.4.2" nlohmann.json = "3.12.0" -# mtest's static case manifest: the cases a test file declares, read as mtest reads them. -mcpplibs.mtest.manifest = "0.5.2" # The tests are mtest's (mtest docs/layout.md): tests/unit, tests/e2e, tests/cts and tests/bench, each # directory one program whose files declare their cases; what every test shares is testsupport/, a diff --git a/mide-plugins/clion/README.md b/mide-plugins/clion/README.md index a021d5c..b3fd303 100644 --- a/mide-plugins/clion/README.md +++ b/mide-plugins/clion/README.md @@ -99,7 +99,7 @@ An individual packaged plugin can be checked without rebuilding the backend: ```bash xvfb-run -a gradle --no-daemon -p integration-tests test \ - -PpluginZip=/absolute/path/mcpp-clion-1.4.0.zip \ + -PpluginZip=/absolute/path/mcpp-clion-1.5.0.zip \ -PmcpplsZip=/absolute/path/mcppls-clion-0.21.0.zip \ -PideVersion=2026.1.5.2 ``` diff --git a/mide-plugins/clion/gradle.properties b/mide-plugins/clion/gradle.properties index 2bdbf07..91014c7 100644 --- a/mide-plugins/clion/gradle.properties +++ b/mide-plugins/clion/gradle.properties @@ -2,7 +2,7 @@ # where it installs (2026.1 and later); the verifier checks both ends. platformType = CL platformVersion = 2026.2.3 -pluginVersion = 1.4.0 +pluginVersion = 1.5.0 # The mcppls CLion plugin this one depends on (io.github.sunrisepeak.mcppls): # its release's one package for every system, fetched for the sandbox, the tests and the verifier. mcpplsPluginVersion = 0.21.0 diff --git a/mide-plugins/clion/src/main/kotlin/io/github/mcppcommunity/mcppclion/McppSession.kt b/mide-plugins/clion/src/main/kotlin/io/github/mcppcommunity/mcppclion/McppSession.kt index 0a94d9f..b3d1760 100644 --- a/mide-plugins/clion/src/main/kotlin/io/github/mcppcommunity/mcppclion/McppSession.kt +++ b/mide-plugins/clion/src/main/kotlin/io/github/mcppcommunity/mcppclion/McppSession.kt @@ -63,14 +63,14 @@ class McppSession(private val project: Project) : Disposable { if (key == "mcpp.ide.backendPath") restart() else pushSettings() }) on("manifest/changed") { forgetSources() } - // A build ended: a test program can list the cases its macros generate (§7.3). + // A build or a test ended: the programs are listed again (mcpp test spec B3). on("job/finished") { p -> val o = p.asJsonObject if (o["status"]?.asString == "ok" && o["kind"]?.asString in setOf("build", "test")) { cases.keys.toList().forEach { key -> key.split('\u0000').takeIf { it.size == 2 }?.let { refreshCases(it[0], it[1], list = true) } } } } - // A C++ source saved: what was read from it (main, a framework's cases) is read again. + // A C++ source saved: what was read from it (main) is read again. project.messageBus.connect(this).subscribe(com.intellij.openapi.vfs.VirtualFileManager.VFS_CHANGES, object : com.intellij.openapi.vfs.newvfs.BulkFileListener { override fun after(events: MutableList) { @@ -297,6 +297,8 @@ class McppSession(private val project: Project) : Disposable { // when unknown, and the editor's markers recomputed when the answer arrives. private val sources = java.util.concurrent.ConcurrentHashMap() private val cases = java.util.concurrent.ConcurrentHashMap() + // The programs whose cases are being asked for: one listing at a time each. + private val asking = java.util.concurrent.ConcurrentHashMap.newKeySet() fun entriesFor(path: String): JsonObject? { val known = sources[path] @@ -322,30 +324,33 @@ class McppSession(private val project: Project) : Disposable { fun casesFor(root: String, test: String): JsonObject? { val known = cases["$root\u0000$test"] - if (known == null) refreshCases(root, test, list = false) + if (known == null) refreshCases(root, test, list = true) return known } fun refreshCases(root: String, test: String, list: Boolean) { + val key = "$root\u0000$test" + if (!asking.add(key)) return request("test/cases", JsonObject().apply { addProperty("root", root) addProperty("test", test) addProperty("list", list && TrustedProjects.isProjectTrusted(project)) - }).thenAccept { r -> - cases["$root\u0000$test"] = r.asJsonObject + }).whenComplete { r, error -> + asking.remove(key) + if (error != null) return@whenComplete + cases[key] = r.asJsonObject restartMarkers() } } - /** A source changed: what was read from it is read again on its next use. */ + /** A source changed: what was read from it is read again on its next use. Its program's cases + * stay as they were listed until a build or a test lists them again. */ fun forgetSource(path: String) { sources.remove(path) - cases.keys.removeIf { it.split('\u0000').getOrNull(1)?.let { test -> path.replace('\\', '/').endsWith("/tests/$test.cpp") } == true } } private fun forgetSources() { sources.clear() - cases.clear() } private fun restartMarkers() = com.intellij.openapi.application.ApplicationManager.getApplication().invokeLater { diff --git a/mide-plugins/clion/src/main/kotlin/io/github/mcppcommunity/mcppclion/run/McppProducers.kt b/mide-plugins/clion/src/main/kotlin/io/github/mcppcommunity/mcppclion/run/McppProducers.kt index aab7ff7..5ce3769 100644 --- a/mide-plugins/clion/src/main/kotlin/io/github/mcppcommunity/mcppclion/run/McppProducers.kt +++ b/mide-plugins/clion/src/main/kotlin/io/github/mcppcommunity/mcppclion/run/McppProducers.kt @@ -1,9 +1,10 @@ // Run configurations from context, and the ▶ in the gutter (mcpp design 2026-10-30 part 2 §6.3, // §7.4). What a file is -- a program's `main`, a test's, or neither -- is the backend's answer // (source/entries); where `main` is, the language service's: the ▶ stands on the PSI element `main` -// on the line the backend read. A test program of a framework has a ▶ on each case's macro -// (test/cases), which runs that case. Every ▶ comes from the same producer, so Run and Debug are -// offered alike. +// on the line the backend read. A test program that implements the test protocol has a ▶ on the +// first element of the line each case is declared on, as the program lists its cases (test/cases, +// mcpp test spec B3), which runs that case. Every ▶ comes from the same producer, so Run and Debug +// are offered alike. package io.github.mcppcommunity.mcppclion.run import com.google.gson.JsonObject @@ -17,6 +18,8 @@ import com.intellij.icons.AllIcons import com.intellij.openapi.util.Ref import com.intellij.openapi.vfs.VirtualFile import com.intellij.psi.PsiElement +import com.intellij.psi.PsiWhiteSpace +import com.intellij.psi.util.PsiTreeUtil import io.github.mcppcommunity.mcppclion.McppSession import java.io.File @@ -40,14 +43,24 @@ private fun lineOf(element: PsiElement): Int? { internal fun entriesOf(session: McppSession, file: VirtualFile): JsonObject? = session.entriesFor(file.path)?.takeIf { it["kind"]?.asString in setOf("program", "test") } -/** The case written at `line` of a test's source, by its full name, with its framework. */ -internal fun caseAt(session: McppSession, entries: JsonObject, line: Int): Pair? { +/** The case declared at `line` of a test's source, by its identifier, as its program lists it. */ +internal fun caseAt(session: McppSession, entries: JsonObject, line: Int): String? { val root = entries["root"]?.asString ?: return null val test = entries["test"]?.asString ?: return null + val file = entries["file"]?.asString ?: return null val cases = session.casesFor(root, test) ?: return null - val framework = cases["framework"]?.takeIf { !it.isJsonNull }?.asString ?: return null - val hit = cases.getAsJsonArray("cases")?.firstOrNull { it.asJsonObject["line"]?.asInt == line } ?: return null - return hit.asJsonObject["name"].asString to framework + if (cases["protocol"]?.takeIf { !it.isJsonNull } == null) return null + val hit = cases.getAsJsonArray("cases")?.map { it.asJsonObject }?.firstOrNull { c -> + c["line"]?.asInt == line && (c["file"]?.asString?.let { File(it).canonicalPath == File(file).canonicalPath } ?: true) + } ?: return null + return hit["name"].asString +} + +/** Whether an element is the first of its line: a case's ▶ stands on it. */ +private fun firstOfLine(element: PsiElement, line: Int): Boolean { + var before = PsiTreeUtil.prevLeaf(element) + while (before is PsiWhiteSpace) before = PsiTreeUtil.prevLeaf(before) + return before == null || lineOf(before) != line } class McppConfigurationProducer : LazyRunConfigurationProducer() { @@ -67,9 +80,8 @@ class McppConfigurationProducer : LazyRunConfigurationProducer { val program = p["program"]?.asString ?: p["test"].asString case(program, p["part"]?.asString, p, p["output"]?.takeIf { it.isJsonPrimitive }?.asString ?: "") diff --git a/mide-plugins/nvim/lua/mcpp/init.lua b/mide-plugins/nvim/lua/mcpp/init.lua index edea0e2..b11428f 100644 --- a/mide-plugins/nvim/lua/mcpp/init.lua +++ b/mide-plugins/nvim/lua/mcpp/init.lua @@ -128,7 +128,7 @@ function M.setup(opts) vim.api.nvim_create_autocmd({ 'BufReadPost', 'BufWritePost' }, { group = group, pattern = { '*.cpp', '*.cc', '*.cxx', '*.cppm', '*.ixx', '*.mpp' }, callback = function(a) - require('mcpp.tests').signs(a.buf) + require('mcpp.tests').signs(a.buf, a.event == 'BufReadPost') end, }) -- Once the backend is ready (McppStatus fires on every change of the statusline; the signs are @@ -146,7 +146,7 @@ function M.setup(opts) end for _, buf in ipairs(vim.api.nvim_list_bufs()) do if vim.api.nvim_buf_is_loaded(buf) then - require('mcpp.tests').signs(buf) + require('mcpp.tests').signs(buf, true) end end end, diff --git a/mide-plugins/nvim/lua/mcpp/session.lua b/mide-plugins/nvim/lua/mcpp/session.lua index b3be537..923473e 100644 --- a/mide-plugins/nvim/lua/mcpp/session.lua +++ b/mide-plugins/nvim/lua/mcpp/session.lua @@ -13,7 +13,7 @@ local uv = vim.uv or vim.loop local is_windows = vim.fn.has('win32') == 1 -- mcpp-nvim's version: mcpp-ide's, whose release carries it. -M.version = '1.4.0' +M.version = '1.5.0' -- Neovim's version, as its release names it. function M.nvim_version() diff --git a/mide-plugins/nvim/lua/mcpp/tests.lua b/mide-plugins/nvim/lua/mcpp/tests.lua index d2df27c..bc259c4 100644 --- a/mide-plugins/nvim/lua/mcpp/tests.lua +++ b/mide-plugins/nvim/lua/mcpp/tests.lua @@ -255,7 +255,7 @@ function M.finish() vim.fn.setqflist({}, ' ', { title = 'mcpp test', items = M.failures or {} }) for _, buf in ipairs(vim.api.nvim_list_bufs()) do if vim.api.nvim_buf_is_loaded(buf) then - M.signs(buf) + M.signs(buf, true) end end end @@ -278,8 +278,10 @@ local function place(buf, line, status, text) end -- The signs of a buffer: ▶ on a program's or a test's `main` and on each case its file declares, or --- the last result, with what the case's line says. -function M.signs(buf) +-- the last result, with what the case's line says. The cases are the program's own listing (mcpp +-- test spec B3): `list` lists the project again when it changed since; without it the last listing +-- answers, so that writing a file builds nothing. +function M.signs(buf, list) local name = vim.api.nvim_buf_get_name(buf) if name == '' or vim.bo[buf].buftype ~= '' or not session.state.ready then return @@ -296,7 +298,7 @@ function M.signs(buf) if entries.kind ~= 'test' then return end - session.request('test/cases', { root = entries.root, test = entries.test }, function(cerr, cases) + session.request('test/cases', { root = entries.root, test = entries.test, list = list or false }, function(cerr, cases) if cerr or not cases or not vim.api.nvim_buf_is_valid(buf) then return end diff --git a/mide-plugins/vscode/data/config-registry.json b/mide-plugins/vscode/data/config-registry.json index cd46686..ad646b4 100644 --- a/mide-plugins/vscode/data/config-registry.json +++ b/mide-plugins/vscode/data/config-registry.json @@ -307,17 +307,21 @@ "description": "A CodeLens above the main of a program or a test: Run and Debug. Where C++ Modules serves the file its position is the language service's; otherwise the line is read from the text." }, { - "key": "mcpp.test.macros", - "type": "array", - "default": [], + "key": "mcpp.test.session", + "type": "string", + "default": "auto", + "enum": [ + "auto", + "off" + ], "scope": "resource", "group": "tasks", "order": 150, "tier": "advanced", "applies": "immediate", - "since": "1.1.0", - "title": "Add a test framework's macros", - "description": "Macros that write a test framework's cases, each as framework:MACRO (gtest:MY_TEST). A test program's cases are listed only when its source calls one of its framework's macros." + "since": "1.5.0", + "title": "Run cases through the framework's test session", + "description": "auto: when the project's test framework states a test session (mcpp test --guide), cases are run, added and cancelled through it; off: each run of cases is a run of mcpp test." }, { "key": "mcpp.languageService.menuItems", diff --git a/mide-plugins/vscode/data/i18n/zh-cn.json b/mide-plugins/vscode/data/i18n/zh-cn.json index 31acddf..6eab4d9 100644 --- a/mide-plugins/vscode/data/i18n/zh-cn.json +++ b/mide-plugins/vscode/data/i18n/zh-cn.json @@ -683,5 +683,6 @@ "Schedule": "调度", "worker {0}": "工作线程 {0}", "Critical path": "关键路径", - "Metrics appear when the project's mcpp states them (test-metrics, mcpp#917).": "项目的 mcpp 声明指标后才会显示（test-metrics，mcpp#917）。" + "Metrics appear when the project's mcpp states them (test-metrics, mcpp#917).": "项目的 mcpp 声明指标后才会显示（test-metrics，mcpp#917）。", + "fatal: the rest of the case did not run": "致命：该用例余下的部分未运行" } diff --git a/mide-plugins/vscode/docs/settings.md b/mide-plugins/vscode/docs/settings.md index 9291c46..0f75f91 100644 --- a/mide-plugins/vscode/docs/settings.md +++ b/mide-plugins/vscode/docs/settings.md @@ -254,19 +254,19 @@ A CodeLens above the main of a program or a test: Run and Debug. Where C++ Modul | Applies | Takes effect immediately | | Since | 1.1.0 | -### `mcpp.test.macros` — advanced +### `mcpp.test.session` — advanced -Macros that write a test framework's cases, each as framework:MACRO (gtest:MY_TEST). A test program's cases are listed only when its source calls one of its framework's macros. +auto: when the project's test framework states a test session (mcpp test --guide), cases are run, added and cancelled through it; off: each run of cases is a run of mcpp test. -书写测试框架用例的宏，每项写作 框架:宏名（gtest:MY_TEST）。测试程序的源文件调用了其框架的宏时才列举它的用例。 +auto：工程的测试框架声明了测试会话（mcpp test --guide）时，经由它运行、加入和取消用例；off：每次运行用例都是一次 mcpp test。 | | | |---|---| -| Type | `array` | -| Default | `[]` | +| Type | `string`: `auto` \| `off` | +| Default | `"auto"` | | Scope | per folder | | Applies | Takes effect immediately | -| Since | 1.1.0 | +| Since | 1.5.0 | ## C++ Modules language service diff --git a/mide-plugins/vscode/package-lock.json b/mide-plugins/vscode/package-lock.json index 13da408..9b018ab 100644 --- a/mide-plugins/vscode/package-lock.json +++ b/mide-plugins/vscode/package-lock.json @@ -1,12 +1,12 @@ { "name": "mcpp-vscode", - "version": "1.4.0", + "version": "1.5.0", "lockfileVersion": 3, "requires": true, "packages": { "": { "name": "mcpp-vscode", - "version": "1.4.0", + "version": "1.5.0", "license": "Apache-2.0", "devDependencies": { "@mermaid-js/tiny": "12.2.0", diff --git a/mide-plugins/vscode/package.json b/mide-plugins/vscode/package.json index dbe6474..87074d4 100644 --- a/mide-plugins/vscode/package.json +++ b/mide-plugins/vscode/package.json @@ -2,7 +2,7 @@ "name": "mcpp-vscode", "displayName": "%displayName%", "description": "%description%", - "version": "1.4.0", + "version": "1.5.0", "publisher": "mcpp-community", "license": "Apache-2.0", "icon": "media/logo.png", @@ -647,13 +647,17 @@ "description": "%mcpp.codeLens.enable.title%", "markdownDescription": "%mcpp.codeLens.enable.description%" }, - "mcpp.test.macros": { - "type": "array", - "default": [], + "mcpp.test.session": { + "type": "string", + "default": "auto", + "enum": [ + "auto", + "off" + ], "scope": "resource", "order": 20150, - "description": "%mcpp.test.macros.title%", - "markdownDescription": "%mcpp.test.macros.description%" + "description": "%mcpp.test.session.title%", + "markdownDescription": "%mcpp.test.session.description%" }, "mcpp.languageService.menuItems": { "type": "boolean", diff --git a/mide-plugins/vscode/package.nls.json b/mide-plugins/vscode/package.nls.json index 1465565..e29ae96 100644 --- a/mide-plugins/vscode/package.nls.json +++ b/mide-plugins/vscode/package.nls.json @@ -275,6 +275,6 @@ "command.mcpp.exportReport.title": "Export IDE Report", "mcpp.codeLens.enable.title": "Show Run and Debug above main", "mcpp.codeLens.enable.description": "A CodeLens above the main of a program or a test: Run and Debug. Where C++ Modules serves the file its position is the language service's; otherwise the line is read from the text.", - "mcpp.test.macros.title": "Add a test framework's macros", - "mcpp.test.macros.description": "Macros that write a test framework's cases, each as framework:MACRO (gtest:MY_TEST). A test program's cases are listed only when its source calls one of its framework's macros." + "mcpp.test.session.title": "Run cases through the framework's test session", + "mcpp.test.session.description": "auto: when the project's test framework states a test session (mcpp test --guide), cases are run, added and cancelled through it; off: each run of cases is a run of mcpp test." } diff --git a/mide-plugins/vscode/package.nls.zh-cn.json b/mide-plugins/vscode/package.nls.zh-cn.json index 1e39c5c..5bd5886 100644 --- a/mide-plugins/vscode/package.nls.zh-cn.json +++ b/mide-plugins/vscode/package.nls.zh-cn.json @@ -275,6 +275,6 @@ "command.mcpp.exportReport.title": "导出 IDE 报告", "mcpp.codeLens.enable.title": "在 main 上方显示运行与调试", "mcpp.codeLens.enable.description": "在程序或测试的 main 上方显示 CodeLens：运行、调试。C++ Modules 服务该文件时位置取自语言服务，否则从文本读出行号。", - "mcpp.test.macros.title": "追加测试框架的宏", - "mcpp.test.macros.description": "书写测试框架用例的宏，每项写作 框架:宏名（gtest:MY_TEST）。测试程序的源文件调用了其框架的宏时才列举它的用例。" + "mcpp.test.session.title": "通过框架的测试会话运行用例", + "mcpp.test.session.description": "auto：工程的测试框架声明了测试会话（mcpp test --guide）时，经由它运行、加入和取消用例；off：每次运行用例都是一次 mcpp test。" } diff --git a/mide-plugins/vscode/src/cli/controller.ts b/mide-plugins/vscode/src/cli/controller.ts index 83f6b6f..6eb93e8 100644 --- a/mide-plugins/vscode/src/cli/controller.ts +++ b/mide-plugins/vscode/src/cli/controller.ts @@ -22,6 +22,7 @@ import { runMcppCommand } from "./process"; import { McppOperationRegistry, classifyTaskExit, + projectTaskIsExclusive, projectTaskPlan, TASK_ARGUMENT_SETTINGS, shouldRefreshLanguageServerAfterTask, @@ -355,11 +356,17 @@ export class McppCliController { // `mcpp.runtime.concurrency` picks the mutual-exclusion scope: one task at a // time per project (the default), or one task at a time for the whole window. + // Only a task that removes or replaces what others read is held to it + // (`clean`, `pack`): mcpp builds one configuration at a time itself and + // releases its directory before a program or a test runs, so a build, a + // run and a test proceed beside each other (design 2026-11-10 supplement, T14). + const exclusive = projectTaskIsExclusive(kind); const globalScope = read("mcpp.runtime.concurrency") === "global"; const token: OperationToken = {}; - const active = globalScope - ? this.operations.beginGlobal(token) - : this.operations.beginProject(project.root, token); + const active = !exclusive ? undefined + : globalScope + ? this.operations.beginGlobal(token) + : this.operations.beginProject(project.root, token); if (active !== undefined) { const labels = controllerLabels(); const choice = await vscode.window.showWarningMessage( @@ -388,7 +395,9 @@ export class McppCliController { ); this.appendTaskCompletion(project.root, plan.title, plan.args, completion); } finally { - if (globalScope) { + if (!exclusive) { + // nothing was held + } else if (globalScope) { this.operations.finishGlobal(token); } else { this.operations.finishProject(project.root, token); diff --git a/mide-plugins/vscode/src/cli/tasks.ts b/mide-plugins/vscode/src/cli/tasks.ts index a51deb6..995b5d6 100644 --- a/mide-plugins/vscode/src/cli/tasks.ts +++ b/mide-plugins/vscode/src/cli/tasks.ts @@ -1,6 +1,16 @@ import { t } from "../i18n/t"; export type ProjectTaskKind = "build" | "run" | "test" | "clean" | "pack"; + +/** + * Whether a project task is held to `mcpp.runtime.concurrency`: one that removes or replaces what + * other tasks read (`clean` its target/, `pack` the dist tree). A build, a run and a test are not: + * mcpp builds one configuration at a time and releases its directory before a program or a test + * runs (mcpp#915). + */ +export function projectTaskIsExclusive(kind: ProjectTaskKind): boolean { + return kind === "clean" || kind === "pack"; +} export type TaskState = "succeeded" | "failed" | "cancelled"; export interface ProjectTaskPlan { diff --git a/mide-plugins/vscode/src/ide/debug.ts b/mide-plugins/vscode/src/ide/debug.ts index a23d8e1..7bef1d7 100644 --- a/mide-plugins/vscode/src/ide/debug.ts +++ b/mide-plugins/vscode/src/ide/debug.ts @@ -69,7 +69,8 @@ export class Debugging implements vscode.DebugConfigurationProvider, vscode.Disp root: project.root, // A test's program (mcpp test --print-launch), where the project's mcpp describes one. ...(config.test ? { test: config.test } : {}), - ...(config.framework ? { framework: config.framework, cases: Array.isArray(config.cases) ? config.cases : [] } : {}), + // Some of its cases (mcpp test spec P4), as the program lists them. + ...(Array.isArray(config.cases) && config.cases.length > 0 ? { cases: config.cases } : {}), ...(config.target !== undefined ? { target: config.target } : {}), ...(config.profile ? { profile: config.profile } : {}), ...(config.engine ? { engine: config.engine } : {}), diff --git a/mide-plugins/vscode/src/ide/mapping.ts b/mide-plugins/vscode/src/ide/mapping.ts index a52700d..31726f5 100644 --- a/mide-plugins/vscode/src/ide/mapping.ts +++ b/mide-plugins/vscode/src/ide/mapping.ts @@ -24,7 +24,7 @@ export function testId(root: string, member: string, test: string): string { return `${root}::${member}::${test}`; } -/** What separates a case's name from its program's id: a name is the framework's, and may hold "::". */ +/** What separates a case's name from its program's id: a case's identifier may hold "::". */ const CASE = "\u001f"; /** A case of a test program (design 2026-10-30 part 2 §7): its program's id, then its full name. */ @@ -39,16 +39,6 @@ export function parseCaseId(id: string): { root: string; member: string; test: s return program === undefined ? undefined : { ...program, name: id.slice(at + 1) }; } -/** A case result's name as its source names it: GoogleTest's instance prefix and parameter dropped. */ -export function baseCaseName(framework: string | null, name: string): string { - if (framework !== "gtest") return name; - const dot = name.indexOf("."); - if (dot < 0) return name; - const suiteParts = name.slice(0, dot).split("/").filter((part) => part !== "" && !/^\d+$/.test(part)); - const test = name.slice(dot + 1).split("/")[0]; - return `${suiteParts[suiteParts.length - 1] ?? ""}.${test}`; -} - /** VS Code's SymbolKind.Function (LSP numbers it 12). */ const FUNCTION_SYMBOL = 11; @@ -118,15 +108,21 @@ export function metricText(name: string, value: number, unit: string | undefined } /** What follows a case's line after a run: its verdict, its duration, its metrics against their bounds. */ -export function caseSummary(o: Pick): string { +export function caseSummary(o: Pick): string { const mark = o.status === "passed" ? "✓" : o.status === "skipped" ? "○" : "✗"; const parts = [`${mark}${o.expectedFailure ? " xfail" : ""}`]; if (o.durationMs !== undefined && o.status !== "skipped") parts.push(`${o.durationMs} ms`); + if (o.checks && o.checks.failed > 0) parts.push(checksText(o.checks)); for (const m of o.metrics ?? []) parts.push(metricText(m.name, m.value, m.unit, o.bounds?.find((b) => b.metric === m.name))); return parts.join(" · "); } +/** A case's checks (mcpp test spec O3): `2 of 5 checks failed`. */ +export function checksText(checks: { held: number; failed: number }): string { + return `${checks.failed} of ${checks.held + checks.failed} checks failed`; +} + /** The test tags of a case: `kind:` and each of its own (VS Code's `@tag` filter selects by them). */ export function caseTags(kind: string | undefined, tags: readonly string[] | undefined): string[] { return [...(kind ? [`kind:${kind}`] : []), ...(tags ?? [])]; diff --git a/mide-plugins/vscode/src/ide/protocol.ts b/mide-plugins/vscode/src/ide/protocol.ts index 247a481..b2a9872 100644 --- a/mide-plugins/vscode/src/ide/protocol.ts +++ b/mide-plugins/vscode/src/ide/protocol.ts @@ -180,7 +180,6 @@ export interface SourceEntries { test?: string; /** A file of a program of several files: the part of the program it is (`unit/a`); `test` is the program. */ part?: string; - framework?: string; main?: { line: number; approximate?: boolean }; } @@ -199,11 +198,11 @@ export interface Metric { better?: "lower" | "higher"; } -/** A case of `test/cases`: its place, and what the test protocol states of it (design 2026-11-10). */ +/** A case of `test/cases`, as its program lists it (mcpp test spec B3): its place and what it states. */ export interface TestCase { name: string; line?: number; - source: "static" | "listed"; + source: "listed"; /** The file that declares it, in a program of several files. */ file?: string; kind?: string; @@ -212,13 +211,18 @@ export interface TestCase { bounds?: MetricBound[]; } -/** `test/cases` (§7.3): a test program's cases, from its sources and, once listed, from the program. */ +/** + * `test/cases`: a test program's cases as it lists them (mcpp test spec B3). `protocol` is the version + * of the test protocol it implements, or null: a program of none is one case. `framework` is the name + * the project's framework states (B6), shown beside the program; nothing is decided by it. + */ export interface TestCases { root: string; test: string; file: string; /** The files of a program of several files. */ files?: string[]; + protocol?: number | null; framework: string | null; listed?: boolean; main?: { line: number; approximate?: boolean }; @@ -235,13 +239,15 @@ export interface TestProgram { layout?: "file" | "directory"; } -/** A failure of a case: its place (0-based line) and, when stated, both sides. */ +/** A failure of a case (mcpp test spec O3): its place (0-based line), and when stated both sides, the statement and whether it ended the case. */ export interface CaseFailure { path: string; line: number; message: string; expected?: string; actual?: string; + source?: string; + fatal?: boolean; } /** A case as it ended, as the backend maps the test protocol's statuses. */ @@ -259,6 +265,10 @@ export interface CaseOutcome { tags?: string[]; expectedFailure?: boolean; unexpectedPass?: boolean; + /** Of the checks the case ran, those that held and those that did not (O3). */ + checks?: { held: number; failed: number }; + /** What the case wrote (O8): its own output, as its program stated it. */ + output?: string; } /** `test/caseEvent`: a case began or ended while its program runs (test-case-stream, mcpp#917). */ @@ -272,7 +282,7 @@ export interface CaseEvent extends Partial { tMs?: number; } -/** `test/caseResult`: one case's outcome, read from the framework's report. */ +/** `test/caseResult`: one case's outcome, from mcpp's records. */ /** Where a failure is: the first `file:line` its output names, the line 0-based. */ export interface FailureLocation { path: string; @@ -289,7 +299,7 @@ export interface CaseResult extends CaseOutcome { status: "passed" | "failed" | "skipped" | "errored"; durationMs: number; message: string; - /** The case's own output (GoogleTest's console between `[ RUN ]` and its result); empty without boundaries. */ + /** The case's own output (O8), or a failure's text; empty when neither. */ output?: string; outputTruncated?: boolean; location?: FailureLocation | null; diff --git a/mide-plugins/vscode/src/ide/testing.ts b/mide-plugins/vscode/src/ide/testing.ts index c45c534..a8c7632 100644 --- a/mide-plugins/vscode/src/ide/testing.ts +++ b/mide-plugins/vscode/src/ide/testing.ts @@ -2,26 +2,31 @@ * mcpp's tests in VS Code's Testing view (design 2026-10-30 part 2 §7; design 2026-11-10). The tests of * each project come from `test/list` (mcpp test --list: no build), each a program located at its * `main`; a program of several files (`[test] programs = "directory"`) has its files as children, - * each the part of the program a run gives a process. A program of a test framework has its cases as - * children of the file that declares them (`test/cases`, read from the sources), and after a build the - * program lists them (only in a trusted workspace). A case carries the test tags `kind:` and its + * each the part of the program a run gives a process. A program that implements the test protocol + * has its cases as children of the file that declares them, as the program lists them (`test/cases`, + * `mcpp test --cases`; only in a trusted workspace). A case carries the test tags `kind:` and its * own tags, so `@kind:bench` filters the view. * * A run of whole programs, or of one file's part, is one test job; a run of cases is `test/runCases`. * Results arrive per case as the program writes them when the project's mcpp states them - * (`test/caseEvent`, test-case-stream), else with each part's record (`test/result` and its cases). A - * failure that states both sides is a diff; after a run each case's line ends with its verdict, its - * duration and its metrics against their bounds. + * (`test/caseEvent`, test-case-stream), else with each part's record (`test/result` and its cases). + * Each failed check is a message at its place: a diff when it states both sides, with its statement + * and whether it ended the case; a case's own output is shown with it; after a run each case's line + * ends with its verdict, its duration, the checks that failed and its metrics against their bounds. + * + * NOTHING HERE KNOWS A TEST FRAMEWORK (design 2026-11-10 supplement, T14). Cases are what a program + * that implements the test protocol lists (mcpp test spec B3) and are run by identifier (P4); a program + * of no protocol is run whole. Tests run beside each other: each is a job of its own. */ import * as path from "node:path"; import * as vscode from "vscode"; import { t } from "../i18n/t"; import type { Jobs } from "./jobs"; -import { baseCaseName, caseId, caseSummary, caseTags, fileId, parseCaseId, parseFileId, parseTestId, testId } from "./mapping"; +import { caseId, caseSummary, caseTags, fileId, parseCaseId, parseFileId, parseTestId, testId } from "./mapping"; import type { IdeProjects } from "./projects"; import type { Session } from "./session"; -import type { CaseEvent, CaseOutcome, CaseResult, FailureLocation, JobFinished, TestCases, TestProgram, TestResult } from "./protocol"; +import type { CaseEvent, CaseFailure, CaseOutcome, CaseResult, FailureLocation, JobFinished, TestCases, TestProgram, TestResult } from "./protocol"; /** Text for the test terminal, whose lines end in `\r\n`. */ function terminal(text: string): string { @@ -48,21 +53,46 @@ function attribute(run: vscode.TestRun, item: vscode.TestItem, output: string | if (truncated) run.appendOutput(terminal(`${t("(the output was longer; its head and tail are shown)")}\n`), at, item); } -/** A failed case's message: a diff when the failure states both sides (VS Code compares them). */ -function failureMessage(root: string, o: CaseOutcome, item: vscode.TestItem): vscode.TestMessage { - const first = o.failures?.[0]; - const text = o.unexpectedPass ? t("The case passed, though it is expected to fail.") : o.message || o.reason || o.status; - const message = first?.expected !== undefined && first.actual !== undefined - ? vscode.TestMessage.diff(first.message || text, first.expected, first.actual) - : new vscode.TestMessage(text); - message.location = failureAt(root, o.location, item); - return message; +/** One failed check, as a person reads it: what did not hold, the statement, and whether the case went on. */ +function checkText(f: CaseFailure): string { + const lines = [f.message]; + if (f.source && f.source !== f.message) lines.push(f.source); + if (f.fatal) lines.push(t("fatal: the rest of the case did not run")); + return lines.join("\n"); +} + +/** + * A failed case's messages (mcpp test spec O3): one per check that did not hold, at its place, a diff + * when it states both sides (VS Code compares them); without a failure, the case's reason. + */ +function failureMessages(root: string, o: CaseOutcome, item: vscode.TestItem): vscode.TestMessage[] { + if (o.unexpectedPass) { + const m = new vscode.TestMessage(t("The case passed, though it is expected to fail.")); + m.location = failureAt(root, o.location, item); + return [m]; + } + const failures = o.failures ?? []; + if (failures.length === 0) { + const m = new vscode.TestMessage(o.message || o.reason || o.status); + m.location = failureAt(root, o.location, item); + return [m]; + } + return failures.map((f) => { + const m = f.expected !== undefined && f.actual !== undefined + ? vscode.TestMessage.diff(checkText(f), f.expected, f.actual) + : new vscode.TestMessage(checkText(f)); + m.location = failureAt(root, { path: f.path, line: f.line }, item); + return m; + }); } interface Program { root: string; member: string; test: string; + /** The program implements the test protocol: its cases are listed and run by identifier. */ + protocol: boolean; + /** The framework's name the project states, shown beside the program. */ framework: string | null; /** The cases by name, and the files' items by part. */ cases: Map; @@ -114,25 +144,41 @@ export class Testing implements vscode.Disposable { private readonly decorations = new CaseDecorations(); private readonly disposables: vscode.Disposable[] = [this.controller, this.decorations]; private readonly programs = new Map(); + /** A discovery the events asked for, run once they settle: a save, a build's end and a changed result arrive together. */ + private pending: { timer: ReturnType; list: boolean } | undefined; constructor(private readonly session: Session, private readonly jobs: Jobs, private readonly projects: IdeProjects) { - this.controller.resolveHandler = async (item) => { if (!item) await this.discover(); }; + this.controller.resolveHandler = async (item) => { if (!item) await this.discover(vscode.workspace.isTrusted); }; this.controller.refreshHandler = async () => { await this.discover(true); }; this.controller.createRunProfile(t("Run"), vscode.TestRunProfileKind.Run, (request, token) => this.run(request, token), true); this.disposables.push( - session.on("manifest/changed", () => void this.discover()), - session.on("result/changed", () => void this.discover()), - session.onDidStart.event(() => void this.discover()), - // A build ended: the programs can list the cases their macros generate. + session.on("manifest/changed", () => this.discoverSoon()), + session.on("result/changed", () => this.discoverSoon()), + session.onDidStart.event(() => this.discoverSoon(vscode.workspace.isTrusted)), + // A build ended: the programs list their cases (mcpp test spec B3). session.on("job/finished", (p: JobFinished) => { - if (p.status === "ok" && (p.kind === "build" || p.kind === "test") && vscode.workspace.isTrusted) void this.discover(true); + if (p.status === "ok" && (p.kind === "build" || p.kind === "test") && vscode.workspace.isTrusted) this.discoverSoon(true); }), vscode.workspace.onDidSaveTextDocument((doc) => { - if (doc.languageId === "cpp" && /[\\/]tests[\\/]/.test(doc.uri.fsPath)) void this.discover(); + if (doc.languageId === "cpp" && /[\\/]tests[\\/]/.test(doc.uri.fsPath)) this.discoverSoon(); }), + { dispose: () => { if (this.pending) clearTimeout(this.pending.timer); } }, ); } + /** A discovery after the events of a moment settle (400 ms), listing when any of them asked for it. */ + private discoverSoon(list = false): void { + const wanted = list || (this.pending?.list ?? false); + if (this.pending) clearTimeout(this.pending.timer); + this.pending = { + list: wanted, + timer: setTimeout(() => { + this.pending = undefined; + void this.discover(wanted); + }, 400), + }; + } + /** The program a test's `main` is, or the item of one file's part of it (the CodeLens and the Run menu run it). */ item(root: string, test: string, part?: string): vscode.TestItem | undefined { const program = this.programs.get(this.findProgramId(root, test) ?? ""); @@ -178,19 +224,20 @@ export class Testing implements vscode.Disposable { const id = testId(root, test.member, test.test); const home = test.main || test.files?.[0] || ""; const item = this.controller.createTestItem(id, test.member ? `${test.member} · ${test.test}` : test.test, home ? vscode.Uri.file(home) : undefined); - const program: Program = { root, member: test.member, test: test.test, framework: null, cases: new Map(), parts: new Map() }; + const program: Program = { root, member: test.member, test: test.test, protocol: false, framework: null, cases: new Map(), parts: new Map() }; let cases: TestCases | undefined; try { cases = await this.session.request("test/cases", { root, test: test.test, list }); } catch { cases = undefined; } - // The program at its `main` (F11); a framework's program at its first case when it has none. + // The program at its `main` (F11); a program of cases at its first case when it has none. const mainLine = cases?.main?.line ?? cases?.cases.find((c) => c.line !== undefined)?.line ?? 0; if (home) item.range = new vscode.Range(mainLine, 0, mainLine, 0); - if (cases?.framework) { + if (cases?.protocol) { + program.protocol = true; program.framework = cases.framework; - item.description = cases.framework; + if (cases.framework) item.description = cases.framework; } const caseItem = (c: TestCases["cases"][number]): vscode.TestItem => { const file = c.file || home; @@ -208,12 +255,12 @@ export class Testing implements vscode.Disposable { const child = this.controller.createTestItem(fileId(root, test.member, test.test, part.name), path.basename(part.file), vscode.Uri.file(part.file)); const own = (cases?.cases ?? []).filter((c) => c.file === part.file); child.range = new vscode.Range(own.find((c) => c.line !== undefined)?.line ?? 0, 0, own.find((c) => c.line !== undefined)?.line ?? 0, 0); - child.children.replace(cases?.framework ? own.map(caseItem) : []); + child.children.replace(program.protocol ? own.map(caseItem) : []); program.parts.set(part.name, child); return child; })); - } else if (cases?.framework) { - item.children.replace(cases.cases.map(caseItem)); + } else if (program.protocol) { + item.children.replace((cases?.cases ?? []).map(caseItem)); } this.programs.set(id, program); return item; @@ -221,19 +268,19 @@ export class Testing implements vscode.Disposable { /** A case's outcome, marked on its item; its line then ends with what the run said. */ private markCase(run: vscode.TestRun, program: Program, o: CaseOutcome, seen: Set, output?: string, truncated?: boolean): void { - const item = program.cases.get(o.case) ?? program.cases.get(baseCaseName(program.framework, o.case)); + const item = program.cases.get(o.case); if (!item) return; seen.add(item); const at = o.status === "failed" ? failureAt(program.root, o.location, item) : undefined; - // Every case's own output reaches its item, passed or not (D16). - attribute(run, item, output, truncated, at); + // Every case's own output reaches its item, passed or not (D16): what it wrote (O8), else what the record carries. + attribute(run, item, o.output || output, truncated, at); const ms = o.durationMs; if (o.status === "passed") run.passed(item, ms); else if (o.status === "skipped") { if (o.reason) run.appendOutput(terminal(`${o.reason}\n`), undefined, item); run.skipped(item); - } else if (o.status === "errored") run.errored(item, failureMessage(program.root, o, item), ms); - else run.failed(item, failureMessage(program.root, o, item), ms); + } else if (o.status === "errored") run.errored(item, failureMessages(program.root, o, item), ms); + else run.failed(item, failureMessages(program.root, o, item), ms); this.decorations.set(item, caseSummary(o)); } @@ -275,18 +322,18 @@ export class Testing implements vscode.Disposable { private async runCases(run: vscode.TestRun, token: vscode.CancellationToken, owner: string, items: vscode.TestItem[]): Promise { const program = this.programs.get(owner); - if (!program?.framework) return; + if (!program?.protocol) return; items.forEach((i) => run.enqueued(i)); // Every case of the program asked for: the filter is left out and the program runs them all. const names = items.length === program.cases.size ? [] : items.map((i) => parseCaseId(i.id)!.name); const seen = new Set(); const handle = await this.jobs.start( - { root: program.root, test: program.test, framework: program.framework, cases: names, color: true }, + { root: program.root, test: program.test, cases: names, color: true }, { output: (text) => run.appendOutput(terminal(text)), caseEvent: (e: CaseEvent) => this.onCaseEvent(run, program, e, seen), caseResult: (r: CaseResult) => { - const item = program.cases.get(r.case) ?? program.cases.get(baseCaseName(program.framework, r.case)); + const item = program.cases.get(r.case); if (item && seen.has(item)) return; this.markCase(run, program, r, seen, r.output, r.outputTruncated); }, diff --git a/mide-plugins/vscode/test/config/wiring.test.ts b/mide-plugins/vscode/test/config/wiring.test.ts index 371e43c..ceca121 100644 --- a/mide-plugins/vscode/test/config/wiring.test.ts +++ b/mide-plugins/vscode/test/config/wiring.test.ts @@ -46,7 +46,7 @@ const READ_INDIRECTLY: ReadonlyArray<{ key: string; file: string; evidence: stri evidence: "for (const key of Object.keys(props)) out[key] = config.get(key);", }, { - key: "mcpp.test.macros", + key: "mcpp.test.session", file: "src/ide/session.ts", evidence: "for (const key of Object.keys(props)) out[key] = config.get(key);", }, diff --git a/mide-plugins/vscode/test/ide/mapping.test.ts b/mide-plugins/vscode/test/ide/mapping.test.ts index 22fcb4a..361bb51 100644 --- a/mide-plugins/vscode/test/ide/mapping.test.ts +++ b/mide-plugins/vscode/test/ide/mapping.test.ts @@ -2,7 +2,7 @@ import * as assert from "node:assert/strict"; import { test } from "node:test"; -import { availableDebuggers, baseCaseName, caseId, caseSummary, caseTags, fileId, format, mainLineOf, measured, metricText, parseCaseId, parseFileId, parseTestId, severityOf, testId, zeroBased } from "../../src/ide/mapping"; +import { availableDebuggers, caseId, checksText, caseSummary, caseTags, fileId, format, mainLineOf, measured, metricText, parseCaseId, parseFileId, parseTestId, severityOf, testId, zeroBased } from "../../src/ide/mapping"; test("positions are 1-based in mcpp and 0-based in VS Code", () => { assert.deepEqual(zeroBased({ line: 1, column: 21 }), { line: 0, character: 20 }); @@ -33,11 +33,10 @@ test("a case's id holds its program's and its name, which may hold ::", () => { assert.equal(parseCaseId(testId("/w/p", "", "unit/math")), undefined); }); -test("a GoogleTest result is matched to its source's case without instance and parameter", () => { - assert.equal(baseCaseName("gtest", "Inst/Param.Works/1"), "Param.Works"); - assert.equal(baseCaseName("gtest", "Typed/0.Holds"), "Typed.Holds"); - assert.equal(baseCaseName("gtest", "Math.Adds"), "Math.Adds"); - assert.equal(baseCaseName("catch2", "a/b.c"), "a/b.c"); +test("a case's checks read as the count that failed of those it ran", () => { + assert.equal(checksText({ held: 3, failed: 2 }), "2 of 5 checks failed"); + assert.equal(caseSummary({ status: "failed", durationMs: 4, checks: { held: 3, failed: 2 } }), "✗ · 4 ms · 2 of 5 checks failed"); + assert.equal(caseSummary({ status: "passed", durationMs: 4, checks: { held: 3, failed: 0 } }), "✓ · 4 ms"); }); test("main is the language service's top-level function of that name", () => { diff --git a/protocol/protocol.schema.json b/protocol/protocol.schema.json index aa0ae6a..f04b69e 100644 --- a/protocol/protocol.schema.json +++ b/protocol/protocol.schema.json @@ -101,23 +101,23 @@ "result": { "$ref": "#/$defs/languageService" } }, "mcpp-ide/source/entries": { - "description": "What a C++ file is to its project (part 2 §6.3): a test's main (tests/**, named as mcpp names it), a program's main, or neither, with the line of main read from the text (approximate) and, for a test, its framework. Nothing is built or run.", + "description": "What a C++ file is to its project (part 2 §6.3): a test's main (tests/**, named as mcpp names it), a program's main, or neither, with the line of main read from the text (approximate). Nothing is built or run.", "params": { "type": "object", "required": ["file"], "properties": { "file": { "type": "string" }, "text": { "type": "string", "description": "The editor's unsaved text" }, "boundary": { "type": "string" } } }, "result": { "type": "object", "required": ["file", "kind"], "properties": { "file": { "type": "string" }, "root": { "type": "string" }, "kind": { "enum": ["program", "test", "none"] }, "test": { "type": "string" }, "part": { "type": "string", "description": "A file of a program of several files: the part of the program the file is (`unit/a`); `test` is the program (design 2026-11-10)" }, - "framework": { "type": "string" }, "main": { "type": "object", "required": ["line"], "properties": { "line": { "type": "integer" }, "approximate": { "type": "boolean" } } } } + "main": { "type": "object", "required": ["line"], "properties": { "line": { "type": "integer" }, "approximate": { "type": "boolean" } } } } } }, "mcpp-ide/test/cases": { - "description": "A test program's cases (part 2 §7.3): as its sources name them (a framework's macros; every file of a program of several files, each case with its file), and with `list` -- trusted, and only for a source that calls its framework's macros -- as the program lists them. Without a framework, no cases: the program is the unit. A case of the test protocol carries its kind, tags, expectation and the bounds on its metrics when its source or its listing states them (design 2026-11-10).", + "description": "A test program's cases as the program lists them (mcpp test spec B3, `mcpp test --cases`), in a trusted workspace and for a project whose mcpp states test-protocol: one listing of the project, kept. `list` lists again when the project changed since the last listing; without it the last listing answers, or none. A program that implements no protocol has no cases: it is the unit. A case carries its file, line, kind, tags, expectation and the bounds on its metrics as the listing states them; `framework` is the name the project's framework states of itself (B6), else empty.", "params": { "type": "object", "properties": { "root": { "type": "string" }, "test": { "type": "string" }, "file": { "type": "string" }, "list": { "type": "boolean" }, "target": { "type": "string" } } }, "result": { "type": "object", "required": ["root", "test", "framework", "cases"], "properties": { "root": { "type": "string" }, "test": { "type": "string" }, "file": { "type": "string" }, "framework": { "type": ["string", "null"] }, "listed": { "type": "boolean" }, "main": { "type": "object" }, "files": { "type": "array", "items": { "type": "string" } }, - "cases": { "type": "array", "items": { "type": "object", "required": ["name", "source"], "properties": { "name": { "type": "string" }, "line": { "type": "integer" }, "source": { "enum": ["static", "listed"] }, + "cases": { "type": "array", "items": { "type": "object", "required": ["name", "source"], "properties": { "name": { "type": "string" }, "line": { "type": "integer" }, "source": { "const": "listed" }, "file": { "type": "string" }, "kind": { "type": "string" }, "tags": { "type": "array", "items": { "type": "string" } }, "expect": { "type": "string" }, "bounds": { "$ref": "#/$defs/bounds" } } } } } } }, @@ -140,8 +140,8 @@ "features": { "type": "object", "properties": { "parts": { "type": "boolean" }, "live": { "type": "boolean" }, "metrics": { "type": "boolean" } } } } } }, "mcpp-ide/test/runCases": { - "description": "Runs some of a test program's cases (all of them when `cases` is empty): a test job whose program receives the framework's filter and report arguments after `--`. Each case's outcome is sent as test/caseResult when the job ends.", - "params": { "type": "object", "required": ["root", "test", "framework"], "properties": { "root": { "type": "string" }, "test": { "type": "string" }, "framework": { "type": "string" }, "cases": { "type": "array", "items": { "type": "string" } }, "profile": { "type": "string" }, "target": { "type": "string" }, "color": { "type": "boolean" } } }, + "description": "Runs some of a test program's cases (all of them when `cases` is empty), by identifier (mcpp test spec P4). Through the project's test session when its framework states one and its mcpp states test-launch-records (O9; `mcpp.test.session`): a job whose cases are sent as test/caseEvent as they begin and end. Else `mcpp test --case ...`: each case's outcome is sent as test/caseResult when the job ends. A `framework` is accepted and not read.", + "params": { "type": "object", "required": ["root", "test"], "properties": { "root": { "type": "string" }, "test": { "type": "string" }, "framework": { "type": "string" }, "cases": { "type": "array", "items": { "type": "string" } }, "profile": { "type": "string" }, "target": { "type": "string" }, "color": { "type": "boolean" } } }, "result": { "type": "object", "required": ["jobId"] } }, "mcpp-ide/projects/list": { @@ -497,17 +497,18 @@ "mcpp-ide/manifest/changed": { "params": { "type": "object", "required": ["root"] } }, "mcpp-ide/manifest/diagnostics": { "params": { "type": "object", "required": ["root", "diagnostics"] } }, "mcpp-ide/project/mcppChanged": { "description": "A job's envelope named another mcpp than the project's was asked as: it is asked again on its next use.", "params": { "type": "object", "required": ["root", "version"] } }, - "mcpp-ide/test/caseResult": { "description": "One case's outcome when a test/runCases job ended: read from the framework's report, or from each record of a protocol program (one per part of a program of several files), with the case's own output (GoogleTest's console between [ RUN ] and its result line; a protocol case's first failure) and a failure's location.", "params": { "allOf": [{ "$ref": "#/$defs/caseOutcome" }, { "type": "object", "required": ["jobId", "root", "test", "case", "status", "output", "outputTruncated"], "properties": { "test": { "type": "string" }, "program": { "type": "string" }, "part": { "type": "string" }, "output": { "type": "string" }, "outputTruncated": { "type": "boolean" } } }] } }, - "mcpp-ide/test/caseEvent": { "description": "A case began or ended while its program runs, as mcpp writes it (test-case-stream, mcpp#917): `begin` names the case; `end` carries its outcome. `program` is the program the part belongs to.", "params": { "allOf": [{ "type": "object", "required": ["jobId", "root", "program", "part", "case", "event"], "properties": { "program": { "type": "string" }, "part": { "type": "string" }, "member": { "type": "string" }, "event": { "enum": ["begin", "end"] }, "tMs": { "type": "number" } } }, { "$ref": "#/$defs/caseOutcomePartial" }] } }, + "mcpp-ide/test/caseResult": { "description": "One case's outcome when a test/runCases job ended, from each record of the program (one per part of a program of several files), with a failure's location and `output`: the case's first failure as text.", "params": { "allOf": [{ "$ref": "#/$defs/caseOutcome" }, { "type": "object", "required": ["jobId", "root", "test", "case", "status", "output", "outputTruncated"], "properties": { "test": { "type": "string" }, "program": { "type": "string" }, "part": { "type": "string" }, "output": { "type": "string" }, "outputTruncated": { "type": "boolean" } } }] } }, + "mcpp-ide/test/caseEvent": { "description": "A case began or ended while its program runs, as mcpp writes it (test-case-stream) or as the test session states it (O9): `begin` names the case; `end` carries its outcome. `program` is the program the part belongs to.", "params": { "allOf": [{ "type": "object", "required": ["jobId", "root", "program", "part", "case", "event"], "properties": { "program": { "type": "string" }, "part": { "type": "string" }, "member": { "type": "string" }, "event": { "enum": ["begin", "end"] }, "tMs": { "type": "number" } } }, { "$ref": "#/$defs/caseOutcomePartial" }] } }, "mcpp-ide/languageService/changed": { "description": "mcppls answered differently from the last time it was asked: another program or version, other actions, or another state of the workspace. An editor that lists the commands asks for them again.", "params": { "type": "object", "required": ["root", "state", "workspaceState"] } } }, "$defs": { "storedRun": { "type": "object", "required": ["id", "status"], "properties": { "id": { "type": "string" }, "started": { "type": "string" }, "status": { "type": "string" }, "ms": { "type": "number" }, "jobs": { "type": "integer" }, "counts": { "type": "object" } } }, "bounds": { "description": "The bounds a case states on its metrics (mtest protocol P6.5)", "type": "array", "items": { "type": "object", "required": ["metric"], "properties": { "metric": { "type": "string" }, "max": { "type": "number" }, "min": { "type": "number" } } } }, "metrics": { "description": "What a case measured (mtest protocol P6), when mcpp states it (test-metrics)", "type": "array", "items": { "type": "object", "required": ["name", "value"], "properties": { "name": { "type": "string" }, "value": { "type": "number" }, "unit": { "type": "string" }, "samples": { "type": "array", "items": { "type": "number" } }, "better": { "enum": ["lower", "higher"] }, "precision": { "type": "string" } } } }, - "caseFailure": { "type": "object", "required": ["path", "line", "message"], "properties": { "path": { "type": "string" }, "line": { "type": "integer", "minimum": 0 }, "message": { "type": "string" }, "expected": { "type": "string" }, "actual": { "type": "string" } } }, - "caseOutcome": { "description": "A case as it ended: pass is passed, fail failed, skip skipped, xfail passed with expectedFailure, xpass failed with unexpectedPass, invalid (a measurement that cannot be judged) errored. Lines are 0-based; message is the first failure with its place, else the reason.", "type": "object", "required": ["case", "status"], "properties": { "case": { "type": "string" }, "status": { "enum": ["passed", "failed", "skipped", "errored"] }, "durationMs": { "type": "integer" }, "reason": { "type": "string" }, "message": { "type": "string" }, "location": { "$ref": "#/$defs/failureLocation" }, "failures": { "type": "array", "items": { "$ref": "#/$defs/caseFailure" } }, "metrics": { "$ref": "#/$defs/metrics" }, "bounds": { "$ref": "#/$defs/bounds" }, "kind": { "type": "string" }, "tags": { "type": "array", "items": { "type": "string" } }, "expectedFailure": { "type": "boolean" }, "unexpectedPass": { "type": "boolean" } } }, - "caseOutcomePartial": { "type": "object", "properties": { "case": { "type": "string" }, "status": { "enum": ["passed", "failed", "skipped", "errored"] }, "durationMs": { "type": "integer" }, "failures": { "type": "array", "items": { "$ref": "#/$defs/caseFailure" } }, "metrics": { "$ref": "#/$defs/metrics" }, "bounds": { "$ref": "#/$defs/bounds" } } }, + "caseFailure": { "type": "object", "required": ["path", "line", "message"], "properties": { "path": { "type": "string" }, "line": { "type": "integer", "minimum": 0 }, "message": { "type": "string" }, "expected": { "type": "string" }, "actual": { "type": "string" }, "source": { "type": "string", "description": "The statement that failed (O3)" }, "fatal": { "type": "boolean", "description": "The case did not go on after it (O3)" } } }, + "caseChecks": { "type": "object", "description": "The checks a case ran (O3)", "properties": { "held": { "type": "integer", "minimum": 0 }, "failed": { "type": "integer", "minimum": 0 } } }, + "caseOutcome": { "description": "A case as it ended: pass is passed, fail failed, skip skipped, xfail passed with expectedFailure, xpass failed with unexpectedPass, invalid (a measurement that cannot be judged) errored. Lines are 0-based; message is the first failure with its place, else the reason.", "type": "object", "required": ["case", "status"], "properties": { "case": { "type": "string" }, "status": { "enum": ["passed", "failed", "skipped", "errored"] }, "durationMs": { "type": "integer" }, "reason": { "type": "string" }, "message": { "type": "string" }, "location": { "$ref": "#/$defs/failureLocation" }, "failures": { "type": "array", "items": { "$ref": "#/$defs/caseFailure" } }, "checks": { "$ref": "#/$defs/caseChecks" }, "output": { "type": "string", "description": "What the case wrote (O8)" }, "metrics": { "$ref": "#/$defs/metrics" }, "bounds": { "$ref": "#/$defs/bounds" }, "kind": { "type": "string" }, "tags": { "type": "array", "items": { "type": "string" } }, "expectedFailure": { "type": "boolean" }, "unexpectedPass": { "type": "boolean" } } }, + "caseOutcomePartial": { "type": "object", "properties": { "case": { "type": "string" }, "status": { "enum": ["passed", "failed", "skipped", "errored"] }, "durationMs": { "type": "integer" }, "failures": { "type": "array", "items": { "$ref": "#/$defs/caseFailure" } }, "checks": { "$ref": "#/$defs/caseChecks" }, "output": { "type": "string" }, "metrics": { "$ref": "#/$defs/metrics" }, "bounds": { "$ref": "#/$defs/bounds" } } }, "failureLocation": { "description": "Where a failure is: the first file:line (or file(line):) its output names, line 0-based; null when it names none.", "oneOf": [{ "type": "null" }, { "type": "object", "required": ["path", "line"], "properties": { "path": { "type": "string" }, "line": { "type": "integer", "minimum": 0 } } }] }, "settingValues": { "type": "object", "additionalProperties": true }, "resolvedSettings": { "type": "object", "required": ["values", "rejected", "renames", "deprecated"] }, @@ -563,7 +564,7 @@ "action": { "type": "object", "required": ["kind"], "properties": { "kind": { "enum": ["job", "run", "debug", "cache", "panel", "cancel", "selectTarget", "installToolchain", "selectDefaultToolchain", "newProject", "quickMenu", "selfCheck", "rpc", "languageService", "forward", "editor", "problems", "report"] } } } } }, - "launchParams": { "type": "object", "required": ["root"], "properties": { "root": { "type": "string" }, "target": { "type": "string" }, "profile": { "type": "string" }, "runner": { "type": "string", "description": "Explicit named runner; empty selects the default and disables automatic debuggee selection." }, "bin": { "type": "string" }, "test": { "type": "string" }, "framework": { "type": "string" }, "cases": { "type": "array", "items": { "type": "string" } }, "args": { "type": "array", "items": { "type": "string" } } } }, + "launchParams": { "type": "object", "required": ["root"], "properties": { "root": { "type": "string" }, "target": { "type": "string" }, "profile": { "type": "string" }, "runner": { "type": "string", "description": "Explicit named runner; empty selects the default and disables automatic debuggee selection." }, "bin": { "type": "string" }, "test": { "type": "string" }, "cases": { "type": "array", "items": { "type": "string" } }, "args": { "type": "array", "items": { "type": "string" } } } }, "graph": { "type": "object", "required": ["nodes", "edges"], "properties": { "nodes": { "type": "array" }, "edges": { "type": "array" } } }, "diagnostic": { "type": "object", diff --git a/spec/i18n/manifest.en.json b/spec/i18n/manifest.en.json index 1465565..e29ae96 100644 --- a/spec/i18n/manifest.en.json +++ b/spec/i18n/manifest.en.json @@ -275,6 +275,6 @@ "command.mcpp.exportReport.title": "Export IDE Report", "mcpp.codeLens.enable.title": "Show Run and Debug above main", "mcpp.codeLens.enable.description": "A CodeLens above the main of a program or a test: Run and Debug. Where C++ Modules serves the file its position is the language service's; otherwise the line is read from the text.", - "mcpp.test.macros.title": "Add a test framework's macros", - "mcpp.test.macros.description": "Macros that write a test framework's cases, each as framework:MACRO (gtest:MY_TEST). A test program's cases are listed only when its source calls one of its framework's macros." + "mcpp.test.session.title": "Run cases through the framework's test session", + "mcpp.test.session.description": "auto: when the project's test framework states a test session (mcpp test --guide), cases are run, added and cancelled through it; off: each run of cases is a run of mcpp test." } diff --git a/spec/i18n/manifest.zh-cn.json b/spec/i18n/manifest.zh-cn.json index 1e39c5c..5bd5886 100644 --- a/spec/i18n/manifest.zh-cn.json +++ b/spec/i18n/manifest.zh-cn.json @@ -275,6 +275,6 @@ "command.mcpp.exportReport.title": "导出 IDE 报告", "mcpp.codeLens.enable.title": "在 main 上方显示运行与调试", "mcpp.codeLens.enable.description": "在程序或测试的 main 上方显示 CodeLens：运行、调试。C++ Modules 服务该文件时位置取自语言服务，否则从文本读出行号。", - "mcpp.test.macros.title": "追加测试框架的宏", - "mcpp.test.macros.description": "书写测试框架用例的宏，每项写作 框架:宏名（gtest:MY_TEST）。测试程序的源文件调用了其框架的宏时才列举它的用例。" + "mcpp.test.session.title": "通过框架的测试会话运行用例", + "mcpp.test.session.description": "auto：工程的测试框架声明了测试会话（mcpp test --guide）时，经由它运行、加入和取消用例；off：每次运行用例都是一次 mcpp test。" } diff --git a/spec/settings.json b/spec/settings.json index cd46686..ad646b4 100644 --- a/spec/settings.json +++ b/spec/settings.json @@ -307,17 +307,21 @@ "description": "A CodeLens above the main of a program or a test: Run and Debug. Where C++ Modules serves the file its position is the language service's; otherwise the line is read from the text." }, { - "key": "mcpp.test.macros", - "type": "array", - "default": [], + "key": "mcpp.test.session", + "type": "string", + "default": "auto", + "enum": [ + "auto", + "off" + ], "scope": "resource", "group": "tasks", "order": 150, "tier": "advanced", "applies": "immediate", - "since": "1.1.0", - "title": "Add a test framework's macros", - "description": "Macros that write a test framework's cases, each as framework:MACRO (gtest:MY_TEST). A test program's cases are listed only when its source calls one of its framework's macros." + "since": "1.5.0", + "title": "Run cases through the framework's test session", + "description": "auto: when the project's test framework states a test session (mcpp test --guide), cases are run, added and cancelled through it; off: each run of cases is a run of mcpp test." }, { "key": "mcpp.languageService.menuItems", diff --git a/spec/test-frameworks.json b/spec/test-frameworks.json deleted file mode 100644 index 48bc055..0000000 --- a/spec/test-frameworks.json +++ /dev/null @@ -1,64 +0,0 @@ -{ - "description": "The test frameworks whose cases the editors show (mcpp design 2026-10-30 part 2 §7.2). A test program belongs to a framework when its plan holds one of the framework's packages and its source calls one of the framework's macros. `name` says how a case's full name is made from the macro's arguments; `list` lists the cases (run only for a program whose source calls a macro); `filter` selects cases, `{}` being the cases joined by `join`, each with the characters of `escape` preceded by a backslash (a GoogleTest name is two identifiers, which hold none of its filter's special characters); `report` writes the results to the file `{}`, read as `reportFormat`. A framework with `protocol` implements the mtest test protocol (mcpp 2026.11.8.1, feature `test-protocol`): its cases are its `\"name\"_test` literals, listed by `mcpp test --cases`, run by `mcpp test --case` and reported in mcpp's records; a program of any framework that implements the protocol (GoogleTest through mcpplibs.mtest.gtest) is listed and run so. boost-ext.ut is not here: its runner takes no filter from the command line, so its tests run as programs.", - "frameworks": [ - { - "id": "mtest", - "title": "mtest", - "packages": ["mcpplibs.mtest"], - "protocol": 1, - "macros": [], - "name": "literal", - "list": [], - "listFormat": "lines", - "filter": [], - "join": ",", - "escape": "", - "report": [], - "reportFormat": "" - }, - { - "id": "gtest", - "title": "GoogleTest", - "packages": ["compat.gtest", "gtest", "gmock"], - "macros": ["TEST", "TEST_F", "TEST_P", "TYPED_TEST", "TYPED_TEST_P"], - "name": "suite.case", - "list": ["--gtest_list_tests"], - "listFormat": "gtest-list", - "filter": ["--gtest_filter={}"], - "join": ":", - "escape": "", - "report": ["--gtest_output=json:{}"], - "reportFormat": "gtest-json" - }, - { - "id": "catch2", - "title": "Catch2", - "packages": ["compat.catch2", "catch2"], - "macros": ["TEST_CASE", "TEST_CASE_METHOD", "SCENARIO", "TEMPLATE_TEST_CASE"], - "name": "literal", - "scenarioPrefix": "Scenario: ", - "list": ["--list-tests", "--reporter", "json"], - "listFormat": "catch2-json", - "filter": ["{}"], - "join": ",", - "escape": ",[]*\\", - "report": ["--reporter", "json::out={}"], - "reportFormat": "catch2-json" - }, - { - "id": "doctest", - "title": "doctest", - "packages": ["compat.doctest", "doctest"], - "macros": ["TEST_CASE", "TEST_CASE_FIXTURE", "SCENARIO"], - "name": "literal", - "scenarioPrefix": " Scenario: ", - "list": ["--list-test-cases", "--no-intro", "--no-version"], - "listFormat": "lines", - "filter": ["--test-case={}"], - "join": ",", - "escape": ",", - "report": ["--reporters=junit", "--out={}"], - "reportFormat": "junit" - } - ] -} diff --git a/src/bin/mockmcpp.cpp b/src/bin/mockmcpp.cpp index ed9b201..4577af7 100644 --- a/src/bin/mockmcpp.cpp +++ b/src/bin/mockmcpp.cpp @@ -7,14 +7,18 @@ // MOCK_MCPP_FAIL "build": the build fails with the compiler's diagnostic; // "manifest": it fails on the manifest (also read from the // project's `.mock-mcpp-fail`) -// MOCK_MCPP_SLOW_MS the build waits this long before it ends, so a test can -// cancel it; SIGTERM then leaves the record `cancelled` +// MOCK_MCPP_SLOW_MS the build, or a run of tests, waits this long before it ends, so a +// test can cancel it; SIGTERM then leaves a build's record `cancelled` // MOCK_MCPP_LOG every argv is appended here, one line each, and the value of // XLINGS_PROJECT_DIR the run received (`` when none) // // `.mock-mcpp-tests` in the directory a command runs in selects the programs `mcpp test` states // (protocol/fixtures/test*-.ndjson). // +// AND A TEST SESSION (the mcpp test spec's O9): `.mock-mcpp-session` in the directory makes +// `mcpp test --guide` state a framework whose session is this program run as `session`, which +// answers `test/run` at once and states one case that begins and passes, then the run's end. +// // AND A VERSION MANAGER'S SHIM, per directory (design 2026-10-30 part 2 §4.2): in the directory a // command runs in, `.mock-mcpp-version` names the version that answers (every output's version is // that one), `.mock-mcpp-features` the features its envelopes state (comma-separated; without @@ -152,8 +156,39 @@ int build(bool message) { } // namespace +// The session: one JSON-RPC message per line on the standard streams. +int session() { + std::string line; + int runs = 0; + auto send = [](const std::string& m) { std::cout << m << '\n' << std::flush; }; + while (std::getline(std::cin, line)) { + const auto id_at = line.find("\"id\":"); + std::string id = "null"; + if (id_at != std::string::npos) { + auto end = line.find_first_of(",}", id_at + 5); + id = line.substr(id_at + 5, end - id_at - 5); + } + if (line.find("\"test/run\"") != std::string::npos) { + const auto run = std::format("r{}", ++runs); + send(std::format(R"({{"jsonrpc":"2.0","id":{},"result":{{"run":"{}"}}}})", id, run)); + send(std::format(R"({{"jsonrpc":"2.0","method":"test/build","params":{{"run":"{}","event":"end","ok":true}}}})", run)); + send(std::format(R"({{"jsonrpc":"2.0","method":"test/case","params":{{"run":"{}","program":"unit/parse","id":"parsing/a field","event":"begin","t_ms":1}}}})", run)); + send(std::format(R"({{"jsonrpc":"2.0","method":"test/case","params":{{"run":"{}","program":"unit/parse","id":"parsing/a field","event":"output","stream":"out","text":"said\n"}}}})", run)); + send(std::format(R"({{"jsonrpc":"2.0","method":"test/case","params":{{"run":"{}","program":"unit/parse","id":"parsing/a field","event":"end","status":"pass","ms":2,"checks":{{"held":1,"failed":0}}}}}})", run)); + send(std::format(R"({{"jsonrpc":"2.0","method":"test/run","params":{{"run":"{}","status":"pass","counts":{{"pass":1}}}}}})", run)); + } else if (line.find("\"shutdown\"") != std::string::npos) { + send(std::format(R"({{"jsonrpc":"2.0","id":{},"result":null}})", id)); + return 0; + } else if (id != "null") { + send(std::format(R"({{"jsonrpc":"2.0","id":{},"result":{{}}}})", id)); + } + } + return 0; +} + int main(int argc, char** argv) { std::vector args(argv + 1, argv + argc); + if (!args.empty() && args[0] == "session") return session(); auto log = env("MOCK_MCPP_LOG"); if (log.empty() && std::filesystem::exists(".mock-mcpp-log")) log = ".mock-mcpp-log"; if (!log.empty()) { @@ -194,7 +229,18 @@ int main(int argc, char** argv) { } // A program that implements the test protocol: its listing (`--cases`) and a run of a selection // (`--case`), each a record with the protocol's fields (mcpp docs/50 §8). + if (first(0) == "test" && has("--guide")) { + // The project's framework (B6), with a session when the directory asks for one. + const bool withSession = !here_file(".mock-mcpp-session").empty(); + std::string self = std::filesystem::absolute(argv[0]).generic_string(); + out(std::format(R"({{"guide":{{"frameworks":[{{"package":"mcpplibs.mtest","version":"0.6.0","entry":"mtest/main","protocol":1,"guide":null,"repo":"","capabilities":[{}],"session":{}}}]}}}})", + withSession ? "\"session\"" : "", withSession ? std::format(R"(["{}","session"])", self) : "null") + "\n"); + return 0; + } if (first(0) == "test") { + // A run that takes its time, as a build does (MOCK_MCPP_SLOW_MS): ended by a request to end. + if (const auto slow = env("MOCK_MCPP_SLOW_MS"); !slow.empty() && !has("--list") && !has("--cases")) + if (mcpplibs::os::interrupt::wait(std::chrono::milliseconds(std::stoi(slow)))) return 130; // `.mock-mcpp-tests` names another set of programs: `directory` is a protocol program of two // files and a file program as mcpp 2026.11.9.1 states them; `directory-917` the same with // mcpp#917's parts, case records and metrics (its listing of cases is the first set's). diff --git a/src/jobs.cppm b/src/jobs.cppm index ad0aa46..130ab03 100644 --- a/src/jobs.cppm +++ b/src/jobs.cppm @@ -1,10 +1,14 @@ // Jobs: a run of mcpp that takes time and reports while it runs -- a build, a // test, a pack, a clean, a toolchain install, `mcpp add`. // -// ONE JOB AT A TIME PER PROJECT. A project's jobs run in the order they were -// asked for. A new job of a kind that supersedes (a build asked for again on -// save) replaces the waiting one of that kind, and the running one is -// cancelled when the request says `supersede`. Cancelling sends the polite +// ONE JOB AT A TIME PER PROJECT, A TEST BESIDE IT. A project's jobs run in the +// order they were asked for. A new job of a kind that supersedes (a build asked +// for again on save) replaces the waiting one of that kind, and the running one +// is cancelled when the request says `supersede`. A test runs at once in a lane +// of its own (design 2026-11-10 supplement, T14): mcpp builds one configuration +// at a time and releases the build directory before the tests run (mcpp#915), +// so a second test, or a build, does not wait for a running one, and cancelling +// one ends that one. Cancelling sends the polite // signal: mcpp ends the steps it started and leaves its result record // `cancelled` (mcpp SPEC-003 §3.7). // @@ -82,6 +86,8 @@ private: std::deque> queue; std::shared_ptr running; std::jthread worker; + bool once = false; // a test's lane: its one job, then it ends + bool ended = false; }; void run_lane(std::string key); void execute(const std::shared_ptr& job); @@ -146,15 +152,27 @@ void Jobs::set_client(mcpp::Client client) { std::string Jobs::submit(Request request) { auto job = std::make_shared(); - const auto key = request.root.lexically_normal().generic_string(); + auto key = request.root.lexically_normal().generic_string(); { std::lock_guard lock(mutex_); job->id = std::format("job-{}", next_++); job->request = std::move(request); + // The lanes of tests that ended are let go. + for (auto it = lanes_.begin(); it != lanes_.end();) { + if (it->second.once && it->second.ended) { + if (it->second.worker.joinable()) it->second.worker.join(); + it = lanes_.erase(it); + } else { + ++it; + } + } + const bool once = job->request.kind == Kind::test; + if (once) key += "\x1f" + job->id; auto& lane = lanes_[key]; + lane.once = once; // The same kind asked for again replaces the one still waiting: a save // followed by another save builds once. - if (job->request.kind == Kind::build || job->request.kind == Kind::test) { + if (job->request.kind == Kind::build) { std::erase_if(lane.queue, [&](const std::shared_ptr& waiting) { if (waiting->request.kind != job->request.kind) return false; waiting->cancelled = true; @@ -198,7 +216,8 @@ Json Jobs::list() const { Json out = Json::array(); for (const auto& [key, lane] : lanes_) { auto describe = [&](const std::shared_ptr& j, std::string_view state) { - out.push_back(Json{{"id", j->id}, {"kind", kind_name(j->request.kind)}, {"root", key}, + out.push_back(Json{{"id", j->id}, {"kind", kind_name(j->request.kind)}, + {"root", j->request.root.lexically_normal().generic_string()}, {"state", state}, {"args", j->request.args}}); }; if (lane.running) describe(lane.running, "running"); @@ -235,7 +254,12 @@ void Jobs::run_lane(std::string key) { execute(job); { std::lock_guard lock(mutex_); - lanes_[key].running.reset(); + auto& lane = lanes_[key]; + lane.running.reset(); + if (lane.once) { + lane.ended = true; + return; + } } } } diff --git a/src/scan.cppm b/src/scan.cppm index b9f0163..891a555 100644 --- a/src/scan.cppm +++ b/src/scan.cppm @@ -1,59 +1,24 @@ -// What a C++ source says about where it starts and what it tests, read as text (mcpp design -// 2026-10-30 part 2 §6.3, §7.3): the line of `main`, and the calls of a test framework's macros. +// Where a C++ source starts, read as text (mcpp design 2026-10-30 part 2 §6.3): the line of `main`. +// What a test program's cases are is the program's to say (mcpp test spec B3); nothing of a test +// framework is read here. // // NOTHING IS BUILT AND NOTHING IS RUN. A file is scanned when it is opened, before any build and // without mcppls, so the editors can place ▶ at once. Comments, string and character literals (raw -// strings included) and preprocessor lines are skipped, so a `main(` in a comment or a `TEST(` in a -// string is not one. A position found this way is approximate where a language service has a -// better one: the editors take the language service's position of `main` when it gives one. -// -// A MACRO CALL COUNTS AT THE START OF A DECLARATION: after `;`, `{`, `}`, a preprocessor line or -// the start of the file. That is where a framework's test macros are written (at namespace scope, -// or in a test body for a section), and an expression that merely contains the name is not one. +// strings included) and preprocessor lines are skipped, so a `main(` in a comment or a string is +// not one. A position found this way is approximate where a language service has a better one: the +// editors take the language service's position of `main` when it gives one. export module mcppide.scan; import std; -import mcpplibs.mtest.manifest; export namespace mcppide::scan { -// One call of a macro: its name, where it is (0-based line and column of the name), and its -// arguments as written, split at the top-level commas, each trimmed. -struct Call { - std::string name; - int line = 0; - int column = 0; - std::vector arguments; -}; - -// A case of the mtest library -- an object (`UTest parse{"parse", ...}`, any suite type) or a -// `"name"_test` declared in one -- as mtest's own static manifest reads it -// (mcpplibs.mtest.manifest): its identifier, the names of what declares it and its own joined by -// `/`, and where its name is (0-based); the kind, tags and bounds on its metrics it states (empty -// kind: its program's). -struct Literal { - std::string id; - int line = 0; - int column = 0; - std::string kind; - std::vector tags; - std::vector bounds; -}; - struct Result { std::optional mainLine; // 0-based - std::vector calls; - std::vector cases; // mtest's cases, by mcpplibs.mtest.manifest }; -// Scans `text` for `main` and for calls of the macros named in `macros`. `types` are the suite types -// the project's tests declare elsewhere (`class Fixture : public CTS` in a module beside them). -Result scan(std::string_view text, const std::vector& macros, - const mcpplibs::mtest::manifest::suite_types& types = {}); - -// A string literal's contents, as the program sees them for the plain escapes (\" \\ \n \t); the -// argument itself when it is not one literal. Adjacent literals are joined. -std::string literal_value(std::string_view argument); +// Scans `text` for `main`. +Result scan(std::string_view text); } // namespace mcppide::scan @@ -64,12 +29,6 @@ namespace { bool ident_start(char c) { return std::isalpha(static_cast(c)) || c == '_'; } bool ident_char(char c) { return std::isalnum(static_cast(c)) || c == '_'; } -std::string trim(std::string_view s) { - while (!s.empty() && std::isspace(static_cast(s.front()))) s.remove_prefix(1); - while (!s.empty() && std::isspace(static_cast(s.back()))) s.remove_suffix(1); - return std::string(s); -} - // A cursor over the text that knows its line and steps over what is not code. struct Cursor { std::string_view text; @@ -125,53 +84,12 @@ struct Cursor { } }; -// The arguments of a call whose `(` the cursor is on, split at top-level commas; the cursor ends -// after the `)`. -std::vector arguments(Cursor& c) { - std::vector out; - std::string current; - int depth = 0; - c.step(); // ( - while (!c.done()) { - const char ch = c.peek(); - if (ch == '/' && c.peek(1) == '/') { c.skip_line_comment(); continue; } - if (ch == '/' && c.peek(1) == '*') { c.skip_block_comment(); continue; } - if (ch == '"' || ch == '\'') { - const auto from = c.at; - c.skip_quoted(ch); - current.append(c.text.substr(from, c.at - from)); - continue; - } - if (ch == '(' || ch == '[' || ch == '{' || ch == '<') { if (ch != '<') ++depth; } - if (ch == ')' || ch == ']' || ch == '}') { - if (depth == 0) { - c.step(); - out.push_back(trim(current)); - return out; - } - --depth; - } - if (ch == ',' && depth == 0) { - out.push_back(trim(current)); - current.clear(); - c.step(); - continue; - } - current.push_back(ch); - c.step(); - } - out.push_back(trim(current)); - return out; -} - } // namespace -Result scan(std::string_view text, const std::vector& macros, - const mcpplibs::mtest::manifest::suite_types& types) { +Result scan(std::string_view text) { Result r; Cursor c{text}; int depth = 0; // braces - bool atDeclaration = true; // the next identifier starts a declaration bool lineHasCode = false; // a `#` is a directive only first on its line std::string previous; // the previous identifier, for `int main(` while (!c.done()) { @@ -180,19 +98,18 @@ Result scan(std::string_view text, const std::vector& macros, if (std::isspace(static_cast(ch))) { c.step(); continue; } if (ch == '/' && c.peek(1) == '/') { c.skip_line_comment(); continue; } if (ch == '/' && c.peek(1) == '*') { c.skip_block_comment(); continue; } - if (ch == '#' && !lineHasCode) { c.skip_directive(); atDeclaration = true; previous.clear(); continue; } + if (ch == '#' && !lineHasCode) { c.skip_directive(); previous.clear(); continue; } lineHasCode = true; - if (c.on_raw()) { c.skip_raw(); atDeclaration = false; previous.clear(); continue; } + if (c.on_raw()) { c.skip_raw(); previous.clear(); continue; } if (ch == '"') { c.skip_quoted(ch); - atDeclaration = false; previous.clear(); continue; } - if (ch == '\'') { c.skip_quoted(ch); atDeclaration = false; previous.clear(); continue; } + if (ch == '\'') { c.skip_quoted(ch); previous.clear(); continue; } if (ident_start(ch)) { const auto from = c.at; - const int line = c.line, column = c.column(); + const int line = c.line; while (!c.done() && ident_char(c.peek())) c.step(); const std::string word(text.substr(from, c.at - from)); // What follows the name, past blanks and comments on the way. @@ -204,61 +121,18 @@ Result scan(std::string_view text, const std::vector& macros, break; } const bool call = look.peek() == '('; - if (call && atDeclaration && std::ranges::find(macros, word) != macros.end()) { - c = look; - r.calls.push_back(Call{word, line, column, arguments(c)}); - atDeclaration = false; - previous.clear(); - continue; - } if (call && word == "main" && depth == 0 && !r.mainLine && (previous == "int" || previous == "auto")) r.mainLine = line; previous = word; - atDeclaration = false; continue; } - if (ch == '{') { - ++depth; - atDeclaration = true; - } else if (ch == '}') { - if (depth > 0) --depth; - atDeclaration = true; - } - else if (ch == ';') atDeclaration = true; - else atDeclaration = false; + if (ch == '{') ++depth; + else if (ch == '}' && depth > 0) --depth; // `std::main`, `x.main`: not the program's. if (ch != ':' ) previous.clear(); c.step(); } - // mtest's cases as mtest reads them: its manifest is the one reader of its declarations. - for (const auto& found : mcpplibs::mtest::manifest::scan(text, types)) - r.cases.push_back(Literal{found.id, found.line - 1, found.column - 1, found.kind, found.tags, found.bounds}); return r; } -std::string literal_value(std::string_view argument) { - std::string out; - std::size_t i = 0; - bool any = false; - while (i < argument.size()) { - while (i < argument.size() && std::isspace(static_cast(argument[i]))) ++i; - if (i >= argument.size()) break; - if (argument[i] != '"') return std::string(argument); - ++i; - while (i < argument.size() && argument[i] != '"') { - if (argument[i] == '\\' && i + 1 < argument.size()) { - const char e = argument[i + 1]; - out.push_back(e == 'n' ? '\n' : e == 't' ? '\t' : e); - i += 2; - continue; - } - out.push_back(argument[i++]); - } - if (i >= argument.size()) return std::string(argument); - ++i; - any = true; - } - return any ? out : std::string(argument); -} - } // namespace mcppide::scan diff --git a/src/server.cppm b/src/server.cppm index 6673ac5..6927c3b 100644 --- a/src/server.cppm +++ b/src/server.cppm @@ -26,22 +26,22 @@ import mcppide.documents; import mcppide.spec; import mcppide.languageservice; import mcppide.scan; -import mcpplibs.mtest.manifest; import mcppide.testing; import mcppide.teststore; +import mcppide.testsession; import mcpplibs.os; export namespace mcppide::server { using Json = nlohmann::json; -inline constexpr std::string_view kVersion = "1.4.0"; +inline constexpr std::string_view kVersion = "1.5.0"; // THE SERIES (mcpp design 2026-11-07, D15): `X.Y` of `X.Y.Z`, shared by the backend and every editor // plugin; components of one series work together. A plugin refuses a backend of another series, // naming both versions. The backend is always `X.Y.0`: the VS Code and CLion packages carry it, so a // change of the backend is a change of every package, and it begins a series (1.3.0). `Z` is an // editor plugin's own patch. -inline constexpr std::string_view kSeries = "1.4"; +inline constexpr std::string_view kSeries = "1.5"; static_assert(kVersion.starts_with(kSeries) && kVersion.size() == kSeries.size() + 2 && kVersion.ends_with(".0")); inline constexpr int kProtocol = 2; @@ -121,18 +121,32 @@ private: std::vector programs_of(const std::filesystem::path& root); std::optional program_named(const std::filesystem::path& root, std::string_view name); std::optional program_of_file(const std::filesystem::path& root, const std::filesystem::path& file); - std::optional> listed_cases(const std::filesystem::path& root, const std::string& test, const Json& p, - const std::vector& statics); - std::optional> framework_listing(const std::filesystem::path& root, const std::string& test, - const Json& p, const testing::Framework& f); + // Some cases run through the project's test session (O9), when its framework states one and + // mcpp gives the programs' launches (B5); nothing when either is not so. + std::optional run_through_session(const std::filesystem::path& root, const std::string& test, + const std::vector& cases, const Json& p); + void on_session(const std::filesystem::path& root, std::string_view method, const Json& params); + // A program's cases as it lists them (mcpp test spec B3), from one listing of the project. + std::optional> listed_cases(const std::filesystem::path& root, const std::string& test, + const Json& p, bool ask); + // The records of `mcpp test --cases` for the whole project. The last listing is kept: `ask` lists + // again when the project has changed since, and without it the last listing answers as it is, so + // that the cases shown stay until a listing replaces them. + std::optional> project_listing(const std::filesystem::path& root, const Json& p, bool ask); + // What the project's test frameworks state of themselves (mcpp test spec B6, `mcpp test --guide`): + // the name the editors show, and the command of a session one of them offers (O9), else empty. + struct Frameworks { + std::string name; + std::vector session; + }; + Frameworks frameworks_of(const std::filesystem::path& root, const Json& p); + std::string framework_name(const std::filesystem::path& root, const Json& p) { return frameworks_of(root, p).name; } // A record of a test job, with the program and part it belongs to, and its cases as the editors // receive them (design 2026-11-10). Json test_result_of(const std::filesystem::path& root, const Json& record, Json n); void protocol_case_results(std::string_view jobId, const std::filesystem::path& root, const std::string& test, const Json& record); // The store's last runs (kept per project; a run is read once). std::vector store_runs(const std::filesystem::path& root, std::size_t count); - // `mcpp.test.macros` as testing reads it: {"gtest": ["MY_TEST"]}. - Json test_macros() const; // The packages of a project's plan, from its build database when one was read. std::vector plan_packages(const std::filesystem::path& root, bool read); Result project_mcpp(const Json& p); @@ -169,6 +183,10 @@ private: Json setting(std::string_view key) const; // `--target ` for a request: its own `target`, else `mcpp.build.target`; nothing for the host. std::vector target_args(const Json& p) const; + // `mcpp test --message-format json` with the profile, the target and `mcpp.task.testArgs`: one + // set of arguments for every test command, so that a listing and a run plan one configuration + // (design 2026-11-10 supplement, T14). + std::vector test_args(const Json& p) const; std::filesystem::path any_root(const Json& p) const; void locate_mcpp(); void on_changed(const std::filesystem::path& root, std::string_view what, const Json& detail); @@ -206,6 +224,23 @@ private: Json searchCache_; std::chrono::steady_clock::time_point searchAt_{}; std::map buildDatabases_; + struct Listing { + std::vector records; + bool current = true; // false once the project changed + }; + std::map listings_; // project_listing + std::map frameworks_; // frameworks_of + // THE TEST SESSIONS (mcpp test spec O9): one per project whose framework states one, and the + // runs through them, each answered to the editor as a job of its own. + struct SessionRun { + std::string jobId; + std::filesystem::path root; + std::map cases; // per case: its failures and its output, as they came + }; + std::map> testSessions_; + std::map sessionRuns_; // by the session's run + std::map> sessionJobs_; // job -> project, run + std::uint64_t sessionJobNext_ = 1; // The project and kind of each job, for the notifications its records become. std::map, std::less<>> jobOf_; // The whole output of the last test record of each running test job, for a case run to divide @@ -348,6 +383,19 @@ Json Session::setting(std::string_view key) const { return nullptr; } +std::vector Session::test_args(const Json& p) const { + std::vector args{"test", "--message-format", "json"}; + const auto profile = p.value("profile", setting("mcpp.build.profile").is_string() ? setting("mcpp.build.profile").get() : "dev"); + if (profile == "release") args.push_back("--release"); + else if (!profile.empty() && profile != "dev") { + args.push_back("--profile"); + args.push_back(profile); + } + for (const auto& a : target_args(p)) args.push_back(a); + for (const auto& a : strings_of(setting("mcpp.task.testArgs"))) args.push_back(a); + return args; +} + std::vector Session::target_args(const Json& p) const { std::string target = p.contains("target") && p["target"].is_string() ? p["target"].get() : std::string{}; if (!p.contains("target")) { @@ -413,6 +461,8 @@ void Session::on_changed(const std::filesystem::path& root, std::string_view wha { std::lock_guard lock(mutex_); buildDatabases_.erase(root); + if (auto it = listings_.find(root); it != listings_.end()) it->second.current = false; + if (what != "result") frameworks_.erase(root); if (what != "result") probes_.erase(root); // a manifest may state another mcpp floor or pin if (what != "result") programs_.erase(root); // and other tests } @@ -754,12 +804,9 @@ std::expected Session::job_start(const Json& p, std::function< case jobs::Kind::test: // The programs, so that each part's record is attributed to its program (design 2026-11-10). (void)programs_of(root); - r.args = {"test", "--message-format", "json"}; - if (release) r.args.push_back("--release"); - append(target); + r.args = test_args(p); if (p.contains("filter") && p["filter"].is_string() && !p["filter"].get().empty()) r.args.push_back(p["filter"].get()); - append(strings_of(setting("mcpp.task.testArgs"))); r.mode = jobs::Mode::ndjson; break; case jobs::Kind::pack: { @@ -815,8 +862,8 @@ std::expected Session::job_start(const Json& p, std::function< } // The mcpp command that builds a program and describes its launch: `mcpp run --print-launch`, or -// `mcpp test --print-launch` for a test (mcpp's `test-launch`), with its framework's filter -// after `--` when cases are asked for. `stream` asks a run for the build's events before the +// `mcpp test --print-launch` for a test (mcpp's `test-launch`); the cases asked for are +// selected by the protocol's variables (debug_configuration). `stream` asks a run for the build's events before the // envelope (mcpp's `launch-stream`), and `programArgs` are the program's own arguments, after `--`. std::vector Session::launch_args(const Json& p, bool stream, bool programArgs) { const auto test = p.value("test", ""); @@ -838,11 +885,6 @@ std::vector Session::launch_args(const Json& p, bool stream, bool p } if (!test.empty()) { args.push_back(test); - if (const auto* f = testing::framework(p.value("framework", ""))) - if (const auto filter = testing::filter_args(*f, strings_of(p.value("cases", Json::array()))); !filter.empty()) { - args.push_back("--"); - for (const auto& a : filter) args.push_back(a); - } } else { if (auto bin = p.value("bin", ""); !bin.empty()) args.push_back(bin); if (const auto program = strings_of(p.value("args", Json::array())); programArgs && !program.empty()) { @@ -912,6 +954,24 @@ std::expected Session::debug_configuration(const Json& p, cons const std::string program = argv.empty() ? l.value("program", "") : argv.front(); std::vector args(argv.size() > 1 ? argv.begin() + 1 : argv.end(), argv.end()); Json env = l.value("env", Json::object()); + // SOME CASES OF A PROGRAM OF THE TEST PROTOCOL, DEBUGGED (mcpp test spec P2, P4): the debugger + // starts the program as a runner would, with the protocol's variables and the cases selected; + // the report goes to a file of this session's own. Any program the launch says implements the + // protocol is debugged so, whatever framework wrote it. + if (const auto cases = strings_of(p.value("cases", Json::array())); !cases.empty() && l.value("protocol", Json(nullptr)).is_number()) { + const auto dir = host_path(mcpplibs::os::fs::utf8(mcpplibs::os::dirs::temp())) / + std::format("mcpp-ide-debug-{}", std::random_device{}()); + std::error_code ec; + std::filesystem::create_directories(dir / "tmp", ec); + std::string lines; + for (const auto& c : cases) lines += c + "\n"; + std::ofstream(dir / "select.txt", std::ios::binary) << lines; + env["MTEST_PROTOCOL"] = std::to_string(l["protocol"].get()); + env["MTEST_MODE"] = "run"; + env["MTEST_SELECT"] = (dir / "select.txt").generic_string(); + env["MTEST_REPORT"] = (dir / "report.ndjson").generic_string(); + env["MTEST_TMPDIR"] = (dir / "tmp").generic_string(); + } const bool through = !l.value("via", Json(nullptr)).is_null(); const auto jit = std::string("settings set plugin.jit-loader.gdb.enable on"); Json cfg{{"name", std::format("mcpp: {}", std::filesystem::path(l.value("program", program)).filename().string())}, @@ -1449,16 +1509,6 @@ std::expected Session::test_list(const Json& p) { return Json{{"root", root.generic_string()}, {"tests", tests}}; } -Json Session::test_macros() const { - Json out = Json::object(); - for (const auto& entry : strings_of(setting("mcpp.test.macros"))) { - const auto colon = entry.find(':'); - if (colon == std::string::npos || colon == 0 || colon + 1 == entry.size()) continue; - out[entry.substr(0, colon)].push_back(entry.substr(colon + 1)); - } - return out; -} - std::vector Session::plan_packages(const std::filesystem::path& root, bool read) { Json db; { @@ -1509,41 +1559,6 @@ std::optional test_name_of(const std::filesystem::path& root, const return below.generic_string(); } -// The suite types the project's tests declare (`class Fixture : public CTS` in a module beside them), -// read across the files of tests/ as mtest's static manifest reads them. They are read again only -// when the files changed: a walk that only stats them (their count, latest change and total size) -// decides, so a source/entries or test/cases call does not read every file of tests/. -mcpplibs::mtest::manifest::suite_types test_suite_types(const std::filesystem::path& root) { - static const std::set cxx{".cpp", ".cc", ".cxx", ".cppm", ".ixx", ".mpp"}; - struct Kept { - std::tuple signature; - mcpplibs::mtest::manifest::suite_types types; - }; - static std::mutex mutex; - static std::map kept; - std::vector files; - std::tuple signature{}; - std::error_code ec; - for (std::filesystem::recursive_directory_iterator it(root / "tests", ec), end; !ec && it != end; it.increment(ec)) { - if (!it->is_regular_file(ec) || !cxx.contains(it->path().extension().string())) continue; - files.push_back(it->path()); - auto& [count, latest, size] = signature; - ++count; - latest = std::max(latest, it->last_write_time(ec)); - size += it->file_size(ec); - } - { - std::lock_guard lock(mutex); - if (auto it = kept.find(root); it != kept.end() && it->second.signature == signature) return it->second.types; - } - std::vector sources; - for (const auto& f : files) sources.push_back(read_text(f)); - auto types = mcpplibs::mtest::manifest::types_of(sources); - std::lock_guard lock(mutex); - kept[root] = Kept{signature, types}; - return types; -} - // The source of a test by its name: tests/.cpp (or .cc, .cxx). std::filesystem::path test_source(const std::filesystem::path& root, std::string_view name) { for (const auto* ext : {".cpp", ".cc", ".cxx"}) { @@ -1566,12 +1581,11 @@ std::vector Session::list_programs(const std::filesystem::path if (!client.ready()) return programs; mcpp::RunOptions o; o.cwd = root; - o.args = {"test", "--list", "--message-format", "json"}; - for (const auto& a : target_args(p)) o.args.push_back(a); + o.args = test_args(p); + o.args.push_back("--list"); o.deny = deny(); o.client = client_label(); - const auto types = test_suite_types(root); - const auto definesMain = [&](const std::filesystem::path& f) { return scan::scan(read_text(f), {}, types).mainLine.has_value(); }; + const auto definesMain = [&](const std::filesystem::path& f) { return scan::scan(read_text(f)).mainLine.has_value(); }; if (auto r = client.records(o); r && r->exitCode == 0) for (const auto& rec : r->records) if (rec.is_object() && rec.contains("test") && rec["test"].is_string()) @@ -1612,8 +1626,7 @@ std::expected Session::source_entries(const Json& p) { auto proj = projects::project_for(file.generic_string(), p.value("boundary", "")); if (!proj) return Json{{"file", file.generic_string()}, {"kind", "none"}}; const auto text = p.contains("text") && p["text"].is_string() ? p["text"].get() : read_text(file); - const auto extra = test_macros(); - const auto found = scan::scan(text, testing::macros(extra), test_suite_types(proj->root)); + const auto found = scan::scan(text); Json out{{"file", file.generic_string()}, {"root", proj->root.generic_string()}, {"kind", "none"}}; if (auto name = test_name_of(proj->root, file)) { out["kind"] = "test"; @@ -1623,7 +1636,6 @@ std::expected Session::source_entries(const Json& p) { out["test"] = program->name; out["part"] = testing::part_of_file(*program, file); } - if (const auto* f = testing::framework_of(plan_packages(proj->root, false), found, extra)) out["framework"] = f->id; } else if (found.mainLine) { out["kind"] = "program"; } @@ -1653,55 +1665,42 @@ std::expected Session::test_cases(const Json& p) { else if (!file.empty()) files.push_back(file); else if (auto source = test_source(root, test); !source.empty()) files.push_back(source); const auto shown = program && !program->main.empty() ? std::filesystem::path(program->main) : files.empty() ? file : files.front(); - Json out{{"root", root.generic_string()}, {"test", test}, {"file", shown.generic_string()}, {"framework", nullptr}, {"cases", Json::array()}}; + Json out{{"root", root.generic_string()}, {"test", test}, {"file", shown.generic_string()}, {"framework", nullptr}, + {"protocol", nullptr}, {"cases", Json::array()}}; if (files.empty()) return out; if (program && program->layout == "directory") out["files"] = program->files; + // `main`, read from the text (approximate); nothing of any framework is read. + if (const auto found = scan::scan(read_text(shown)); found.mainLine) + out["main"] = Json{{"line", *found.mainLine}, {"approximate", true}}; bool trusted; { std::lock_guard lock(mutex_); trusted = trusted_; } - const auto extra = test_macros(); - const auto types = test_suite_types(root); - const auto packages = plan_packages(root, trusted); - const testing::Framework* f = nullptr; - std::vector cases; - for (const auto& one : files) { - const auto found = scan::scan(read_text(one), testing::macros(extra), types); - if (found.mainLine && one == shown) out["main"] = Json{{"line", *found.mainLine}, {"approximate", true}}; - const auto* framework = testing::framework_of(packages, found, extra); - if (!framework || (f && framework != f)) continue; - f = framework; - for (auto c : testing::static_cases(*f, found, extra)) { - c.file = one.generic_string(); - if (c.kind.empty() && f->protocol) c.kind = testing::kind_of_path(c.file); - cases.push_back(std::move(c)); - } - } - if (f) out["framework"] = f->id; - // A PROGRAM THAT IMPLEMENTS THE TEST PROTOCOL LISTS ITS OWN CASES (mcpp's `test-protocol`): - // `mcpp test --cases`, whatever framework wrote it; its cases then run by `--case`. - if (p.value("list", false) && trusted && has_feature(root, "test-protocol")) - if (auto listed = listed_cases(root, test, p, cases)) { - cases = std::move(*listed); - out["framework"] = "mtest"; - out["listed"] = true; - f = testing::framework("mtest"); - } - if (!f) return out; - if (!out.value("listed", false) && p.value("list", false) && trusted && !f->protocol && files.size() == 1) - if (auto names = framework_listing(root, test, p, *f)) { - cases = testing::merge(cases, *names, *f); - out["listed"] = true; - } - for (const auto& c : cases) out["cases"].push_back(testing::case_json(c)); + // THE CASES ARE THE PROGRAM'S (mcpp test spec B2, B3): a program that implements the test + // protocol lists them (`mcpp test --cases`), whatever framework or language wrote it, and its + // cases run by `--case`. A program that implements none is one case, judged by its exit status. + // `list` asks for a listing when the project changed since the last; without it the last one + // answers, or none. + if (!trusted || !has_feature(root, "test-protocol")) return out; + auto listed = listed_cases(root, test, p, p.value("list", false)); + if (!listed) return out; + out["protocol"] = 1; + out["framework"] = framework_name(root, p); + out["listed"] = true; + for (const auto& c : *listed) out["cases"].push_back(testing::case_json(c)); return out; } -// A protocol program's own listing (`mcpp test --cases`), each case placed where its source -// places it when the source was read, with what either states of its kind, tags and bounds. -std::optional> Session::listed_cases(const std::filesystem::path& root, const std::string& test, - const Json& p, const std::vector& statics) { +// The project's listing (mcpp test spec B3): one `mcpp test --cases` for every program, kept until +// the project changes. A listing per program planned the project once for each; the test fast path +// (mcpp 2026.11.10.1) answers an unchanged project without planning. +std::optional> Session::project_listing(const std::filesystem::path& root, const Json& p, bool ask) { + { + std::lock_guard lock(mutex_); + if (auto it = listings_.find(root); it != listings_.end() && (it->second.current || !ask)) return it->second.records; + } + if (!ask) return std::nullopt; mcpp::Client client; { std::lock_guard lock(mutex_); @@ -1710,121 +1709,227 @@ std::optional> Session::listed_cases(const std::files if (!client.ready()) return std::nullopt; mcpp::RunOptions o; o.cwd = root; - o.args = {"test", "--message-format", "json"}; - for (const auto& a : target_args(p)) o.args.push_back(a); - o.args.push_back(test); + o.args = test_args(p); o.args.push_back("--cases"); o.deny = deny(); o.client = client_label(); auto r = client.records(o); if (!r) return std::nullopt; - for (const auto& rec : r->records) { - if (!rec.is_object() || rec.value("test", "") != test || !rec.contains("listed") || !rec["protocol"].is_number()) continue; - std::vector listed; - for (const auto& c : testing::protocol_listing(rec)) { - const auto known = std::ranges::find_if(statics, [&](const testing::Case& s) { return s.name == c.id; }); - const bool placed = known != statics.end() && known->line >= 0; - testing::Case one{c.id, placed ? known->line : c.line, "listed"}; - one.file = !c.file.empty() ? c.file : placed ? known->file : std::string{}; - one.kind = !c.kind.empty() ? c.kind : known != statics.end() ? known->kind : std::string{}; - one.tags = !c.tags.empty() || known == statics.end() ? c.tags : known->tags; - one.expect = c.expect; - one.bounds = !c.bounds.empty() || known == statics.end() ? c.bounds : known->bounds; - listed.push_back(std::move(one)); - } - return listed; - } - return std::nullopt; + std::vector records; + for (const auto& rec : r->records) + if (rec.is_object() && rec.contains("test")) records.push_back(rec); + std::lock_guard lock(mutex_); + listings_[root] = Listing{records, true}; + return records; } -// The names a framework's own list command prints (`mcpp test -- `), for a program that -// does not implement the protocol. -std::optional> Session::framework_listing(const std::filesystem::path& root, const std::string& test, - const Json& p, const testing::Framework& f) { +Session::Frameworks Session::frameworks_of(const std::filesystem::path& root, const Json& p) { + { + std::lock_guard lock(mutex_); + if (auto it = frameworks_.find(root); it != frameworks_.end()) return it->second; + } + Frameworks out{"protocol", {}}; mcpp::Client client; { std::lock_guard lock(mutex_); client = client_; } - if (!client.ready()) return std::nullopt; + if (!client.ready()) return out; mcpp::RunOptions o; o.cwd = root; - o.args = {"test", "--message-format", "json"}; - for (const auto& a : target_args(p)) o.args.push_back(a); - o.args.push_back(test); - o.args.push_back("--"); - for (const auto& a : f.list) o.args.push_back(a); + o.args = test_args(p); + o.args.push_back("--guide"); o.deny = deny(); o.client = client_label(); - auto r = client.records(o); - if (!r) return std::nullopt; - for (const auto& rec : r->records) - if (rec.is_object() && rec.value("test", "") == test && rec.contains("run_output")) - return testing::listed(f, rec.value("run_output", "")); + if (auto r = client.records(o)) + for (const auto& rec : r->records) { + if (!rec.is_object() || !rec.contains("guide")) continue; + const auto list = rec["guide"].value("frameworks", Json::array()); + if (list.size() == 1) out.name = list[0].value("package", out.name); + for (const auto& f : list) { + const auto caps = f.value("capabilities", Json::array()); + const bool session = std::ranges::find(caps, Json("session")) != caps.end(); + if (session && f.contains("session") && f["session"].is_array() && !f["session"].empty()) + out.session = strings_of(f["session"]); + } + } + std::lock_guard lock(mutex_); + frameworks_[root] = out; + return out; +} + +std::optional> Session::listed_cases(const std::filesystem::path& root, const std::string& test, + const Json& p, bool ask) { + auto records = project_listing(root, p, ask); + if (!records) return std::nullopt; + for (const auto& rec : *records) { + if (rec.value("test", "") != test || !rec.contains("listed") || !rec["protocol"].is_number()) continue; + std::vector listed; + for (const auto& c : testing::protocol_listing(rec)) { + testing::Case one{c.id, c.line, "listed"}; + one.file = c.file; + one.kind = c.kind; + one.tags = c.tags; + one.expect = c.expect; + one.bounds = c.bounds; + listed.push_back(std::move(one)); + } + return listed; + } return std::nullopt; } -// Runs some of a test program's cases (§7.3): `mcpp test -- `, the report -// written to a directory of this run's own, read when the job ends -- each case's outcome sent as -// `test/caseResult` -- and removed. +// Runs some of a test program's cases (mcpp test spec P4): `mcpp test --case ...`, each +// case's outcome sent as `test/caseResult` from the program's records when the job ends (and as it +// happens through `test/caseEvent`, test-case-stream). Any program that implements the test +// protocol is run so, whatever framework wrote it; the `framework` a client names is not read. std::expected Session::test_run_cases(const Json& p) { const auto root = root_of(p); const auto test = p.value("test", ""); - const auto* f = testing::framework(p.value("framework", "")); - if (root.empty() || test.empty() || !f) return std::unexpected(invalid("test/runCases names a project, a test and its framework")); + if (root.empty() || test.empty()) return std::unexpected(invalid("test/runCases names a project and a test")); const auto cases = strings_of(p.value("cases", Json::array())); - const auto dir = host_path(mcpplibs::os::fs::utf8(mcpplibs::os::dirs::temp())) / - std::format("mcpp-ide-cases-{}-{}", std::random_device{}(), std::chrono::steady_clock::now().time_since_epoch().count()); - std::error_code ec; - std::filesystem::create_directories(dir, ec); - if (ec) return std::unexpected(rpc::Error{rpc::REQUEST_FAILED, ec.message()}); - const auto report = dir / "report"; + if (auto through = run_through_session(root, test, cases, p)) return *through; Json args = Json::array(); - if (f->protocol) { - // The protocol selects by identifier, and the record reports each case (mcpp docs/08). - for (const auto& c : cases) { args.push_back("--case"); args.push_back(c); } - } else { - args.push_back("--"); - for (const auto& a : testing::run_args(*f, cases, report)) args.push_back(a); + for (const auto& c : cases) { + args.push_back("--case"); + args.push_back(c); } Json job{{"kind", "test"}, {"root", root.generic_string()}, {"filter", test}, {"args", args}, {"color", p.value("color", true)}}; for (const auto* key : {"profile", "target"}) if (p.contains(key)) job[key] = p[key]; - const std::string framework = f->id; - return job_start(job, [this, root, test, framework, dir, report](std::string_view jobId, const Json&) { - const auto* fw = testing::framework(framework); - std::ifstream in(report, std::ios::binary); - const std::string text(std::istreambuf_iterator(in), {}); - in.close(); - std::string programOutput; + return job_start(job, [this, root, test](std::string_view jobId, const Json&) { + // Every record of the job: a program of several files reports one per part. + std::vector records; { std::lock_guard lock(mutex_); - if (auto it = programOutput_.find(jobId); it != programOutput_.end()) programOutput = it->second; + if (auto it = testRecords_.find(jobId); it != testRecords_.end()) records = it->second; } - if (fw && fw->protocol) { - // Every record of the job: a program of several files reports one per part. - std::vector records; + for (const auto& record : records) protocol_case_results(jobId, root, test, record); + }); +} + +std::optional Session::run_through_session(const std::filesystem::path& root, const std::string& test, + const std::vector& cases, const Json& p) { + const auto chosen = setting("mcpp.test.session"); + if (chosen.is_string() && chosen.get() == "off") return std::nullopt; + bool trusted; + mcpp::Client client; + { + std::lock_guard lock(mutex_); + trusted = trusted_; + client = client_; + } + if (!trusted || !client.ready() || !has_feature(root, "test-launch-records")) return std::nullopt; + const auto command = frameworks_of(root, p).session; + if (command.empty()) return std::nullopt; + testsession::Client* session = nullptr; + { + std::lock_guard lock(mutex_); + if (auto it = testSessions_.find(root); it != testSessions_.end() && it->second->alive()) session = it->second.get(); + } + if (!session) { + auto started = testsession::Client::start( + command, root, {{"MTEST_MCPP", client.program().string()}}, + [this, root](std::string_view method, const Json& params) { on_session(root, method, params); }); + if (!started) { + rpc_.notify("window/logMessage", Json{{"type", 2}, {"message", started.error()}}); + return std::nullopt; + } + std::lock_guard lock(mutex_); + session = (testSessions_[root] = std::move(*started)).get(); + } + // The configuration a run of mcpp plans, as every test command states it (test_args). + auto args = test_args(p); + args.erase(args.begin(), args.begin() + 3); // `test --message-format json` + Json selection = Json::array({Json{{"program", test}, {"cases", cases}}}); + std::string jobId; + { + std::lock_guard lock(mutex_); + jobId = std::format("job-s{}", sessionJobNext_++); + } + // The run is known, and its job started, before the session's first notification of it. + auto answered = session->request( + "test/run", Json{{"project", root.generic_string()}, {"selection", selection}, {"options", Json{{"args", args}}}}, + std::chrono::seconds{60}, [this, root, test, jobId](const Json& result) { + if (!result.is_object() || !result.contains("run") || !result["run"].is_string()) return; + const auto run = result["run"].get(); { std::lock_guard lock(mutex_); - if (auto it = testRecords_.find(jobId); it != testRecords_.end()) records = it->second; + sessionRuns_[run] = SessionRun{jobId, root, {}}; + sessionJobs_[jobId] = {root, run}; } - for (const auto& record : records) protocol_case_results(jobId, root, test, record); - } else if (fw) { - const auto outputs = testing::case_outputs(*fw, programOutput); - for (const auto& o : testing::read_report(*fw, text)) { - // Each case with its own output (passed or not) and a failure's place (D16). - const auto own = outputs.find(o.name); - bool truncated = false; - auto output = views::bounded_output(own != outputs.end() ? own->second : std::string_view{}, truncated); - const Json location = o.status == "failed" ? views::failure_location(o.message.empty() ? output : o.message) : Json(nullptr); - rpc_.notify("mcpp-ide/test/caseResult", Json{{"jobId", jobId}, {"root", root.generic_string()}, {"test", test}, - {"case", o.name}, {"status", o.status}, {"durationMs", o.durationMs}, - {"message", o.message}, {"output", std::move(output)}, - {"outputTruncated", truncated}, {"location", location}}); + rpc_.notify("mcpp-ide/job/started", Json{{"jobId", jobId}, {"kind", "test"}, {"root", root.generic_string()}, + {"argv", Json::array({"test/run", test})}}); + }); + if (!answered || !answered->contains("run")) return std::nullopt; + return Json{{"jobId", jobId}}; +} + +// What the session states, as the editors receive a job's: a case's beginning and end +// (`test/caseEvent`, with its failures and output as they came), a program's output where it has no +// case of its own, and the run's end (`job/finished`). +void Session::on_session(const std::filesystem::path& root, std::string_view method, const Json& params) { + const auto run = params.value("run", ""); + std::string jobId; + { + std::lock_guard lock(mutex_); + auto it = sessionRuns_.find(run); + if (it == sessionRuns_.end()) return; + jobId = it->second.jobId; + } + const auto program = params.value("program", ""); + if (method == "test/case") { + const auto id = params.value("id", ""); + const auto event = params.value("event", ""); + if (event == "begin") { + Json e{{"jobId", jobId}, {"root", root.generic_string()}, {"member", ""}, {"program", program}, {"part", program}, + {"case", id}, {"event", "begin"}}; + if (params.contains("t_ms")) e["tMs"] = params["t_ms"]; + rpc_.notify("mcpp-ide/test/caseEvent", e); + } else if (event == "output" || event == "failure") { + std::lock_guard lock(mutex_); + auto& c = sessionRuns_[run].cases[id]; + if (!c.is_object()) c = Json::object(); + if (event == "output") c["output"] = c.value("output", std::string{}) + params.value("text", std::string{}); + else { + Json f = params; + for (const auto* k : {"run", "program", "event", "id"}) f.erase(k); + c["failures"].push_back(std::move(f)); } + } else if (event == "end") { + Json c{{"id", id}, {"status", params.value("status", std::string("fail"))}, {"duration_ms", params.value("ms", 0.0)}, + {"reason", params.value("reason", std::string{})}}; + if (params.contains("checks")) c["checks"] = params["checks"]; + { + std::lock_guard lock(mutex_); + const auto& kept = sessionRuns_[run].cases[id]; + if (kept.contains("failures")) c["failures"] = kept["failures"]; + if (kept.contains("output")) c["output"] = kept["output"]; + } + Json e = testing::protocol_case(c); + e["jobId"] = jobId; + e["root"] = root.generic_string(); + e["member"] = ""; + e["program"] = program; + e["part"] = program; + e["event"] = "end"; + rpc_.notify("mcpp-ide/test/caseEvent", e); } - std::error_code ignored; - std::filesystem::remove_all(dir, ignored); - }); + } else if (method == "test/program" || method == "test/build") { + std::string text; + if (params.contains("output") && params["output"].is_string()) text = params["output"].get(); + if (params.contains("error") && params["error"].is_string()) text += params["error"].get() + "\n"; + if (!text.empty()) rpc_.notify("mcpp-ide/job/output", Json{{"jobId", jobId}, {"text", text}}); + } else if (method == "test/run") { + const auto status = params.value("status", std::string("fail")); + { + std::lock_guard lock(mutex_); + sessionRuns_.erase(run); + sessionJobs_.erase(jobId); + } + rpc_.notify("mcpp-ide/job/finished", Json{{"jobId", jobId}, {"kind", "test"}, {"root", root.generic_string()}, + {"status", status == "pass" ? "ok" : status == "cancelled" ? "cancelled" : "failed"}, + {"exitCode", status == "pass" ? 0 : 1}}); + } } Json Session::test_result_of(const std::filesystem::path& root, const Json& record, Json n) { @@ -2096,7 +2201,20 @@ void Session::install() { s.on("mcpp-ide/result/last", [](const Json& p, const rpc::Context&) -> Result { return projects::last_result(root_of(p)); }); s.on("mcpp-ide/job/start", [this](const Json& p, const rpc::Context&) { return job_start(p); }); s.on("mcpp-ide/job/cancel", [this](const Json& p, const rpc::Context&) -> Result { - return Json{{"cancelled", jobs_->cancel(p.value("jobId", ""))}}; + const auto jobId = p.value("jobId", ""); + testsession::Client* client = nullptr; + std::string run; + { + std::lock_guard lock(mutex_); + if (auto it = sessionJobs_.find(jobId); it != sessionJobs_.end()) + if (auto c = testSessions_.find(it->second.first); c != testSessions_.end()) { + client = c->second.get(); + run = it->second.second; + } + } + // A run through a test session is cancelled by the session: it ends the processes (O9). + if (client) return Json{{"cancelled", client->request("test/cancel", Json{{"run", run}}).has_value()}}; + return Json{{"cancelled", jobs_->cancel(jobId)}}; }); s.on("mcpp-ide/job/list", [this](const Json&, const rpc::Context&) -> Result { return jobs_->list(); }); s.on("mcpp-ide/run/launch", [this](const Json& p, const rpc::Context& c) { return run_launch(p, c); }); diff --git a/src/spec.cppm b/src/spec.cppm index 2654f59..2dd80b7 100644 --- a/src/spec.cppm +++ b/src/spec.cppm @@ -32,8 +32,6 @@ const Json& toml_schema(); const Json& buildscript_api(); // spec/commands.json: the command table the editors offer (groups, commands). const Json& commands(); -// spec/test-frameworks.json: the frameworks whose cases the editors show. -const Json& test_frameworks(); // Whether an editor (the protocol's `editor`: vscode, jetbrains, nvim, cli) offers a command: the // row's `editors` when it has them (`omitted` says why an editor is left out), else every editor; // a forward to one of mcppls's actions in every editor (the backend offers it where mcppls names an @@ -116,11 +114,6 @@ const Json& commands() { return c; } -const Json& test_frameworks() { - static const Json t = Json::parse(generated::kTestFrameworksJson); - return t; -} - bool offered_in(const Json& c, std::string_view editor) { // A deprecated id is VS Code's alone: its manifest keeps declaring it for the settings and // keybindings of earlier releases, and no other editor declares it. diff --git a/src/testing.cppm b/src/testing.cppm index 4e81729..45cc3c3 100644 --- a/src/testing.cppm +++ b/src/testing.cppm @@ -1,61 +1,29 @@ -// A test program's cases (mcpp design 2026-10-30 part 2 §7): which framework a program uses, its -// cases as its source names them and as the program lists them, the arguments that run some of -// them and write a report, and the report read back. +// A test program's cases as mcpp states them (the mcpp test spec, mcpp docs/specs/test.md): the +// programs of `mcpp test --list`, each program's cases as it lists them (B3, `--cases`), and each +// case's outcome from mcpp's records, with the details of its checks (O3) and its own output (O8). // -// mcpp's own unit is the program (`mcpp test `, judged by its exit code; F17). A case is the -// framework's, so everything here is the framework's convention, from spec/test-frameworks.json, -// and reaches the program after `--`, as written. -// -// A PROGRAM BELONGS TO A FRAMEWORK WHEN BOTH HOLD: its plan holds one of the framework's packages, -// and its source calls one of the framework's macros. A program that depends on a framework and -// writes its own `main` must not be asked to list its cases: GoogleTest's `--gtest_list_tests` -// would run such a program's tests there and then. +// NO FRAMEWORK IS KNOWN HERE (design 2026-11-10 supplement, T14). A program that implements the +// test protocol lists, selects and reports its cases, whatever framework or language wrote it; a +// program that implements none is one case, judged by its exit status. The editors read mcpp's +// records and nothing of a framework's own output, flags or report formats. + export module mcppide.testing; import std; import nlohmann.json; -import mcppide.scan; -import mcppide.spec; export namespace mcppide::testing { using Json = nlohmann::json; -struct Framework { - std::string id, title; - std::vector packages, macros; - std::string name; // "suite.case" | "literal" - std::string scenarioPrefix; // what SCENARIO puts before its name - std::vector list; - std::string listFormat; // "gtest-list" | "catch2-json" | "lines" - std::vector filter, report; - std::string join, escape, reportFormat; - // The version of the mtest test protocol the framework's programs implement (0: none). Their - // cases are their `"name"_test` literals in the source, listed by `mcpp test --cases`, run by - // `mcpp test --case` and reported in mcpp's records (mcpp's feature `test-protocol`). - int protocol = 0; -}; - -// spec/test-frameworks.json, in its order. -const std::vector& frameworks(); -const Framework* framework(std::string_view id); - -// The macro names to look for: every framework's, and those `mcpp.test.macros` adds -// ({"gtest": ["MY_TEST"]}). -std::vector macros(const Json& extra); - -// The framework of a program whose plan holds `packages` and whose source scanned as `found`; none -// when no framework has both. -const Framework* framework_of(const std::vector& packages, const scan::Result& found, const Json& extra); - +// A case as its program lists it (B3, P3): its identifier, its place (0-based line; -1 when the +// program names none), the file that declares it (a program of several files has one per case), its +// kind (`unit`, `e2e`, `cts`, `bench`; empty when it states none), its tags, `fail` when it is +// expected to fail, and the bounds on its metrics ([{metric, max?, min?}]). struct Case { - std::string name; // the framework's full name: Suite.Name, or the test's name - int line = -1; // 0-based, in the program's source; -1 when only the program listed it - std::string source; // "static" (its source) | "listed" (the program listed it) - // Of a program that implements the test protocol (mtest's manifest, or the program's listing): - // the file that declares it (a program of several files has one per case), its kind (`unit`, - // `e2e`, `cts`, `bench`; empty when neither it nor its directory states one), its tags, `fail` - // when it is expected to fail, and the bounds on its metrics ([{metric, max?, min?}]). + std::string name; + int line = -1; + std::string source = "listed"; std::string file; std::string kind; std::vector tags; @@ -70,26 +38,6 @@ Json case_json(const Case& c); // the first of `unit`, `e2e`, `cts`, `bench` below a `tests` directory in its path; empty otherwise. std::string kind_of_path(std::string_view file); -// The cases the source names, in order. -std::vector static_cases(const Framework& f, const scan::Result& found, const Json& extra); - -// The names the program's list command printed. -std::vector listed(const Framework& f, std::string_view output); - -// The name a listed case has in its source: GoogleTest's instantiated and typed names lose the -// instance prefix and the parameter (`Inst/Suite.Name/0` and `Suite/0.Name` are `Suite.Name`). -std::string base_name(const Framework& f, std::string_view name); - -// The cases after a listing: a listed name whose base is a static case's gives that case its -// listed name (the first occurrence; a second name of one base is a case of its own at the same -// line); a name with no static case is added without a line. -std::vector merge(const std::vector& statics, const std::vector& names, const Framework& f); - -// What follows `--` to run `cases` (all of them when empty) and write the report to `report`. -std::vector run_args(const Framework& f, const std::vector& cases, const std::filesystem::path& report); -// The filter alone: what a debugged program receives to run `cases`. -std::vector filter_args(const Framework& f, const std::vector& cases); - struct Outcome { std::string name; std::string status; // passed | failed | skipped @@ -97,15 +45,6 @@ struct Outcome { std::string message; }; -// Each case's own output in its program's output, by the framework's boundaries (design 2026-11-07, -// D16): GoogleTest prints `[ RUN ] Suite.Name` before a case and `[ OK ]`, `[ FAILED ]` or -// `[ SKIPPED ]` with the name after it, and the case's output is those lines and what is between -// them. Empty for a framework whose console names no case boundary (Catch2 and doctest report a case -// in their report; a failure's expression and place are its message). -std::map case_outputs(const Framework& f, std::string_view output); - -// A report the framework wrote. -std::vector read_report(const Framework& f, std::string_view text); // The cases of a test program that implements the test protocol, as a record of `mcpp test // --message-format json` states them (mcpp docs/50 §8): `listed` (from `--cases`), each with its @@ -134,9 +73,9 @@ CaseStatus case_status(std::string_view protocol); // One case of a record's `cases` (or of a `case` record that ended), as the editors receive it: // {case, status, durationMs, reason, message, location, failures: [{path, line, message, expected?, -// actual?}], and when stated metrics, bounds, kind, tags, expectedFailure, unexpectedPass}. A line is -// 0-based; `message` is the first failure with its place, else the reason; `location` is the first -// failure's place, or null. +// actual?, source?, fatal?}], and when stated checks {held, failed} (O3), output (O8), metrics, +// bounds, kind, tags, expectedFailure, unexpectedPass}. A line is 0-based; `message` is the first +// failure with its place, else the reason; `location` is the first failure's place, or null. Json protocol_case(const Json& c); struct ProtocolOutcome { @@ -225,371 +164,8 @@ double number_of(const Json& c, const char* key, const char* other) { } // The macros of `f`, with what `extra` adds for it. -std::vector macros_of(const Framework& f, const Json& extra) { - auto out = f.macros; - if (extra.is_object() && extra.contains(f.id)) - for (const auto& m : strings(extra[f.id])) out.push_back(m); - return out; -} - -std::string name_of(const Framework& f, const scan::Call& call) { - if (f.name == "suite.case") - return call.arguments.size() >= 2 ? call.arguments[0] + "." + call.arguments[1] : std::string{}; - for (const auto& a : call.arguments) - if (!a.empty() && a.front() == '"') { - const auto value = scan::literal_value(a); - return call.name == "SCENARIO" ? f.scenarioPrefix + value : value; - } - return {}; -} - -std::int64_t milliseconds(std::string_view seconds) { - double v = 0; - std::from_chars(seconds.data(), seconds.data() + seconds.size(), v); - return static_cast(v * 1000.0 + 0.5); -} - -// The value of an XML attribute in `tag` (the text between `<` and `>`), with the five entities. -std::string attribute(std::string_view tag, std::string_view name) { - const auto key = std::string(" ") + std::string(name) + "=\""; - const auto at = tag.find(key); - if (at == std::string_view::npos) return {}; - const auto from = at + key.size(); - const auto end = tag.find('"', from); - std::string raw(tag.substr(from, end == std::string_view::npos ? std::string_view::npos : end - from)); - std::string out; - for (std::size_t i = 0; i < raw.size(); ++i) { - if (raw[i] != '&') { out.push_back(raw[i]); continue; } - const auto semi = raw.find(';', i); - if (semi == std::string::npos) { out.push_back('&'); continue; } - const auto entity = std::string_view(raw).substr(i + 1, semi - i - 1); - out += entity == "lt" ? "<" : entity == "gt" ? ">" : entity == "quot" ? "\"" : entity == "apos" ? "'" : entity == "amp" ? "&" : ""; - i = semi; - } - return out; -} - -std::vector gtest_report(const Json& doc) { - std::vector out; - for (const auto& suite : doc.value("testsuites", Json::array())) { - const auto suiteName = suite.value("name", ""); - for (const auto& t : suite.value("testsuite", Json::array())) { - Outcome o; - o.name = suiteName + "." + t.value("name", ""); - o.durationMs = milliseconds(t.value("time", "0s")); - const auto failures = t.value("failures", Json::array()); - if (t.value("result", "") == "SKIPPED" || t.value("status", "") == "NOTRUN") o.status = "skipped"; - else o.status = failures.empty() ? "passed" : "failed"; - for (const auto& f : failures) { - if (!o.message.empty()) o.message += "\n"; - o.message += f.value("failure", ""); - } - out.push_back(std::move(o)); - } - } - return out; -} - -std::vector catch2_report(const Json& doc) { - std::vector out; - const auto run = doc.value("test-run", Json::object()); - for (const auto& tc : run.value("test-cases", Json::array())) { - Outcome o; - o.name = tc.value("test-info", Json::object()).value("name", ""); - const auto totals = tc.value("totals", Json::object()); - const auto assertions = totals.value("assertions", Json::object()); - const auto failed = assertions.value("failed", 0); - const auto skipped = totals.value("test-cases", Json::object()).value("skipped", 0) + assertions.value("skipped", 0); - o.status = failed > 0 ? "failed" : skipped > 0 ? "skipped" : "passed"; - // The failed assertions' expressions, as the reporter wrote them. - std::function walk = [&](const Json& node) { - if (node.is_object()) { - if (node.value("kind", "") == "assertion" && node.value("status", true) == false) { - if (!o.message.empty()) o.message += "\n"; - const auto loc = node.value("source-location", Json::object()); - o.message += std::format("{}:{}: {}", loc.value("filename", ""), loc.value("line", 0), node.value("expression", node.value("message", ""))); - } - for (const auto& v : node) walk(v); - } else if (node.is_array()) { - for (const auto& v : node) walk(v); - } - }; - walk(tc.value("runs", Json::array())); - if (o.status == "failed" && o.message.empty()) o.message = std::format("{} assertion(s) failed", failed); - out.push_back(std::move(o)); - } - return out; -} - -std::vector junit_report(std::string_view text) { - std::vector out; - std::size_t at = 0; - while ((at = text.find("', at); - if (close == std::string_view::npos) break; - const auto tag = text.substr(at, close - at); - Outcome o; - o.name = attribute(tag, "name"); - o.durationMs = milliseconds(attribute(tag, "time")); - o.status = attribute(tag, "status") == "skipped" ? "skipped" : "passed"; - const bool empty = close > 0 && text[close - 1] == '/'; - if (!empty) { - const auto end = text.find("", close); - const auto body = text.substr(close + 1, end == std::string_view::npos ? std::string_view::npos : end - close - 1); - if (const auto f = body.find("', f); - o.message = attribute(body.substr(f, fclose - f), "message"); - } else if (body.find("& frameworks() { - static const std::vector all = [] { - std::vector out; - for (const auto& f : spec::test_frameworks().value("frameworks", Json::array())) { - Framework v; - v.id = f.value("id", ""); - v.title = f.value("title", v.id); - v.packages = strings(f.value("packages", Json::array())); - v.macros = strings(f.value("macros", Json::array())); - v.name = f.value("name", "literal"); - v.scenarioPrefix = f.value("scenarioPrefix", ""); - v.list = strings(f.value("list", Json::array())); - v.listFormat = f.value("listFormat", "lines"); - v.filter = strings(f.value("filter", Json::array())); - v.report = strings(f.value("report", Json::array())); - v.join = f.value("join", ","); - v.escape = f.value("escape", ""); - v.reportFormat = f.value("reportFormat", ""); - v.protocol = f.value("protocol", 0); - out.push_back(std::move(v)); - } - return out; - }(); - return all; -} - -const Framework* framework(std::string_view id) { - for (const auto& f : frameworks()) if (f.id == id) return &f; - return nullptr; -} - -std::vector macros(const Json& extra) { - std::vector out; - for (const auto& f : frameworks()) - for (const auto& m : macros_of(f, extra)) - if (std::ranges::find(out, m) == out.end()) out.push_back(m); - return out; -} - -const Framework* framework_of(const std::vector& packages, const scan::Result& found, const Json& extra) { - for (const auto& f : frameworks()) { - const bool linked = std::ranges::any_of(f.packages, [&](const std::string& p) { - return std::ranges::any_of(packages, [&](const std::string& in) { return in == p || in.starts_with(p + ":"); }); - }); - if (!linked) continue; - if (f.protocol && !found.cases.empty()) return &f; - const auto names = macros_of(f, extra); - if (std::ranges::any_of(found.calls, [&](const scan::Call& c) { return std::ranges::find(names, c.name) != names.end(); })) - return &f; - } - return nullptr; -} - -std::vector static_cases(const Framework& f, const scan::Result& found, const Json& extra) { - std::vector out; - if (f.protocol) { - for (const auto& c : found.cases) { - Case one{c.id, c.line, "static"}; - one.kind = c.kind; - one.tags = c.tags; - for (const auto& b : c.bounds) one.bounds.push_back(bound_json(b.metric, b.max, b.min)); - out.push_back(std::move(one)); - } - return out; - } - const auto names = macros_of(f, extra); - for (const auto& call : found.calls) { - if (std::ranges::find(names, call.name) == names.end()) continue; - auto name = name_of(f, call); - if (!name.empty()) out.push_back(Case{std::move(name), call.line, "static"}); - } - return out; -} - -std::vector listed(const Framework& f, std::string_view output) { - std::vector out; - if (f.listFormat == "gtest-list") { - std::string suite; - for (const auto& raw : lines(output)) { - const std::string_view line(raw); - const auto comment = line.find(" #"); - const auto text = trim(comment == std::string_view::npos ? line : line.substr(0, comment)); - if (text.empty()) continue; - if (!line.empty() && line.front() != ' ') { - if (text.back() == '.') suite = text; - continue; - } - if (!suite.empty()) out.push_back(suite + text); - } - } else if (f.listFormat == "catch2-json") { - const auto start = output.find('{'); - const Json doc = Json::parse(start == std::string_view::npos ? std::string_view{} : output.substr(start), nullptr, false); - if (doc.is_object()) - for (const auto& t : doc.value("listings", Json::object()).value("tests", Json::array())) - if (t.contains("name") && t["name"].is_string()) out.push_back(t["name"].get()); - } else { - for (const auto& raw : lines(output)) { - const auto text = trim(raw); - if (text.empty() || text.starts_with("[doctest]") || text.starts_with("===")) continue; - out.push_back(raw.starts_with(" Scenario: ") ? raw : text); - } - } - return out; -} - -std::string base_name(const Framework& f, std::string_view name) { - if (f.name != "suite.case") return std::string(name); - const auto dot = name.find('.'); - if (dot == std::string_view::npos) return std::string(name); - auto suite = name.substr(0, dot); - auto test = name.substr(dot + 1); - // TEST_P: Inst/Suite.Name/0; TYPED_TEST: Suite/0.Name; TYPED_TEST_P: Inst/Suite/0.Name. - if (const auto slash = test.find('/'); slash != std::string_view::npos) test = test.substr(0, slash); - const auto parts = [](std::string_view s) { - std::vector v; - std::size_t at = 0; - while (true) { - const auto next = s.find('/', at); - v.push_back(s.substr(at, next == std::string_view::npos ? std::string_view::npos : next - at)); - if (next == std::string_view::npos) break; - at = next + 1; - } - return v; - }(suite); - // The suite is the last part that is not a number. - for (auto it = parts.rbegin(); it != parts.rend(); ++it) - if (!it->empty() && !std::ranges::all_of(*it, [](char c) { return std::isdigit(static_cast(c)); })) { - suite = *it; - break; - } - return std::format("{}.{}", suite, test); -} - -std::vector merge(const std::vector& statics, const std::vector& names, const Framework& f) { - std::vector out = statics; - std::vector taken(out.size(), false); - for (const auto& name : names) { - const auto base = base_name(f, name); - bool placed = false; - for (std::size_t i = 0; i < statics.size() && !placed; ++i) { - if (statics[i].name != base) continue; - if (!taken[i]) { - out[i].name = name; - out[i].source = "listed"; - taken[i] = true; - } else { - out.push_back(Case{name, statics[i].line, "listed"}); - } - placed = true; - } - if (!placed) out.push_back(Case{name, -1, "listed"}); - } - return out; -} - -std::vector run_args(const Framework& f, const std::vector& cases, const std::filesystem::path& report) { - auto out = filter_args(f, cases); - for (const auto& a : f.report) { - std::string arg = a; - if (const auto at = arg.find("{}"); at != std::string::npos) arg.replace(at, 2, report.string()); - out.push_back(std::move(arg)); - } - return out; -} - -std::vector filter_args(const Framework& f, const std::vector& cases) { - std::vector out; - if (!cases.empty()) { - // A GoogleTest case named as its source names it stands for its instances too: - // Inst/Suite.Name/N (TEST_P) and Suite/N.Name (TYPED_TEST). - std::vector patterns; - for (const auto& c : cases) { - patterns.push_back(c); - if (f.name == "suite.case" && c.find('/') == std::string::npos) - if (const auto dot = c.find('.'); dot != std::string::npos) { - patterns.push_back("*/" + c + "/*"); - patterns.push_back(c.substr(0, dot) + "/*." + c.substr(dot + 1)); - } - } - std::string joined; - for (const auto& c : patterns) { - if (!joined.empty()) joined += f.join; - for (char ch : c) { - if (f.escape.find(ch) != std::string::npos) joined.push_back('\\'); - joined.push_back(ch); - } - } - for (const auto& a : f.filter) { - std::string arg = a; - if (const auto at = arg.find("{}"); at != std::string::npos) arg.replace(at, 2, joined); - out.push_back(std::move(arg)); - } - } - return out; -} - -std::map case_outputs(const Framework& f, std::string_view output) { - std::map out; - if (f.id != "gtest") return out; - std::string current, text; - std::size_t at = 0; - while (at < output.size()) { - auto end = output.find('\n', at); - const auto stop = end == std::string_view::npos ? output.size() : end + 1; - const auto line = output.substr(at, stop - at); - at = stop; - // `[ RUN ] Suite.Name`; the result lines name it again, then ` (N ms)`. - const auto tagged = [&](std::string_view tag) -> std::optional { - if (!line.starts_with(tag)) return std::nullopt; - auto name = line.substr(tag.size()); - while (!name.empty() && (name.back() == '\n' || name.back() == '\r')) name.remove_suffix(1); - if (const auto paren = name.find(" ("); paren != std::string_view::npos) name = name.substr(0, paren); - return std::string(name); - }; - if (auto started = tagged("[ RUN ] ")) { - current = *started; - text = std::string(line); - continue; - } - if (current.empty()) continue; - text += line; - for (std::string_view tag : {"[ OK ] ", "[ FAILED ] ", "[ SKIPPED ] "}) - if (auto ended = tagged(tag); ended && *ended == current) { - out[current] = std::move(text); - current.clear(); - text.clear(); - break; - } - } - // A case the program ended in (a crash): what it printed until then. - if (!current.empty()) out[current] = std::move(text); - return out; -} - std::vector protocol_listing(const Json& record) { std::vector out; for (const auto& c : record.value("listed", Json::array())) { @@ -625,8 +201,9 @@ Json protocol_case(const Json& c) { if (!f.is_object()) continue; const int line = f.value("line", 0); Json one{{"path", f.value("file", "")}, {"line", line > 0 ? line - 1 : 0}, {"message", f.value("message", "")}}; - for (const auto* key : {"expected", "actual"}) + for (const auto* key : {"expected", "actual", "source"}) if (f.contains(key) && f[key].is_string()) one[key] = f[key]; + if (f.value("fatal", false)) one["fatal"] = true; out["failures"].push_back(std::move(one)); } if (!out["failures"].empty()) { @@ -638,6 +215,8 @@ Json protocol_case(const Json& c) { } else { out["message"] = out["reason"]; } + if (auto k = c.find("checks"); k != c.end() && k->is_object()) out["checks"] = *k; + if (auto o = c.find("output"); o != c.end() && o->is_string() && !o->get().empty()) out["output"] = *o; if (auto m = c.find("metrics"); m != c.end() && m->is_array() && !m->empty()) out["metrics"] = *m; if (auto b = bounds_of(c); !b.empty()) out["bounds"] = std::move(b); if (auto k = c.value("kind", ""); !k.empty()) out["kind"] = k; @@ -749,13 +328,4 @@ std::string part_of_file(const Program& program, const std::filesystem::path& fi return {}; } -std::vector read_report(const Framework& f, std::string_view text) { - if (f.reportFormat == "junit") return junit_report(text); - const Json doc = Json::parse(text, nullptr, false); - if (!doc.is_object()) return {}; - if (f.reportFormat == "gtest-json") return gtest_report(doc); - if (f.reportFormat == "catch2-json") return catch2_report(doc); - return {}; -} - } // namespace mcppide::testing diff --git a/src/testsession.cppm b/src/testsession.cppm new file mode 100644 index 0000000..ffa2b86 --- /dev/null +++ b/src/testsession.cppm @@ -0,0 +1,163 @@ +// A test session (the mcpp test spec's O9, mcpp docs/specs/test.md): the command a project's test +// framework states (`test-session`, B6), started once per project and spoken to in JSON-RPC 2.0 on +// its standard streams, one message per line. The backend runs a project's cases through it when +// one is stated: a run is answered at once, a case added joins it, and a cancel ends the processes +// of what it names. Nothing here knows which framework states it; the methods and notifications are +// the spec's. +export module mcppide.testsession; + +import std; +import nlohmann.json; +import mcpplibs.os; + +export namespace mcppide::testsession { + +using Json = nlohmann::json; + +class Client { +public: + // A notification of the session: its method (`test/build`, `test/case`, `test/program`, + // `test/run`) and its parameters. Called from the session's reading thread. + using notify_fn = std::function; + + // The session started from `argv` in `cwd`, with `env` over this program's environment; an + // error names why not. + static std::expected, std::string> + start(const std::vector& argv, const std::filesystem::path& cwd, + const std::vector>& env, notify_fn notify); + + ~Client(); + Client(const Client&) = delete; + Client& operator=(const Client&) = delete; + + // Called with a request's `result` on the reading thread, before any message the session sent + // after its answer: what an answer names (a run) is known before its notifications arrive. + using answered_fn = std::function; + + // A request and its answer (`result`), or why there is none: the session's `error`, its end, + // or `limit` passing. + std::expected request(std::string_view method, const Json& params, + std::chrono::milliseconds limit = std::chrono::seconds{60}, + answered_fn answered = {}); + // Whether the session still runs. + bool alive() const; + +private: + Client() = default; + void read_loop(); + + mcpplibs::os::process::process proc_; + notify_fn notify_; + mutable std::mutex mutex_; + std::condition_variable answered_; + std::map answers_; + std::map hooks_; + std::int64_t next_ = 1; + bool ended_ = false; + std::jthread reader_; +}; + +} // namespace mcppide::testsession + +namespace mcppide::testsession { + +namespace os = mcpplibs::os; + +std::expected, std::string> +Client::start(const std::vector& argv, const std::filesystem::path& cwd, + const std::vector>& env, notify_fn notify) { + if (argv.empty()) return std::unexpected(std::string("the session names no command")); + os::process::spawn_options o; + o.program = argv.front(); + o.args.assign(argv.begin() + 1, argv.end()); + o.cwd = cwd; + o.in = os::process::stdio::pipe; + o.out = os::process::stdio::pipe; + o.err = os::process::stdio::null; + // Ending it ends the programs it started. It ends with the backend by its input closing (a + // session ends at the end of its input); a lifetime bound by the system is not asked, as on + // Linux it follows the thread that started it, and a request's thread ends with the request. + o.own_unit = true; + for (const auto& [k, v] : env) o.env.emplace_back(k, v); + auto p = os::process::process::spawn(o); + if (!p) return std::unexpected(std::format("{} did not start: {}", argv.front(), p.error().message())); + std::unique_ptr c(new Client()); + c->proc_ = std::move(*p); + c->notify_ = std::move(notify); + c->reader_ = std::jthread([raw = c.get()] { raw->read_loop(); }); + return c; +} + +Client::~Client() { + if (proc_.valid()) { + (void)proc_.write(Json{{"jsonrpc", "2.0"}, {"method", "shutdown"}, {"id", 0}}.dump() + "\n"); + proc_.close_input(); + if (auto w = proc_.wait_for(std::chrono::seconds{3}); !w || !*w) proc_.kill(); + } + if (reader_.joinable()) reader_.join(); +} + +bool Client::alive() const { + std::lock_guard g(mutex_); + return !ended_; +} + +void Client::read_loop() { + std::string pending; + for (;;) { + auto chunk = proc_.read_output(); + if (!chunk || chunk->empty()) break; + pending += *chunk; + for (auto nl = pending.find('\n'); nl != std::string::npos; nl = pending.find('\n')) { + const auto line = pending.substr(0, nl); + pending.erase(0, nl + 1); + auto m = Json::parse(line, nullptr, false); + if (!m.is_object()) continue; + if (m.contains("id") && m["id"].is_number_integer() && (m.contains("result") || m.contains("error"))) { + const auto id = m["id"].get(); + answered_fn hook; + { + std::lock_guard g(mutex_); + if (auto h = hooks_.find(id); h != hooks_.end()) { + hook = std::move(h->second); + hooks_.erase(h); + } + } + if (hook && m.contains("result")) hook(m["result"]); + std::lock_guard g(mutex_); + answers_[id] = std::move(m); + answered_.notify_all(); + } else if (m.contains("method") && m["method"].is_string() && notify_) { + notify_(m["method"].get(), m.value("params", Json::object())); + } + } + } + std::lock_guard g(mutex_); + ended_ = true; + answered_.notify_all(); +} + +std::expected Client::request(std::string_view method, const Json& params, + std::chrono::milliseconds limit, answered_fn answered) { + std::int64_t id; + { + std::lock_guard g(mutex_); + if (ended_) return std::unexpected(std::string("the test session has ended")); + id = next_++; + if (answered) hooks_[id] = std::move(answered); + } + const Json m{{"jsonrpc", "2.0"}, {"id", id}, {"method", std::string(method)}, {"params", params}}; + if (auto w = proc_.write(m.dump() + "\n"); !w) return std::unexpected(std::string("the test session does not read")); + std::unique_lock l(mutex_); + if (!answered_.wait_for(l, limit, [&] { return ended_ || answers_.contains(id); })) { + hooks_.erase(id); + return std::unexpected(std::format("the test session did not answer {} in time", method)); + } + if (!answers_.contains(id)) return std::unexpected(std::string("the test session has ended")); + auto a = std::move(answers_[id]); + answers_.erase(id); + if (a.contains("error")) return std::unexpected(a["error"].value("message", std::string("the session refused it"))); + return a.value("result", Json::object()); +} + +} // namespace mcppide::testsession diff --git a/tests/bench/backend.cpp b/tests/bench/backend.cpp index 1669871..c7be2c1 100644 --- a/tests/bench/backend.cpp +++ b/tests/bench/backend.cpp @@ -81,11 +81,10 @@ Bench keystroke{"a keystroke in a manifest of 400 dependencies", at_most{"manife }); }}; -Bench testFile{"the entries and cases of a test file of 2000 cases", at_most{"tests.scan", 0.1}, [] { +Bench testFile{"where a test file of 2000 cases starts", at_most{"tests.scan", 0.1}, [] { const auto text = test_file(2000); bench::measure("tests.scan", [&] { - const auto found = scan::scan(text, {}); - testsupport::check(found.cases.size() == 2200); + const auto found = scan::scan(text); bench::keep(found); }, {.samples = 10}); }}; diff --git a/tests/e2e/jobs.cpp b/tests/e2e/jobs.cpp index dfe04b6..42de3d7 100644 --- a/tests/e2e/jobs.cpp +++ b/tests/e2e/jobs.cpp @@ -82,6 +82,34 @@ E2E jobQueue{"jobs", [] { if (mcpplibs::os::host() == mcpplibs::os::platform::windows) check(record.find("\"ok\"") == std::string::npos); else check(record.find("\"cancelled\"") != std::string::npos); + // Tests run beside each other (design 2026-11-10 supplement, T14): a test asked for while + // another of the project runs starts at once, and cancelling one leaves the other running. + jobs::Request first{.kind = jobs::Kind::test, .root = root, .args = {"test", "--message-format", "json"}, .mode = jobs::Mode::ndjson}; + first.env = fixtures; + first.env.push_back({"MOCK_MCPP_SLOW_MS", "20000"}); + auto second = first; + const auto a = runner.submit(first); + const auto b = runner.submit(second); + bool both = false; + for (int i = 0; i < 200 && !both; ++i) { + std::this_thread::sleep_for(std::chrono::milliseconds(10)); + const auto list = runner.list(); + both = std::ranges::count_if(list, [](const Json& j) { return j.value("state", "") == "running"; }) == 2; + } + check(both); + check(runner.cancel(a)); + std::this_thread::sleep_for(std::chrono::milliseconds(300)); + { + const auto list = runner.list(); + check(std::ranges::any_of(list, [&](const Json& j) { return j.value("id", "") == b && j.value("state", "") == "running"; })); + } + check(runner.cancel(b)); + runner.drain(); + { + std::lock_guard g(seen.lock); + check(seen.finished[a]["status"] == "cancelled" && seen.finished[b]["status"] == "cancelled"); + } + std::error_code ec; std::filesystem::remove_all(root, ec); }}; diff --git a/tests/e2e/protocol.cpp b/tests/e2e/protocol.cpp index a65e62a..6ed9fa4 100644 --- a/tests/e2e/protocol.cpp +++ b/tests/e2e/protocol.cpp @@ -340,10 +340,13 @@ E2E protocol{"protocol", [] { << "import mcpplibs.mtest;\nusing namespace mcpplibs::mtest;\nsuite parsing{\"parsing\", [] {\n" " \"an empty line\"_test = [] {};\n \"a field\"_test = [] {};\n}};\n"; const auto listing = result_of("mcpp-ide/test/cases", Json{{"root", protocolRoot.generic_string()}, {"test", "unit/parse"}, {"list", true}}); - check(listing["framework"] == "mtest" && listing.value("listed", false)); + check(listing["protocol"] == 1 && listing.value("listed", false) && listing["framework"].is_string()); check(listing["cases"].size() == 2 && listing["cases"][0]["name"] == "parsing/an empty line" && listing["cases"][0]["line"] == 3); + // Without `list` the last listing answers: the cases shown stay until a listing replaces them. + const auto kept = result_of("mcpp-ide/test/cases", Json{{"root", protocolRoot.generic_string()}, {"test", "unit/parse"}}); + check(kept.value("listed", false) && kept["cases"] == listing["cases"]); const auto caseResultsBefore = wire.notifications("mcpp-ide/test/caseResult").size(); - const auto run = result_of("mcpp-ide/test/runCases", Json{{"root", protocolRoot.generic_string()}, {"test", "unit/parse"}, {"framework", "mtest"}, + const auto run = result_of("mcpp-ide/test/runCases", Json{{"root", protocolRoot.generic_string()}, {"test", "unit/parse"}, {"cases", Json::array({"parsing/an empty line", "parsing/a field"})}}); check(run.contains("jobId")); for (int i = 0; i < 2000 && wire.notifications("mcpp-ide/test/caseResult").size() < caseResultsBefore + 2; ++i) @@ -358,6 +361,36 @@ E2E protocol{"protocol", [] { for (const auto& n : caseResults) check(conforms(n, protocol["notifications"]["mcpp-ide/test/caseResult"]["params"], protocol, "mcpp-ide/test/caseResult")); } + // THROUGH A TEST SESSION (the mcpp test spec's O9): a project whose framework states one + // (`mcpp test --guide`, B6) and whose mcpp gives launches (B5) runs its cases through it: the + // run is a job of its own, its cases arrive as they begin and end with their checks and output, + // and its end is the job's. + { + const auto dir = root.parent_path() / (root.filename().string() + "-session"); + std::filesystem::create_directories(dir / "tests" / "unit"); + std::ofstream(dir / "mcpp.toml") << "[package]\nname = \"app\"\nversion = \"0.1.0\"\n"; + std::ofstream(dir / ".mock-mcpp-features") << "test-protocol,test-launch-records"; + std::ofstream(dir / ".mock-mcpp-session") << "on"; + std::ofstream(dir / "tests" / "unit" / "parse.cpp") << "int main() { return 0; }\n"; + const auto events = wire.notifications("mcpp-ide/test/caseEvent").size(); + const auto ran = result_of("mcpp-ide/test/runCases", Json{{"root", dir.generic_string()}, {"test", "unit/parse"}, + {"cases", Json::array({"parsing/a field"})}}); + const auto job = ran.value("jobId", ""); + check(job.starts_with("job-s")); + auto finished = [&] { + for (const auto& n : wire.notifications("mcpp-ide/job/finished")) if (n.value("jobId", "") == job) return n; + return Json(); + }; + for (int i = 0; i < 2000 && finished().is_null(); ++i) std::this_thread::sleep_for(std::chrono::milliseconds(5)); + check(finished().value("status", "") == "ok"); + const auto all = wire.notifications("mcpp-ide/test/caseEvent"); + check(all.size() >= events + 2); + if (all.size() >= events + 2) { + check(all[events]["event"] == "begin" && all[events]["case"] == "parsing/a field" && all[events]["jobId"] == job); + const auto& end = all[events + 1]; + check(end["event"] == "end" && end["status"] == "passed" && end["checks"]["held"] == 1 && end["output"] == "said\n"); + } + } // A program of several files (design 2026-11-10): as mcpp 2026.11.9.1 states it (its parts // derived), and as mcpp#917 states it (parts, case records, metrics). A run's part records are // attributed to their program, each part's cases reported, and the store's runs read back. @@ -389,12 +422,11 @@ E2E protocol{"protocol", [] { } const auto entries = result_of("mcpp-ide/source/entries", Json{{"file", b}}); check(entries["kind"] == "test" && entries["test"] == "unit" && entries["part"] == "unit/b"); - // Before a listing, the program's files (its cases are read from them where its plan holds - // mtest; the mock's plan does not); the listing places each case in its file. + // Before a listing, the program's files and no case; the listing places each case in its file. const auto statics = result_of("mcpp-ide/test/cases", Json{{"root", dir.generic_string()}, {"test", "unit"}}); check(statics["test"] == "unit" && statics["files"].size() == 2 && same(statics["files"][0], a) && same(statics["files"][1], b)); const auto listedCases = result_of("mcpp-ide/test/cases", Json{{"root", dir.generic_string()}, {"test", "unit"}, {"list", true}}); - check(listedCases.value("listed", false) && listedCases["framework"] == "mtest" && listedCases["cases"].size() == 2); + check(listedCases.value("listed", false) && listedCases["protocol"] == 1 && listedCases["cases"].size() == 2); if (listedCases["cases"].size() == 2) { const auto& alpha = listedCases["cases"][0]; check(alpha["name"] == "alpha" && same(alpha["file"], a) && alpha["line"] == 3 && alpha["kind"] == "bench" && alpha["bounds"][0]["max"] == 15); @@ -429,7 +461,7 @@ E2E protocol{"protocol", [] { // record states them, not the job's filter. const auto caseResultsBefore = wire.notifications("mcpp-ide/test/caseResult").size(); const auto finishedCases = wire.notifications("mcpp-ide/job/finished").size(); - result_of("mcpp-ide/test/runCases", Json{{"root", dir.generic_string()}, {"test", "unit/a"}, {"framework", "mtest"}, + result_of("mcpp-ide/test/runCases", Json{{"root", dir.generic_string()}, {"test", "unit/a"}, {"cases", Json::array({"alpha"})}}); for (int i = 0; i < 2000 && wire.notifications("mcpp-ide/job/finished").size() <= finishedCases; ++i) std::this_thread::sleep_for(std::chrono::milliseconds(5)); diff --git a/tests/unit/testing.cpp b/tests/unit/testing.cpp index 11f153e..b6d7758 100644 --- a/tests/unit/testing.cpp +++ b/tests/unit/testing.cpp @@ -1,14 +1,12 @@ -// A source read as text (design 2026-10-30 part 2 §6.3, §7.3): `main` and a framework's macros, -// never inside a comment, a string or a raw string; a program's framework needs both its package -// and its macros; cases named as each framework names them, listed names attached to them, the -// filter escaped by the framework's rules, and the three report formats read back. And the -// backend's source/entries and test/cases over a project on disk. +// What the editors read of a test program (mcpp test spec, mcpp docs/specs/test.md): where a source +// starts (`main`, never inside a comment, a string or a raw string), a program's cases as it lists +// them (B3), each case's outcome with the details of its checks (O3) and its own output (O8), and the +// backend's source/entries and test/cases over a project on disk. No test framework is known. import std; import mcppide.testsupport; import mcpplibs.mtest; import nlohmann.json; import mcppide.scan; -import mcpplibs.mtest.manifest; import mcppide.testing; import mcppide.rpc; import mcppide.server; @@ -19,16 +17,6 @@ using namespace mcpplibs::mtest; using namespace mcppide; using Json = nlohmann::json; -namespace { - -const testing::Framework& fw(std::string_view id) { - const auto* f = testing::framework(id); - check(f != nullptr); - return *f; -} - -} // namespace - UTest sources{"sources", [] { // main: the first `int main(` or `auto main(` at file scope, outside comments and literals. "main is found outside comments and literals"_test = [&] { @@ -41,146 +29,20 @@ const char* raw = R"x(int main() namespace n { int main(); } int main(int argc, char** argv) { return 0; } -)", {}); +)"); check(r.mainLine && *r.mainLine == 8); - check(!scan::scan("void f() { obj.main(); std::main(); }", {}).mainLine); - check(scan::scan("export auto main() -> int { return 0; }", {}).mainLine == 0); - }; - // Macro calls at the start of a declaration, with their arguments. - const auto macros = testing::macros(Json::object()); - "macro calls and their arguments"_test = [&] { - const auto r = scan::scan(R"(#include -TEST(Math, Adds) { EXPECT_EQ(1 + 1, 2); } -// TEST(Not, This) -TEST_P(Param, Works) {} -void helper() { auto x = TEST(1, 2); } -TEST(Math, - Spans) {} -)", macros); - check(r.calls.size() == 3); - check(r.calls[0].name == "TEST" && r.calls[0].line == 1 && r.calls[0].arguments == std::vector({"Math", "Adds"})); - check(r.calls[1].name == "TEST_P" && r.calls[1].line == 3); - check(r.calls[2].line == 5 && r.calls[2].arguments[1] == "Spans"); - check(scan::literal_value(R"("a \"b\"" " c")") == "a \"b\" c"); - }; - - // A program's framework: its package in the plan and its macros in its source. - "a framework needs its package and its macros"_test = [&] { - const auto gtestSource = scan::scan("TEST(A, B) {}\nTEST_F(F, C) {}\n", macros); - check(testing::framework_of({"app", "compat.gtest"}, gtestSource, Json::object()) == &fw("gtest")); - check(testing::framework_of({"app"}, gtestSource, Json::object()) == nullptr); - const auto ownMain = scan::scan("int main() { return 0; }\n", macros); - check(testing::framework_of({"compat.gtest"}, ownMain, Json::object()) == nullptr); - // A macro of the project's own, named by mcpp.test.macros. - const Json extra{{"gtest", Json::array({"MY_TEST"})}}; - const auto own = scan::scan("MY_TEST(S, N) {}\n", testing::macros(extra)); - check(testing::framework_of({"compat.gtest"}, own, extra) == &fw("gtest")); - const auto cases = testing::static_cases(fw("gtest"), own, extra); - check(cases.size() == 1 && cases[0].name == "S.N" && cases[0].line == 0); - // Catch2 and doctest share TEST_CASE; the package decides. - const auto both = scan::scan("TEST_CASE(\"adds\", \"[math]\") {}\nSCENARIO(\"grows\") {}\n", macros); - check(testing::framework_of({"compat.catch2"}, both, Json::object()) == &fw("catch2")); - check(testing::framework_of({"compat.doctest"}, both, Json::object()) == &fw("doctest")); - const auto catchCases = testing::static_cases(fw("catch2"), both, Json::object()); - check(catchCases.size() == 2 && catchCases[0].name == "adds" && catchCases[1].name == "Scenario: grows"); - const auto doctestCases = testing::static_cases(fw("doctest"), both, Json::object()); - check(doctestCases[1].name == " Scenario: grows"); - }; - - // Listings, base names and the merge. - "listings, base names and the merge"_test = [&] { - const auto names = testing::listed(fw("gtest"), "Math.\n Adds\n Spans\nInst/Param.\n Works/0 # GetParam() = 1\n Works/1 # GetParam() = 2\nTyped/0. # TypeParam = int\n Holds\n"); - check(names == std::vector({"Math.Adds", "Math.Spans", "Inst/Param.Works/0", "Inst/Param.Works/1", "Typed/0.Holds"})); - check(testing::base_name(fw("gtest"), "Inst/Param.Works/1") == "Param.Works"); - check(testing::base_name(fw("gtest"), "Typed/0.Holds") == "Typed.Holds"); - const std::vector statics{{"Math.Adds", 1, "static"}, {"Param.Works", 3, "static"}, {"Math.Spans", 5, "static"}}; - const auto merged = testing::merge(statics, names, fw("gtest")); - check(merged[0].name == "Math.Adds" && merged[0].source == "listed"); - check(merged[1].name == "Inst/Param.Works/0" && merged[1].line == 3); - check(std::ranges::any_of(merged, [](const auto& c) { return c.name == "Inst/Param.Works/1" && c.line == 3; })); - check(std::ranges::any_of(merged, [](const auto& c) { return c.name == "Typed/0.Holds" && c.line == -1; })); - const auto listedCatch = testing::listed(fw("catch2"), R"({"version": 1, "listings": {"tests": [{"name": "adds"}, {"name": "a, b"}]}})"); - check(listedCatch == std::vector({"adds", "a, b"})); - const auto listedDoctest = testing::listed(fw("doctest"), "[doctest] doctest version is \"2.4.12\"\n===\nadds\n Scenario: grows\n===\n[doctest] unskipped test cases passing the current filters: 2\n"); - check(listedDoctest == std::vector({"adds", " Scenario: grows"})); - }; - - // The arguments after `--`: the filter escaped by the framework's rules, and the report. - "the filter and the report arguments"_test = [&] { - const auto g = testing::run_args(fw("gtest"), {"Math.Adds", "Inst/Param.Works/0"}, "/r/out"); - check(g == std::vector({"--gtest_filter=Math.Adds:*/Math.Adds/*:Math/*.Adds:Inst/Param.Works/0", "--gtest_output=json:/r/out"})); - const auto c = testing::run_args(fw("catch2"), {"a, b", "x[1]"}, "/r/out"); - check(c == std::vector({"a\\, b,x\\[1\\]", "--reporter", "json::out=/r/out"})); - const auto d = testing::run_args(fw("doctest"), {}, "/r/out"); - check(d == std::vector({"--reporters=junit", "--out=/r/out"})); + check(!scan::scan("void f() { obj.main(); std::main(); }").mainLine); + check(scan::scan("export auto main() -> int { return 0; }").mainLine == 0); }; - // Reports. - "reports"_test = [&] { - const auto g = testing::read_report(fw("gtest"), R"({"testsuites": [{"name": "Math", "testsuite": [ - {"name": "Adds", "status": "RUN", "result": "COMPLETED", "time": "0.002s"}, - {"name": "Spans", "status": "RUN", "result": "COMPLETED", "time": "0s", "failures": [{"failure": "a.cpp:3\nExpected 2"}]}, - {"name": "Later", "status": "RUN", "result": "SKIPPED", "time": "0s"}]}]})"); - check(g.size() == 3 && g[0].name == "Math.Adds" && g[0].status == "passed" && g[0].durationMs == 2); - check(g[1].status == "failed" && g[1].message.find("Expected 2") != std::string::npos); - check(g[2].status == "skipped"); - const auto c = testing::read_report(fw("catch2"), R"({"version": 1, "test-run": {"test-cases": [ - {"test-info": {"name": "adds"}, "runs": [], "totals": {"assertions": {"passed": 1, "failed": 0}}}, - {"test-info": {"name": "grows"}, "runs": [{"path": [{"kind": "assertion", "status": false, "source-location": {"filename": "g.cpp", "line": 7}}]}], - "totals": {"assertions": {"passed": 0, "failed": 1}}}]}})"); - check(c.size() == 2 && c[0].status == "passed" && c[1].status == "failed" && c[1].message.starts_with("g.cpp:7")); - const auto d = testing::read_report(fw("doctest"), R"( - - - - -)"); - check(d.size() == 3 && d[0].status == "passed" && d[0].durationMs == 1); - check(d[1].name == "a & b" && d[1].status == "failed" && d[1].message.find("NOT correct") != std::string::npos); - check(d[2].status == "skipped"); - - // Each case's own output, by GoogleTest's boundaries (D16); none for a framework without them. - const auto outputs = testing::case_outputs(fw("gtest"), "Running main() from gtest_main.cc\n" - "[==========] Running 3 tests from 1 test suite.\n" - "[ RUN ] Math.Adds\nadding\n[ OK ] Math.Adds (2 ms)\n" - "[ RUN ] Math.Spans\na.cpp:3: Failure\nExpected 2\n[ FAILED ] Math.Spans (0 ms)\n" - "[ RUN ] Math.Crashes\nhalf way\n"); - check(outputs.size() == 3); - check(outputs.at("Math.Adds") == "[ RUN ] Math.Adds\nadding\n[ OK ] Math.Adds (2 ms)\n"); - check(outputs.at("Math.Spans").contains("a.cpp:3: Failure") && outputs.at("Math.Spans").ends_with("[ FAILED ] Math.Spans (0 ms)\n")); - check(outputs.at("Math.Crashes") == "[ RUN ] Math.Crashes\nhalf way\n"); - check(testing::case_outputs(fw("catch2"), "[ RUN ] x\n").empty()); - }; - - // mtest's cases as mtest's static manifest reads them: each object (`UTest`, `Suite`, ...) is a - // case, and so is each `"name"_test` declared in one, under its name; never a literal in a - // comment or one not followed by `_test`. The framework is claimed by its package and its - // cases, and a record of `mcpp test` lists and reports them (the test protocol). - "mtest cases by its manifest"_test = [&] { - const auto r = scan::scan(R"(import mcpplibs.mtest; -using namespace mcpplibs::mtest; -// "commented"_test -UTest empty{"an empty line", [] { expect(true); }}; -Suite parsing{"parsing", tag{"p"}, [] { - "one field"_test[tag{"x"}] = [] {}; - const char* s = "not a case"; -}}; -mcpplibs::mtest::E2E other{"other", [] { "after"_test = [] {}; }}; -int main() { "top"_test = [] {}; return report(); } -)", {}); - std::vector ids; - for (const auto& c : r.cases) ids.push_back(c.id); - check(ids == std::vector{"an empty line", "parsing", "parsing/one field", "other", "other/after", "top"}); - check(r.cases.size() == 6 && r.cases[0].line == 3 && r.cases[2].line == 5); - check(testing::framework_of({"app", "mcpplibs.mtest"}, r, Json::object()) == &fw("mtest")); - check(testing::framework_of({"app"}, r, Json::object()) == nullptr); - const auto cases = testing::static_cases(fw("mtest"), r, Json::object()); - check(cases.size() == 6 && cases[0].name == "an empty line" && cases[0].line == 3); - + // A program's cases as it lists them and as its records report them (the test protocol). + "cases as a program lists and reports them"_test = [&] { const auto record = Json::parse(R"({"test":"parse","protocol":1, "listed":[{"id":"parsing/an empty line","file":"tests/parse.cpp","line":5}], - "cases":[{"id":"a","status":"pass","duration_ms":1.5}, - {"id":"b","status":"fail","duration_ms":2,"failures":[{"file":"tests/parse.cpp","line":9,"message":"1 == 2","expected":"2","actual":"1"}]}, + "cases":[{"id":"a","status":"pass","duration_ms":1.5,"output":"said by a\n"}, + {"id":"b","status":"fail","duration_ms":2,"checks":{"held":3,"failed":1}, + "failures":[{"file":"tests/parse.cpp","line":9,"message":"1 == 2","expected":"2","actual":"1", + "source":"expect(eq(x, 2));","fatal":true}]}, {"id":"c","status":"skip","duration_ms":0,"reason":"later"}, {"id":"d","status":"xfail","duration_ms":0}]})"); const auto listed = testing::protocol_listing(record); @@ -188,27 +50,17 @@ int main() { "top"_test = [] {}; return report(); } const auto outcomes = testing::protocol_outcomes(record); check(outcomes.size() == 4); check(outcomes.size() == 4 && outcomes[0].outcome.status == "passed" && outcomes[0].outcome.durationMs == 1); + check(outcomes.size() == 4 && outcomes[0].detail["output"] == "said by a\n"); check(outcomes.size() == 4 && outcomes[1].outcome.status == "failed" && outcomes[1].location["line"] == 8 && outcomes[1].outcome.message.contains("expected 2, actual 1")); + check(outcomes.size() == 4 && outcomes[1].detail["checks"]["failed"] == 1 + && outcomes[1].detail["failures"][0]["source"] == "expect(eq(x, 2));" && outcomes[1].detail["failures"][0]["fatal"] == true); check(outcomes.size() == 4 && outcomes[2].outcome.status == "skipped" && outcomes[2].outcome.message == "later"); check(outcomes.size() == 4 && outcomes[3].outcome.status == "passed"); }; - // A suite type the project's tests declare in a module beside them is a suite type in every - // file that uses it: the server reads the types across tests/ before it scans a file. - "suite types across sources"_test = [&] { - const std::vector sources{ - "export module cts;\nimport mcpplibs.mtest;\nexport class Fixture : public mcpplibs::mtest::CTS {\n" - "public:\n using CTS::CTS;\n};\n"}; - const auto types = mcpplibs::mtest::manifest::types_of(sources); - const std::string_view cases = "import cts;\nFixture one{\"one\"};\nFixture two{\"two@warm\"};\n"; - const auto r = scan::scan(cases, {}, types); - check(r.cases.size() == 2 && r.cases[0].id == "one" && r.cases[1].id == "two@warm" && r.cases[1].line == 2); - check(scan::scan(cases, {}).cases.empty()); - }; - - // source/entries and test/cases over a project on disk (no mcpp: the plan's packages are not - // known, so no framework is claimed, and the static entries still answer). + // source/entries and test/cases over a project on disk (no mcpp: nothing is listed, and the + // entries still answer). "source entries and cases on disk"_test = [&] { const auto base = testsupport::temp() / std::format("mcppide-testing-{}", std::random_device{}()); std::filesystem::create_directories(base / "src"); @@ -245,7 +97,7 @@ int main() { "top"_test = [] {}; return report(); } auto test = call("mcpp-ide/source/entries", Json{{"file", (base / "tests" / "unit" / "math.cpp").generic_string()}}); check(test["kind"] == "test" && test["test"] == "unit/math" && !test.contains("framework")); auto cases = call("mcpp-ide/test/cases", Json{{"root", base.generic_string()}, {"test", "unit/math"}}); - check(cases["framework"].is_null() && cases["cases"].empty()); + check(cases["framework"].is_null() && cases["protocol"].is_null() && cases["cases"].empty()); std::error_code ignored; std::filesystem::remove_all(base, ignored); }; diff --git a/tests/unit/teststore.cpp b/tests/unit/teststore.cpp index e06974c..b54019e 100644 --- a/tests/unit/teststore.cpp +++ b/tests/unit/teststore.cpp @@ -7,7 +7,6 @@ import std; import mcppide.testsupport; import mcpplibs.mtest; import nlohmann.json; -import mcppide.scan; import mcppide.testing; import mcppide.teststore; @@ -123,14 +122,17 @@ UTest teststore{"teststore", [] { check(plain["message"] == "no network" && plain["location"].is_null() && !plain.contains("metrics") && !plain.contains("expectedFailure")); }; - // A case's kind and the bounds on its metrics, as its source states them (mtest's manifest). - "a case as its source states it"_test = [&] { - const auto found = scan::scan("import mcpplibs.mtest;\nusing namespace mcpplibs::mtest;\n" - "Bench alpha{\"alpha\", at_most{\"ready\", 15}, [] {}};\nUTest beta{\"beta\", [] {}};\n", {}); - const auto cases = testing::static_cases(*testing::framework("mtest"), found, Json::object()); - check(cases.size() == 2 && cases[0].kind == "bench" && cases[0].bounds.size() == 1 && cases[0].bounds[0]["max"] == 15); - check(cases.size() == 2 && cases[1].kind == "unit" && cases[1].bounds.empty()); - const auto row = testing::case_json(cases[0]); + // A case as a program lists it (mcpp test spec B3): its kind and the bounds on its metrics. + "a case as its program lists it"_test = [&] { + const auto listed = testing::protocol_listing(Json::parse(R"({"test":"b","protocol":1, + "listed":[{"id":"alpha","file":"t.cpp","line":3,"kind":"bench","bounds":[{"metric":"ready","max":15}]}, + {"id":"beta","file":"t.cpp","line":4,"kind":"unit"}]})")); + check(listed.size() == 2 && listed[0].kind == "bench" && listed[0].bounds.size() == 1 && listed[0].bounds[0]["max"] == 15); + check(listed.size() == 2 && listed[1].kind == "unit" && listed[1].bounds.empty() && listed[1].line == 3); + testing::Case c{listed[0].id, listed[0].line}; + c.kind = listed[0].kind; + c.bounds = listed[0].bounds; + const auto row = testing::case_json(c); check(row["kind"] == "bench" && row["bounds"][0]["metric"] == "ready" && !row.contains("tags") && !row.contains("file")); }; }};