Repository navigation
feat(evalboard): Task Failure Analysis panel from the skill-doctor report - #264
CarlesUIPath wants to merge 13 commits into
Conversation
…port The run page reads the skill-doctor report under <run>/skill-doctor/ (contract v1): status.json (refetched after 60 s), the triage file it names once done, and <fixes_dir>/fixes.json plus its patches (cached for good). Names are checked against their patterns before any fetch; a schema_version other than 1, or unparsable JSON, is treated as absent, and readRunDoctor never throws into the page. When the report is done, analysis.md is cut at its first `## Findings` line (outside code fences): the AI summary keeps the top, and a native React panel replaces Findings and below (counts by layer and class, new vs recurring, diagnosis cost, Actions, a collapsible task table, Fixes with patch download through /api/file). Otherwise the page is unchanged, plus a one-line status for queued, running, failed, skipped and retracted reports. PR links render only for UiPath/skills pull URLs and build links only for dev.azure.com URLs; anything else is plain text. Internal edition and the skills source only; EVALBOARD_TASK_ANALYSIS=off turns it off. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- readRunDoctor is the one place that decides which doctor files a status implies and parses each once; ensureRunDoctorFile only downloads. - Parsed types carry only what the page renders; the id validators move to a Node-free lib/ids.ts shared by blob.ts and doctor.ts. - One predicate (patchIds, capped at 10, ids checked) picks the patches to fetch and to link; /api/file fetches a patch on demand and serves only <fixes_dir>/<id>.patch from skill-doctor/. - The header counts the per-task triage state (new / changed / known) and shows vs_history only when there is a base run. - A done report gets the panel even without a `## Findings` split or an analysis.md. - parseFixes requires schema_version 1; a file that does not parse is logged by name; the kill switch accepts "off" in any case. - Tests for the task-link check, the text fallbacks, the fixes.json version gate, the patch cap and the cold-cache patch fetch. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… /api/file - /api/file serves a skill-doctor patch only when this run's status names its fixes dir and fixes.json lists it (patchIds), loading both through readRunDoctor on a cold cache: one fetch path, and a made-up name costs no blob call. - The skill-doctor branch returns nothing when the panel is off (OSS edition, another source, or EVALBOARD_TASK_ANALYSIS=off). - ensureRunDoctorFile reuses the ids.ts validator; doctor.ts constants nothing imports are module-private; stale comments fixed. - Tests: the download guard, row badges, unlisted patches, the kill switch on /api/file, and a storage error on the patch fetch. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
Claude finished @CarlesUIPath's task in 1m 21s —— View job Code Review in Progress
|
…ke the doctor's page The block now sits under the AI summary and opens like it: a bordered section with a header button, collapsed by default. The header carries "<N> errors · <M> proposed PRs ready", or the doctor's state when the report is not done. Inside, in the order of the skill-doctor page: the stats row and the failures-by-layer bar, Fix first (the ranked actions, with an "N PR ready" badge when an action's tasks have a pr-ready fix), then Skill failures: layer chips, search, Expand all / Collapse all, and one collapsed accordion per skill with a card per task (pills, diagnosis, the proposed PR row with outcome, patch, gates and cost, and the Fix / Failing check / Evidence / Passed before / PRs / Related / Owner list). Tasks stopped at a pre-run check collapse into one block per cause. The separate task and fix tables are gone. States, gating, link allowlists, patch rules and the kill switch are unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ailures Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…dates Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- A pr-ready fix candidate reads "PR with fix" with its title, patch, gates and cost; the title expands to the prepared PR body (<fixes_dir>/<id>.md, loaded by name for ids in patchIds, rendered with ReactMarkdown without raw HTML; only allowlisted links, images as alt text), then the patch download. A doctor PR shows as "PR with fix" linking to GitHub instead. Other outcomes stay "Fix candidate". - "Proposed PR": an open, allowlisted PR from the task's existing PRs that its fix text names (#N or /pull/N). - Header, skill accordions and Fix first badges count distinct PRs with fix (both kinds); skill and card sorts use the same model. - The Owner row is gone and owners are no longer parsed. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…Rs; no costs - "PR with fix: #N ↗" links the doctor's opened PR (<fixes_dir>/opened.json, fetched by name and refreshed like status.json, allowlisted URLs only) or the triage's doctor_pr, with the gate verdicts. - A pr-ready fix with no opened PR reads "Fix prepared, PR not opened yet" with the patch download; it is not a PR with fix and sorts right after linked PRs. - "PRs with fix" (header, skill accordions, Fix first badges, both sorts) counts only linked PRs: opened or doctor PRs, and third-party open PRs the fix text names. - Cards drop "Open PRs on these files" and "Related skill PRs"; related_skill_prs is no longer parsed. Fix first drops open-pr actions. - The inline PR body and its .md loading are gone; /api/file still serves .patch only. - No costs anywhere; cost fields are no longer parsed. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- Layer and solvable labels are Maps and opened.json lookups use Object.hasOwn, so report keys such as "constructor" or "__proto__" cannot reach the object prototype; parseOpened builds through a Map. - A task covered by a pr-ready candidate whose PR was opened gets that PR (opened.json is keyed by the candidate's primary task) instead of "Fix prepared". - The doctor's own PR is filtered out of the Proposed PR list and counted once. - PR URLs (existing_prs, doctor_pr, opened.json) are allowlisted in the parsers; the view links them without re-checking. - Cards drop the Related row (no longer parsed) and the "none yet" placeholder. - One layerCounts() helper and one FixRow component; Body split into Stats / LayerBar / FixFirst / SkillFailures; fixTier uses early returns; SkillGroup.ready is now fixPrs; dead "—" sentinels and the ExternalLink fallback are gone. - Tests: prototype keys, the covered-task PR, own PR vs proposed, the opened.json refresh, parser allowlisting. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…d pr_checks The "fix text names #N" rule is gone. A third-party PR now shows only when the triage's fixed_by_pr says "fixes" and the PR is open in the row's existing PRs, joined with fixes.json pr_checks for that row and number: - verified: "Proposed PR" with the replay evidence; counts as a PR with fix and sorts after the doctor's own PR; - untested or no check: "Open PR may fix this (not verified)" with the diagnosis's basis, the evidence, and "needs a Windows run" when the check says so; not counted, sorted after prepared fixes; - not_verified, may_help, unrelated: nothing. Both fields are optional and untrusted: malformed entries parse to nothing, pr_checks URLs are allowlisted and must match the number, and task ids are own keys. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…h the evidence head_sha is optional (string, null or absent); only a 40-hex value is kept. Tests cover it and the untested evidence texts of the final contract. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- Fix candidates with outcome not-run are hidden and no longer lift a card's sort tier. - A pr_checks entry that does not validate is kept as null, so its task shows no PR row instead of "may fix (not verified)". - rows_executed must be a non-negative integer. - One own() helper for report-keyed lookups; one existingPr field (verified or not) instead of proposed + maybe; Patch absorbs PatchCell; one ExtA anchor; DoctorPrCheck.url is validated but not kept; doc comments updated. - Tests: not-run candidates, a malformed not_verified check, task ids named constructor / __proto__ / toString with opened.json and pr_checks entries, rows_executed. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
Claude finished @CarlesUIPath's task in 1m 22s —— View job Code Review in Progress
|
uipreliga
left a comment
There was a problem hiding this comment.
Review: coder_eval — pr:264
Scope: pr:264 · branch feat/evalboard-task-failure-analysis (PR #264 → main) · 0636adc · 2026-11-20T01:30Z · workflow variant
Change class: complex — new evalboard feature that parses untrusted external JSON, adds a new gated file-serving branch to /api/file (resolveSafePath), and a large new UI panel
The codebase is healthy at 9.3/10: the architecture and API surface are clean, there are no critical or high findings, and no finding can change a task's score or final_status for the same agent output; the real risks are in the evalboard skill-doctor feature, where a path-traversal gap lets /api/file read one level above the runs directory, storage failures show as 403/404 instead of 5xx, and the doctor contract has no schema or staleness check, so fix the security gap first, then the error handling and the test gaps.
Summary
| Axis | Score | 🔴 | 🟠 | 🟡 | 🔵 | Top Issue |
|---|---|---|---|---|---|---|
| 1. Code Quality & Style | 8.8 / 10 | 0 | 0 | 1 | 7 | Closed doctor-contract value sets (fix outcome, PR state, etc.) typed as free string and matched by magic literals |
| 2. Type Safety | 9.2 / 10 | 0 | 0 | 1 | 3 | parseTriage coerces a missing or malformed summary.rows_failed to 0, so the doctor header shows "0 errors" and no drift warning is logged |
| 3. Test Health | 8.6 / 10 | 0 | 0 | 2 | 4 | The doctor-state gate is tested for the triage only. Fix and patch fetch and /api/file patch serving for a non-done status (retracted/failed/skipped) that still names fixes_dir are not tested |
| 4. Security | 9.3 / 10 | 0 | 0 | 1 | 2 | isValidId (evalboard/lib/ids.ts:7/13) accepts "." and "..", so /api/file?run=..&path=.env serves files one level above the runs dir (this is older than the PR, and the PR copies it into the new ids.ts and adds readRunDoctor as a new sink) |
| 5. Architecture & Design | 9.9 / 10 | 0 | 0 | 0 | 1 | A new multi-file doctor subsystem is added to the 3111-line lib/runs.ts module |
| 6. Error Handling & Resilience | 9.4 / 10 | 0 | 0 | 1 | 1 | /api/file doctor-patch branch turns auth, network and IMDS blob failures into 403 'forbidden' or 404 'not found'. This breaks the blob-layer rule that only a 404 means absent. |
| 7. API Surface & Maintainability | 9.8 / 10 | 0 | 0 | 0 | 2 | Internal-only doctor gate is recomputed at each caller (page and /api/file) instead of in the reader; kill switch accepts only 'off' |
| 8. Evaluation Harness Quality | 9.3 / 10 | 0 | 0 | 1 | 2 | The doctor triage is not checked against the run.json it describes, but it still replaces the AI summary's Findings |
Overall Score: 9.3 / 10 · Weakest Axis: Test Health at 8.6 / 10
Totals: 🔴 0 · 🟠 0 · 🟡 7 · 🔵 22 across 8 axes.
Blockers
None.
Non-blocking, but please consider before merge
-
[Axis 1] Closed doctor-contract value sets (fix outcome, PR state, etc.) typed as free string and matched by magic literals (
evalboard/app/runs/[id]/task-failure-analysis.tsx:56) —DoctorFix.outcome: string(lib/doctor.ts:123),DoctorTask.layer: string(doctor.ts:62) andsolvable: string | null(doctor.ts:63) stay untyped. The view then special-cases bare literals that tsc cannot check:const isFixable = (t: DoctorTask) => /^yes/.test(t.solvable ?? "");(tfa:56),const isReady = (c: DoctorFix) => c.outcome === "pr-ready";(tfa:57),c.outcome !== "not-run"(tfa:195),[count("env"), "environment"]/[count("agent"), "agent mistakes"](tfa:691/693),layerOf("env")(tfa:589), and in the parsera.kind !== "open-pr"(doctor.ts:286). Thesolvablefield is also read in two different ways: a regex prefix inisFixable(tfa:56) andSOLVABLE.get(t.solvable.split(" ")[0])(tfa:415). If you misspell one of these literals, or the doctor renames a value, the code still compiles and the result is quietly wrong (for example, a wrong count in the 'fixable in the repo' stat). doctor.ts:26-38 already has the generic fix (DOCTOR_STATESas const +isDoctorState). Use the same pattern for layer, outcome, solvable and action kind: parse each into a closed literal union in doctor.ts, with oneother/unknown bucket. Then makeLAYERS/SOLVABLERecord<Union, ...>so the compiler checks every comparison. -
[Axis 2] parseTriage coerces a missing or malformed summary.rows_failed to 0, so the doctor header shows "0 errors" and no drift warning is logged (
evalboard/lib/doctor.ts:279) —int()(lines 157-159:return typeof v === "number" && Number.isInteger(v) && v >= 0 ? v : dflt;withdflt = 0) turns an absent, string-valued, negative or fractional count into a real-looking 0.parseTriageuses it at line 279 (rowsFailed: int(s.rows_failed),) and line 280 (rowsDiagnosed: int(s.rows_diagnosed),). Six lines earlier (273-278), the same function narrowsrun.rows_executedtonumber | null, so the two count fields are handled differently. Failure case: a triage file with"summary": {}or"rows_failed": "5"passesisObj(j.summary)and parses.digest()(task-failure-analysis.tsx:145,const errors =${triage.rowsFailed} error...``) then shows "0 errors" in the always-visible header of a run that has failures. Because the parse succeeds, thedid not parse as contract v1warning in readRunDoctor (runs.ts) never fires, so this kind of contract drift stays hidden. The same coercion turns `gates.mutants` `{n: 3.5}` into "mutants 0/0" (line 300) and turns a bad `actions[].count` into 0 (line 287). Fix: type the counts as `number | null` (or reject the triage when `summary.rows_failed` is not a non-negative integer, as the module header promises for files that do not parse), and render null as "—". Static check: a vitest case that gives parseTriage `summary: {}` and expects null or a null count. -
[Axis 3] The doctor-state gate is tested for the triage only. Fix and patch fetch and /api/file patch serving for a non-done status (retracted/failed/skipped) that still names fixes_dir are not tested (
evalboard/lib/__tests__/doctor-blob.test.ts:131) — The only non-done readRunDoctor test also clears the fixes pointer:seedBlobs(statusWith({ state: "running", fixes_dir: null }));(line 131). This means the branch that matters is never exercised. In lib/runs.ts the fixes path has no state check:const fixesRel = status.fixesDir ?${status.fixesDir}/fixes.json: null;(line 2740) andif (status.fixesDir) {(line 2753, which fetches every patchIds() patch plus opened.json). resolveSafePath then serves a patch when onlydoctor.status.fixesDir !== patch.fixesDir || !patchIds(doctor.fixes).has(patch.id)(lines 3061-3062) is false. So aretractedstatus.json that keepsfixes_dirstill downloads fixes.json and the patches, and /api/file still serves the retracted report's patch through a direct link, even though planAnalysis/TaskFailureAnalysis show no body (and no patch links) for that state. No test decides which behavior is intended. Add atest.each(["queued","running","failed","skipped","retracted"])case that seeds the fixture status withfixes_dir: "fixes-1000001"kept. Assert the exactstate.downloadslist and theresolveSafePath(RUN, "skill-doctor/fixes-1000001/example-grader-fix.patch")result. If the decision is that a retracted (or any non-done) report serves nothing, gate fixesRel/patches on the state in readRunDoctor and let the test pin that. -
[Axis 3] Cross-repo doctor contract pinned only by hand-written fixtures, with no schema parity check (
evalboard/lib/__tests__/doctor.test.ts:20) — lib/doctor.ts says the types "are written by hand against the doctor's status_v1 / triage_v1 / fixes_v1 schemas" (lines 10-11). Every parser test reads fixtures that the same change wrote by hand:const FIX = path.join(__dirname, "fixtures", "skill-doctor");andconst TRIAGE = read("triage-1000001.json");(lines 20-23). The tests therefore prove parser ↔ fixture, not parser ↔ producer. The producer lives in another repo. Optional fields degrade field by field to null (fixedByPr: parseFixedByPr(v.fixed_by_pr),rowsRunfromrun.rows_executed,headShafromhead_sha). Theconsole.warnin readRunDoctor'sloadfires only when a whole file returns null. So if the doctor renamesfixed_by_prorrows_executed, the board silently drops the Proposed PR rows or the "rows run" stat, and no test fails. This is the event-reassembled-record shape: vendor the doctor's v1 JSON Schemas (or a golden file the producer generates) under lib/tests/fixtures/skill-doctor/. Add a test that validates the fixtures against them, and a parity test asserting every schema property the parsers read (fixed_by_pr,pr_checks.*.head_sha,run.rows_executed,existing_prs,doctor_pr,env_group, ...) exists in the schema. That way a producer-side rename fails here. -
[Axis 4] isValidId (evalboard/lib/ids.ts:7/13) accepts "." and "..", so /api/file?run=..&path=.env serves files one level above the runs dir (this is older than the PR, and the PR copies it into the new ids.ts and adds readRunDoctor as a new sink) (
evalboard/lib/ids.ts:7) — The new file ids.ts says 'Run / task IDs get reflected into filesystem paths and blob prefixes, so reject anything outside a narrow whitelist', butconst ID_RE = /^[\w.-]+$/;(line 7) andisValidId(line 13) accept "." and "..". In lib/runs.ts:3031if (!isValidId(runId)) return null;is the only runId check. Thenconst candidate = path.resolve(baseReal, relPath);(runs.ts:3074) resolves under<RUNS_DIR>/... I confirmed this with a vitest probe in local mode (EVALBOARD_LOCAL_RUNS_DIR=/runs):resolveSafePath("..", ".env")returned<x>/.env. In blob mode the parent is the app cwd, because RUNS_DIR defaults tocwd/runs-remote. The exploit there depends on Azure returning 404 for the blob../run.json, which ensureRunSummary swallows. The defect is older than this PR. This PR moves the validator into a new module that claims it is safe, and adds another runId-to-path sink:path.join(dir, runId, DOCTOR_DIR)in readRunDoctor (runs.ts:2713), plus ensureRunDoctorFile downloads to<RUNS_DIR>/../skill-doctor/.... Fix: make isValidId reject ids that are only dots, e.g.ID_RE.test(id) && !/^\.+$/.test(id), and make isValidTaskId reject dot-only segments too. Then delete the special case in clearRunCacheDir that exists only because of this gap. Add a vitest case that assertsresolveSafePath("..", ".env") === null. CVSS:3.1/AV:N/AC:H/PR:L/UI:N/S:U/C:H/I:N/A:N -
[Axis 6] /api/file doctor-patch branch turns auth, network and IMDS blob failures into 403 'forbidden' or 404 'not found'. This breaks the blob-layer rule that only a 404 means absent. (
evalboard/lib/runs.ts:3058) — resolveSafePath's new skill-doctor branch callsconst doctor = await readRunDoctor(runId, source);(runs.ts:3058). readRunDoctor never throws, because itsgethelper catches every error, not only a 404:} catch (err) { console.error(skill-doctor: fetching ${rel} failed for run ${runId}:, err); }(runs.ts:2717-2718). On a cold cache, a 500, auth or IMDS failure on status.json or fixes.json makes readRunDoctor return null or an empty patchIds set. resolveSafePath then returns null, and app/api/file/route.ts:22-23 answersforbidden(403). A transient failure on the patch blob itself givesnot found(404). The test at doctor-blob.test.ts:305-318 fixes this behavior in place ('/api/file then answers 404 for the missing file'). Every other branch of resolveSafePath goes through ensureTaskDir or ensureRunSummary, which rethrow non-404 errors (if (!isNotFound(err)) throw err;, blob.ts:217 and others). The blob layer states the rule at blob.ts:211-213: 'Auth, network, and IMDS failures must propagate so they don't masquerade as "not found" in the UI.' The 'never throws' contract is right for the page panel, which is optional. It is wrong for the file route, where a user clicks a link and must be able to tell 'not listed' from 'storage down'. The same root cause shows in the panel digest: when the triage fetch fails transiently, task-failure-analysis.tsx:143 shows"report done, but its triage file is missing or did not parse", which blames a contract problem for an infra error. Fix: give readRunDoctor a mode, or a separate loader for resolveSafePath, that lets non-404 fetch errors propagate so /api/file returns 500. Or havegetrecord a fetch-failed flag on RunDoctor so the digest can say 'could not be fetched'. Change the doctor-blob test at 305-318 to expect a thrown error or a 5xx, not a 404. Static check: a vitest case that runs /api/file withfailWith: {statusCode: 500}and expects a 5xx response. -
[Axis 8] The doctor triage is not checked against the run.json it describes, but it still replaces the AI summary's Findings (
evalboard/lib/doctor.ts:271) —parseTriagereads onlyrows_executedfrom the triage'srunblock:const run = isObj(j.run) ? j.run : {};(doctor.ts:271). The producer also writesrun.run_idandrun.run_json_etag(see fixtures/skill-doctor/triage-1000001.json), and the parser ignores both.readRunDoctortreats the triage as immutable (runs.ts:2699// fetched again. Everything it points at is immutable.). page.tsx:101const plan = planAnalysis(analysis, doctor);then removes analysis.md's## Findingssection whenever a report isdone. Failure case: a run.json is re-graded in place (evaluate <run_dir>, then re-uploaded) or re-uploaded after the doctor ran, or a triage lands under the wrong run prefix. The board then shows failed rows, layer counts and 'fixable' stats from the old outcomes, and it hides the AI summary's own Findings. The two views can disagree with the grid on the same page. This does not change any score, but the run report becomes wrong for cross-run triage. Fix: inparseTriage, keeprun.run_idand drop the triage (or mark it stale) when it is not equal torunId. In page.tsx, intersecttriage.taskswith the failing rows intasksfrom run.json, and show rows that now pass as stale instead of silently counting them. If the board can get the run.json blob etag, compare it torun_json_etag. Add a vitest case for a mismatched run_id and for a triage row that passes in run.json.
Nits
-
[Axis 1] taskFixes is re-derived many times per task on each render, including inside a sort comparator, and groupBySkill is not memoized (
evalboard/app/runs/[id]/task-failure-analysis.tsx:342) —const tier = (t: DoctorTask) => taskFixes(doctor, t).tier;(tfa:342) runs insidecards.sort(...), sotaskFixes(which filters every candidate and resolves PRs) runs O(n log n) times per skill.countFixPrs(doctor, items)(tfa:354) andcountPrepared(doctor, items)(tfa:355) re-derive it for every task,TaskCardre-derives it again (tfa:399), anddigest(tfa:144) and everyFixFirstaction (tfa:752) re-derive it once more.const groups = groupBySkill(doctor, active, query);(tfa:790) is not memoized, so each keystroke in the search box runs all of this again, plus aJSON.stringify(t)for each task. Compute aMap<taskId, TaskFixes>once perdoctor(oneuseMemoat the TaskFailureAnalysis root, or one pass in lib/doctor.ts) and pass it down. Then countFixPrs/countPrepared/fixTier read from the map. -
[Axis 1] Free-text search matches JSON field names and literal
null, so common queries match every task (evalboard/app/runs/[id]/task-failure-analysis.tsx:325) —(!q || JSON.stringify(t).toLowerCase().includes(q))(tfa:325) searches the serialized object, which includes keys such as"taskId","layer","why","fix","evidence","fixedByPr"and the literalnull. A search for "fix", "why", "layer", "null" or "skill" matches every task, and the search also opens every accordion (tfa:855). Search only the value fields the placeholder promises ("Search task, cause, PR…"): taskId, skill, why, fix, criterion, evidence, and the PR urls/titles. -
[Axis 1] Doctor-contract domain rules (PR-with-fix counting, sort tiers, grouping, digest) live in the 981-line client view, not lib/doctor.ts (
evalboard/app/runs/[id]/task-failure-analysis.tsx:220) — The file mixes UI with exported pure logic:export function existingPrFix(tfa:220),export function taskFixes(tfa:276),countPrepared/countFixPrs(tfa:300/305),export function groupBySkill(tfa:316),export function digest(tfa:129) andfixesForTask(tfa:193). lib/doctor.ts says it is 'Pure data + string helpers' and is already where the panel's other rules live (patchIds, planAnalysis). Move this derivation logic into lib/doctor.ts, or into a lib/doctor-view.ts sibling. The component file then keeps only rendering, and the PR-counting contract is tested next to the parsers that feed it. -
[Axis 1] Collapsible header markup copied from AnalysisPanel, and a second hand-built /api/file URL (
evalboard/app/runs/[id]/task-failure-analysis.tsx:941) — The block says it is 'shaped like the AI summary panel' (tfa:909). It copies analysis-panel.tsx:32-45 almost verbatim:className="border border-studio-blue/30 bg-studio-blue/5 rounded-lg", thew-full px-4 py-3 flex ... hover:bg-studio-blue/10 rounded-lg transition-colorsbutton, and{open ? "Hide" : "Read analysis →"}(tfa:941-958). If someone restyles one panel, the two drift apart. Extract a small sharedCollapsiblePanel({label, extra, children})shell that both use. In the same way,patchHref(tfa:153-158) rebuilds`/api/file?run=${encodeURIComponent(runId)}&path=${encodeURIComponent(...)}`+withSource, which [...task]/_sections.tsx:1764-1768 already builds. Add onefileHref(runId, relPath, sourceId)helper. -
[Axis 1] Id validators moved to ids.ts but kept behind a blob.ts re-export shim and a doctor.ts alias; storage layer now depends on the doctor module (
evalboard/lib/blob.ts:17) —export { isValidId, isValidTaskId };(blob.ts:17) is a re-export shim. ids.ts:1-3 says it exists 'for its existing importers', which are only lib/runs.ts:14-15 and lib/tests/blob.test.ts:2. CLAUDE.md says 'Greenfield project — no backward-compatibility burden', so point those two importers at ./ids and remove the shim.export const isLinkableTaskId = isValidTaskId;(doctor.ts:396) adds a second name for the same predicate, for one call site (tfa:386), so tests now cover the same function under two names. Also,if (!isValidTaskId(relPath) || relPath.split("/").length > 2)(blob.ts:338) uses a task-id validator as a blob-path validator. Give it its own named check, or reuse the doctor's TRIAGE_FILE_RE/FIXES_DIR_RE/patchIdFromPath patterns, which already describe the allowed paths. -
[Axis 1] Positive-integer and verdict-whitelist checks repeated inline in the parsers, even though an
int()helper exists (evalboard/lib/doctor.ts:273) —int()(doctor.ts:157-159) exists, but the same check is written out again:typeof run.rows_executed === "number" && Number.isInteger(run.rows_executed) && run.rows_executed >= 0 ? run.rows_executed : null(doctor.ts:273-278), andif (typeof n !== "number" || !Number.isInteger(n) || n < 1) return null;twice (doctor.ts:203 and 212). The verdict whitelists are inline||chains (doctor.ts:204, 214), not the const-array guard used for DOCTOR_STATES. AddnonNegInt(v): number | null/prNumber(v): number | nullhelpers and a genericoneOf(values)(v)guard. Then each parser line states only its field. -
[Axis 1] Narrative design comments that restate the code or record rationale/history, against the 'comments are a last resort' principle (
evalboard/lib/runs.ts:3041) — The 7-line comment at runs.ts:3041-3047 ('skill-doctor/ belongs to the doctor contract ... readRunDoctor is the one fetch path ... a made-up name costs no blob call. Off with the panel.') is design rationale for the next 15 lines of code. The 7-line comment at blob.ts:322-328 repeats the same story. page.tsx:49-50 ('internal-only, skills nightly only, with an env kill switch (EVALBOARD_TASK_ANALYSIS=off). readRunDoctor never throws.') restates the taskAnalysisEnabled docstring (doctor.ts:428-431) and the readRunDoctor header (runs.ts:2702-2707). page.tsx:99-100 restates the planAnalysis docstring (doctor.ts:451-457). Keep one contract docstring per function. Move the fetch-path rationale to one place (the readRunDoctor docstring, or .claude/notes/), and remove the copies at the call sites. -
[Axis 2] The PR-URL allowlist is guaranteed only by a comment: isSkillsPrUrl narrows to plain
string(evalboard/lib/doctor.ts:387) —export function isSkillsPrUrl(u: unknown): u is string {narrows tostring, so the result of the check is lost in the type.DoctorPr.url,DoctorTask.doctorPr,RunDoctor.openedvalues andTaskFixes.ownPrare allstring. The sinkExtA({ href }: { href: string ... })(task-failure-analysis.tsx:99-110) depends on the docstring "Callers pass only allowlisted URLs", and doctor.ts:50-51 says the same thing in a comment ("Every PR URL the parsers hand out has passed isSkillsPrUrl: views can link them without checking again"). tsc does not stop a new code path that puts an unvalidated report string (for examplet.fix) intoExtA. Fix: add a branded type (type SkillsPrUrl = string & { readonly __brand: "SkillsPrUrl" }), declare the guard asu is SkillsPrUrl, type the parsed fields with it, and makeExtA.hrefacceptSkillsPrUrl | AdoUrl. tsc then enforces the invariant. Test code that setsdoctorPr = "https://..."directly would need the guard or an explicit cast, which makes the bypass visible. -
[Axis 2] Report-keyed Record types say every key is present; prototype safety depends on always calling own() (
evalboard/lib/doctor.ts:133) —prChecks: Record<string, DoctorPrCheck | null>;(line 133) andopened?: Record<string, string>;(line 144) are indexed by task ids taken from untrusted JSON. tsconfig.json does not enablenoUncheckedIndexedAccess, soprChecks[k]has typeDoctorPrCheck | null. At runtime a missing key givesundefined, and"constructor"givesObject.prototype.constructor. Production code avoids this only by convention, throughown()(line 235). The PR's own tests index the Record directly (task-failure-analysis.test.tsx:297...d.fixes!.prChecks["example-t2"]!,), which shows how easily the helper gets skipped. Also,openedis optional (?), but readRunDoctor always sets it (let opened: Record<string, string> = {};), so the optional marker is looser than the data. Fix: type these fields asPartial<Record<string, T>>(or{ readonly [k: string]: T | undefined }), or useReadonlyMap(React server components serialize Map). Makeopenedrequired. Alternatively, enablenoUncheckedIndexedAccessfor evalboard/ as a tsc gate. -
[Axis 2] Body re-asserts
doctor.triage!instead of receiving the triage the caller already narrowed (evalboard/app/runs/[id]/task-failure-analysis.tsx:875) —const triage = doctor.triage!;(line 875) relies on the caller's separate checkconst hasBody = doctor.status.state === "done" && doctor.triage !== null;(line 923). The type does not carry the invariant "done ⇒ triage", which only appears as a comment at doctor.ts:139 ("Set only when the status isdoneand its triage file parsed."). If a future caller renders<Body>without that guard, tsc accepts it and the component crashes ontriage.tasks. Fix: passtriage: DoctorTriageas a prop from the narrowed branch ({hasBody && open && doctor.triage && <Body triage={doctor.triage} .../>}), or model RunDoctor as a discriminated union onstatus.state.by.get(skill)!at line 334 is safe because the loop iteratesby.keys(); it could use[...by.entries()]to drop the assertion. -
[Axis 3] Empty-state rendering of the new panel is not asserted: a done triage with zero failed rows, and rowsRun null (
evalboard/app/runs/[id]/__tests__/task-failure-analysis.test.tsx:98) — Every render test uses the 5-task fixture, and no test setstasks = []orrowsRun = null. With an empty triage,LayerBarreturns early (if (n === 0) return null;, task-failure-analysis.tsx:710). SkillFailures then showsNo failures match this filter.(line 848) when no filter is applied, which is a misleading message for a clean run. The.filter(([v]) => v !== null)that hides the "rows run" stat when the run did not report it (line 698) is also never exercised from the render side. Add a test that renders{...doctor(), triage: {...triage, tasks: [], actions: [], rowsFailed: 0}}, opens the block, and asserts the digest ("0 errors"), the stats, and the skill-section text (and fix the copy if "match this filter" is not intended). Also add a case withrowsRun: nullthat asserts "rows run" is absent. -
[Axis 3] The 'never throws' test does not reach readRunDoctor's outer catch, which no test covers and which is effectively unreachable (
evalboard/lib/__tests__/doctor-blob.test.ts:236) —test("never throws: a storage error is no report", ...)setsstate.failWith = { statusCode: 500 }. That error is swallowed by the innergethelper in runs.ts (catch (err) { console.error(skill-doctor: fetching ${rel} failed ...) }), andloadthen returns null for status.json. The outer} catch (err) { console.error(readRunDoctor failed for run ${runId}:, err); return null; }(runs.ts lines 2762-2765, reported uncovered) is never entered. The test also does not assert thaterrwas called. So the documented "never throws" contract is tested only through the inner swallow. Either make a parser or fs read throw (e.g.vi.spyOn(fs, "readFile")rejecting with a non-ENOENT error after status loads) and assert the outer path, or delete the outer try/catch if it is unreachable by construction. Assertexpect(err).toHaveBeenCalled()in the existing test. -
[Axis 3] Test redundancy and a misleading comment in doctor-blob opened.json tests (
evalboard/lib/__tests__/doctor-blob.test.ts:229) —// Cached copy from before is fresh, so drop it to read the bad one.(line 229) is followed by anfs.rm(..., { force: true }). This step does nothing: the earlier read in the same test ran afterdelete state.blobs[...opened.json], got a 404, and cached nothing. The comment tells a reader that a stale cached file exists when it does not. Separately,opened.json maps opened PRs by task; a bad host or a problem entry does not(lines 208-219) repeats the allowlist/problem-entry filtering already pinned by doctor.test.tsparseOpened: allowlisted PR URLs map by task; the rest is skipped(lines 287-306). Keep the blob-level test to the wiring (opened.json is fetched from<fixes_dir>/and passed to parseOpened). Drop the redundant rm and comment. -
[Axis 3] The page-level wiring of the doctor panel (taskAnalysisEnabled gate → readRunDoctor, planAnalysis → AnalysisPanel/TaskFailureAnalysis) has no test (
evalboard/app/runs/[id]/page.tsx:51) — The pieces are unit-tested (taskAnalysisEnabled, planAnalysis, the component), but the composition in page.tsx is not:const doctorOn = taskAnalysisEnabled(isInternal, source.id, process.env.EVALBOARD_TASK_ANALYSIS);,doctorOn ? readRunDoctor(id, source) : Promise.resolve(null), and the swap from{analysis && <AnalysisPanel markdown={analysis} />}to{plan.summary && <AnalysisPanel markdown={plan.summary} />}. Some regressions would pass every current test: passinganalysisin place ofplan.summary(the Findings section shows twice), or calling readRunDoctor without the gate (OSS or non-skills sources fetch doctor blobs). No test for this page exists today, so a full server-component test may be too much. A small extracted helper (for example,loadDoctorPlan(analysis, isInternal, source, env)) with a test for the off/OSS/non-skills cases would pin it. -
[Axis 4] The /api/file skill-doctor gate checks only the raw first path segment, so './skill-doctor/...' or 'activation/../skill-doctor/...' skips the edition check, the kill switch and the patch allowlist (
evalboard/lib/runs.ts:3040) — The gate isconst parts = relPath.split("/"); if (parts[0] === DOCTOR_DIR) { ... }(runs.ts:3039-3040). It runs on the raw relPath, but the file is chosen later bypath.resolve(baseReal, relPath)(3074), which normalizes the path. I confirmed this with a vitest probe in both the oss and the internal edition.skill-doctor/status.jsonreturns null as intended. But./skill-doctor/status.json,./skill-doctor/fixes-1/zz.patchandactivation/../skill-doctor/status.jsonall resolve to the file through theelsebranch. That branch skips taskAnalysisEnabled (the internal-only check and EVALBOARD_TASK_ANALYSIS=off) and patchIds. So the statement in the comment ('Only a fix patch is served from it') and in the README ('/api/fileserves only the patches fixes.json lists') is false for any file that is already cached or on disk: status, triage, fixes and opened files, patches beyond the allowlist, and every file in local mode. The impact is low: in blob mode only internal readers fill the cache, and the page already shows the same data. Fix: normalize first, e.g.const norm = path.posix.normalize(relPath), reject a result that starts with../, and dispatch onnorm.split("/"). Or, after resolving, reject any candidate inside<base>/skill-doctor/that the DOCTOR_DIR branch did not approve. Add a regression test for./skill-doctor/status.json. CVSS:3.1/AV:N/AC:H/PR:L/UI:N/S:U/C:L/I:N/A:N -
[Axis 4] ADO_RE accepts any dev.azure.com org or path, so model-written report text can render a clickable link to an attacker-owned ADO project (
evalboard/lib/doctor.ts:385) — The comment at doctor.ts:382-383 says 'a URL in it is shown as a link only when it has exactly the expected shape', and RichText describes the allowlist as 'a UiPath/skills PR or a dev.azure.com build'. Butconst ADO_RE = /^https:\/\/dev\.azure\.com\/[^\s"'<>]*$/;accepts any organization and any path. Free text from the triage file (why, fix, evidence), which comes from a model and the run's logs, is passed to RichText. Through prompt injection in an agent's output, that text can includehttps://dev.azure.com/<attacker-org>/.... The board then renders it as a trusted link next to the skills PR links.status.triggergoes through BuildLink with the same check. Fix: pin ADO_RE to the real organization and the build-results path, e.g./^https:\/\/dev\.azure\.com\/<org>\/[^/\s]+\/_build\/results\?buildId=\d+$/, as tightly as SKILLS_PR_RE pinsUiPath/skills/pull/\d+. CVSS:3.1/AV:N/AC:H/PR:N/UI:R/S:U/C:L/I:N/A:N -
[Axis 5] A new multi-file doctor subsystem is added to the 3111-line lib/runs.ts module (
evalboard/lib/runs.ts:2700) — runs.ts is already 3111 lines and mixes many concerns. This PR adds a new subsystem to it:const DOCTOR_STATUS_REFRESH_MS = 60_000;(line 2700),readRunDoctor(line 2708, about 70 lines), and a doctor branch inresolveSafePath(line 3040). That subsystem has a 5-file contract, its own refresh policy and its own logging. This follows the existing pattern (readRunAnalysis and readRunMeta live there too), so it is not a new regression. But lib/doctor.ts must stay free of Node builtins for the client, so the I/O half now lives in a different module from its contract. Put the server half in a server-onlylib/doctor-read.ts(readRunDoctor plus the refresh constant), imported by runs.ts and page.tsx. Then the whole doctor feature can be found, and removed, as two lib files. -
[Axis 6] readRunDoctor's catch-all fetch handler also swallows its own input-validation errors, then reads the filesystem with the rejected run id (
evalboard/lib/runs.ts:2713) — readRunDoctor buildsconst base = path.join(dir, runId, DOCTOR_DIR);(runs.ts:2713) before it validates runId. The only validation is inside ensureRunDoctorFile (assertValidId(runId, "runId")and theInvalid skill-doctor paththrow, blob.ts). That throw lands in the same blanket catch as transient blob errors (runs.ts:2717-2718). The catch logs the error as 'fetching status.json failed' and execution continues tofs.readFile(path.join(base, rel), "utf-8")(runs.ts:2724) with the id that validation rejected. So a first-party guard failure is reported as a network problem, and the guard does not stop the read. No current caller reaches this. page.tsx calls readRunSummary in parallel, and its ensureRunSummary assert rejects the Promise.all. resolveSafePath checksisValidId(runId)first (runs.ts:3031). But readRunDoctor is exported with a documented 'never throws' contract, so a future caller has no protection. Fix: addif (!isValidId(runId)) return null;at the top of readRunDoctor, as resolveSafePath does. Then narrowgetso it catches only fetch errors, and let a validation error from a pattern-checked relPath (a programmer error) propagate to the outer catch, where it is logged as a contract failure. -
[Axis 7] Internal-only doctor gate is recomputed at each caller (page and /api/file) instead of in the reader; kill switch accepts only 'off' (
evalboard/lib/doctor.ts:440) —(env ?? "").trim().toLowerCase() !== "off"(doctor.ts:440) meansEVALBOARD_TASK_ANALYSIS=0,=falseor=disabledleaves the panel and the /api/file patch route ON without a warning. The sibling knob in the same README section uses a different form: README.md:209 says 'EVALBOARD_WARM_INTERVAL_SECONDSsets the interval;0turns it off'. An operator who copies that form gets no kill switch. Also, page.tsx:51-55 and runs.ts:3048-3053 both calltaskAnalysisEnabled(isInternal, source.id, process.env.EVALBOARD_TASK_ANALYSIS)with the same arguments. Fix: accept the usual falsy spellings (off/0/false/no), or log a warning on an unknown value. Add a zero-argumenttaskAnalysisOn(sourceId)wrapper so the env read happens in one place. README.md:219 is also a 121-character line in a section that wraps at about 80 characters. Move the kill-switch sentence onto its own line. -
[Axis 7] RunDoctor type contract drifts from the reader:
openedis optional but readRunDoctor always sets it, and rowsDiagnosed is parsed but never read (evalboard/lib/doctor.ts:144) —opened?: Record<string, string>;(doctor.ts:144) is optional, but readRunDoctor (runs.ts:2752-2763) always returns{ status, triage, fixes, opened }, with opened defaulting to{}. Consumers must handleundefinedfor no reason, and a fixture that leaves the field out type-checks. Make the field required. Also,rowsDiagnosed: number;(doctor.ts:108), filled at doctor.ts:280rowsDiagnosed: int(s.rows_diagnosed),, has no reader anywhere in app/ or lib/ (a grep finds only these two lines). The module header (doctor.ts:10-11) says the types 'carry only the fields the run page renders'. Remove the field, or render it in Stats. -
[Axis 8] Two failure counts in one panel come from different fields and can disagree (
evalboard/app/runs/[id]/task-failure-analysis.tsx:145) — The header digest uses the summary count:const errors =${triage.rowsFailed} error${...}`` (line 145). The Stats block uses the parsed row list:[triage.tasks.length, "failed rows"],(line 690). `parseTask` drops any row that has no `task_id`, and `summary.rows_failed` is an independent producer field, so the closed header and the opened body can show different failure totals for the same run. Use one source for both (prefer `triage.tasks.length`, or show both with clear labels). Add a test where `rows_failed` is not equal to `tasks.length`. -
[Axis 8] resolveSafePath reserves the skill-doctor/ prefix in every edition, shadowing an experiment arm named 'skill-doctor' (
evalboard/lib/runs.ts:3040) —if (parts[0] === DOCTOR_DIR) {(runs.ts:3040) runs before the variant branch for all sources. WhentaskAnalysisEnabledis false, the function returns null. Because coder_eval does not reserve the variant id 'skill-doctor' (isValidVariantId accepts it), artifacts of a real arm with that name (on Scribe or in local mode too) can no longer be served by /api/file. Run listings are not affected: listRunIdsRemote reads only top-level prefixes, variants come from run.json, and ensureTaskDir lists${runId}/${variant}/${task}/. Either reserve the name on the producer side (reject variant_id 'skill-doctor' in coder_eval, as 'activation' is excluded here by convention) or apply the doctor branch only to sources where the panel is on. Record the reserved name in docs/REPORT_SCHEMA.md or the run-layout note so the cross-repo producer knows about it.
What's Missing
Parallel paths:
- 🔵 The evalboard now reserves
<run>/skill-doctor/in /api/file (runs.ts:3040), but the coder_eval producer side was not updated to match.variant_idin src/coder_eval/models/experiment.py still accepts "skill-doctor". The run-layout contract (.claude/shared/run-layout.md and its verbatim mirror plugins/coder-eval/reference/run-layout.md) and docs/REPORT_SCHEMA.md do not listskill-doctor/(oranalysis.md) as a reserved sibling of the variant dirs. (trigger: evalboard/lib/runs.ts) (restates: Axis 8: resolveSafePath reserves the skill-doctor/ prefix in every edition)
Tests:
- 🟡 The new module ids.ts has no test file of its own. Its validators are tested only through the blob.ts re-export (lib/tests/blob.test.ts:2). No case pins "." or ".." for isValidId. No case covers the new runId sinks either:
resolveSafePath("..", ".env")and readRunDoctor'spath.join(dir, runId, DOCTOR_DIR)(runs.ts:2713). (trigger: evalboard/lib/ids.ts) (restates: Axis 4: isValidId accepts "." and "..") - 🔵 No route-level test covers app/api/file/route.ts for the new skill-doctor branch. The resolveSafePath tests (doctor-blob.test.ts:250-336) assert only null vs path. They do not pin the HTTP status: 403 for an unlisted patch or the feature off, 404 for a listed patch not on disk, and 5xx (expected) for a storage error. (trigger: evalboard/lib/runs.ts) (restates: Axis 6: /api/file doctor-patch branch turns blob failures into 403/404)
Display & mapping dicts:
- 🔵 The LAYERS / SOLVABLE display maps (tfa:30-55) have fallbacks that no test exercises. An unknown layer shows its raw key in grey. An unknown
yes-*solvable value is counted as fixable byisFixable(regex), butSOLVABLE.getreturns no pill for it. As a result, the 'fixable' stat and the card pills can disagree. The fixture uses only grader/skill/env and yes-grader/yes-skill/env-fix, so no render test covers an off-map value. (trigger: evalboard/app/runs/[id]/task-failure-analysis.tsx) (restates: Axis 1: Closed doctor-contract value sets typed as free string)
Downstream consumers:
- 🔵 The same run page now shows three failure counts from three sources: the run.json grid, the digest header (
triage.rowsFailed, tfa:145) and the Stats 'failed rows' (triage.tasks.length, tfa:690). Nothing reconciles them, and no test asserts that they agree. (trigger: evalboard/app/runs/[id]/task-failure-analysis.tsx) (restates: Axis 8: Two failure counts in one panel come from different fields) - 🔵
splitAnalysis/planAnalysisnow split analysis.md on the exact## Findingsheading and hide everything below it when a doctor report is done. That heading comes from the analysis.md template in plugins/coder-eval/skills/analyze/SKILL.md:269, which does not record that the board depends on it. No test ties producer and consumer together. If the skill renames the heading, the split stops silently and the board falls back to the full analysis plus the block, so the Findings show twice. (trigger: evalboard/lib/doctor.ts)
Daily/nightly:
- 🔵 The PR body does not say what happens to the skills nightly run page when a run is re-graded in place (
evaluate <run_dir>) or re-uploaded after the doctor ran. The triage is cached for good and is not checked against run.json's run_id or run_json_etag, but it still replaces the AI summary's Findings. The PR also does not say which pipeline owns the<run>/skill-doctor/prefix in the nightly blob upload. For example, a re-upload with destination-delete semantics would remove it. (trigger: evalboard/app/runs/[id]/page.tsx) (restates: Axis 8: The doctor triage is not checked against the run.json it describes)
Harness & Lint Improvements
Static checks (lint / type):
- [ce-lint] Add a TS source-scan guard,
evalboard/lib/__tests__/id-validators.test.ts. It does two things. (a) It fails on any^[\w.-]+-shaped regex literal used as a path or blob id validator outsidelib/ids.tsandlib/variants.ts. Today it would flagids.ts:7ID_RE, plusruns.ts:2792SERIES_KEY_RE, which copiesseries.ts:97. (b) It imports every exportedisValid*/assertValid*fromlib/ids.tsandlib/variants.tsand asserts that each one rejects ".", ".." and "...". It follows the vitest-scan pattern oflib/__tests__/no-zero-coalesce.test.ts. It claims no CE id, because the CE runner reads Python ASTs only. Blind spot: a validator written with a different character class is not seen. Prevents: A4 medium: isValidId accepts "..", so/api/file?run=..&path=.envreads<RUNS_DIR>/../.envand readRunDoctor gets a new runId-to-path sink (ids.ts:7,runs.ts:2713/3031/3074). It would also remove the need for the special case in clearRunCacheDir. - [ce-lint] Add a TS source scan for the blob-layer rule that only a 404 means absent. In
lib/*.ts, acatch (err)block whosetrycalls anensure*blob helper must referenceisNotFound(or rethrow. Each exception goes in a reasoned allowlist that is keyed on the trimmed source line, as in no-zero-coalesce.test.ts. The rule is in the comment at blob.ts:211-213, and today nothing enforces it. Blind spot: a helper that wraps the ensure call one level down is not seen. Prevents: A6 medium: readRunDoctor'sget(runs.ts:2717) swallows auth, network and IMDS errors, so /api/file answers 403 or 404 when storage is down, and the digest blames the contract. A6 low: the same catch-all swallows ensureRunDoctorFile's own assertValidId throw and then reads the filesystem with the rejected id. If the panel's never-throws path goes in the allowlist, its author must write down what that means for the file route. - Make the doctor contract types closed (type-checker tier; for evalboard this is tsc, not pyright). In
lib/doctor.ts, parselayer,outcome,solvableand actionkindintoas constliteral unions with oneotherbucket, in the same way asDOCTOR_STATES/isDoctorState. Make LAYERS/SOLVABLERecord<Union, …>. tsc then reports TS2367 for a misspelled=== "pr-ready"/count("env")and TS2741 for a missing map entry. Also brand the allowlisted URLs:isSkillsPrUrl(u): u is SkillsPrUrl, withExtA.href: SkillsPrUrl | AdoUrl. To keep this from regressing, add a short vitest scan that fails if aDoctor*interface field named layer/outcome/solvable/kind/state is typedstring, or ifExtA's href is typedstring. Prevents: A1/A2 medium: free-string fields with magic-literal matching (task-failure-analysis.tsx:56/57/195/415/589/691/693,doctor.ts:62/63/123/286), and the two different readings ofsolvable. A2 low: the PR-URL allowlist that only a comment guarantees (doctor.ts:387, ExtA sink at tfa:99-110). - Turn on
noUncheckedIndexedAccess(tsc tier) through a scopedevalboard/tsconfig.strict.jsonthat extends the base config. At first it includes onlylib/doctor.ts,lib/ids.tsandapp/runs/[id]/task-failure-analysis.tsx, andpnpm typecheckruns it. Grow it file by file. A repo-wide flip is not practical now: I measured 450 errors (about 112 outside tests). The doctor files have 13 (doctor.ts 8, tfa 5), so a scoped flip is cheap. With the flag on,prChecks[k]andopened[k]are typedT | undefined, so a direct index withoutown()must handle a missing key. Prevents: A2 low: report-keyedRecord<string, …>types (doctor.ts:133/144) claim every key is present, and prototype safety depends on rememberingown(). The test already indexesprChecks["example-t2"]!directly. - [ce-lint] Extend
lib/__tests__/no-zero-coalesce.test.ts: addlib/doctor.tsto COVERED, and add a pattern for a parse helper whose numeric parameter defaults to zero (dflt = 0,(v: unknown, … = 0)). Fields parsed from an external contract must benumber | null, so a missing value does not render as a measured zero. This is the TS counterpart to CE049 and CE069. Blind spot: inlinetypeof v === "number" ? v : 0ternaries are not matched. Prevents: A2 medium:int()(doctor.ts:157-159) turns a missing or malformedsummary.rows_failedinto 0. The header then shows "0 errors", and the did-not-parse warning never fires (doctor.ts:279/280/287). - [ce-lint] Add a TS source scan for path-prefix dispatch on a raw path. In any
lib/function that callspath.resolve(…, relPath), fail onrelPath.split("/"),relPath.startsWith(orparts[0] ===unless the same function first assignspath.posix.normalize(relPath)and dispatches on the normalized value. Blind spot: the scan keys on the identifier namerelPath, so a renamed variable is not seen. The runtime property test in harness_improvements covers that gap. Prevents: A4 low: the/api/fileskill-doctor gate (runs.ts:3039-3040) checks the raw first segment, so./skill-doctor/…oractivation/../skill-doctor/…skips the edition check, the kill switch and the patch allowlist. - [ce-lint] Add a TS source scan that bans compatibility shims under
evalboard/lib/, because the project is greenfield and has no backward-compatibility burden. It fails onexport { a, b };when it re-exports names imported from a sibling module, and onexport const X = Y;whenYis a bare imported identifier (an alias). Exceptions go in a reasoned allowlist (for example, thelib/overview.tsre-export oftaskCarriesRepoTag, if it is still needed). Prevents: A1/A5/A7 low: theblob.ts:17re-export shim over the moved id validators, and thedoctor.ts:396aliasisLinkableTaskId = isValidTaskId. - [ce-lint] Add a cheap TS source scan that fails on
JSON.stringify(<x>).toLowerCase().includes((and onJSON.stringify(...)inside a.filter(search predicate) underevalboard/app/. Free-text search must go over named value fields, not a serialized object that contains key names and the literalnull. Prevents: A1 low: the search attask-failure-analysis.tsx:325matches every task for "fix", "why", "layer" or "null", and opens every accordion. - Set up eslint with typescript-eslint in
evalboard/now. The promotion bar in.claude/harness-candidates.mdwas already met ("stand up eslint next timeevalboard/is touched substantively"), and this PR adds a 981-line view plus a 5-file contract. Wire it intopnpm verify, somake evalboard-verifyand theevalboardCI job run it. Turn on@typescript-eslint/no-non-null-assertion(allowlist with reasons),@typescript-eslint/switch-exhaustiveness-checkand@typescript-eslint/no-unnecessary-condition. Move the vitest source scans above into custom rules over time, so they parse TypeScript and do not regex lines. Prevents: A2 low:const triage = doctor.triage!(task-failure-analysis.tsx:875) re-asserts an invariant (done implies triage) that only a comment states, andby.get(skill)!(tfa:334). It also removes the regex blind spots in every TS source-scan guard proposed here.
Harness improvements (not statically reachable):
- Add a cross-repo contract golden for skill-doctor status_v1, triage_v1 and fixes_v1. The producer repo publishes its JSON Schemas (or a golden document the producer generates) as a versioned artifact, and the evalboard CI fetches the current copy and does not vendor a stale one. A vitest parity test then (a) validates
lib/__tests__/fixtures/skill-doctor/*against those schemas, and (b) asserts that every property the parsers read (fixed_by_pr,existing_prs,env_group,doctor_pr,pr_checks.*.head_sha,run.rows_executed,run.run_id,run_json_etag) exists in the schema. Also log a runtimeconsole.warnwhen a known optional key is missing from a v1 document, so drift shows in the logs before the next sync. Why not static: The producer is in another repo. A schema copied into evalboard drifts silently, so the check needs the producer's current schema at CI time, which is a network or cross-repo fetch, not an AST property of this tree. Prevents: A3/A8 medium: the doctor contract is pinned only by hand-written fixtures, and a producer-side rename silently drops the Proposed PR rows or the 'rows run' stat (doctor.ts:10-11/220/259-262/274-277). It also lets the board userun_id/run_json_etag, which are sent today and ignored. - Add a doctor-state matrix test that iterates the
DOCTOR_STATESconst array (it does not list states by hand), so a new state cannot go untested. For each state, withfixes_dirset, assert the exactstate.downloadslist from readRunDoctor and the result ofresolveSafePath(RUN, "skill-doctor/<fixes_dir>/<id>.patch"). This forces a decision on whether a retracted or failed report still serves patches. Why not static: Whether fixes and patches are fetched or served for a state depends on runtime control flow against mocked blob storage. Only an executed test sees it. Prevents: A3 medium: the state gate is tested for the triage only. A retracted, failed or skipped status that keepsfixes_dirstill downloads fixes and patches and serves them through /api/file (runs.ts:2740/2753/3061-3062;doctor-blob.test.ts:131). - Add a path-normalization property test for
resolveSafePath. For every gated prefix (skill-doctor,activation, variant dirs), assert that./<p>/x,a/../<p>/xand<p>/./xgive the same decision as the canonical<p>/x. Also assert that a dot-only run id ("..",".") returns null, for exampleresolveSafePath("..", ".env") === null, in both local and blob mode. Why not static: The final file comes frompath.resolveagainst the real filesystem and the storage mode. Only executing the resolver shows which file is selected. The source scan above catches only therelPathnaming convention. Prevents: A4 medium (dot-only run id reads outside RUNS_DIR) and A4 low (the raw-segment skill-doctor gate can be bypassed). - Add a route-level storage-failure contract test. Call
/api/filewith the blob mock set tofailWith: {statusCode: 500}(and also auth or IMDS errors), and assert a 5xx response, never 403 or 404. Replace the test atdoctor-blob.test.ts:305-318, which now fixes the masking behavior in place. Add a digest test that asserts a transient triage-fetch failure renders as 'could not be fetched', not 'missing or did not parse'. Why not static: The HTTP status comes from runtime error propagation through readRunDoctor, resolveSafePath and route.ts, against injected storage faults. Prevents: A6 medium: storage failures show as 403 'forbidden' or 404 'not found' on the file route, and the panel blames contract drift for infra errors. - Add a triage-to-run consistency check, with tests. parseTriage keeps
run.run_id(and the board comparesrun_json_etagwhen it can get it), and drops the triage or marks it stale on a mismatch. page.tsx intersectstriage.taskswith the failing rows in run.json. Add tests for a mismatched run_id, for a triage row that now passes, and forsummary.rows_failed !== tasks.length, which makes the header and the body use one count. Why not static: Staleness and count disagreement are properties of two data files at runtime (re-grade plus re-upload, or a producer writing to the wrong prefix). No source pattern shows them. Prevents: A8 medium: an unverified triage replaces the AI summary's Findings and can disagree with the grid (doctor.ts:271,page.tsx:101). A8 low: the header uses rowsFailed and the body uses tasks.length (task-failure-analysis.tsx:145/690). - Extract
loadDoctorPlan(analysis, isInternal, source, env)from page.tsx, with a unit test for the off, OSS, non-skills and enabled cases. It asserts that readRunDoctor is not called when the panel is gated off and thatplan.summary, not the rawanalysis, goes to AnalysisPanel. Put the env read and the gate in this one helper, so /api/file and the page share the same predicate. Make the kill switch accept off/0/false/no, or warn on an unknown value. Why not static: Whether the panel is on depends on the edition, the source id and the env at runtime. The composition bugs (wrong prop, missing gate) are type-correct. Prevents: A3 low: no test for the page-level wiring. A7 low: the gate is recomputed at each caller, and the kill switch accepts only 'off' (doctor.ts:440,page.tsx:51-55,runs.ts:3048-3053). - Add empty-state and degraded-field render tests for TaskFailureAnalysis: a done triage with
tasks: [],actions: []androwsFailed: 0; a case withrowsRun: null; and a case where readRunDoctor's outer catch is reached (a parser or fs read throws after status loads) withexpect(err).toHaveBeenCalled(). If the outer catch cannot be reached, delete it. Why not static: The rendered copy (for example the misleading 'No failures match this filter.') and whether the catch can be reached are runtime and render-tree properties. Prevents: A3 low: the empty state is not asserted, and the never-throws test does not reach readRunDoctor's outer catch (task-failure-analysis.tsx:698/710/848,runs.ts:2762-2765). - Record the reserved run-dir name
skill-doctoron the producer side. coder_eval's variant-id validator rejectsskill-doctor(andactivation), and.claude/shared/run-layout.mdand docs/REPORT_SCHEMA.md list the reserved names. Add a pytest that asserts every name the evalboard reserves inresolveSafePathis rejected as a variant_id. Why not static: The reservation crosses a language boundary (a TS resolver and a Python validator). Keeping the two lists in step needs a parity test that reads both sides, in the same way as status-parity.test.ts. Prevents: A8 low:runs.ts:3040shadows the artifacts of an experiment arm named 'skill-doctor' in every edition.
Top 5 Priority Actions
- No finding changes a task's score or final_status, so security comes first: make isValidId reject ids that are only dots (evalboard/lib/ids.ts:7, the way isValidVariantId in lib/variants.ts:25-26 does), normalize relPath before the skill-doctor dispatch (evalboard/lib/runs.ts:3039-3040) so './skill-doctor/...' and 'activation/../skill-doctor/...' cannot skip the gate, and add regression tests that
resolveSafePath("..", ".env")and./skill-doctor/status.jsonboth return null. - Stop the /api/file doctor-patch branch from turning auth, network and IMDS blob failures into 403 or 404 (evalboard/lib/runs.ts:3058 calls readRunDoctor, whose
getat runs.ts:2717-2718 catches every error): let non-404 errors propagate so the route returns 5xx, and change the doctor-blob.test.ts:305-318 case to expect that. - Decide whether a non-done doctor status (retracted, failed, skipped) that still names fixes_dir may fetch and serve patches, gate fixesRel and patches on the state if it may not (evalboard/lib/runs.ts:2740/2753 and 3059-3063), and pin the decision with a
test.eachover the non-done states that keepsfixes_dirset (evalboard/lib/tests/doctor-blob.test.ts:131). - Check the triage against the run.json it describes: keep
run.run_idandrun_json_etagin parseTriage (evalboard/lib/doctor.ts:271) and mark the triage stale on a mismatch, so a re-graded or wrongly-placed triage does not replace the AI summary's Findings (page.tsx:101) and show wrong failure counts. - Pin the cross-repo doctor contract: add a producer-published schema or golden file that CI refreshes, plus a parity test for every field the parsers read (evalboard/lib/tests/doctor.test.ts:20), and make missing or malformed counts
number | nullinstead of a false 0 (evalboard/lib/doctor.ts:279,int()at 157-159), so producer drift fails a test or logs a warning instead of showing "0 errors".
Stats: 0 🔴 · 0 🟠 · 7 🟡 · 22 🔵 across 8 axes reviewed.
- isValidId / isValidTaskId reject dot-only ids, so /api/file?run=.. can no longer read above the runs dir - resolveSafePath dispatches on the normalized path: ./skill-doctor/... and activation/../skill-doctor/... no longer skip the skill-doctor gate - /api/file: a storage error on a doctor patch propagates (5xx) instead of reading as 403/404; a patch link fetches only that patch; the page still never throws and says "could not be fetched" - only a done report fetches or serves fixes (a retracted one keeps fixes_dir but serves nothing) - a triage or fixes.json that names another run is ignored - a triage with a missing or malformed rows_failed / rows_diagnosed does not parse (no false "0 errors"); mutant counts must be integers - copies of the doctor's v1 schemas under the test fixtures, with a parity test for every field the parsers read and the closed value sets (states, layers, verdicts, the open-pr action kind) Not in this round: comparing run_json_etag with run.json's etag (a run.json re-graded in place after the doctor ran); fixes_v1 does not pin the candidate fields covers / patch_lines / title / gates yet. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Adds a Task failure analysis block to the run page. It is built from the skill-doctor report that sits next to a run's
analysis.md, in<runId>/skill-doctor/. It shows only in the internal edition, for the skills source, and has a kill switch.What the page shows
<X> errors · <Y> PRs with fix, with the Y part omitted at 0;<fixes_dir>/opened.jsonor the triage'sdoctor_pr.fixed_by_pr.verdict == "fixes") and verified by replaying stored runs (pr_checks), with the evidence shown.Data and safety
readRunDoctorreads it in one pass:status.json, refreshed every 60 s;<fixes_dir>/fixes.json, thenopened.json(also refreshed every 60 s) and the listed patches (at most 10).schema_version !== 1counts as absent;https://github.com/UiPath/skills/pull/<n>only) and ADO links are allowlisted too;Object.hasOwn;/api/file: only<fixes_dir>/<id>.patchfor an id that the run'sfixes.jsonlists, fetched on a cold cache. Everything else underskill-doctor/returns 403. With the feature off, nothing there is served or fetched.isInternal, the skills source, andEVALBOARD_TASK_ANALYSISnotoff.skill-doctor/never appears as a run.Tests
npm run verifypasses: tsc, vitest (65 files, 1049 tests) andnext build./coder-eval-code-review-wf(8 axes, adversarial verify). The final passes scored 9.8 and 9.6, with no open critical, high or medium findings.🤖 Generated with Claude Code