Skip to content

feat(evalboard): Task Failure Analysis panel from the skill-doctor report - #264

Open
CarlesUIPath wants to merge 13 commits into
mainfrom
feat/evalboard-task-failure-analysis
Open

CarlesUIPath wants to merge 13 commits into
mainfrom
feat/evalboard-task-failure-analysis

Conversation

@CarlesUIPath

@CarlesUIPath CarlesUIPath commented Nov 19, 2026 •

Copy link
Copy Markdown
Collaborator

Adds a Task failure analysis block to the run page. It is built from the skill-doctor report that sits next to a run's analysis.md, in <runId>/skill-doctor/. It shows only in the internal edition, for the skills source, and has a kill switch.

What the page shows

  • Placement and header:
    • a collapsed block styled like the AI summary, directly under it;
    • header: <X> errors · <Y> PRs with fix, with the Y part omitted at 0;
    • while the report is not done, the header shows its state instead: running since…, failed with the reason, or skipped.
  • Inside, following the order of the doctor's own report page:
    • a stats row and a failures-by-layer bar;
    • Fix first: the doctor's ranked actions;
    • Skill failures: layer chips, search, expand and collapse all, and one collapsed accordion per skill, sorted by PRs with fix.
  • Task cards show the task link, pills, the diagnosis, then Fix, Failing check, Evidence and Passed before. Tasks that stopped at the pre-run check are folded per cause.
  • PR rows on a card:
    • PR with fix: #N ↗: the doctor's own draft PR, from <fixes_dir>/opened.json or the triage's doctor_pr.
    • Fix prepared, PR not opened yet: a fix that passed the doctor's gates, with a patch download.
    • Proposed PR: an open PR by someone else that the doctor judged a fix (fixed_by_pr.verdict == "fixes") and verified by replaying stored runs (pr_checks), with the evidence shown.
    • Open PR may fix this (not verified): the same, when the doctor could not verify it.
    • Nothing is inferred from prose. Candidates the doctor never tried are not shown.
  • Left out: no costs and no owner names.

Data and safety

  • Reading the report: readRunDoctor reads it in one pass:
    • status.json, refreshed every 60 s;
    • the immutable triage file it names;
    • <fixes_dir>/fixes.json, then opened.json (also refreshed every 60 s) and the listed patches (at most 10).
  • Trust: all of this JSON is treated as untrusted:
    • file names are pattern-checked before any fetch;
    • schema_version !== 1 counts as absent;
    • every PR URL field is allowlisted in the parser (https://github.com/UiPath/skills/pull/<n> only) and ADO links are allowlisted too;
    • lookups keyed by report data use Maps or Object.hasOwn;
    • nothing renders raw HTML;
    • a doctor file problem never breaks the page, and parse failures are logged by file name only.
  • Patch downloads via /api/file: only <fixes_dir>/<id>.patch for an id that the run's fixes.json lists, fetched on a cold cache. Everything else under skill-doctor/ returns 403. With the feature off, nothing there is served or fetched.
  • Gating: isInternal, the skills source, and EVALBOARD_TASK_ANALYSIS not off.
  • Listings: runs, variants and tasks are still listed from top-level prefixes only, so skill-doctor/ never appears as a run.

Tests

  • npm run verify passes: tsc, vitest (65 files, 1049 tests) and next build.
  • All fixtures are synthetic.
  • Reviewed five times with this repo's /coder-eval-code-review-wf (8 axes, adversarial verify). The final passes scored 9.8 and 9.6, with no open critical, high or medium findings.

🤖 Generated with Claude Code

CarlesUIPath and others added 3 commits November 19, 2026 14:13
…port

The run page reads the skill-doctor report under <run>/skill-doctor/
(contract v1): status.json (refetched after 60 s), the triage file it
names once done, and <fixes_dir>/fixes.json plus its patches (cached for
good). Names are checked against their patterns before any fetch; a
schema_version other than 1, or unparsable JSON, is treated as absent, and
readRunDoctor never throws into the page.

When the report is done, analysis.md is cut at its first `## Findings`
line (outside code fences): the AI summary keeps the top, and a native
React panel replaces Findings and below (counts by layer and class, new vs
recurring, diagnosis cost, Actions, a collapsible task table, Fixes with
patch download through /api/file). Otherwise the page is unchanged, plus a
one-line status for queued, running, failed, skipped and retracted
reports. PR links render only for UiPath/skills pull URLs and build links
only for dev.azure.com URLs; anything else is plain text.

Internal edition and the skills source only; EVALBOARD_TASK_ANALYSIS=off
turns it off.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- readRunDoctor is the one place that decides which doctor files a status
  implies and parses each once; ensureRunDoctorFile only downloads.
- Parsed types carry only what the page renders; the id validators move
  to a Node-free lib/ids.ts shared by blob.ts and doctor.ts.
- One predicate (patchIds, capped at 10, ids checked) picks the patches to
  fetch and to link; /api/file fetches a patch on demand and serves only
  <fixes_dir>/<id>.patch from skill-doctor/.
- The header counts the per-task triage state (new / changed / known)
  and shows vs_history only when there is a base run.
- A done report gets the panel even without a `## Findings` split or an
  analysis.md.
- parseFixes requires schema_version 1; a file that does not parse is
  logged by name; the kill switch accepts "off" in any case.
- Tests for the task-link check, the text fallbacks, the fixes.json
  version gate, the patch cap and the cold-cache patch fetch.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… /api/file

- /api/file serves a skill-doctor patch only when this run's status names
  its fixes dir and fixes.json lists it (patchIds), loading both through
  readRunDoctor on a cold cache: one fetch path, and a made-up name costs
  no blob call.
- The skill-doctor branch returns nothing when the panel is off (OSS
  edition, another source, or EVALBOARD_TASK_ANALYSIS=off).
- ensureRunDoctorFile reuses the ids.ts validator; doctor.ts constants
  nothing imports are module-private; stale comments fixed.
- Tests: the download guard, row badges, unlisted patches, the kill
  switch on /api/file, and a storage error on the patch fetch.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@github-actions

github-actions Bot commented Nov 19, 2026 •

Copy link
Copy Markdown

Claude finished @CarlesUIPath's task in 1m 21s —— View job


Code Review in Progress

Review Checklist:

  • Read .github/code_review.md for review standards
  • Read CLAUDE.md for project conventions
  • Run git diff to see full changes
  • Review each changed file with full context
  • Check cross-file consistency
  • Analyze what might be missing
  • Post comprehensive review

Starting review now...

@CarlesUIPath
CarlesUIPath marked this pull request as draft November 19, 2026 16:12
CarlesUIPath and others added 9 commits November 19, 2026 16:20
…ke the doctor's page

The block now sits under the AI summary and opens like it: a bordered
section with a header button, collapsed by default. The header carries
"<N> errors · <M> proposed PRs ready", or the doctor's state when the
report is not done.

Inside, in the order of the skill-doctor page: the stats row and the
failures-by-layer bar, Fix first (the ranked actions, with an "N PR
ready" badge when an action's tasks have a pr-ready fix), then Skill
failures: layer chips, search, Expand all / Collapse all, and one
collapsed accordion per skill with a card per task (pills, diagnosis,
the proposed PR row with outcome, patch, gates and cost, and the Fix /
Failing check / Evidence / Passed before / PRs / Related / Owner list).
Tasks stopped at a pre-run check collapse into one block per cause. The
separate task and fix tables are gone. States, gating, link allowlists,
patch rules and the kill switch are unchanged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ailures

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…dates

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- A pr-ready fix candidate reads "PR with fix" with its title, patch,
  gates and cost; the title expands to the prepared PR body
  (<fixes_dir>/<id>.md, loaded by name for ids in patchIds, rendered with
  ReactMarkdown without raw HTML; only allowlisted links, images as alt
  text), then the patch download. A doctor PR shows as "PR with fix"
  linking to GitHub instead. Other outcomes stay "Fix candidate".
- "Proposed PR": an open, allowlisted PR from the task's existing PRs
  that its fix text names (#N or /pull/N).
- Header, skill accordions and Fix first badges count distinct PRs with
  fix (both kinds); skill and card sorts use the same model.
- The Owner row is gone and owners are no longer parsed.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…Rs; no costs

- "PR with fix: #N ↗" links the doctor's opened PR (<fixes_dir>/opened.json,
  fetched by name and refreshed like status.json, allowlisted URLs only)
  or the triage's doctor_pr, with the gate verdicts.
- A pr-ready fix with no opened PR reads "Fix prepared, PR not opened
  yet" with the patch download; it is not a PR with fix and sorts right
  after linked PRs.
- "PRs with fix" (header, skill accordions, Fix first badges, both sorts)
  counts only linked PRs: opened or doctor PRs, and third-party open PRs
  the fix text names.
- Cards drop "Open PRs on these files" and "Related skill PRs";
  related_skill_prs is no longer parsed. Fix first drops open-pr actions.
- The inline PR body and its .md loading are gone; /api/file still serves
  .patch only.
- No costs anywhere; cost fields are no longer parsed.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- Layer and solvable labels are Maps and opened.json lookups use
  Object.hasOwn, so report keys such as "constructor" or "__proto__"
  cannot reach the object prototype; parseOpened builds through a Map.
- A task covered by a pr-ready candidate whose PR was opened gets that
  PR (opened.json is keyed by the candidate's primary task) instead of
  "Fix prepared".
- The doctor's own PR is filtered out of the Proposed PR list and counted
  once.
- PR URLs (existing_prs, doctor_pr, opened.json) are allowlisted in the
  parsers; the view links them without re-checking.
- Cards drop the Related row (no longer parsed) and the "none yet"
  placeholder.
- One layerCounts() helper and one FixRow component; Body split into
  Stats / LayerBar / FixFirst / SkillFailures; fixTier uses early
  returns; SkillGroup.ready is now fixPrs; dead "—" sentinels and the
  ExternalLink fallback are gone.
- Tests: prototype keys, the covered-task PR, own PR vs proposed, the
  opened.json refresh, parser allowlisting.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…d pr_checks

The "fix text names #N" rule is gone. A third-party PR now shows only
when the triage's fixed_by_pr says "fixes" and the PR is open in the
row's existing PRs, joined with fixes.json pr_checks for that row and
number:
- verified: "Proposed PR" with the replay evidence; counts as a PR with
  fix and sorts after the doctor's own PR;
- untested or no check: "Open PR may fix this (not verified)" with the
  diagnosis's basis, the evidence, and "needs a Windows run" when the
  check says so; not counted, sorted after prepared fixes;
- not_verified, may_help, unrelated: nothing.
Both fields are optional and untrusted: malformed entries parse to
nothing, pr_checks URLs are allowlisted and must match the number, and
task ids are own keys.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…h the evidence

head_sha is optional (string, null or absent); only a 40-hex value is kept.
Tests cover it and the untested evidence texts of the final contract.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- Fix candidates with outcome not-run are hidden and no longer lift a
  card's sort tier.
- A pr_checks entry that does not validate is kept as null, so its task
  shows no PR row instead of "may fix (not verified)".
- rows_executed must be a non-negative integer.
- One own() helper for report-keyed lookups; one existingPr field
  (verified or not) instead of proposed + maybe; Patch absorbs PatchCell;
  one ExtA anchor; DoctorPrCheck.url is validated but not kept; doc
  comments updated.
- Tests: not-run candidates, a malformed not_verified check, task ids
  named constructor / __proto__ / toString with opened.json and
  pr_checks entries, rows_executed.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@CarlesUIPath
CarlesUIPath marked this pull request as ready for review November 20, 2026 00:38
@github-actions

github-actions Bot commented Nov 20, 2026 •

Copy link
Copy Markdown

Claude finished @CarlesUIPath's task in 1m 22s —— View job


Code Review in Progress

Todo List:

  • Read .github/code_review.md for review standards
  • Read CLAUDE.md for project conventions
  • Run git diff to see full changes
  • Review each changed file with full context
  • Check cross-file consistency
  • Perform "what's missing" analysis
  • Validate security considerations
  • Provide structured review feedback

Starting code review...

@uipreliga uipreliga left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review: coder_eval — pr:264

Scope: pr:264 · branch feat/evalboard-task-failure-analysis (PR #264 → main) · 0636adc · 2026-11-20T01:30Z · workflow variant

Change class: complex — new evalboard feature that parses untrusted external JSON, adds a new gated file-serving branch to /api/file (resolveSafePath), and a large new UI panel

The codebase is healthy at 9.3/10: the architecture and API surface are clean, there are no critical or high findings, and no finding can change a task's score or final_status for the same agent output; the real risks are in the evalboard skill-doctor feature, where a path-traversal gap lets /api/file read one level above the runs directory, storage failures show as 403/404 instead of 5xx, and the doctor contract has no schema or staleness check, so fix the security gap first, then the error handling and the test gaps.

Summary

Axis Score 🔴 🟠 🟡 🔵 Top Issue
1. Code Quality & Style 8.8 / 10 0 0 1 7 Closed doctor-contract value sets (fix outcome, PR state, etc.) typed as free string and matched by magic literals
2. Type Safety 9.2 / 10 0 0 1 3 parseTriage coerces a missing or malformed summary.rows_failed to 0, so the doctor header shows "0 errors" and no drift warning is logged
3. Test Health 8.6 / 10 0 0 2 4 The doctor-state gate is tested for the triage only. Fix and patch fetch and /api/file patch serving for a non-done status (retracted/failed/skipped) that still names fixes_dir are not tested
4. Security 9.3 / 10 0 0 1 2 isValidId (evalboard/lib/ids.ts:7/13) accepts "." and "..", so /api/file?run=..&path=.env serves files one level above the runs dir (this is older than the PR, and the PR copies it into the new ids.ts and adds readRunDoctor as a new sink)
5. Architecture & Design 9.9 / 10 0 0 0 1 A new multi-file doctor subsystem is added to the 3111-line lib/runs.ts module
6. Error Handling & Resilience 9.4 / 10 0 0 1 1 /api/file doctor-patch branch turns auth, network and IMDS blob failures into 403 'forbidden' or 404 'not found'. This breaks the blob-layer rule that only a 404 means absent.
7. API Surface & Maintainability 9.8 / 10 0 0 0 2 Internal-only doctor gate is recomputed at each caller (page and /api/file) instead of in the reader; kill switch accepts only 'off'
8. Evaluation Harness Quality 9.3 / 10 0 0 1 2 The doctor triage is not checked against the run.json it describes, but it still replaces the AI summary's Findings

Overall Score: 9.3 / 10 · Weakest Axis: Test Health at 8.6 / 10
Totals: 🔴 0 · 🟠 0 · 🟡 7 · 🔵 22 across 8 axes.

Blockers

None.

Non-blocking, but please consider before merge

  1. [Axis 1] Closed doctor-contract value sets (fix outcome, PR state, etc.) typed as free string and matched by magic literals (evalboard/app/runs/[id]/task-failure-analysis.tsx:56) — DoctorFix.outcome: string (lib/doctor.ts:123), DoctorTask.layer: string (doctor.ts:62) and solvable: string | null (doctor.ts:63) stay untyped. The view then special-cases bare literals that tsc cannot check: const isFixable = (t: DoctorTask) => /^yes/.test(t.solvable ?? ""); (tfa:56), const isReady = (c: DoctorFix) => c.outcome === "pr-ready"; (tfa:57), c.outcome !== "not-run" (tfa:195), [count("env"), "environment"] / [count("agent"), "agent mistakes"] (tfa:691/693), layerOf("env") (tfa:589), and in the parser a.kind !== "open-pr" (doctor.ts:286). The solvable field is also read in two different ways: a regex prefix in isFixable (tfa:56) and SOLVABLE.get(t.solvable.split(" ")[0]) (tfa:415). If you misspell one of these literals, or the doctor renames a value, the code still compiles and the result is quietly wrong (for example, a wrong count in the 'fixable in the repo' stat). doctor.ts:26-38 already has the generic fix (DOCTOR_STATES as const + isDoctorState). Use the same pattern for layer, outcome, solvable and action kind: parse each into a closed literal union in doctor.ts, with one other/unknown bucket. Then make LAYERS/SOLVABLE Record<Union, ...> so the compiler checks every comparison.
  2. [Axis 2] parseTriage coerces a missing or malformed summary.rows_failed to 0, so the doctor header shows "0 errors" and no drift warning is logged (evalboard/lib/doctor.ts:279) — int() (lines 157-159: return typeof v === "number" && Number.isInteger(v) && v >= 0 ? v : dflt; with dflt = 0) turns an absent, string-valued, negative or fractional count into a real-looking 0. parseTriage uses it at line 279 (rowsFailed: int(s.rows_failed),) and line 280 (rowsDiagnosed: int(s.rows_diagnosed),). Six lines earlier (273-278), the same function narrows run.rows_executed to number | null, so the two count fields are handled differently. Failure case: a triage file with "summary": {} or "rows_failed": "5" passes isObj(j.summary) and parses. digest() (task-failure-analysis.tsx:145, const errors = ${triage.rowsFailed} error...``) then shows "0 errors" in the always-visible header of a run that has failures. Because the parse succeeds, the did not parse as contract v1 warning in readRunDoctor (runs.ts) never fires, so this kind of contract drift stays hidden. The same coercion turns `gates.mutants` `{n: 3.5}` into "mutants 0/0" (line 300) and turns a bad `actions[].count` into 0 (line 287). Fix: type the counts as `number | null` (or reject the triage when `summary.rows_failed` is not a non-negative integer, as the module header promises for files that do not parse), and render null as "—". Static check: a vitest case that gives parseTriage `summary: {}` and expects null or a null count.
  3. [Axis 3] The doctor-state gate is tested for the triage only. Fix and patch fetch and /api/file patch serving for a non-done status (retracted/failed/skipped) that still names fixes_dir are not tested (evalboard/lib/__tests__/doctor-blob.test.ts:131) — The only non-done readRunDoctor test also clears the fixes pointer: seedBlobs(statusWith({ state: "running", fixes_dir: null })); (line 131). This means the branch that matters is never exercised. In lib/runs.ts the fixes path has no state check: const fixesRel = status.fixesDir ? ${status.fixesDir}/fixes.json : null; (line 2740) and if (status.fixesDir) { (line 2753, which fetches every patchIds() patch plus opened.json). resolveSafePath then serves a patch when only doctor.status.fixesDir !== patch.fixesDir || !patchIds(doctor.fixes).has(patch.id) (lines 3061-3062) is false. So a retracted status.json that keeps fixes_dir still downloads fixes.json and the patches, and /api/file still serves the retracted report's patch through a direct link, even though planAnalysis/TaskFailureAnalysis show no body (and no patch links) for that state. No test decides which behavior is intended. Add a test.each(["queued","running","failed","skipped","retracted"]) case that seeds the fixture status with fixes_dir: "fixes-1000001" kept. Assert the exact state.downloads list and the resolveSafePath(RUN, "skill-doctor/fixes-1000001/example-grader-fix.patch") result. If the decision is that a retracted (or any non-done) report serves nothing, gate fixesRel/patches on the state in readRunDoctor and let the test pin that.
  4. [Axis 3] Cross-repo doctor contract pinned only by hand-written fixtures, with no schema parity check (evalboard/lib/__tests__/doctor.test.ts:20) — lib/doctor.ts says the types "are written by hand against the doctor's status_v1 / triage_v1 / fixes_v1 schemas" (lines 10-11). Every parser test reads fixtures that the same change wrote by hand: const FIX = path.join(__dirname, "fixtures", "skill-doctor"); and const TRIAGE = read("triage-1000001.json"); (lines 20-23). The tests therefore prove parser ↔ fixture, not parser ↔ producer. The producer lives in another repo. Optional fields degrade field by field to null (fixedByPr: parseFixedByPr(v.fixed_by_pr), rowsRun from run.rows_executed, headSha from head_sha). The console.warn in readRunDoctor's load fires only when a whole file returns null. So if the doctor renames fixed_by_pr or rows_executed, the board silently drops the Proposed PR rows or the "rows run" stat, and no test fails. This is the event-reassembled-record shape: vendor the doctor's v1 JSON Schemas (or a golden file the producer generates) under lib/tests/fixtures/skill-doctor/. Add a test that validates the fixtures against them, and a parity test asserting every schema property the parsers read (fixed_by_pr, pr_checks.*.head_sha, run.rows_executed, existing_prs, doctor_pr, env_group, ...) exists in the schema. That way a producer-side rename fails here.
  5. [Axis 4] isValidId (evalboard/lib/ids.ts:7/13) accepts "." and "..", so /api/file?run=..&path=.env serves files one level above the runs dir (this is older than the PR, and the PR copies it into the new ids.ts and adds readRunDoctor as a new sink) (evalboard/lib/ids.ts:7) — The new file ids.ts says 'Run / task IDs get reflected into filesystem paths and blob prefixes, so reject anything outside a narrow whitelist', but const ID_RE = /^[\w.-]+$/; (line 7) and isValidId (line 13) accept "." and "..". In lib/runs.ts:3031 if (!isValidId(runId)) return null; is the only runId check. Then const candidate = path.resolve(baseReal, relPath); (runs.ts:3074) resolves under <RUNS_DIR>/... I confirmed this with a vitest probe in local mode (EVALBOARD_LOCAL_RUNS_DIR=/runs): resolveSafePath("..", ".env") returned <x>/.env. In blob mode the parent is the app cwd, because RUNS_DIR defaults to cwd/runs-remote. The exploit there depends on Azure returning 404 for the blob ../run.json, which ensureRunSummary swallows. The defect is older than this PR. This PR moves the validator into a new module that claims it is safe, and adds another runId-to-path sink: path.join(dir, runId, DOCTOR_DIR) in readRunDoctor (runs.ts:2713), plus ensureRunDoctorFile downloads to <RUNS_DIR>/../skill-doctor/.... Fix: make isValidId reject ids that are only dots, e.g. ID_RE.test(id) && !/^\.+$/.test(id), and make isValidTaskId reject dot-only segments too. Then delete the special case in clearRunCacheDir that exists only because of this gap. Add a vitest case that asserts resolveSafePath("..", ".env") === null. CVSS:3.1/AV:N/AC:H/PR:L/UI:N/S:U/C:H/I:N/A:N
  6. [Axis 6] /api/file doctor-patch branch turns auth, network and IMDS blob failures into 403 'forbidden' or 404 'not found'. This breaks the blob-layer rule that only a 404 means absent. (evalboard/lib/runs.ts:3058) — resolveSafePath's new skill-doctor branch calls const doctor = await readRunDoctor(runId, source); (runs.ts:3058). readRunDoctor never throws, because its get helper catches every error, not only a 404: } catch (err) { console.error(skill-doctor: fetching ${rel} failed for run ${runId}:, err); } (runs.ts:2717-2718). On a cold cache, a 500, auth or IMDS failure on status.json or fixes.json makes readRunDoctor return null or an empty patchIds set. resolveSafePath then returns null, and app/api/file/route.ts:22-23 answers forbidden (403). A transient failure on the patch blob itself gives not found (404). The test at doctor-blob.test.ts:305-318 fixes this behavior in place ('/api/file then answers 404 for the missing file'). Every other branch of resolveSafePath goes through ensureTaskDir or ensureRunSummary, which rethrow non-404 errors (if (!isNotFound(err)) throw err;, blob.ts:217 and others). The blob layer states the rule at blob.ts:211-213: 'Auth, network, and IMDS failures must propagate so they don't masquerade as "not found" in the UI.' The 'never throws' contract is right for the page panel, which is optional. It is wrong for the file route, where a user clicks a link and must be able to tell 'not listed' from 'storage down'. The same root cause shows in the panel digest: when the triage fetch fails transiently, task-failure-analysis.tsx:143 shows "report done, but its triage file is missing or did not parse", which blames a contract problem for an infra error. Fix: give readRunDoctor a mode, or a separate loader for resolveSafePath, that lets non-404 fetch errors propagate so /api/file returns 500. Or have get record a fetch-failed flag on RunDoctor so the digest can say 'could not be fetched'. Change the doctor-blob test at 305-318 to expect a thrown error or a 5xx, not a 404. Static check: a vitest case that runs /api/file with failWith: {statusCode: 500} and expects a 5xx response.
  7. [Axis 8] The doctor triage is not checked against the run.json it describes, but it still replaces the AI summary's Findings (evalboard/lib/doctor.ts:271) — parseTriage reads only rows_executed from the triage's run block: const run = isObj(j.run) ? j.run : {}; (doctor.ts:271). The producer also writes run.run_id and run.run_json_etag (see fixtures/skill-doctor/triage-1000001.json), and the parser ignores both. readRunDoctor treats the triage as immutable (runs.ts:2699 // fetched again. Everything it points at is immutable.). page.tsx:101 const plan = planAnalysis(analysis, doctor); then removes analysis.md's ## Findings section whenever a report is done. Failure case: a run.json is re-graded in place (evaluate <run_dir>, then re-uploaded) or re-uploaded after the doctor ran, or a triage lands under the wrong run prefix. The board then shows failed rows, layer counts and 'fixable' stats from the old outcomes, and it hides the AI summary's own Findings. The two views can disagree with the grid on the same page. This does not change any score, but the run report becomes wrong for cross-run triage. Fix: in parseTriage, keep run.run_id and drop the triage (or mark it stale) when it is not equal to runId. In page.tsx, intersect triage.tasks with the failing rows in tasks from run.json, and show rows that now pass as stale instead of silently counting them. If the board can get the run.json blob etag, compare it to run_json_etag. Add a vitest case for a mismatched run_id and for a triage row that passes in run.json.

Nits

  1. [Axis 1] taskFixes is re-derived many times per task on each render, including inside a sort comparator, and groupBySkill is not memoized (evalboard/app/runs/[id]/task-failure-analysis.tsx:342) — const tier = (t: DoctorTask) => taskFixes(doctor, t).tier; (tfa:342) runs inside cards.sort(...), so taskFixes (which filters every candidate and resolves PRs) runs O(n log n) times per skill. countFixPrs(doctor, items) (tfa:354) and countPrepared(doctor, items) (tfa:355) re-derive it for every task, TaskCard re-derives it again (tfa:399), and digest (tfa:144) and every FixFirst action (tfa:752) re-derive it once more. const groups = groupBySkill(doctor, active, query); (tfa:790) is not memoized, so each keystroke in the search box runs all of this again, plus a JSON.stringify(t) for each task. Compute a Map<taskId, TaskFixes> once per doctor (one useMemo at the TaskFailureAnalysis root, or one pass in lib/doctor.ts) and pass it down. Then countFixPrs/countPrepared/fixTier read from the map.
  2. [Axis 1] Free-text search matches JSON field names and literal null, so common queries match every task (evalboard/app/runs/[id]/task-failure-analysis.tsx:325) — (!q || JSON.stringify(t).toLowerCase().includes(q)) (tfa:325) searches the serialized object, which includes keys such as "taskId", "layer", "why", "fix", "evidence", "fixedByPr" and the literal null. A search for "fix", "why", "layer", "null" or "skill" matches every task, and the search also opens every accordion (tfa:855). Search only the value fields the placeholder promises ("Search task, cause, PR…"): taskId, skill, why, fix, criterion, evidence, and the PR urls/titles.
  3. [Axis 1] Doctor-contract domain rules (PR-with-fix counting, sort tiers, grouping, digest) live in the 981-line client view, not lib/doctor.ts (evalboard/app/runs/[id]/task-failure-analysis.tsx:220) — The file mixes UI with exported pure logic: export function existingPrFix (tfa:220), export function taskFixes (tfa:276), countPrepared/countFixPrs (tfa:300/305), export function groupBySkill (tfa:316), export function digest (tfa:129) and fixesForTask (tfa:193). lib/doctor.ts says it is 'Pure data + string helpers' and is already where the panel's other rules live (patchIds, planAnalysis). Move this derivation logic into lib/doctor.ts, or into a lib/doctor-view.ts sibling. The component file then keeps only rendering, and the PR-counting contract is tested next to the parsers that feed it.
  4. [Axis 1] Collapsible header markup copied from AnalysisPanel, and a second hand-built /api/file URL (evalboard/app/runs/[id]/task-failure-analysis.tsx:941) — The block says it is 'shaped like the AI summary panel' (tfa:909). It copies analysis-panel.tsx:32-45 almost verbatim: className="border border-studio-blue/30 bg-studio-blue/5 rounded-lg", the w-full px-4 py-3 flex ... hover:bg-studio-blue/10 rounded-lg transition-colors button, and {open ? "Hide" : "Read analysis →"} (tfa:941-958). If someone restyles one panel, the two drift apart. Extract a small shared CollapsiblePanel({label, extra, children}) shell that both use. In the same way, patchHref (tfa:153-158) rebuilds `/api/file?run=${encodeURIComponent(runId)}&path=${encodeURIComponent(...)}` + withSource, which [...task]/_sections.tsx:1764-1768 already builds. Add one fileHref(runId, relPath, sourceId) helper.
  5. [Axis 1] Id validators moved to ids.ts but kept behind a blob.ts re-export shim and a doctor.ts alias; storage layer now depends on the doctor module (evalboard/lib/blob.ts:17) — export { isValidId, isValidTaskId }; (blob.ts:17) is a re-export shim. ids.ts:1-3 says it exists 'for its existing importers', which are only lib/runs.ts:14-15 and lib/tests/blob.test.ts:2. CLAUDE.md says 'Greenfield project — no backward-compatibility burden', so point those two importers at ./ids and remove the shim. export const isLinkableTaskId = isValidTaskId; (doctor.ts:396) adds a second name for the same predicate, for one call site (tfa:386), so tests now cover the same function under two names. Also, if (!isValidTaskId(relPath) || relPath.split("/").length > 2) (blob.ts:338) uses a task-id validator as a blob-path validator. Give it its own named check, or reuse the doctor's TRIAGE_FILE_RE/FIXES_DIR_RE/patchIdFromPath patterns, which already describe the allowed paths.
  6. [Axis 1] Positive-integer and verdict-whitelist checks repeated inline in the parsers, even though an int() helper exists (evalboard/lib/doctor.ts:273) — int() (doctor.ts:157-159) exists, but the same check is written out again: typeof run.rows_executed === "number" && Number.isInteger(run.rows_executed) && run.rows_executed >= 0 ? run.rows_executed : null (doctor.ts:273-278), and if (typeof n !== "number" || !Number.isInteger(n) || n < 1) return null; twice (doctor.ts:203 and 212). The verdict whitelists are inline || chains (doctor.ts:204, 214), not the const-array guard used for DOCTOR_STATES. Add nonNegInt(v): number | null / prNumber(v): number | null helpers and a generic oneOf(values)(v) guard. Then each parser line states only its field.
  7. [Axis 1] Narrative design comments that restate the code or record rationale/history, against the 'comments are a last resort' principle (evalboard/lib/runs.ts:3041) — The 7-line comment at runs.ts:3041-3047 ('skill-doctor/ belongs to the doctor contract ... readRunDoctor is the one fetch path ... a made-up name costs no blob call. Off with the panel.') is design rationale for the next 15 lines of code. The 7-line comment at blob.ts:322-328 repeats the same story. page.tsx:49-50 ('internal-only, skills nightly only, with an env kill switch (EVALBOARD_TASK_ANALYSIS=off). readRunDoctor never throws.') restates the taskAnalysisEnabled docstring (doctor.ts:428-431) and the readRunDoctor header (runs.ts:2702-2707). page.tsx:99-100 restates the planAnalysis docstring (doctor.ts:451-457). Keep one contract docstring per function. Move the fetch-path rationale to one place (the readRunDoctor docstring, or .claude/notes/), and remove the copies at the call sites.
  8. [Axis 2] The PR-URL allowlist is guaranteed only by a comment: isSkillsPrUrl narrows to plain string (evalboard/lib/doctor.ts:387) — export function isSkillsPrUrl(u: unknown): u is string { narrows to string, so the result of the check is lost in the type. DoctorPr.url, DoctorTask.doctorPr, RunDoctor.opened values and TaskFixes.ownPr are all string. The sink ExtA({ href }: { href: string ... }) (task-failure-analysis.tsx:99-110) depends on the docstring "Callers pass only allowlisted URLs", and doctor.ts:50-51 says the same thing in a comment ("Every PR URL the parsers hand out has passed isSkillsPrUrl: views can link them without checking again"). tsc does not stop a new code path that puts an unvalidated report string (for example t.fix) into ExtA. Fix: add a branded type (type SkillsPrUrl = string & { readonly __brand: "SkillsPrUrl" }), declare the guard as u is SkillsPrUrl, type the parsed fields with it, and make ExtA.href accept SkillsPrUrl | AdoUrl. tsc then enforces the invariant. Test code that sets doctorPr = "https://..." directly would need the guard or an explicit cast, which makes the bypass visible.
  9. [Axis 2] Report-keyed Record types say every key is present; prototype safety depends on always calling own() (evalboard/lib/doctor.ts:133) — prChecks: Record<string, DoctorPrCheck | null>; (line 133) and opened?: Record<string, string>; (line 144) are indexed by task ids taken from untrusted JSON. tsconfig.json does not enable noUncheckedIndexedAccess, so prChecks[k] has type DoctorPrCheck | null. At runtime a missing key gives undefined, and "constructor" gives Object.prototype.constructor. Production code avoids this only by convention, through own() (line 235). The PR's own tests index the Record directly (task-failure-analysis.test.tsx:297 ...d.fixes!.prChecks["example-t2"]!,), which shows how easily the helper gets skipped. Also, opened is optional (?), but readRunDoctor always sets it (let opened: Record<string, string> = {};), so the optional marker is looser than the data. Fix: type these fields as Partial<Record<string, T>> (or { readonly [k: string]: T | undefined }), or use ReadonlyMap (React server components serialize Map). Make opened required. Alternatively, enable noUncheckedIndexedAccess for evalboard/ as a tsc gate.
  10. [Axis 2] Body re-asserts doctor.triage! instead of receiving the triage the caller already narrowed (evalboard/app/runs/[id]/task-failure-analysis.tsx:875) — const triage = doctor.triage!; (line 875) relies on the caller's separate check const hasBody = doctor.status.state === "done" && doctor.triage !== null; (line 923). The type does not carry the invariant "done ⇒ triage", which only appears as a comment at doctor.ts:139 ("Set only when the status is done and its triage file parsed."). If a future caller renders <Body> without that guard, tsc accepts it and the component crashes on triage.tasks. Fix: pass triage: DoctorTriage as a prop from the narrowed branch ({hasBody && open && doctor.triage && <Body triage={doctor.triage} .../>}), or model RunDoctor as a discriminated union on status.state. by.get(skill)! at line 334 is safe because the loop iterates by.keys(); it could use [...by.entries()] to drop the assertion.
  11. [Axis 3] Empty-state rendering of the new panel is not asserted: a done triage with zero failed rows, and rowsRun null (evalboard/app/runs/[id]/__tests__/task-failure-analysis.test.tsx:98) — Every render test uses the 5-task fixture, and no test sets tasks = [] or rowsRun = null. With an empty triage, LayerBar returns early (if (n === 0) return null;, task-failure-analysis.tsx:710). SkillFailures then shows No failures match this filter. (line 848) when no filter is applied, which is a misleading message for a clean run. The .filter(([v]) => v !== null) that hides the "rows run" stat when the run did not report it (line 698) is also never exercised from the render side. Add a test that renders {...doctor(), triage: {...triage, tasks: [], actions: [], rowsFailed: 0}}, opens the block, and asserts the digest ("0 errors"), the stats, and the skill-section text (and fix the copy if "match this filter" is not intended). Also add a case with rowsRun: null that asserts "rows run" is absent.
  12. [Axis 3] The 'never throws' test does not reach readRunDoctor's outer catch, which no test covers and which is effectively unreachable (evalboard/lib/__tests__/doctor-blob.test.ts:236) — test("never throws: a storage error is no report", ...) sets state.failWith = { statusCode: 500 }. That error is swallowed by the inner get helper in runs.ts (catch (err) { console.error(skill-doctor: fetching ${rel} failed ...) }), and load then returns null for status.json. The outer } catch (err) { console.error(readRunDoctor failed for run ${runId}:, err); return null; } (runs.ts lines 2762-2765, reported uncovered) is never entered. The test also does not assert that err was called. So the documented "never throws" contract is tested only through the inner swallow. Either make a parser or fs read throw (e.g. vi.spyOn(fs, "readFile") rejecting with a non-ENOENT error after status loads) and assert the outer path, or delete the outer try/catch if it is unreachable by construction. Assert expect(err).toHaveBeenCalled() in the existing test.
  13. [Axis 3] Test redundancy and a misleading comment in doctor-blob opened.json tests (evalboard/lib/__tests__/doctor-blob.test.ts:229) — // Cached copy from before is fresh, so drop it to read the bad one. (line 229) is followed by an fs.rm(..., { force: true }). This step does nothing: the earlier read in the same test ran after delete state.blobs[...opened.json], got a 404, and cached nothing. The comment tells a reader that a stale cached file exists when it does not. Separately, opened.json maps opened PRs by task; a bad host or a problem entry does not (lines 208-219) repeats the allowlist/problem-entry filtering already pinned by doctor.test.ts parseOpened: allowlisted PR URLs map by task; the rest is skipped (lines 287-306). Keep the blob-level test to the wiring (opened.json is fetched from <fixes_dir>/ and passed to parseOpened). Drop the redundant rm and comment.
  14. [Axis 3] The page-level wiring of the doctor panel (taskAnalysisEnabled gate → readRunDoctor, planAnalysis → AnalysisPanel/TaskFailureAnalysis) has no test (evalboard/app/runs/[id]/page.tsx:51) — The pieces are unit-tested (taskAnalysisEnabled, planAnalysis, the component), but the composition in page.tsx is not: const doctorOn = taskAnalysisEnabled(isInternal, source.id, process.env.EVALBOARD_TASK_ANALYSIS);, doctorOn ? readRunDoctor(id, source) : Promise.resolve(null), and the swap from {analysis && <AnalysisPanel markdown={analysis} />} to {plan.summary && <AnalysisPanel markdown={plan.summary} />}. Some regressions would pass every current test: passing analysis in place of plan.summary (the Findings section shows twice), or calling readRunDoctor without the gate (OSS or non-skills sources fetch doctor blobs). No test for this page exists today, so a full server-component test may be too much. A small extracted helper (for example, loadDoctorPlan(analysis, isInternal, source, env)) with a test for the off/OSS/non-skills cases would pin it.
  15. [Axis 4] The /api/file skill-doctor gate checks only the raw first path segment, so './skill-doctor/...' or 'activation/../skill-doctor/...' skips the edition check, the kill switch and the patch allowlist (evalboard/lib/runs.ts:3040) — The gate is const parts = relPath.split("/"); if (parts[0] === DOCTOR_DIR) { ... } (runs.ts:3039-3040). It runs on the raw relPath, but the file is chosen later by path.resolve(baseReal, relPath) (3074), which normalizes the path. I confirmed this with a vitest probe in both the oss and the internal edition. skill-doctor/status.json returns null as intended. But ./skill-doctor/status.json, ./skill-doctor/fixes-1/zz.patch and activation/../skill-doctor/status.json all resolve to the file through the else branch. That branch skips taskAnalysisEnabled (the internal-only check and EVALBOARD_TASK_ANALYSIS=off) and patchIds. So the statement in the comment ('Only a fix patch is served from it') and in the README ('/api/file serves only the patches fixes.json lists') is false for any file that is already cached or on disk: status, triage, fixes and opened files, patches beyond the allowlist, and every file in local mode. The impact is low: in blob mode only internal readers fill the cache, and the page already shows the same data. Fix: normalize first, e.g. const norm = path.posix.normalize(relPath), reject a result that starts with ../, and dispatch on norm.split("/"). Or, after resolving, reject any candidate inside <base>/skill-doctor/ that the DOCTOR_DIR branch did not approve. Add a regression test for ./skill-doctor/status.json. CVSS:3.1/AV:N/AC:H/PR:L/UI:N/S:U/C:L/I:N/A:N
  16. [Axis 4] ADO_RE accepts any dev.azure.com org or path, so model-written report text can render a clickable link to an attacker-owned ADO project (evalboard/lib/doctor.ts:385) — The comment at doctor.ts:382-383 says 'a URL in it is shown as a link only when it has exactly the expected shape', and RichText describes the allowlist as 'a UiPath/skills PR or a dev.azure.com build'. But const ADO_RE = /^https:\/\/dev\.azure\.com\/[^\s"'<>]*$/; accepts any organization and any path. Free text from the triage file (why, fix, evidence), which comes from a model and the run's logs, is passed to RichText. Through prompt injection in an agent's output, that text can include https://dev.azure.com/<attacker-org>/.... The board then renders it as a trusted link next to the skills PR links. status.trigger goes through BuildLink with the same check. Fix: pin ADO_RE to the real organization and the build-results path, e.g. /^https:\/\/dev\.azure\.com\/<org>\/[^/\s]+\/_build\/results\?buildId=\d+$/, as tightly as SKILLS_PR_RE pins UiPath/skills/pull/\d+. CVSS:3.1/AV:N/AC:H/PR:N/UI:R/S:U/C:L/I:N/A:N
  17. [Axis 5] A new multi-file doctor subsystem is added to the 3111-line lib/runs.ts module (evalboard/lib/runs.ts:2700) — runs.ts is already 3111 lines and mixes many concerns. This PR adds a new subsystem to it: const DOCTOR_STATUS_REFRESH_MS = 60_000; (line 2700), readRunDoctor (line 2708, about 70 lines), and a doctor branch in resolveSafePath (line 3040). That subsystem has a 5-file contract, its own refresh policy and its own logging. This follows the existing pattern (readRunAnalysis and readRunMeta live there too), so it is not a new regression. But lib/doctor.ts must stay free of Node builtins for the client, so the I/O half now lives in a different module from its contract. Put the server half in a server-only lib/doctor-read.ts (readRunDoctor plus the refresh constant), imported by runs.ts and page.tsx. Then the whole doctor feature can be found, and removed, as two lib files.
  18. [Axis 6] readRunDoctor's catch-all fetch handler also swallows its own input-validation errors, then reads the filesystem with the rejected run id (evalboard/lib/runs.ts:2713) — readRunDoctor builds const base = path.join(dir, runId, DOCTOR_DIR); (runs.ts:2713) before it validates runId. The only validation is inside ensureRunDoctorFile (assertValidId(runId, "runId") and the Invalid skill-doctor path throw, blob.ts). That throw lands in the same blanket catch as transient blob errors (runs.ts:2717-2718). The catch logs the error as 'fetching status.json failed' and execution continues to fs.readFile(path.join(base, rel), "utf-8") (runs.ts:2724) with the id that validation rejected. So a first-party guard failure is reported as a network problem, and the guard does not stop the read. No current caller reaches this. page.tsx calls readRunSummary in parallel, and its ensureRunSummary assert rejects the Promise.all. resolveSafePath checks isValidId(runId) first (runs.ts:3031). But readRunDoctor is exported with a documented 'never throws' contract, so a future caller has no protection. Fix: add if (!isValidId(runId)) return null; at the top of readRunDoctor, as resolveSafePath does. Then narrow get so it catches only fetch errors, and let a validation error from a pattern-checked relPath (a programmer error) propagate to the outer catch, where it is logged as a contract failure.
  19. [Axis 7] Internal-only doctor gate is recomputed at each caller (page and /api/file) instead of in the reader; kill switch accepts only 'off' (evalboard/lib/doctor.ts:440) — (env ?? "").trim().toLowerCase() !== "off" (doctor.ts:440) means EVALBOARD_TASK_ANALYSIS=0, =false or =disabled leaves the panel and the /api/file patch route ON without a warning. The sibling knob in the same README section uses a different form: README.md:209 says 'EVALBOARD_WARM_INTERVAL_SECONDS sets the interval; 0 turns it off'. An operator who copies that form gets no kill switch. Also, page.tsx:51-55 and runs.ts:3048-3053 both call taskAnalysisEnabled(isInternal, source.id, process.env.EVALBOARD_TASK_ANALYSIS) with the same arguments. Fix: accept the usual falsy spellings (off/0/false/no), or log a warning on an unknown value. Add a zero-argument taskAnalysisOn(sourceId) wrapper so the env read happens in one place. README.md:219 is also a 121-character line in a section that wraps at about 80 characters. Move the kill-switch sentence onto its own line.
  20. [Axis 7] RunDoctor type contract drifts from the reader: opened is optional but readRunDoctor always sets it, and rowsDiagnosed is parsed but never read (evalboard/lib/doctor.ts:144) — opened?: Record<string, string>; (doctor.ts:144) is optional, but readRunDoctor (runs.ts:2752-2763) always returns { status, triage, fixes, opened }, with opened defaulting to {}. Consumers must handle undefined for no reason, and a fixture that leaves the field out type-checks. Make the field required. Also, rowsDiagnosed: number; (doctor.ts:108), filled at doctor.ts:280 rowsDiagnosed: int(s.rows_diagnosed),, has no reader anywhere in app/ or lib/ (a grep finds only these two lines). The module header (doctor.ts:10-11) says the types 'carry only the fields the run page renders'. Remove the field, or render it in Stats.
  21. [Axis 8] Two failure counts in one panel come from different fields and can disagree (evalboard/app/runs/[id]/task-failure-analysis.tsx:145) — The header digest uses the summary count: const errors = ${triage.rowsFailed} error${...}`` (line 145). The Stats block uses the parsed row list: [triage.tasks.length, "failed rows"], (line 690). `parseTask` drops any row that has no `task_id`, and `summary.rows_failed` is an independent producer field, so the closed header and the opened body can show different failure totals for the same run. Use one source for both (prefer `triage.tasks.length`, or show both with clear labels). Add a test where `rows_failed` is not equal to `tasks.length`.
  22. [Axis 8] resolveSafePath reserves the skill-doctor/ prefix in every edition, shadowing an experiment arm named 'skill-doctor' (evalboard/lib/runs.ts:3040) — if (parts[0] === DOCTOR_DIR) { (runs.ts:3040) runs before the variant branch for all sources. When taskAnalysisEnabled is false, the function returns null. Because coder_eval does not reserve the variant id 'skill-doctor' (isValidVariantId accepts it), artifacts of a real arm with that name (on Scribe or in local mode too) can no longer be served by /api/file. Run listings are not affected: listRunIdsRemote reads only top-level prefixes, variants come from run.json, and ensureTaskDir lists ${runId}/${variant}/${task}/. Either reserve the name on the producer side (reject variant_id 'skill-doctor' in coder_eval, as 'activation' is excluded here by convention) or apply the doctor branch only to sources where the panel is on. Record the reserved name in docs/REPORT_SCHEMA.md or the run-layout note so the cross-repo producer knows about it.

What's Missing

Parallel paths:

  • 🔵 The evalboard now reserves <run>/skill-doctor/ in /api/file (runs.ts:3040), but the coder_eval producer side was not updated to match. variant_id in src/coder_eval/models/experiment.py still accepts "skill-doctor". The run-layout contract (.claude/shared/run-layout.md and its verbatim mirror plugins/coder-eval/reference/run-layout.md) and docs/REPORT_SCHEMA.md do not list skill-doctor/ (or analysis.md) as a reserved sibling of the variant dirs. (trigger: evalboard/lib/runs.ts) (restates: Axis 8: resolveSafePath reserves the skill-doctor/ prefix in every edition)

Tests:

  • 🟡 The new module ids.ts has no test file of its own. Its validators are tested only through the blob.ts re-export (lib/tests/blob.test.ts:2). No case pins "." or ".." for isValidId. No case covers the new runId sinks either: resolveSafePath("..", ".env") and readRunDoctor's path.join(dir, runId, DOCTOR_DIR) (runs.ts:2713). (trigger: evalboard/lib/ids.ts) (restates: Axis 4: isValidId accepts "." and "..")
  • 🔵 No route-level test covers app/api/file/route.ts for the new skill-doctor branch. The resolveSafePath tests (doctor-blob.test.ts:250-336) assert only null vs path. They do not pin the HTTP status: 403 for an unlisted patch or the feature off, 404 for a listed patch not on disk, and 5xx (expected) for a storage error. (trigger: evalboard/lib/runs.ts) (restates: Axis 6: /api/file doctor-patch branch turns blob failures into 403/404)

Display & mapping dicts:

  • 🔵 The LAYERS / SOLVABLE display maps (tfa:30-55) have fallbacks that no test exercises. An unknown layer shows its raw key in grey. An unknown yes-* solvable value is counted as fixable by isFixable (regex), but SOLVABLE.get returns no pill for it. As a result, the 'fixable' stat and the card pills can disagree. The fixture uses only grader/skill/env and yes-grader/yes-skill/env-fix, so no render test covers an off-map value. (trigger: evalboard/app/runs/[id]/task-failure-analysis.tsx) (restates: Axis 1: Closed doctor-contract value sets typed as free string)

Downstream consumers:

  • 🔵 The same run page now shows three failure counts from three sources: the run.json grid, the digest header (triage.rowsFailed, tfa:145) and the Stats 'failed rows' (triage.tasks.length, tfa:690). Nothing reconciles them, and no test asserts that they agree. (trigger: evalboard/app/runs/[id]/task-failure-analysis.tsx) (restates: Axis 8: Two failure counts in one panel come from different fields)
  • 🔵 splitAnalysis/planAnalysis now split analysis.md on the exact ## Findings heading and hide everything below it when a doctor report is done. That heading comes from the analysis.md template in plugins/coder-eval/skills/analyze/SKILL.md:269, which does not record that the board depends on it. No test ties producer and consumer together. If the skill renames the heading, the split stops silently and the board falls back to the full analysis plus the block, so the Findings show twice. (trigger: evalboard/lib/doctor.ts)

Daily/nightly:

  • 🔵 The PR body does not say what happens to the skills nightly run page when a run is re-graded in place (evaluate <run_dir>) or re-uploaded after the doctor ran. The triage is cached for good and is not checked against run.json's run_id or run_json_etag, but it still replaces the AI summary's Findings. The PR also does not say which pipeline owns the <run>/skill-doctor/ prefix in the nightly blob upload. For example, a re-upload with destination-delete semantics would remove it. (trigger: evalboard/app/runs/[id]/page.tsx) (restates: Axis 8: The doctor triage is not checked against the run.json it describes)

Harness & Lint Improvements

Static checks (lint / type):

  • [ce-lint] Add a TS source-scan guard, evalboard/lib/__tests__/id-validators.test.ts. It does two things. (a) It fails on any ^[\w.-]+-shaped regex literal used as a path or blob id validator outside lib/ids.ts and lib/variants.ts. Today it would flag ids.ts:7 ID_RE, plus runs.ts:2792 SERIES_KEY_RE, which copies series.ts:97. (b) It imports every exported isValid* / assertValid* from lib/ids.ts and lib/variants.ts and asserts that each one rejects ".", ".." and "...". It follows the vitest-scan pattern of lib/__tests__/no-zero-coalesce.test.ts. It claims no CE id, because the CE runner reads Python ASTs only. Blind spot: a validator written with a different character class is not seen. Prevents: A4 medium: isValidId accepts "..", so /api/file?run=..&path=.env reads <RUNS_DIR>/../.env and readRunDoctor gets a new runId-to-path sink (ids.ts:7, runs.ts:2713/3031/3074). It would also remove the need for the special case in clearRunCacheDir.
  • [ce-lint] Add a TS source scan for the blob-layer rule that only a 404 means absent. In lib/*.ts, a catch (err) block whose try calls an ensure* blob helper must reference isNotFound( or rethrow. Each exception goes in a reasoned allowlist that is keyed on the trimmed source line, as in no-zero-coalesce.test.ts. The rule is in the comment at blob.ts:211-213, and today nothing enforces it. Blind spot: a helper that wraps the ensure call one level down is not seen. Prevents: A6 medium: readRunDoctor's get (runs.ts:2717) swallows auth, network and IMDS errors, so /api/file answers 403 or 404 when storage is down, and the digest blames the contract. A6 low: the same catch-all swallows ensureRunDoctorFile's own assertValidId throw and then reads the filesystem with the rejected id. If the panel's never-throws path goes in the allowlist, its author must write down what that means for the file route.
  • Make the doctor contract types closed (type-checker tier; for evalboard this is tsc, not pyright). In lib/doctor.ts, parse layer, outcome, solvable and action kind into as const literal unions with one other bucket, in the same way as DOCTOR_STATES/isDoctorState. Make LAYERS/SOLVABLE Record<Union, …>. tsc then reports TS2367 for a misspelled === "pr-ready" / count("env") and TS2741 for a missing map entry. Also brand the allowlisted URLs: isSkillsPrUrl(u): u is SkillsPrUrl, with ExtA.href: SkillsPrUrl | AdoUrl. To keep this from regressing, add a short vitest scan that fails if a Doctor* interface field named layer/outcome/solvable/kind/state is typed string, or if ExtA's href is typed string. Prevents: A1/A2 medium: free-string fields with magic-literal matching (task-failure-analysis.tsx:56/57/195/415/589/691/693, doctor.ts:62/63/123/286), and the two different readings of solvable. A2 low: the PR-URL allowlist that only a comment guarantees (doctor.ts:387, ExtA sink at tfa:99-110).
  • Turn on noUncheckedIndexedAccess (tsc tier) through a scoped evalboard/tsconfig.strict.json that extends the base config. At first it includes only lib/doctor.ts, lib/ids.ts and app/runs/[id]/task-failure-analysis.tsx, and pnpm typecheck runs it. Grow it file by file. A repo-wide flip is not practical now: I measured 450 errors (about 112 outside tests). The doctor files have 13 (doctor.ts 8, tfa 5), so a scoped flip is cheap. With the flag on, prChecks[k] and opened[k] are typed T | undefined, so a direct index without own() must handle a missing key. Prevents: A2 low: report-keyed Record<string, …> types (doctor.ts:133/144) claim every key is present, and prototype safety depends on remembering own(). The test already indexes prChecks["example-t2"]! directly.
  • [ce-lint] Extend lib/__tests__/no-zero-coalesce.test.ts: add lib/doctor.ts to COVERED, and add a pattern for a parse helper whose numeric parameter defaults to zero (dflt = 0, (v: unknown, … = 0)). Fields parsed from an external contract must be number | null, so a missing value does not render as a measured zero. This is the TS counterpart to CE049 and CE069. Blind spot: inline typeof v === "number" ? v : 0 ternaries are not matched. Prevents: A2 medium: int() (doctor.ts:157-159) turns a missing or malformed summary.rows_failed into 0. The header then shows "0 errors", and the did-not-parse warning never fires (doctor.ts:279/280/287).
  • [ce-lint] Add a TS source scan for path-prefix dispatch on a raw path. In any lib/ function that calls path.resolve(…, relPath), fail on relPath.split("/"), relPath.startsWith( or parts[0] === unless the same function first assigns path.posix.normalize(relPath) and dispatches on the normalized value. Blind spot: the scan keys on the identifier name relPath, so a renamed variable is not seen. The runtime property test in harness_improvements covers that gap. Prevents: A4 low: the /api/file skill-doctor gate (runs.ts:3039-3040) checks the raw first segment, so ./skill-doctor/… or activation/../skill-doctor/… skips the edition check, the kill switch and the patch allowlist.
  • [ce-lint] Add a TS source scan that bans compatibility shims under evalboard/lib/, because the project is greenfield and has no backward-compatibility burden. It fails on export { a, b }; when it re-exports names imported from a sibling module, and on export const X = Y; when Y is a bare imported identifier (an alias). Exceptions go in a reasoned allowlist (for example, the lib/overview.ts re-export of taskCarriesRepoTag, if it is still needed). Prevents: A1/A5/A7 low: the blob.ts:17 re-export shim over the moved id validators, and the doctor.ts:396 alias isLinkableTaskId = isValidTaskId.
  • [ce-lint] Add a cheap TS source scan that fails on JSON.stringify(<x>).toLowerCase().includes( (and on JSON.stringify(...) inside a .filter( search predicate) under evalboard/app/. Free-text search must go over named value fields, not a serialized object that contains key names and the literal null. Prevents: A1 low: the search at task-failure-analysis.tsx:325 matches every task for "fix", "why", "layer" or "null", and opens every accordion.
  • Set up eslint with typescript-eslint in evalboard/ now. The promotion bar in .claude/harness-candidates.md was already met ("stand up eslint next time evalboard/ is touched substantively"), and this PR adds a 981-line view plus a 5-file contract. Wire it into pnpm verify, so make evalboard-verify and the evalboard CI job run it. Turn on @typescript-eslint/no-non-null-assertion (allowlist with reasons), @typescript-eslint/switch-exhaustiveness-check and @typescript-eslint/no-unnecessary-condition. Move the vitest source scans above into custom rules over time, so they parse TypeScript and do not regex lines. Prevents: A2 low: const triage = doctor.triage! (task-failure-analysis.tsx:875) re-asserts an invariant (done implies triage) that only a comment states, and by.get(skill)! (tfa:334). It also removes the regex blind spots in every TS source-scan guard proposed here.

Harness improvements (not statically reachable):

  • Add a cross-repo contract golden for skill-doctor status_v1, triage_v1 and fixes_v1. The producer repo publishes its JSON Schemas (or a golden document the producer generates) as a versioned artifact, and the evalboard CI fetches the current copy and does not vendor a stale one. A vitest parity test then (a) validates lib/__tests__/fixtures/skill-doctor/* against those schemas, and (b) asserts that every property the parsers read (fixed_by_pr, existing_prs, env_group, doctor_pr, pr_checks.*.head_sha, run.rows_executed, run.run_id, run_json_etag) exists in the schema. Also log a runtime console.warn when a known optional key is missing from a v1 document, so drift shows in the logs before the next sync. Why not static: The producer is in another repo. A schema copied into evalboard drifts silently, so the check needs the producer's current schema at CI time, which is a network or cross-repo fetch, not an AST property of this tree. Prevents: A3/A8 medium: the doctor contract is pinned only by hand-written fixtures, and a producer-side rename silently drops the Proposed PR rows or the 'rows run' stat (doctor.ts:10-11/220/259-262/274-277). It also lets the board use run_id/run_json_etag, which are sent today and ignored.
  • Add a doctor-state matrix test that iterates the DOCTOR_STATES const array (it does not list states by hand), so a new state cannot go untested. For each state, with fixes_dir set, assert the exact state.downloads list from readRunDoctor and the result of resolveSafePath(RUN, "skill-doctor/<fixes_dir>/<id>.patch"). This forces a decision on whether a retracted or failed report still serves patches. Why not static: Whether fixes and patches are fetched or served for a state depends on runtime control flow against mocked blob storage. Only an executed test sees it. Prevents: A3 medium: the state gate is tested for the triage only. A retracted, failed or skipped status that keeps fixes_dir still downloads fixes and patches and serves them through /api/file (runs.ts:2740/2753/3061-3062; doctor-blob.test.ts:131).
  • Add a path-normalization property test for resolveSafePath. For every gated prefix (skill-doctor, activation, variant dirs), assert that ./<p>/x, a/../<p>/x and <p>/./x give the same decision as the canonical <p>/x. Also assert that a dot-only run id ("..", ".") returns null, for example resolveSafePath("..", ".env") === null, in both local and blob mode. Why not static: The final file comes from path.resolve against the real filesystem and the storage mode. Only executing the resolver shows which file is selected. The source scan above catches only the relPath naming convention. Prevents: A4 medium (dot-only run id reads outside RUNS_DIR) and A4 low (the raw-segment skill-doctor gate can be bypassed).
  • Add a route-level storage-failure contract test. Call /api/file with the blob mock set to failWith: {statusCode: 500} (and also auth or IMDS errors), and assert a 5xx response, never 403 or 404. Replace the test at doctor-blob.test.ts:305-318, which now fixes the masking behavior in place. Add a digest test that asserts a transient triage-fetch failure renders as 'could not be fetched', not 'missing or did not parse'. Why not static: The HTTP status comes from runtime error propagation through readRunDoctor, resolveSafePath and route.ts, against injected storage faults. Prevents: A6 medium: storage failures show as 403 'forbidden' or 404 'not found' on the file route, and the panel blames contract drift for infra errors.
  • Add a triage-to-run consistency check, with tests. parseTriage keeps run.run_id (and the board compares run_json_etag when it can get it), and drops the triage or marks it stale on a mismatch. page.tsx intersects triage.tasks with the failing rows in run.json. Add tests for a mismatched run_id, for a triage row that now passes, and for summary.rows_failed !== tasks.length, which makes the header and the body use one count. Why not static: Staleness and count disagreement are properties of two data files at runtime (re-grade plus re-upload, or a producer writing to the wrong prefix). No source pattern shows them. Prevents: A8 medium: an unverified triage replaces the AI summary's Findings and can disagree with the grid (doctor.ts:271, page.tsx:101). A8 low: the header uses rowsFailed and the body uses tasks.length (task-failure-analysis.tsx:145/690).
  • Extract loadDoctorPlan(analysis, isInternal, source, env) from page.tsx, with a unit test for the off, OSS, non-skills and enabled cases. It asserts that readRunDoctor is not called when the panel is gated off and that plan.summary, not the raw analysis, goes to AnalysisPanel. Put the env read and the gate in this one helper, so /api/file and the page share the same predicate. Make the kill switch accept off/0/false/no, or warn on an unknown value. Why not static: Whether the panel is on depends on the edition, the source id and the env at runtime. The composition bugs (wrong prop, missing gate) are type-correct. Prevents: A3 low: no test for the page-level wiring. A7 low: the gate is recomputed at each caller, and the kill switch accepts only 'off' (doctor.ts:440, page.tsx:51-55, runs.ts:3048-3053).
  • Add empty-state and degraded-field render tests for TaskFailureAnalysis: a done triage with tasks: [], actions: [] and rowsFailed: 0; a case with rowsRun: null; and a case where readRunDoctor's outer catch is reached (a parser or fs read throws after status loads) with expect(err).toHaveBeenCalled(). If the outer catch cannot be reached, delete it. Why not static: The rendered copy (for example the misleading 'No failures match this filter.') and whether the catch can be reached are runtime and render-tree properties. Prevents: A3 low: the empty state is not asserted, and the never-throws test does not reach readRunDoctor's outer catch (task-failure-analysis.tsx:698/710/848, runs.ts:2762-2765).
  • Record the reserved run-dir name skill-doctor on the producer side. coder_eval's variant-id validator rejects skill-doctor (and activation), and .claude/shared/run-layout.md and docs/REPORT_SCHEMA.md list the reserved names. Add a pytest that asserts every name the evalboard reserves in resolveSafePath is rejected as a variant_id. Why not static: The reservation crosses a language boundary (a TS resolver and a Python validator). Keeping the two lists in step needs a parity test that reads both sides, in the same way as status-parity.test.ts. Prevents: A8 low: runs.ts:3040 shadows the artifacts of an experiment arm named 'skill-doctor' in every edition.

Top 5 Priority Actions

  1. No finding changes a task's score or final_status, so security comes first: make isValidId reject ids that are only dots (evalboard/lib/ids.ts:7, the way isValidVariantId in lib/variants.ts:25-26 does), normalize relPath before the skill-doctor dispatch (evalboard/lib/runs.ts:3039-3040) so './skill-doctor/...' and 'activation/../skill-doctor/...' cannot skip the gate, and add regression tests that resolveSafePath("..", ".env") and ./skill-doctor/status.json both return null.
  2. Stop the /api/file doctor-patch branch from turning auth, network and IMDS blob failures into 403 or 404 (evalboard/lib/runs.ts:3058 calls readRunDoctor, whose get at runs.ts:2717-2718 catches every error): let non-404 errors propagate so the route returns 5xx, and change the doctor-blob.test.ts:305-318 case to expect that.
  3. Decide whether a non-done doctor status (retracted, failed, skipped) that still names fixes_dir may fetch and serve patches, gate fixesRel and patches on the state if it may not (evalboard/lib/runs.ts:2740/2753 and 3059-3063), and pin the decision with a test.each over the non-done states that keeps fixes_dir set (evalboard/lib/tests/doctor-blob.test.ts:131).
  4. Check the triage against the run.json it describes: keep run.run_id and run_json_etag in parseTriage (evalboard/lib/doctor.ts:271) and mark the triage stale on a mismatch, so a re-graded or wrongly-placed triage does not replace the AI summary's Findings (page.tsx:101) and show wrong failure counts.
  5. Pin the cross-repo doctor contract: add a producer-published schema or golden file that CI refreshes, plus a parity test for every field the parsers read (evalboard/lib/tests/doctor.test.ts:20), and make missing or malformed counts number | null instead of a false 0 (evalboard/lib/doctor.ts:279, int() at 157-159), so producer drift fails a test or logs a warning instead of showing "0 errors".

Stats: 0 🔴 · 0 🟠 · 7 🟡 · 22 🔵 across 8 axes reviewed.

@uipreliga
uipreliga self-requested a review November 20, 2026 04:13

@uipreliga uipreliga left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fix & 🚢

- isValidId / isValidTaskId reject dot-only ids, so /api/file?run=.. can no
  longer read above the runs dir
- resolveSafePath dispatches on the normalized path: ./skill-doctor/... and
  activation/../skill-doctor/... no longer skip the skill-doctor gate
- /api/file: a storage error on a doctor patch propagates (5xx) instead of
  reading as 403/404; a patch link fetches only that patch; the page still
  never throws and says "could not be fetched"
- only a done report fetches or serves fixes (a retracted one keeps
  fixes_dir but serves nothing)
- a triage or fixes.json that names another run is ignored
- a triage with a missing or malformed rows_failed / rows_diagnosed does not
  parse (no false "0 errors"); mutant counts must be integers
- copies of the doctor's v1 schemas under the test fixtures, with a parity
  test for every field the parsers read and the closed value sets (states,
  layers, verdicts, the open-pr action kind)

Not in this round: comparing run_json_etag with run.json's etag (a run.json
re-graded in place after the doctor ran); fixes_v1 does not pin the candidate
fields covers / patch_lines / title / gates yet.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants