Skip to content

fix(stack): recover a failed tool's stderr from engine logs when attach drops it - #7682

Open
avallete wants to merge 3 commits into
developfrom
avallete/flaky-podman-attach-stderr
Open

avallete wants to merge 3 commits into
developfrom
avallete/flaky-podman-attach-stderr

Conversation

@avallete

Copy link
Copy Markdown
Member

TL;DR

A one-off tool container can fail while its attach session relays no stderr; the runtime now reports the container's logged stderr in that case.

  • Commands.integration.test.ts › podman runs SQL from stdin, streams a dump, and returns client failure: psql exited 3 on the early-exit stdin case but the caller saw an empty stderr. Output comes only from start --attach --interactive, so an attach session that ends before relaying stderr loses the psql error for every caller, not only the test. When the attached stderr ends empty and the process failed, the runtime now reads logs for the container (before it is removed) and relays its stderr, under a Container.loggedStderr span that records the recovered byte count. A new container integration test drops the attach stderr and checks the failure output still arrives.

ref

spotted in: https://github.com/supabase/cli/actions/runs/44064870746/job/133975841072


Generated by Claude Code

…ch drops it

A one-off container attached through `start --attach --interactive` can exit
with a failure while the attach session relays no stderr, so a psql error
reached the caller as an empty stream. When the attached stderr ends empty
and the process failed, read the container's logged stderr before it is
removed and relay that instead.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MBEk1Pbmd87VhSdE6Rb9aF
@avallete
avallete requested a review from a team as a code owner November 18, 2026 07:41
@avallete
avallete requested a review from kanadgupta November 18, 2026 07:41

Copy link
Copy Markdown
Member Author

/ai-review


Generated by Claude Code

@avallete avallete added the run-flaky-check Run the flaky-test check on this PR label Nov 18, 2026 — with Claude
@github-actions github-actions Bot removed the run-flaky-check Run the flaky-test check on this PR label Nov 18, 2026
@github-actions

Copy link
Copy Markdown
Contributor

CLI benchmark (Lightweight Commands): no changes found

Commit 732a344 against develop at 0d1a3be · 10 runs each · report only, never blocks merging

Command Workload Before After Change
supabase --version version 174 ms 172 ms no real change
supabase --help help 174 ms 178 ms no real change
supabase init init 180 ms 182 ms no real change
supabase migration new bench migration-new 186 ms 185 ms no real change
supabase config convert config-convert 174 ms 187 ms no real change (+7%, within normal variation)
supabase config validate config-validate 188 ms 189 ms no real change
supabase gen types --lang typescript --project-id replayfixtureprojref gen-types 192 ms 195 ms no real change
supabase projects list projects-list 172 ms 169 ms no real change
supabase functions list --project-ref replayfixtureprojref functions-list 186 ms 192 ms no real change

API calls: unchanged for all 9 commands.

Binary size: 135.2 MB, unchanged.

Details: timing ranges, API calls per command, slowest phases

Timing ranges

Command Workload Before range After range Change Verdict
supabase --version version 152-189 ms 162-186 ms -1 ms (-1%) within noise
supabase --help help 169-199 ms 171-195 ms +4 ms (+2%) within noise
supabase init init 176-194 ms 176-188 ms +3 ms (+1%) within noise
supabase migration new bench migration-new 167-201 ms 160-215 ms -1 ms (-1%) within noise
supabase config convert config-convert 169-194 ms 169-206 ms +13 ms (+7%) within noise
supabase config validate config-validate 174-238 ms 173-267 ms +1 ms (+1%) within noise
supabase gen types --lang typescript --project-id replayfixtureprojref gen-types 182-199 ms 187-208 ms +3 ms (+1%) within noise
supabase projects list projects-list 165-206 ms 167-219 ms -3 ms (-1%) within noise
supabase functions list --project-ref replayfixtureprojref functions-list 174-209 ms 173-246 ms +7 ms (+4%) within noise

API calls per command

Command Workload Before After Endpoint changes
supabase --version version 0 0
supabase --help help 0 0
supabase init init 0 0
supabase migration new bench migration-new 0 0
supabase config convert config-convert 0 0
supabase config validate config-validate 0 0
supabase gen types --lang typescript --project-id replayfixtureprojref gen-types 2 2
supabase projects list projects-list 1 1
supabase functions list --project-ref replayfixtureprojref functions-list 2 2

Slowest phases

No command moved beyond noise, so there are no phase deltas.

Workflow run

Comment /bench to run the lightweight benchmark (this comment) again on the latest commit, or /bench full to also run the integration-heavy benchmark, which posts in its own comment.

@github-actions github-actions Bot left a comment •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Superseded by a newer AI review

🤖 AI Review

CLI reference prose guidance: not applicable (no reference sources changed).

Verified all three findings from both independent reviews: one minor log-recovery concern and two nits are confirmed. No duplicates were identified. The log streams drain concurrently, but the timeout can still discard recovered stderr while waiting for replay completion. CLI reference prose review was not applicable.

Findings

Severity Location Category Sources Claim
🟡 MINOR packages/stack/src/runtime/Container.ts:608 error-handling claude The fallback replays the container's entire stdout and stderr under a 10-second timeout. For a failed tool with large logged stdout, replay can exceed that budget and discard all recovered stderr, including stderr already collected.
⚪ NIT packages/stack/src/runtime/Container.ts:612 correctness claude Recovered stderr is decoded as UTF-8 and re-encoded, so invalid UTF-8 bytes are replaced rather than preserved in the Uint8Array stream.
⚪ NIT packages/stack/src/runtime/Container.ts:627 observability codex The recovered-byte counter measures UTF-16 code units rather than bytes, undercounting non-ASCII stderr.

Stats

Claude findings: 2 · Codex findings: 1 · Confirmed: 3 · Refuted: 0 · Uncertain: 0


Models: claude-opus-5-5 + gpt-6.1-sol · Trigger: auto · Workflow run

This review runs once per PR. A maintainer can request another with a /ai-review comment.

Comment thread packages/stack/src/runtime/Container.ts Outdated
Comment on lines +608 to +625
const child = yield* spawner.spawn(command(["logs", name]));
const [, stderr, code] = yield* Effect.all(
[
child.stdout.pipe(Stream.runDrain),
child.stderr.pipe(Stream.decodeText, Stream.mkString),
child.exitCode,
],
{ concurrency: "unbounded" },
);
yield* Effect.annotateCurrentSpan("process.exit_code", Number(code));
return Number(code) === 0 ? stderr : "";
}),
).pipe(
Effect.timeout("10 seconds"),
Effect.tapError((cause) =>
Effect.logDebug(`Could not read logs of container ${name}`, cause),
),
Effect.orElseSucceed(() => ""),

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 MINOR · error-handling · source: claude

The fallback replays the container's entire stdout and stderr under a 10-second timeout. For a failed tool with large logged stdout, replay can exceed that budget and discard all recovered stderr, including stderr already collected.

Evidence: Container.ts:608 invokes logs without --tail or --since. Lines 609-615 concurrently drain stdout, collect stderr, and await exit. Lines 621-625 convert a timeout of that whole operation into an empty string.

Suggested fix: Bound the replay with a limit such as --tail 200, and preserve stderr already collected if the recovery operation times out.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in f7a9da5: stderr chunks are collected into a Ref as they arrive, so a timeout keeps what was already read. I left out --tail, because it counts stdout and stderr lines together and could cut the stderr we need when there is a lot of stdout.


Generated by Claude Code

Comment thread packages/stack/src/runtime/Container.ts Outdated
const [, stderr, code] = yield* Effect.all(
[
child.stdout.pipe(Stream.runDrain),
child.stderr.pipe(Stream.decodeText, Stream.mkString),

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚪ NIT · correctness · source: claude

Recovered stderr is decoded as UTF-8 and re-encoded, so invalid UTF-8 bytes are replaced rather than preserved in the Uint8Array stream.

Evidence: Container.ts:612 collects stderr through Stream.decodeText and Stream.mkString; line 646 emits new TextEncoder().encode(text). ContainerProcess.stderr is a Uint8Array stream at line 81, and the ordinary attached path forwards its byte chunks at lines 639-640.

Suggested fix: Collect and concatenate raw Uint8Array chunks, emit those bytes directly, and use their byteLength for the recovery annotation.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in f7a9da5: the raw stderr bytes are now passed through as-is, with no text decoding.


Generated by Claude Code

Comment thread packages/stack/src/runtime/Container.ts Outdated
),
Effect.orElseSucceed(() => ""),
);
yield* Effect.annotateCurrentSpan("container.stderr_recovered_bytes", text.length);

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚪ NIT · observability · source: codex

The recovered-byte counter measures UTF-16 code units rather than bytes, undercounting non-ASCII stderr.

Evidence: Container.ts:627 records text.length as container.stderr_recovered_bytes, while line 646 emits UTF-8 bytes. For example, é has text.length 1 but encodes to 2 bytes.

Suggested fix: Record the emitted Uint8Array's byteLength, reusing the encoded buffer or collecting raw bytes.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in f7a9da5: the annotation now records the byteLength of the bytes that are emitted.


Generated by Claude Code

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🤖 AI Review

CLI reference prose guidance: not applicable (no reference sources changed).

Verified all four findings as distinct minor issues: engine diagnostics can contaminate recovered stderr, recovery buffers stderr without a size cap, stdout replay can exhaust the recovery timeout, and the byte-count attribute undercounts non-ASCII output. CLI reference prose checks were not applicable.

Findings

Severity Location Category Sources Claim
🟡 MINOR packages/stack/src/runtime/Container.ts:618 error-handling claude Successful engine CLI warnings can be presented as container stderr because loggedStderr returns the logs subprocess's entire stderr without distinguishing engine diagnostics from container output.
🟡 MINOR packages/stack/src/runtime/Container.ts:612 resource-usage claude Recovery collects the complete logged stderr into an unbounded string before emitting it, so large failed-tool logs can cause excessive memory use even when the consumer otherwise retains only a bounded tail.
🟡 MINOR packages/stack/src/runtime/Container.ts:611 performance claude Recovery replays and discards the container's complete stdout history, which can exhaust the 10-second timeout and discard recovered stderr for stdout-heavy failed commands.
🟡 MINOR packages/stack/src/runtime/Container.ts:627 observability codex The recovered stderr byte count is incorrect for non-ASCII diagnostics.

Stats

Claude findings: 3 · Codex findings: 1 · Confirmed: 4 · Refuted: 0 · Uncertain: 0


Models: claude-opus-5-5 + gpt-6.1-sol · Trigger: manual · Workflow run

This review runs once per PR. A maintainer can request another with a /ai-review comment.

Comment thread packages/stack/src/runtime/Container.ts Outdated
{ concurrency: "unbounded" },
);
yield* Effect.annotateCurrentSpan("process.exit_code", Number(code));
return Number(code) === 0 ? stderr : "";

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 MINOR · error-handling · source: claude

Successful engine CLI warnings can be presented as container stderr because loggedStderr returns the logs subprocess's entire stderr without distinguishing engine diagnostics from container output.

Evidence: Container.ts:608-618 invokes logs and returns its unfiltered stderr on exit code 0; lines 645-646 emit it as container stderr. host/CommandRunner.ts:290 forwards this stream directly to input.stderr, and services/Database.ts:699 uses its trimmed contents as the error message.

Suggested fix: Suppress engine-generated warnings for the recovery invocation using supported engine options while preserving container stderr; avoid filtering arbitrary container output by warning-shaped prefixes.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not changed. Neither docker logs nor podman logs has a supported flag that silences engine-side warnings separately from container stderr, and filtering by prefix would be guesswork, as you note. This path only runs when the attach session relayed no stderr and the tool failed, so an engine warning in the message beats an empty one. A maintainer can decide whether that tradeoff is acceptable.


Generated by Claude Code

Comment thread packages/stack/src/runtime/Container.ts Outdated
const [, stderr, code] = yield* Effect.all(
[
child.stdout.pipe(Stream.runDrain),
child.stderr.pipe(Stream.decodeText, Stream.mkString),

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 MINOR · resource-usage · source: claude

Recovery collects the complete logged stderr into an unbounded string before emitting it, so large failed-tool logs can cause excessive memory use even when the consumer otherwise retains only a bounded tail.

Evidence: Container.ts:612 uses Stream.mkString without a size limit, and line 646 allocates an encoded buffer for the complete result. The ordinary run helper bounds retained text at lines 559-569, while Database.ts:685-690 bounds its downstream stderr collection only after recovery has buffered everything.

Suggested fix: Retain a bounded stderr tail during recovery, or stream recovered chunks without collecting the complete log.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 64c5ab6: recovery now keeps only the last 64 KiB of stderr chunks, which matches the run helper's bound.


Generated by Claude Code

const child = yield* spawner.spawn(command(["logs", name]));
const [, stderr, code] = yield* Effect.all(
[
child.stdout.pipe(Stream.runDrain),

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 MINOR · performance · source: claude

Recovery replays and discards the container's complete stdout history, which can exhaust the 10-second timeout and discard recovered stderr for stdout-heavy failed commands.

Evidence: Container.ts:608 invokes logs without a replay limit. Lines 609-615 wait for stdout drainage, stderr collection, and process exit together. Lines 621-625 replace the entire result with an empty string on timeout, including any stderr already collected.

Suggested fix: Bound engine-side replay, for example with a suitable logs --tail limit, or use a supported mechanism that retrieves only container stderr.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Since f7a9da5, a timeout keeps the stderr collected so far instead of dropping it. I left out --tail: it counts stdout and stderr lines together, so with a lot of stdout it could cut exactly the stderr this path exists to recover. Recovery only runs for a one-off tool that failed with no attached stderr at all.


Generated by Claude Code

Comment thread packages/stack/src/runtime/Container.ts Outdated
),
Effect.orElseSucceed(() => ""),
);
yield* Effect.annotateCurrentSpan("container.stderr_recovered_bytes", text.length);

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 MINOR · observability · source: codex

The recovered stderr byte count is incorrect for non-ASCII diagnostics.

Evidence: Container.ts:627 records container.stderr_recovered_bytes using text.length, which counts UTF-16 code units. Line 646 emits UTF-8 bytes using TextEncoder: é records 1 while emitting 2 bytes, and an emoji records 2 while emitting 4 bytes.

Suggested fix: Record the UTF-8 encoded buffer's byteLength, preferably reusing that buffer for emission.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in f7a9da5: the annotation now records the emitted buffer's byteLength.


Generated by Claude Code

@avallete avallete added the run-flaky-check Run the flaky-test check on this PR label Nov 18, 2026 — with Claude
@github-actions github-actions Bot removed the run-flaky-check Run the flaky-test check on this PR label Nov 18, 2026
@avallete avallete added the run-flaky-check Run the flaky-test check on this PR label Nov 18, 2026 — with Claude
@github-actions github-actions Bot removed the run-flaky-check Run the flaky-test check on this PR label Nov 18, 2026
@github-actions

Copy link
Copy Markdown
Contributor

Flaky test check: ⚠️ 2 flaky

Commit 63fc0cb606c7 · focused ×10 (5 executions each), 9928 tests · integration ×10, 7346 tests · unit ×10, 9942 tests · workflow run

Flaky tests (2)

Suite Test File Failed Where First failure
focused managed migration squash > docker squashes through an owned shadow stack apps/cli/src/commands/migration/squash/squash.native.integration.test.ts 1/50 8.4 Error: managed squash failed: ed3f17bf-81ae-4bc0-b908-420d159727cb failed: Engine command 'start' did not finish within 30s; container state: status=running started=2026-11-18T08:01:31.06501425Z pid=7…
focused stops its containers and exits when its whole state root is confirmed gone packages/stack/src/StackHost.container-shutdown.integration.test.ts 1/50 4.4 StackError: b07f22ed-7037-46c3-8979-372f5112d9d3 failed: Engine command 'start' did not finish within 30s; container state: status=running started=2026-11-18T08:02:06.445218204Z pid=745582 error=: Eng…

Each execution is a separate Vitest process; Where lists failing runs, as run.iteration for suites executed more than once per run. The full data is in the flaky-check-report artifact.

Copy link
Copy Markdown
Member Author

The Report flaky tests failure doesn't come from this PR. The test this PR fixes (Commands.integration.test.ts) passed in all 50 focused executions, and the new container test passed as well. Both flakes that were reported (squash.native and StackHost.container-shutdown) fail the same way: the non-attached start engine command stalls past 30s while the container is already running, about a minute apart on two different runners. This PR only changes the attached one-off stderr path, so that stall is outside its scope.


Generated by Claude Code

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants