Skip to content

fix(daemon): shut down detached daemons via RPC before OS signals - #4659

Open
GaspardIV wants to merge 2 commits into
callstack:mainfrom
GaspardIV:fix/windows-graceful-daemon-stop
Open

GaspardIV wants to merge 2 commits into
callstack:mainfrom
GaspardIV:fix/windows-graceful-daemon-stop

Conversation

@GaspardIV

@GaspardIV GaspardIV commented Nov 17, 2026 •

Copy link
Copy Markdown

Summary

  • Add fenced daemon_shutdown RPC admission (packages/daemon/src/sessions/server/daemon-idle-reap.ts) that reuses quiesce auth, publicNetworkOnly, pid/processStartTime fence, embedded refusal, and startingUp guard while draining active sessions and retained leases.
  • Add 'shutdown' to DAEMON_CAPABILITIES (packages/daemon-contracts/src/daemon-resolution.ts) with a compatibleChanges entry in test/wire-compat/ledger.json.
  • Wire createGracefulDaemonShutdown through stopRecordedDaemon (parsing raw DaemonStopIdentity.record via parseDaemonInfo), cleanupDaemonAfterRequest, and readReusableLocalDaemon -> stopDaemonForTakeover, mapping timeout to 'accepted' so slow drains wait termTimeoutMs before SIGKILL (SIGTERM Win32 fallback marked forced).
  • Addresses the Windows daemon cleanup issue tracked in [browserstack] ENGINE_FAILURE (cleanup) on iOS: agent-device fails to send DELETE /session and times out waiting for graceful shutdown report e2e-inc/e2e#1723 (where SIGTERM terminates the process before retained leases finalize).

Validation

Verified locally on Windows 11 (b04b6bb31):

  • pnpm build, pnpm check:quick, scripts/layering/check.ts, scripts/di-seams/check.ts, pnpm check:production-exports, scripts/wire-compat/run.ts: passed.
  • smoke-daemon-http.test.ts + smoke-daemon-clean.test.ts: 4/4 passed.
  • Focused Vitest (16 daemon/contracts/client/CLI/host files): 178/178 passed; local WebDriver mock E2E + old-daemon compat passed.
  • Awaiting fork CI approval for Linux/macOS suites, pnpm check:affected --run, and native device lanes (3 unrelated daemon test files fail locally on Windows due to pre-existing POSIX chmod/SIGTERM/quoting assumptions, and win32 branches rely on mocks where noted).

@thymikee

Copy link
Copy Markdown
Member

I reviewed 6e1b790 and found one change to make before merge. The RPC-first stop route itself looks well owned.

readStopPort and readStopEndpoint in daemon-registration.ts re-implement the daemon.json endpoint parser that already lives in daemon-client-metadata.ts. That covers the token check, the both-ports-absent rule, the 1..65535 port check and the transport/capabilities reads. stopRecordedDaemon then rebuilds a DaemonInfo by hand from the second parse. If a new transport field or a stricter token rule lands only in parseDaemonInfo, the stop route will send daemon_shutdown to an endpoint the rest of the client rejects, or skip one it accepts. The rule to satisfy: one parser turns a daemon.json record into a connectable DaemonInfo. Please keep DaemonStopIdentity carrying only the record that readDaemonRegistration already read, and let daemon-client parse it with the existing parseDaemonInfo. That means exporting a record-taking variant, or moving the parser into daemon-contracts. Then DaemonStopEndpoint, readStopEndpoint and readStopPort can go, which removes about 45 production lines. Would it also work to build the hook in one daemon-client helper shared by stop, cleanup and takeover, so every graceful detached retirement tries the RPC first?

Not blocking, take or leave: DaemonStopEndpoint sits between the JSDoc of DaemonStopIdentity and its type, so that type loses its docs; the remote refusal in leave() still says "Daemon quiesce is not available to remote clients" although it now serves daemon_shutdown too; a timeout result maps to unavailable and goes straight to SIGTERM, so a daemon that admitted the shutdown but answered after 2 s can be killed mid-drain on Windows, and waiting the term budget for exit first (as for accepted) would close that; and the graceful takeover path and the startup retire call stopAndRetireDaemon without the hook, so a graceful Windows takeover of a daemon advertising shutdown still uses TerminateProcess and now reports mode forced.

The only reported check was skipped, so nothing yet attributes a pass or failure to this head. The full CI, including the Linux/macOS suites, check:affected, wire-compat and layering, has not run here, and the diff touches the daemon stop and lifecycle route that the integration smoke tests exercise. The TerminateProcess behavior and the WebDriver-mock before/after table rest on your report. The win32 branches are covered only by a process.platform mock. Also, e2e-inc/e2e#1723 is in another repository, and #1723 here is an unrelated PR. Before merge, please fold the duplicate parser into parseDaemonInfo, then a maintainer needs to approve the fork CI run so the full suites and the wire-compat and layering gates run on the new head.

@GaspardIV

Copy link
Copy Markdown
Author

Thanks @thymikee for the review — addressed the blocking issue and the follow-ups in b04b6bb31:

  1. Deduplicated daemon.json endpoint parsing (blocking)

    • Removed DaemonStopEndpoint, readStopPort, and readStopEndpoint from packages/daemon-contracts/src/daemon-registration.ts (readDaemonStopIdentity now attaches only the raw record already read by readDaemonRegistration, restoring the JSDoc placement on DaemonStopIdentity).
    • Exported parseDaemonInfo in packages/daemon-client/src/daemon-client-metadata.ts and switched stopRecordedDaemon (packages/daemon-client/src/daemon-client-stop.ts) to parse record.record through parseDaemonInfo directly instead of reconstructing DaemonInfo by hand.
    • Updated packages/daemon-contracts/src/daemon-registration.test.ts to assert raw record handling.
  2. Shared graceful-shutdown hook for stop, cleanup, and takeover (and why host is optional)

    • Widened createGracefulDaemonShutdown(info, target, { mode, timeoutMs }) in packages/daemon-client/src/daemon-client-lifecycle-request.ts to accept DaemonInfo | null | undefined and mode?: 'graceful' | 'force', making it the single hook builder shared by stopRecordedDaemon, cleanupDaemonAfterRequest, and stopDaemonForTakeover.
    • Wired stopDaemonForTakeover and retireDaemonForTakeover (packages/daemon-client/src/daemon-client-startup.ts) to accept an optional host?: HostEnvironment parameter, passed explicitly from readReusableLocalDaemon (settings.host) so 'graceful' takeover sends daemon_shutdown RPC first when host is present.
    • Why host is optional rather than switching target to required { paths, host } everywhere:
      1. Layering (R86 process-globals): daemon-client-startup.ts is an internal client module rather than a declared process entry, so calling captureHostEnvironment(process) there is rejected by scripts/layering/check.ts — the caller must supply host.
      2. replaceUnreachable in daemon-client-ensure.ts: replaceUnreachable only runs after observeLocalDaemon(..., { refineUnreachable: true }) has retried 'auto' across all advertised transports and found the daemon unresponsive (replaceIncompatible -> daemon_quiesce handles daemons reachable on an alternate transport). Keeping host optional avoids altering daemon-client-ensure.ts ('force' callers and replaceUnreachable, where sending daemon_shutdown + waiting on timeout to a TCP-hung unresponsive process would add 2 s + 3 s delay) and keeps existing non-RPC takeover test callers intact.
    • Added unit coverage in packages/daemon-client/src/__tests__/daemon-client-stop.test.ts verifying stopDaemonForTakeover sends daemon_shutdown RPC in 'graceful' mode (no signals) and skips it in 'force' mode.
  3. Timeout -> exit wait & remote refusal wording

    • createGracefulDaemonShutdown now returns 'accepted' on result.kind === 'timeout' so slow daemons draining past the 2 s RPC timeout still get the termTimeoutMs exit wait instead of immediate Win32 SIGTERM / TerminateProcess (added an HTTP slow-drain timeout test in daemon-client-stop.test.ts).
    • Updated the publicNetworkOnly error in packages/daemon/src/sessions/server/daemon-idle-reap.ts to report Daemon ${options.requireIdle ? 'quiesce' : 'shutdown'} is not available to remote clients. and added explicit message assertions for both in packages/daemon/src/server/http/daemon-runtime-quiesce.test.ts.
  4. PR cross-reference & Windows checks on b04b6bb31

    • Updated the PR summary to use the full link https://github.com/e2e-inc/e2e/issues/1723 so it isn't confused with callstack/agent-device#1723.
    • Re-ran oxfmt --check, pnpm build, pnpm check:quick, scripts/layering/check.ts, scripts/di-seams/check.ts, pnpm check:production-exports, scripts/wire-compat/run.ts, smoke-daemon-http.test.ts + smoke-daemon-clean.test.ts (4/4), and 16 focused daemon/contracts/client/CLI/host Vitest files (178/178 passed on Windows 11).

Could a maintainer approve the fork CI run for b04b6bb31 so the full Linux/macOS suites, check:affected, and remaining gates run?

@thymikee

Copy link
Copy Markdown
Member

This PR is ready at b04b6bb. The findings from the earlier review (6e1b790, #4659 (comment)) are fixed, and I found nothing new that needs a change.

Not blocking, and you can take or leave these: readReusableLocalDaemon passes settings.host for every non-reuse decision, including an unreachable daemon, so a TCP-hung daemon waits out the 2s RPC timeout and the 3s term window before SIGKILL (https://github.com/callstack/agent-device/blob/b04b6bb/packages/daemon-client/src/daemon-client-startup.ts#L146); passing host only when decision.cause is not unreachable would match the rule in your stopDaemonForTakeover comment. The new mode option and its force guard duplicate stopDaemonProcess, which never calls the hook in force mode (https://github.com/callstack/agent-device/blob/b04b6bb/packages/daemon-client/src/daemon-client-lifecycle-request.ts#L127), so the force half of the takeover test passes without the guard. The takeover test calls stopDaemonForTakeover directly, so deleting settings.host in the startup path keeps every test green (https://github.com/callstack/agent-device/blob/b04b6bb/packages/daemon-client/src/__tests__/daemon-client-stop.test.ts#L478); driving it through readReusableLocalDaemon with an incompatible-version mock daemon would close that gap.

No checks have reported on b04b6bb yet because fork CI is waiting for maintainer approval. The new tests, wire-compat and layering gates have not run yet, and the Windows results rest on your report. A maintainer needs to approve the CI run so the full suites, wire-compat and layering gates can run before merge. There are no conflicts.

@thymikee thymikee added the ready-for-human Valid work that needs human implementation, judgment, or maintainer merge label Nov 18, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ready-for-human Valid work that needs human implementation, judgment, or maintainer merge

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants