Repository navigation
ci(release): run the setup-cli smoke test daily instead of after each release - #7666
Conversation
There was a problem hiding this comment.
🤖 AI Review
Both independent reviews completed: Claude reported no findings and Codex reported two. Confirmed the version-selection monitoring gap; the timeout-notification finding remains uncertain because GitHub's timeout result mapping could not be verified.
Findings
| Severity | Location | Category | Sources | Claim |
|---|---|---|---|---|
| 🟡 MINOR | .github/workflows/setup-cli-smoke-test.yml:58 |
test-coverage |
codex | Mismatched platform dist-tags exclude supported releases from every scheduled smoke-test matrix. |
| 🟡 MINOR | .github/workflows/setup-cli-smoke-test.yml:303 |
error-handling |
codex | A summary-job timeout may suppress the scheduled failure notification. |
Stats
Claude findings: 0 · Codex findings: 2 · Confirmed: 1 · Refuted: 0 · Uncertain: 1
Models: claude-opus-5-5 + gpt-6.1-sol · Trigger: auto · Workflow run
This review runs once per PR. A maintainer can request another with a /ai-review comment.
CLI benchmark (Lightweight Commands): no changes foundCommit
API calls: unchanged for all 9 commands. Binary size: 135.2 MB, unchanged. Details: timing ranges, API calls per command, slowest phasesTiming ranges
API calls per command
Slowest phasesNo command moved beyond noise, so there are no phase deltas. Comment |
… release The post-release smoke test installed through setup-cli seconds after the npm publish, while the registry could still serve an older package list. npm then skipped the new platform package without failing, setup-cli fell back to the GitHub release, and Slack reported a fallback that was only propagation delay. The smoke test now runs on a daily schedule and tests the newest version on each npm dist-tag that ships the @supabase/cli-* platform packages, skipping versions published less than an hour earlier. Scheduled runs post to Slack when a job fails or setup-cli v3 falls back from npm. The release pipeline no longer calls it, and the npm wait script it needed is removed. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…n summary timeouts Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
aa6a7bd to
853d20a
Compare
Coly010
left a comment
There was a problem hiding this comment.
Approving, nice cleanup. I traced the target resolution against the live dist-tags (latest, beta, next, v2.stable all resolve; hotfix is skipped as intended) and the matrix, summary and Slack paths read correctly. A few asks, none blocking:
- If the platform filter or age cutoff leaves
targetsempty, every test job skips and the scheduled run goes green with no coverage. Could the scheduled path fail or alert on an empty list, or at least requirelatestto resolve? - One hour is a tight margin. A past beta's musl package took roughly 55 minutes to show up on npm, so a slow publish could still trigger a false fallback alert. Maybe 3h or more?
- Could the Slack message name the failing version(s)? Right now you have to open the run to find out which one.
- The docs say the daily check covers
v<N>.stable, but older lines without the platform packages are skipped. Fine today sincev2.stableships them, but worth rewording.
Also worth noting somewhere that intermediate next/beta versions released within a day won't be tested, and a bad release now surfaces up to ~24h later.
Fail the daily setup-cli smoke run when no version can be tested or latest lacks the platform packages, raise the npm age cutoff to three hours, name the failing or falling-back versions in Slack, and document the coverage limits of a daily check. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
Thanks for the suggestions. I have implemented all of them in the new commit. |
What
Moves the
supabase/setup-clismoke test out of the release pipeline and into a daily scheduled workflow.Why
The post-release smoke test installed through setup-cli seconds after the npm publish, while the registry could still serve an older package list. npm then skipped the new
@supabase/cli-<platform>optional dependency without failing, the npm shim could not find a binary, and setup-cli fell back to the GitHub release. Slack reported this as a setup-cli npm fallback on most releases, although it was only propagation delay and the release itself was fine.How
setup-cli-smoke-test.ymlruns daily. It resolves the newest version on each npm dist-tag that ships the@supabase/cli-*platform packages (latest,beta,next,v<N>.stable) and skips versions published less than an hour earlier. Each version runs the existing setup-cli v1/v2/v3 matrix on Linux, macOS, Windows and Alpine, plus the forced GitHub release fallback job.workflow_dispatchwith aversiontests one release without posting.setup-cli-smokejob, its fallback output, and the separate fallback Slack notice are removed, along with the fallback notice on success messages.wait-for-npm-version.shis removed, since tested versions are already an hour old.v2.xstill runs the post-release smoke test from its own copy of the release pipeline; a backport that removes thesetup-cli-smokejob there stops the same alerts on maintenance releases.🤖 Generated with Claude Code