Skip to content

ci(release): run the setup-cli smoke test daily instead of after each release - #7666

Merged
Prashansa-K merged 3 commits into
developfrom
ci/npm-fallback-cron
Nov 17, 2026
Merged

Prashansa-K merged 3 commits into
developfrom
ci/npm-fallback-cron

Conversation

@Prashansa-K

Copy link
Copy Markdown
Contributor

What

Moves the supabase/setup-cli smoke test out of the release pipeline and into a daily scheduled workflow.

Why

The post-release smoke test installed through setup-cli seconds after the npm publish, while the registry could still serve an older package list. npm then skipped the new @supabase/cli-<platform> optional dependency without failing, the npm shim could not find a binary, and setup-cli fell back to the GitHub release. Slack reported this as a setup-cli npm fallback on most releases, although it was only propagation delay and the release itself was fine.

How

  • setup-cli-smoke-test.yml runs daily. It resolves the newest version on each npm dist-tag that ships the @supabase/cli-* platform packages (latest, beta, next, v<N>.stable) and skips versions published less than an hour earlier. Each version runs the existing setup-cli v1/v2/v3 matrix on Linux, macOS, Windows and Alpine, plus the forced GitHub release fallback job.
  • Scheduled runs post to Slack when a job fails or when setup-cli v3 falls back from npm. workflow_dispatch with a version tests one release without posting.
  • The release pipeline no longer calls the smoke test: the setup-cli-smoke job, its fallback output, and the separate fallback Slack notice are removed, along with the fallback notice on success messages.
  • wait-for-npm-version.sh is removed, since tested versions are already an hour old.
  • The release process doc describes the daily check.

v2.x still runs the post-release smoke test from its own copy of the release pipeline; a backport that removes the setup-cli-smoke job there stops the same alerts on maintenance releases.

🤖 Generated with Claude Code

@Prashansa-K
Prashansa-K requested review from a team and supabase-oss as code owners November 17, 2026 11:22

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🤖 AI Review

Both independent reviews completed: Claude reported no findings and Codex reported two. Confirmed the version-selection monitoring gap; the timeout-notification finding remains uncertain because GitHub's timeout result mapping could not be verified.

Findings

Severity Location Category Sources Claim
🟡 MINOR .github/workflows/setup-cli-smoke-test.yml:58 test-coverage codex Mismatched platform dist-tags exclude supported releases from every scheduled smoke-test matrix.
🟡 MINOR .github/workflows/setup-cli-smoke-test.yml:303 error-handling codex A summary-job timeout may suppress the scheduled failure notification.

Stats

Claude findings: 0 · Codex findings: 2 · Confirmed: 1 · Refuted: 0 · Uncertain: 1


Models: claude-opus-5-5 + gpt-6.1-sol · Trigger: auto · Workflow run

This review runs once per PR. A maintainer can request another with a /ai-review comment.

@github-actions

Copy link
Copy Markdown
Contributor

CLI benchmark (Lightweight Commands): no changes found

Commit aa6a7bd against develop at 4d215b2 · 10 runs each · report only, never blocks merging

Command Workload Before After Change
supabase --version version 573 ms 553 ms no real change
supabase --help help 509 ms 456 ms no real change (-10%, within normal variation)
supabase init init 524 ms 564 ms no real change (+8%, within normal variation)
supabase migration new bench migration-new 343 ms 360 ms no real change (+5%, within normal variation)
supabase config convert config-convert 415 ms 376 ms no real change (-9%, within normal variation)
supabase config validate config-validate 319 ms 353 ms no real change (+11%, within normal variation)
supabase gen types --lang typescript --project-id replayfixtureprojref gen-types 481 ms 498 ms no real change
supabase projects list projects-list 463 ms 489 ms no real change (+6%, within normal variation)
supabase functions list --project-ref replayfixtureprojref functions-list 422 ms 448 ms no real change (+6%, within normal variation)

API calls: unchanged for all 9 commands.

Binary size: 135.2 MB, unchanged.

Details: timing ranges, API calls per command, slowest phases

Timing ranges

Command Workload Before range After range Change Verdict
supabase --version version 468-725 ms 402-812 ms -20 ms (-4%) within noise
supabase --help help 442-761 ms 406-622 ms -53 ms (-10%) within noise
supabase init init 391-664 ms 392-693 ms +41 ms (+8%) within noise
supabase migration new bench migration-new 309-553 ms 317-483 ms +17 ms (+5%) within noise
supabase config convert config-convert 313-569 ms 303-783 ms -38 ms (-9%) within noise
supabase config validate config-validate 249-537 ms 242-470 ms +34 ms (+11%) within noise
supabase gen types --lang typescript --project-id replayfixtureprojref gen-types 300-736 ms 322-677 ms +17 ms (+4%) within noise
supabase projects list projects-list 281-683 ms 288-715 ms +26 ms (+6%) within noise
supabase functions list --project-ref replayfixtureprojref functions-list 363-534 ms 344-572 ms +25 ms (+6%) within noise

API calls per command

Command Workload Before After Endpoint changes
supabase --version version 0 0
supabase --help help 0 0
supabase init init 0 0
supabase migration new bench migration-new 0 0
supabase config convert config-convert 0 0
supabase config validate config-validate 0 0
supabase gen types --lang typescript --project-id replayfixtureprojref gen-types 2 2
supabase projects list projects-list 1 1
supabase functions list --project-ref replayfixtureprojref functions-list 2 2

Slowest phases

No command moved beyond noise, so there are no phase deltas.

Workflow run

Comment /bench to run the lightweight benchmark (this comment) again on the latest commit, or /bench full to also run the integration-heavy benchmark, which posts in its own comment.

@Prashansa-K
Prashansa-K requested a review from Coly010 November 17, 2026 11:49
Prashansa-K and others added 2 commits November 17, 2026 17:20
… release

The post-release smoke test installed through setup-cli seconds after the
npm publish, while the registry could still serve an older package list.
npm then skipped the new platform package without failing, setup-cli fell
back to the GitHub release, and Slack reported a fallback that was only
propagation delay.

The smoke test now runs on a daily schedule and tests the newest version on
each npm dist-tag that ships the @supabase/cli-* platform packages, skipping
versions published less than an hour earlier. Scheduled runs post to Slack
when a job fails or setup-cli v3 falls back from npm. The release pipeline
no longer calls it, and the npm wait script it needed is removed.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…n summary timeouts

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

@Coly010 Coly010 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approving, nice cleanup. I traced the target resolution against the live dist-tags (latest, beta, next, v2.stable all resolve; hotfix is skipped as intended) and the matrix, summary and Slack paths read correctly. A few asks, none blocking:

  • If the platform filter or age cutoff leaves targets empty, every test job skips and the scheduled run goes green with no coverage. Could the scheduled path fail or alert on an empty list, or at least require latest to resolve?
  • One hour is a tight margin. A past beta's musl package took roughly 55 minutes to show up on npm, so a slow publish could still trigger a false fallback alert. Maybe 3h or more?
  • Could the Slack message name the failing version(s)? Right now you have to open the run to find out which one.
  • The docs say the daily check covers v<N>.stable, but older lines without the platform packages are skipped. Fine today since v2.stable ships them, but worth rewording.

Also worth noting somewhere that intermediate next/beta versions released within a day won't be tested, and a bad release now surfaces up to ~24h later.

Fail the daily setup-cli smoke run when no version can be tested or latest
lacks the platform packages, raise the npm age cutoff to three hours, name
the failing or falling-back versions in Slack, and document the coverage
limits of a daily check.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@Prashansa-K

Copy link
Copy Markdown
Contributor Author

Thanks for the suggestions. I have implemented all of them in the new commit.

@Prashansa-K
Prashansa-K added this pull request to the merge queue Nov 17, 2026
Merged via the queue into develop with commit 9a231c5 Nov 17, 2026
46 checks passed
@Prashansa-K
Prashansa-K deleted the ci/npm-fallback-cron branch November 17, 2026 13:43
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants