Skip to content

Images: lock the image before its manifests and tags, so deletes and tag writes no longer deadlock - #917

Merged
schmitch merged 1 commit into
mainfrom
claude/575-image-delete-lock-order
Nov 17, 2026
Merged

schmitch merged 1 commit into
mainfrom
claude/575-image-delete-lock-order

Conversation

@schmitch

Copy link
Copy Markdown
Member

Fixes #575

The bug

Two groups of transactions lock the same rows of an image in opposite orders:

  • Manifests, then tags: the admin manifest delete and image retention (oci_dao::lock_image_for_retention), each batch of a whole image's delete (lock_manifests), and the registry's manifest DELETE (delete_manifest).
  • Tag, then manifest: a push that moves a tag (set_tag locks the tag, then updates the manifest it points at and the one it leaves) and a tag delete from the admin API or the registry (delete_tag deletes the tag, then updates its manifest).

One of each on the same image can wait for a row the other holds. Postgres aborts one (40P01): the push or the tag delete answers 500, or retention fails its batch. The retry #905 added covers only the admin manifest and image delete, and only when Postgres aborts that side.

The remedy: one lock order, with the image first

The issue offers one lock order on every path, or a retry of 40P01. This is one lock order, but not an order of the rows, because the rows have none that all paths can follow:

  • A push learns which manifest a tag leaves only once it holds the tag, so it cannot lock manifests first without guessing and re-checking.
  • Tags first on the delete side does not work either: a push holds the manifests its index lists FOR SHARE before it reaches its tags.

So every transaction that writes an image's manifests and tags now takes a lock on the image before its first row lock on either: oci_dao::lock_image, a transaction-scoped advisory lock on the repository and the image's name, as proxy_dao::lock_cached_package already does for a cached package. Such transactions take turns per image and never wait for each other's rows.

It is taken by:

  • the push and the import (commit_manifest) and a proxy's cached manifest (store_proxied_manifest);
  • the registry's and the admin API's tag delete and the registry's manifest delete (inside oci_dao::delete_tag / delete_manifest, their first statement);
  • the admin manifest delete and retention (inside lock_image_for_retention), retention's apply transaction, and each batch of a whole image's delete;
  • set_imported_times (a manifest, then its tag).

I preferred this over a retry because a retry leaves the deadlock in place (a second of waiting each time, and a retry can lose again), and the push has no retry-safe wrapper today.

Behaviour to be aware of:

  • Writes to one image's manifests and tags are now serialized. That is the short database transaction at the end of a manifest push or a delete, not blob uploads or pulls. A retention batch or an image-delete batch holds the lock for its transaction, as it already held the image's rows.
  • During a rolling upgrade, an instance of the earlier release writes without the lock, so the old deadlock can still occur until every instance runs this. For that reason Admin image delete: keep a platform manifest an index pushed meanwhile lists #905's retry and its second read of the image stay, with their comments saying so.

Left out

Verification

Reproduced first, on main without the fix, then kept as regression tests. Postgres 18 and versitygw from mirror.gcr.io through testcontainers.

  • crates/server/tests/registry.rs, an_image_delete_and_a_tag_write_wait_for_each_other_instead_of_deadlocking: an admin manifest delete holds the image's manifests and waits at its first tag; then a real push of latest, an admin tag delete and a registry tag DELETE each start. Before the fix all three answered 500 (the push, the admin tag delete and the registry tag delete, one run each); with it they answer 201 / 200 / 202 and the delete 200.
  • crates/db/tests/oci_images.rs, a_tag_write_waits_for_an_image_locked_for_retention: the same two orders as two transactions on the DAO (lock_image_for_retention against delete_tag and against set_tag). With lock_image turned into a no-op it fails with 40P01 deadlock detected; with the lock it passes.
  • cargo test -p forge-server --test registry: 59 passed. Admin image delete: keep a platform manifest an index pushed meanwhile lists #905's first test now waits for the image's lock where it waited for the manifest row; its assertions are unchanged.
  • cargo test -p forge-db: all passed. cargo test -p forge-core --test retention_apply --test retention_worker: 9 and 22 passed.
  • cargo fmt --all --check, cargo clippy --workspace --all-targets -- -D warnings, cargo xtask changelog --check, cargo xtask docs --check, cargo xtask unused, cargo xtask doc-check: clean.
  • cargo sqlx prepare --check --workspace -- --all-targets --all-features against a migrated Postgres 18: clean (one new cached query).

Not run locally: the rest of the workspace's tests, the e2e suites and the web tests; CI runs them. Not tested: a retention run driven end to end against a concurrent tag delete (the DAO test covers the lock retention takes), and the rolling-upgrade case.

The changelog fragment is docs/changes/2.2.0/575.fixed.md; the issue moved from Backlog to the 2.2.0 milestone.


Generated by Claude Code

A manifest delete, a whole-image delete and image retention locked an
image's manifests and then its tags (oci_dao::lock_image_for_retention,
delete_manifest, lock_manifests). A push that moves a tag
(oci_dao::set_tag) and a tag delete (oci_dao::delete_tag) locked the tag
and then the manifest it points at or leaves. Two such transactions on
one image could each wait for a row the other held; Postgres aborted one
(40P01) and it answered 500, or failed its retention batch.

The rows have no order all these paths can follow: a push learns which
manifest a tag leaves only once it holds the tag. So every transaction
that writes an image's manifests and tags now takes a transaction-scoped
advisory lock on the image first (oci_dao::lock_image, as
proxy_dao::lock_cached_package does for a cached package): the push and
the import (commit_manifest), a proxy's cached manifest, the registry's
and the admin API's tag and manifest deletes, each batch of a whole
image's delete, retention's apply and set_imported_times. They take
turns per image and never wait for each other's rows.

The retry of a deadlocked admin delete and its second read of the image
(#560) stay: an instance of an earlier release pushes without the lock
during a rolling upgrade.

Fixes #575

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012e6DJbtdvmZFvSvRWFDdP2
@schmitch
schmitch added this pull request to the merge queue Nov 17, 2026
Merged via the queue into main with commit 8f7dfff Nov 17, 2026
19 checks passed
@schmitch
schmitch deleted the claude/575-image-delete-lock-order branch November 17, 2026 17:21
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Admin image delete deadlocks with a concurrent tag push (opposite lock order), and one side answers 500

2 participants