Repository navigation
Images: lock the image before its manifests and tags, so deletes and tag writes no longer deadlock - #917
Merged
Conversation
A manifest delete, a whole-image delete and image retention locked an image's manifests and then its tags (oci_dao::lock_image_for_retention, delete_manifest, lock_manifests). A push that moves a tag (oci_dao::set_tag) and a tag delete (oci_dao::delete_tag) locked the tag and then the manifest it points at or leaves. Two such transactions on one image could each wait for a row the other held; Postgres aborted one (40P01) and it answered 500, or failed its retention batch. The rows have no order all these paths can follow: a push learns which manifest a tag leaves only once it holds the tag. So every transaction that writes an image's manifests and tags now takes a transaction-scoped advisory lock on the image first (oci_dao::lock_image, as proxy_dao::lock_cached_package does for a cached package): the push and the import (commit_manifest), a proxy's cached manifest, the registry's and the admin API's tag and manifest deletes, each batch of a whole image's delete, retention's apply and set_imported_times. They take turns per image and never wait for each other's rows. The retry of a deadlocked admin delete and its second read of the image (#560) stay: an instance of an earlier release pushes without the lock during a rolling upgrade. Fixes #575 Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012e6DJbtdvmZFvSvRWFDdP2
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #575
The bug
Two groups of transactions lock the same rows of an image in opposite orders:
oci_dao::lock_image_for_retention), each batch of a whole image's delete (lock_manifests), and the registry's manifestDELETE(delete_manifest).set_taglocks the tag, then updates the manifest it points at and the one it leaves) and a tag delete from the admin API or the registry (delete_tagdeletes the tag, then updates its manifest).One of each on the same image can wait for a row the other holds. Postgres aborts one (
40P01): the push or the tag delete answers 500, or retention fails its batch. The retry #905 added covers only the admin manifest and image delete, and only when Postgres aborts that side.The remedy: one lock order, with the image first
The issue offers one lock order on every path, or a retry of
40P01. This is one lock order, but not an order of the rows, because the rows have none that all paths can follow:FOR SHAREbefore it reaches its tags.So every transaction that writes an image's manifests and tags now takes a lock on the image before its first row lock on either:
oci_dao::lock_image, a transaction-scoped advisory lock on the repository and the image's name, asproxy_dao::lock_cached_packagealready does for a cached package. Such transactions take turns per image and never wait for each other's rows.It is taken by:
commit_manifest) and a proxy's cached manifest (store_proxied_manifest);oci_dao::delete_tag/delete_manifest, their first statement);lock_image_for_retention), retention's apply transaction, and each batch of a whole image's delete;set_imported_times(a manifest, then its tag).I preferred this over a retry because a retry leaves the deadlock in place (a second of waiting each time, and a retry can lose again), and the push has no retry-safe wrapper today.
Behaviour to be aware of:
Left out
Backlog): three multi-row statements still lock an image's tags or manifests without the image's lock (expire_tags,complete_config_summary,purge_repository). Found by reading, not reproduced, and not what this issue names.Verification
Reproduced first, on
mainwithout the fix, then kept as regression tests. Postgres 18 and versitygw frommirror.gcr.iothrough testcontainers.crates/server/tests/registry.rs,an_image_delete_and_a_tag_write_wait_for_each_other_instead_of_deadlocking: an admin manifest delete holds the image's manifests and waits at its first tag; then a real push oflatest, an admin tag delete and a registry tagDELETEeach start. Before the fix all three answered 500 (the push, the admin tag delete and the registry tag delete, one run each); with it they answer 201 / 200 / 202 and the delete 200.crates/db/tests/oci_images.rs,a_tag_write_waits_for_an_image_locked_for_retention: the same two orders as two transactions on the DAO (lock_image_for_retentionagainstdelete_tagand againstset_tag). Withlock_imageturned into a no-op it fails with40P01 deadlock detected; with the lock it passes.cargo test -p forge-server --test registry: 59 passed. Admin image delete: keep a platform manifest an index pushed meanwhile lists #905's first test now waits for the image's lock where it waited for the manifest row; its assertions are unchanged.cargo test -p forge-db: all passed.cargo test -p forge-core --test retention_apply --test retention_worker: 9 and 22 passed.cargo fmt --all --check,cargo clippy --workspace --all-targets -- -D warnings,cargo xtask changelog --check,cargo xtask docs --check,cargo xtask unused,cargo xtask doc-check: clean.cargo sqlx prepare --check --workspace -- --all-targets --all-featuresagainst a migrated Postgres 18: clean (one new cached query).Not run locally: the rest of the workspace's tests, the e2e suites and the web tests; CI runs them. Not tested: a retention run driven end to end against a concurrent tag delete (the DAO test covers the lock retention takes), and the rolling-upgrade case.
The changelog fragment is
docs/changes/2.2.0/575.fixed.md; the issue moved fromBacklogto the2.2.0milestone.Generated by Claude Code