Skip to content

CLDSRV-1078 store a replica sent without microVersionId header - #6378

Merged
bert-e merged 1 commit into
development/9.4from
bugfix/CLDSRV-1078/store-replica-without-microversionid-header
Nov 19, 2026
Merged

bert-e merged 1 commit into
development/9.4from
bugfix/CLDSRV-1078/store-replica-without-microversionid-header

Conversation

@nicolas2bert

Copy link
Copy Markdown
Contributor

setup
Bidirectional replication: bucket A replicates to bucket B, and bucket B replicates to bucket A. Site B is new, site A is still old. A customer with one-way replication is not affected.

what happens

  1. A user writes an object through the old cloudserver A. The old backbeat A replicates it to the new cloudserver B. All good.
  2. The user sets a tag, an ACL, a legal hold or a retention on that object, through the old cloudserver A.
  3. The old backbeat A sends the new metadata to the new cloudserver B: PUT /_/backbeat/metadata, with x-scal-replication-content: METADATA and no x-scal-micro-version-id header
  4. The new cloudserver B returns a 500. The change is not written on site B. The old backbeat A retries, then gives up: the object stays PENDING on site A, then FAILED.

It happens for every such change, as long as site A is old and site B is new. Nothing special has to go wrong. It is the normal state between the two upgrades.

why the new cloudserver B refuses it
Bucket B replicates to bucket A. So when the new cloudserver B stores a replica, it wants to mark it "to replicate further". To do that safely it needs the x-scal-micro-version-id header: this is what stops a replication loop. An old backbeat doesn't send that header, so the new cloudserver B returns a 500.

With one-way replication, bucket B has no rule. The new cloudserver B has nowhere to send the replica, so it accepts the change.

A cascade (bucket A to bucket B to bucket C) would hit the same 500, because bucket B has a rule there too. But cascade is not supported on the old version, so no customer upgrading has that setup.

how we fix it
The new cloudserver B no longer refuses a write that has no header. It stores it as a plain replica, and it doesn't mark it to replicate further.

@bert-e

bert-e commented Nov 18, 2026

Copy link
Copy Markdown
Contributor

Hello nicolas2bert,

My role is to assist you with the merge of this
pull request. Please type @bert-e help to get information
on this process, or consult the user documentation.

Available options
name description privileged authored
/after_pull_request Wait for the given pull request id to be merged before continuing with the current one.
/bypass_author_approval Bypass the pull request author's approval ⭐
/bypass_build_status Bypass the build and test status ⭐
/bypass_commit_size Bypass the check on the size of the changeset TBA ⭐
/bypass_incompatible_branch Bypass the check on the source branch prefix ⭐
/bypass_jira_check Bypass the Jira issue check ⭐
/bypass_peer_approval Bypass the pull request peers' approval ⭐
/bypass_leader_approval Bypass the pull request leaders' approval ⭐
/bypass_source_branch_lineage Bypass the cross-branch contamination check ⭐
/approve Instruct Bert-E that the author has approved the pull request. ✍️
/create_pull_requests Allow the creation of integration pull requests.
/create_integration_branches Allow the creation of integration branches.
/no_octopus Prevent Wall-E from doing any octopus merge and use multiple consecutive merge instead
/unanimity Change review acceptance criteria from one reviewer at least to all reviewers
/wait Instruct Bert-E not to run until further notice.
Available commands
name description privileged
/help Print Bert-E's manual in the pull request.
/status Print Bert-E's current status in the pull request.
/clear Remove all comments from Bert-E from the history TBA
/retry Re-start a fresh build TBA
/build Re-start a fresh build TBA
/force_reset Delete integration branches & pull requests, and restart merge process from the beginning.
/reset Try to remove integration branches unless there are commits on them which do not appear on the source branch.

Status report is not available.

@codecov

codecov Bot commented Nov 18, 2026 •

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 86.74%. Comparing base (9a67225) to head (d50ac7a).
⚠️ Report is 1 commits behind head on development/9.4.
✅ All tests successful. No failed tests found.

Additional details and impacted files

Impacted file tree graph

Files with missing lines Coverage Δ
lib/routes/routeBackbeat.js 78.90% <100.00%> (+0.09%) ⬆️

... and 1 file with indirect coverage changes

@@                 Coverage Diff                 @@
##           development/9.4    #6378      +/-   ##
===================================================
+ Coverage            86.73%   86.74%   +0.01%     
===================================================
  Files                  213      213              
  Lines                14631    14630       -1     
===================================================
+ Hits                 12690    12691       +1     
+ Misses                1941     1939       -2     
Flag Coverage Δ
checksums-disabled-tests 35.33% <0.00%> (+<0.01%) ⬆️
file-ft-tests 70.02% <100.00%> (-0.05%) ⬇️
file-ft-tests-null-compat 70.51% <100.00%> (+<0.01%) ⬆️
kmip-ft-tests 28.07% <0.00%> (+<0.01%) ⬆️
mongo-v0-ft-tests 71.15% <100.00%> (-0.05%) ⬇️
mongo-v1-ft-tests 71.18% <100.00%> (+0.03%) ⬆️
multiple-backend 36.11% <100.00%> (+<0.01%) ⬆️
s3c-ft-tests-v0 65.10% <100.00%> (+<0.01%) ⬆️
s3c-ft-tests-v0-null-compat 65.14% <100.00%> (-0.01%) ⬇️
s3c-ft-tests-v1 65.08% <100.00%> (+0.01%) ⬆️
sur-tests 36.60% <0.00%> (-0.03%) ⬇️
sur-tests-inflights 39.46% <0.00%> (+0.03%) ⬆️
unit 74.87% <100.00%> (+0.01%) ⬆️
utapi-v2-tests 35.30% <0.00%> (+<0.01%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

🚀 New features to boost your workflow:
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@bert-e

bert-e commented Nov 18, 2026

Copy link
Copy Markdown
Contributor

Incorrect fix version

The Fix Version/s in issue CLDSRV-1078 contains:

  • None

Considering where you are trying to merge, I ignored possible hotfix versions and I expected to find:

  • 9.4.17

  • 9.5.7

  • 9.6.0

Please check the Fix Version/s of CLDSRV-1078, or the target
branch of this pull request.

Comment on lines +847 to +857
// We replicate further only when the write carries a
// microVersionId, because this is what stops a replication loop.
// A write has no x-scal-micro-version-id header only during an
// upgrade: backbeat sends this header since 9.5.0, when cascaded
// replication was added. An older backbeat does not send it.
// In that case we store the object as a replica and we don't
// replicate it further (CLDSRV-1078).
// Example: site A is not upgraded yet. Its backbeat sends a new
// tag for the object key0, with no header. We store the tag on
// our key0. Our bucket replicates to site A, but we don't send
// key0 back there.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
// We replicate further only when the write carries a
// microVersionId, because this is what stops a replication loop.
// A write has no x-scal-micro-version-id header only during an
// upgrade: backbeat sends this header since 9.5.0, when cascaded
// replication was added. An older backbeat does not send it.
// In that case we store the object as a replica and we don't
// replicate it further (CLDSRV-1078).
// Example: site A is not upgraded yet. Its backbeat sends a new
// tag for the object key0, with no header. We store the tag on
// our key0. Our bucket replicates to site A, but we don't send
// key0 back there.
// Forward only with x-scal-micro-version-id: it prevents replication loops.
// Backbeat < 9.5 doesn't send it (mid-upgrade): store as replica, don't forward (CLDSRV-1078).

Comment on lines +521 to +524
// We send the tag t=1 for key0, like backbeat does when a tag is set on
// the source object. microVersionId is the one of the source object.
// When it is undefined we send no x-scal-micro-version-id header, like
// a backbeat older than 9.5.0.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
// We send the tag t=1 for key0, like backbeat does when a tag is set on
// the source object. microVersionId is the one of the source object.
// When it is undefined we send no x-scal-micro-version-id header, like
// a backbeat older than 9.5.0.
// Tag update as sent by backbeat. No microVersionId: no header, like backbeat < 9.5.

@nicolas2bert

Copy link
Copy Markdown
Contributor Author

@bert-e approve

@bert-e

bert-e commented Nov 18, 2026

Copy link
Copy Markdown
Contributor

Incorrect fix version

The Fix Version/s in issue CLDSRV-1078 contains:

  • None

Considering where you are trying to merge, I ignored possible hotfix versions and I expected to find:

  • 9.4.17

  • 9.5.7

  • 9.6.0

Please check the Fix Version/s of CLDSRV-1078, or the target
branch of this pull request.

The following options are set: approve

@nicolas2bert

Copy link
Copy Markdown
Contributor Author

ping

@bert-e

bert-e commented Nov 18, 2026

Copy link
Copy Markdown
Contributor

Integration data created

I have created the integration data for the additional destination branches.

The following branches will NOT be impacted:

  • development/7.10
  • development/7.4
  • development/7.70
  • development/8.8
  • development/9.0
  • development/9.1
  • development/9.2
  • development/9.3

You can set option create_pull_requests if you need me to create
integration pull requests in addition to integration branches, with:

@bert-e create_pull_requests

The following options are set: approve

@bert-e

bert-e commented Nov 18, 2026

Copy link
Copy Markdown
Contributor

Build failed

The build for commit did not succeed in branch w/9.6/bugfix/CLDSRV-1078/store-replica-without-microversionid-header

The following options are set: approve

@bert-e

bert-e commented Nov 18, 2026

Copy link
Copy Markdown
Contributor

Build failed

The build for commit did not succeed in branch w/9.5/bugfix/CLDSRV-1078/store-replica-without-microversionid-header

The following options are set: approve

A backbeat older than 9.5 sends no x-scal-micro-version-id header.
putMetadata answered a 500 to its metadata-only updates when the
destination bucket replicates further (bidirectional replication), so
tags, ACL, legal hold and retention set on a 9.5 site never reached a
10.1 site (S3C-11679).

- store such a write as a plain replica instead of refusing it
- don't replicate it further: without a microVersionId nothing can
  stop a replication loop
@nicolas2bert
nicolas2bert force-pushed the bugfix/CLDSRV-1078/store-replica-without-microversionid-header branch from 0574080 to d50ac7a Compare November 19, 2026 10:16
@nicolas2bert

Copy link
Copy Markdown
Contributor Author

/reset

@bert-e

bert-e commented Nov 19, 2026

Copy link
Copy Markdown
Contributor

History mismatch

Merge commit #35305c8a9fbdf13b9d23bf637bc3d4bbe2214dc7 on the integration branch
w/9.5/bugfix/CLDSRV-1078/store-replica-without-microversionid-header is merging a branch which is neither the current
branch bugfix/CLDSRV-1078/store-replica-without-microversionid-header nor the development branch
development/9.5.

It is likely due to a rebase of the branch bugfix/CLDSRV-1078/store-replica-without-microversionid-header and the
merge is not possible until all related w/* branches are deleted or updated.

Please use the reset command to have me reinitialize these branches.

The following options are set: approve

// tag for the object key0, with no header. We store the tag on
// our key0. Our bucket replicates to site A, but we don't send
// key0 back there.
const forward = hasNextHop && hasMicroVersionId;

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

When hasNextHop && !hasMicroVersionId, the next hop is now skipped with no log line, so during a mixed-version upgrade nobody can tell why a replica was not sent on. Add a log so this case can be seen.

Suggested change
const forward = hasNextHop && hasMicroVersionId;
const forward = hasNextHop && hasMicroVersionId;
if (hasNextHop && !hasMicroVersionId) {
log.info('putMetadata: no x-scal-micro-version-id, storing replica without next hop', {
bucketName,
objectKey,
});
}

@nicolas2bert

Copy link
Copy Markdown
Contributor Author

ping

@nicolas2bert

Copy link
Copy Markdown
Contributor Author

/status

@bert-e

bert-e commented Nov 19, 2026

Copy link
Copy Markdown
Contributor

Status

check status
➡️ Approvals ☀️
➡️ Integration builds ☀️
➡️ Fix versions ☀️
➡️ History ☀️

The following options are set: approve

@bert-e

bert-e commented Nov 19, 2026

Copy link
Copy Markdown
Contributor

History mismatch

Merge commit #35305c8a9fbdf13b9d23bf637bc3d4bbe2214dc7 on the integration branch
w/9.5/bugfix/CLDSRV-1078/store-replica-without-microversionid-header is merging a branch which is neither the current
branch bugfix/CLDSRV-1078/store-replica-without-microversionid-header nor the development branch
development/9.5.

It is likely due to a rebase of the branch bugfix/CLDSRV-1078/store-replica-without-microversionid-header and the
merge is not possible until all related w/* branches are deleted or updated.

Please use the reset command to have me reinitialize these branches.

The following options are set: approve

@nicolas2bert

Copy link
Copy Markdown
Contributor Author

@bert-e approve

@nicolas2bert

Copy link
Copy Markdown
Contributor Author

/reset

@bert-e

bert-e commented Nov 19, 2026

Copy link
Copy Markdown
Contributor

Reset complete

I have successfully deleted this pull request's integration branches.

The following options are set: approve

@bert-e

bert-e commented Nov 19, 2026

Copy link
Copy Markdown
Contributor

I have successfully merged the changeset of this pull request
into targetted development branches:

  • ✔️ development/9.4

  • ✔️ development/9.5

  • ✔️ development/9.6

The following branches have NOT changed:

  • development/7.10
  • development/7.4
  • development/7.70
  • development/8.8
  • development/9.0
  • development/9.1
  • development/9.2
  • development/9.3

This pull request did not target the following hotfix branch(es) so they
were left untouched:

  • hotfix/7.70.11
  • hotfix/9.0.32
  • hotfix/9.2.36
  • hotfix/7.10.30
  • hotfix/7.10.3
  • hotfix/7.4.9
  • hotfix/7.8.0
  • hotfix/7.9.0
  • hotfix/7.10.49
  • hotfix/7.4.8
  • hotfix/7.4.3
  • hotfix/7.4.10
  • hotfix/7.70.51
  • hotfix/6.4.7
  • hotfix/7.10.15
  • hotfix/7.10.2
  • hotfix/7.10.4
  • hotfix/7.4.1
  • hotfix/7.70.45
  • hotfix/7.7.0
  • hotfix/7.4.4
  • hotfix/9.3.13
  • hotfix/7.10.0
  • hotfix/7.4.6
  • hotfix/9.2.24
  • hotfix/7.4.7
  • hotfix/9.0.7
  • hotfix/7.4.2
  • hotfix/8.8.45
  • hotfix/7.2.0
  • hotfix/7.10.28
  • hotfix/7.4.0
  • hotfix/7.6.0
  • hotfix/7.70.73
  • hotfix/7.10.1
  • hotfix/7.70.21
  • hotfix/7.4.5
  • hotfix/7.10.27
  • hotfix/7.10.8

Please check the status of the associated issue CLDSRV-1078.

Goodbye nicolas2bert.

The following options are set: approve

@bert-e
bert-e merged commit d50ac7a into development/9.4 Nov 19, 2026
36 of 37 checks passed
@bert-e
bert-e deleted the bugfix/CLDSRV-1078/store-replica-without-microversionid-header branch November 19, 2026 13:03
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

6 participants