Add a D/R mode to the mongo-processor by delthas · Pull Request #2833 · scality/backbeat

delthas · 2026-08-27T12:41:18Z

The D/R metadata sink reuses the mongo-processor (ZKOP-562). Ingestion and D/R disagree about what to do with an object's metadata, so those decisions move behind a MetadataPolicy, picked by extensions.mongoProcessor.mode: ingestion, the default, carries today's behaviour verbatim; dr selects PullReplicationMetadataPolicy.

The processor calls the policy hooks, drawn as hexagons, at these points:

flowchart TD
    E["entry from Kafka"] --> T{"put or delete?"}

    T -->|put| S1{{"skipsMetadataFetch"}}
    S1 -->|false| S2{{"targetVersionId"}}
    S2 --> R1["read the stored document"]
    S1 -->|true| A{{"apply"}}
    R1 --> A
    A -->|null| K1["skip"]
    A -->|"content, versionId"| RI["resolve replicationInfo<br/>against this site's bucket"]
    RI --> W["write under versionId and repair the master,<br/>or write the master alone"]

    T -->|delete| D1{{"targetVersionId"}}
    D1 --> R2["read the stored document"]
    R2 -->|missing| K2["skip"]
    R2 --> D2{{"skipsDelete"}}
    D2 -->|true| K2
    D2 -->|false| DEL["delete the version and repair the master,<br/>or delete the master alone"]

	ingestion	dr
stored document read	only with a scal header or an enabled replication rule	always
version read, written and deleted	the scal version when the header is set, the entry's own otherwise; an entry marked null is the master; a restored object whose version is not stored is written under its own id	the document the key names: a version by its id, null versions included, and a master as the master document itself
first write	owner, location and data part made local; ACLs reset	as sent; ACLs reset
update	stored document kept, tags taken	stored document kept, tags and object-lock state taken, always written
object overwritten in place	merged	replaced, when its modification date differs from the stored one
delete	skipped when the object moved location	always applied

"Overwritten in place" is an object with no version of its own, or the master a suspended bucket marks null. An overwrite moves the modification date, which a tag, retention, legal hold or restore update keeps. microVersionId cannot tell an overwrite from an update: tagging, retention, legal hold, ACL and restore completion bump it, and all of those merge, while a plain PUT never sets it.

The stored placement is kept unless the stored location is still on the remote site (isCRR), which is groundwork for clean room. On a deployed PRA sink locationConfig.json is {}, so that branch never fires.

Commits, in order: no health-check whitelist without a server section, as a D/R sink serves no API; two small fixes; the policy, which moves ingestion's behaviour unchanged; the D/R policy; then the overwrite rule; and a none log source for a process that runs no queue populator, so the sink's queuePopulator section needs nothing beside its mongodb client. The sink reads its mongodb client from queuePopulator.mongo and its service account from extensions.ingestion.auth, as the rest of backbeat does; zenko-operator#631 writes those sections.

Known limits:

The sink does not reconcile a restore it performed while replicating. A replaced or deleted object strands the restored data, since nothing garbage-collects it on the sink yet. PRA users are told not to use the sink site while it replicates.
replicationInfo is reset on every write: the source pipeline strips the replication configuration from sink buckets.

Follow-ups: data GC on the sink (BB-813, BB-815), circuit breaking (BB-822), __metastore and Vault entities (OS-1110), transitionInProgress (BB-819), a real locationConfig.json on the sink (ZKOP-566).

Goes with scality/zenko-operator#631, which keys every entry by the document it comes from.

Issue: BB-811

bert-e · 2026-08-27T12:41:21Z

Hello delthas,

My role is to assist you with the merge of this
pull request. Please type @bert-e help to get information
on this process, or consult the user documentation.

Available options

name	description	privileged	authored
`/after_pull_request`	Wait for the given pull request id to be merged before continuing with the current one.
`/bypass_author_approval`	Bypass the pull request author's approval	⭐
`/bypass_build_status`	Bypass the build and test status	⭐
`/bypass_commit_size`	Bypass the check on the size of the changeset `TBA`	⭐
`/bypass_incompatible_branch`	Bypass the check on the source branch prefix	⭐
`/bypass_jira_check`	Bypass the Jira issue check	⭐
`/bypass_peer_approval`	Bypass the pull request peers' approval	⭐
`/bypass_leader_approval`	Bypass the pull request leaders' approval	⭐
`/approve`	Instruct Bert-E that the author has approved the pull request.		✍️
`/create_pull_requests`	Allow the creation of integration pull requests.
`/create_integration_branches`	Allow the creation of integration branches.
`/no_octopus`	Prevent Wall-E from doing any octopus merge and use multiple consecutive merge instead
`/unanimity`	Change review acceptance criteria from `one reviewer at least` to `all reviewers`
`/wait`	Instruct Bert-E not to run until further notice.

Available commands

name	description	privileged
`/help`	Print Bert-E's manual in the pull request.
`/status`	Print Bert-E's current status in the pull request.
`/clear`	Remove all comments from Bert-E from the history `TBA`
`/retry`	Re-start a fresh build `TBA`
`/build`	Re-start a fresh build `TBA`
`/force_reset`	Delete integration branches & pull requests, and restart merge process from the beginning.
`/reset`	Try to remove integration branches unless there are commits on them which do not appear on the source branch.

Status report is not available.

codecov · 2026-08-27T12:41:58Z

Codecov Report

❌ Patch coverage is 99.13043% with 1 line in your changes missing coverage. Please review.
✅ Project coverage is 76.76%. Comparing base (11f56c2) to head (d558470).
⚠️ Report is 8 commits behind head on development/9.6.

Files with missing lines	Patch %	Lines
...rocessor/metadataPolicy/IngestionMetadataPolicy.js	98.00%	1 Missing ⚠️

Additional details and impacted files

Files with missing lines	Coverage Δ
...ns/mongoProcessor/MongoProcessorConfigValidator.js	`100.00% <100.00%> (ø)`
extensions/mongoProcessor/MongoQueueProcessor.js	`80.95% <100.00%> (+6.16%)`	⬆️
...ns/mongoProcessor/metadataPolicy/MetadataPolicy.js	`100.00% <100.00%> (ø)`
...or/metadataPolicy/PullReplicationMetadataPolicy.js	`100.00% <100.00%> (ø)`
extensions/mongoProcessor/metadataPolicy/index.js	`100.00% <100.00%> (ø)`
lib/Config.js	`84.48% <100.00%> (+0.13%)`	⬆️
lib/config.joi.js	`100.00% <100.00%> (ø)`
lib/queuePopulator/QueuePopulator.js	`83.47% <100.00%> (+0.14%)`	⬆️
...rocessor/metadataPolicy/IngestionMetadataPolicy.js	`98.00% <98.00%> (ø)`

... and 1 file with indirect coverage changes

Components	Coverage Δ
Bucket Notification	`80.25% <ø> (ø)`
Core Library	`83.66% <100.00%> (+0.02%)`	⬆️
Ingestion	`78.21% <99.07%> (+2.83%)`	⬆️
Lifecycle	`81.65% <ø> (ø)`
Oplog Populator	`85.83% <ø> (ø)`
Replication	`63.86% <ø> (ø)`
Bucket Scanner	`85.76% <ø> (ø)`

@@                 Coverage Diff                 @@
##           development/9.6    #2833      +/-   ##
===================================================
+ Coverage            76.52%   76.76%   +0.24%     
===================================================
  Files                  207      211       +4     
  Lines                14487    14557      +70     
===================================================
+ Hits                 11086    11175      +89     
+ Misses                3391     3372      -19     
  Partials                10       10

Flag	Coverage Δ
api:retry	`9.32% <24.34%> (+0.16%)`	⬆️
api:routes	`9.10% <24.34%> (+0.16%)`	⬆️
bucket-scanner	`85.76% <ø> (ø)`
ft_test:queuepopulator	`9.83% <26.08%> (+0.17%)`	⬆️
ingestion	`13.41% <95.65%> (+0.74%)`	⬆️
lib	`9.10% <24.34%> (+0.17%)`	⬆️
lifecycle	`20.63% <24.34%> (+0.11%)`	⬆️
notification	`0.97% <0.00%> (-0.01%)`	⬇️
oplogPopulator	`0.13% <0.00%> (-0.01%)`	⬇️
replication	`18.95% <24.34%> (+0.10%)`	⬆️
unit	`57.92% <32.17%> (-0.08%)`	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

bert-e · 2026-08-27T12:42:02Z

Waiting for approval

The following approvals are needed before I can proceed with the merge:

the author
2 peers

bert-e · 2026-08-27T13:37:56Z

Waiting for approval

The following approvals are needed before I can proceed with the merge:

the author
2 peers

delthas · 2026-09-11T15:07:18Z

Requested @maeldonn in place of Sylvain Senechal, who is currently on PTO.

delthas · 2026-09-14T00:32:29Z

Requested @SylvainSenechal in place of Mael Donnart, who is currently on PTO.

francoisferrand · 2026-09-28T07:10:34Z

The D/R pipeline sends every document.

The D/R pipeline should not send every document anymore. Since mongo-processor handles the master "creation/update", the D/R pipeline should now drop the master whenever they have an associated version key (like ingestion populator does).

delthas · 2026-09-29T09:22:37Z

The D/R pipeline should not send every document anymore.

Done in scality/zenko-operator#631: the pipeline keeps a master only when it is the object itself, i.e. with no version of its own, or a suspended bucket's null version.

bert-e · 2026-09-29T09:46:04Z

Waiting for approval

The following approvals are needed before I can proceed with the merge:

the author
2 peers

claude · 2026-09-29T09:55:48Z

-    _updateObjectDataStoreName(entry, location) {
-        entry.setDataStoreName(location);
-    }
+    _getTargetMetadata(log, entry, target, done) {


Async/await migration suggestion: _getTargetMetadata is a new function with a done callback parameter. Per the project's async/await migration rule, new functions should use async/await. It could return { objMD, versionId } (or throw on error), and callers could await or util.callbackify it. Not a blocker given the surrounding callback context.

delthas · 2026-09-29T22:23:02Z

Added a logSource: "none" setting

francoisferrand · 2026-09-30T19:34:51Z

+        return false;
+    }
+
+    targetVersionId(entry) {


note for reviewers: the content of this function could/should probably be moved out of the policy; however

we don't want to change the behavior of OOB here, so keeping the exact behavior of ingestion policy for now

ingestion requires the x-scal-version-id mapping anyway, which does not apply for pull replication

bert-e · 2026-09-30T22:09:49Z

Waiting for approval

The following approvals are needed before I can proceed with the merge:

the author
2 peers

The `server` section is optional, but the whitelist of addresses allowed to reach the health checks was extended unconditionally, so a process serving no API exited before starting: a D/R sink runs the mongo-processor alone, and configuring a section it never reads to get past this is no answer. Issue: BB-811

The mongo-processor reads the object it is about to write, and a first write finds nothing there. Ingestion reaches that path only for a restored object or a bucket carrying replication rules, so both the error log and the error test it leans on went unnoticed; a D/R sink reads the stored document for every entry, which makes a miss the steady state. Log it as the outcome it is, and test it the way the delete path a few lines below already does. `err.NoSuchKey` is arsenal's deprecated comparison, set only while `allowUnsafeErrComp` is on and slated for removal with ARSN-176 -- with it off, every first delivery of every object would be an error. Issue: BB-811

The bucket processor circuit breaker expands a `${location}` template over every location in the config, and its spec restated the expansion by hand, one entry per location: adding a location to the sample config broke it. Build the expectation from the config instead. Issue: BB-811

The mongo-processor was written as the out-of-band ingestion consumer, and the D/R metadata sink is to reuse it. The two disagree about most of what it does to an object's metadata, so put those decisions behind a policy: an abstract MetadataPolicy whose methods assert, an implementation per mode, and an index mapping the configured mode to the class, as the notification extension does for its destinations. The processor hands the policy entries and asks it whether an entry needs the metadata already stored, which version on this site the entry acts on, how the entry applies onto what is stored -- including the version the result is written under, or nothing if the entry changes nothing -- and whether a delete still applies. IngestionMetadataPolicy carries today's behaviour verbatim, so this commit changes nothing. The `x-amz-meta-scal-version-id` header of a restored object is ingestion's alone, so the policy reads it, where the processor did. The version an entry is written under stays apart from the one it is read from: an entry naming a version that is not stored is ingested under its own id. The default lives beside the policy map, so a processor built programmatically gets the same mode as one built from a config file. Issue: BB-811

The D/R metadata sink replicates production's objects, accounts included, so it applies what the source-side pipeline sends rather than rewriting it into something local: a new object is written as it arrives, with its ACLs reset as they are not replicated, and an update merges the entry's tags and object-lock state into the stored document, cleared values included -- removing a legal hold is an update. Every update is written, a replay included, rather than diffed against what is stored. The stored document keeps everything else, and above all its placement. The copy engine rewrites location and dataStoreName to a local location after the first write, and applying the entry's would send reads back to the source and leak a local copy that is never garbage-collected; a version still on the remote site takes the entry's. A restore this site performed is kept the same way. Several things follow that were unreachable before: - the stored document is always read, because it is what tells a first write from an update, and the entry cannot -- an insert is redelivered on replay and overlaps the bootstrap dump, so it is no promise that the object is absent here; - a delete always applies, where the ingestion guard skips one whose object has moved location, which for a replicated object it always has; - a scal version id names a version of the system an object was ingested from, and is ignored. The D/R source keys every entry by the document it comes from, so the sink acts on that very document: a version by its version id, null versions included, and a master as the master document itself, which only reaches the sink for an object with no version of its own. A master document that copies a version, or the latest version mongo returns in place of a missing master, is not what such an entry targets: the entry is written over it. The mock client returns the document it was given, so null versions are tested against mongodb. Issue: BB-811

An object with no version of its own is rewritten in place, and so is the master a versioning suspended bucket marks null. Merging such an entry kept the previous object's content, headers, user metadata and archive on the sink, which then described an object that no longer existed. The Kafka Connect sink this replaces wrote the whole document, so the merge was a regression against it. An overwrite moves the modification date, which a tag, retention, legal hold or restore update keeps: an entry whose date differs from the stored one replaces the document, and any other entry still merges, so a tag change keeps a restore this site performed. The content digest cannot tell an overwrite either, a PUT replacing the whole metadata whatever the bytes do. A version is immutable and always merges. Nothing of a replaced document is kept. A restore this site performed describes bytes that are gone; reclaiming them is left to the D/R garbage collection still to come. Issue: BB-811

Every backbeat process reads its mongodb client from the queuePopulator section, so a process that only needs that client, like the mongo-processor of a D/R sink, had to fill in a cron rule, a zookeeper path, a probe server and a log source for a populator that never runs. The 'none' log source says so: it needs none of those settings, and a queue populator started with it refuses to run. Issue: BB-811

A null master is a version like any other: the source gives it a document of its own on its first metadata update, and streams every later change under that version id. Requesting and writing it as the master would leave those changes on another document, so the sink targets a null master by the internal version id it carries, and only an object with no version id at all is still written in place, told from an overwrite by its modification date. That also leaves one place to decide whether an entry updates the stored document: a targeted version always does, and the guard for a master copying a version has no case left to cover. Issue: BB-811

francoisferrand · 2026-09-30T22:13:04Z

/bypass_author_approval

bert-e · 2026-09-30T22:24:50Z

I have successfully merged the changeset of this pull request
into targetted development branches:

✔️ development/9.6

The following branches have NOT changed:

development/7.10
development/7.4
development/7.70
development/8.6
development/9.0
development/9.1
development/9.2
development/9.3
development/9.4
development/9.5

This pull request did not target the following hotfix branch(es) so they
were left untouched:

hotfix/7.8.0
hotfix/7.10.17
hotfix/7.10.2
hotfix/7.70.1
hotfix/7.4.9
hotfix/7.4.3
hotfix/9.0.7
hotfix/8.2.12
hotfix/7.4.10
hotfix/7.4.4
hotfix/7.9.0
hotfix/7.10.1
hotfix/7.4.8
hotfix/7.10.12
hotfix/7.4.2
hotfix/7.10.4
hotfix/7.4.1
hotfix/7.4.5
hotfix/7.6.0
hotfix/7.2.0
hotfix/7.70.12
hotfix/7.10.3
hotfix/7.4.7
hotfix/7.4.0
hotfix/7.4.6
hotfix/7.10.0
hotfix/7.10.8
hotfix/7.70.15
hotfix/9.0.4
hotfix/7.7.0

Please check the status of the associated issue BB-811.

Goodbye delthas.

The following options are set: bypass_author_approval

scality deleted a comment from bert-e Aug 27, 2026

delthas force-pushed the improvement/BB-811/dr-mode-mongo-processor branch 2 times, most recently from 1a74a03 to 8675518 Compare August 27, 2026 13:37

claude Bot reviewed Aug 27, 2026

View reviewed changes

Comment thread extensions/mongoProcessor/mongoProcessorTask.js Outdated

delthas force-pushed the improvement/BB-811/dr-mode-mongo-processor branch from e77010b to 5112a3a Compare August 31, 2026 08:31

scality deleted a comment from bert-e Aug 31, 2026

delthas force-pushed the improvement/BB-811/dr-mode-mongo-processor branch from 5112a3a to 0eb82b1 Compare August 31, 2026 09:24

claude Bot reviewed Aug 31, 2026

View reviewed changes

Comment thread extensions/mongoProcessor/MongoProcessorConfigValidator.js Outdated

delthas force-pushed the improvement/BB-811/dr-mode-mongo-processor branch 2 times, most recently from 2956946 to d161e55 Compare September 4, 2026 15:18

claude Bot reviewed Sep 4, 2026

View reviewed changes

Comment thread extensions/mongoProcessor/modes/IngestionMode.js Outdated

delthas force-pushed the improvement/BB-811/dr-mode-mongo-processor branch from d161e55 to 486c8e2 Compare September 7, 2026 15:44

delthas mentioned this pull request Sep 7, 2026

Cover D/R deletion and in-place overwrite in the PRA test scality/Zenko#2529

Closed

delthas marked this pull request as ready for review September 7, 2026 15:55

delthas force-pushed the improvement/BB-811/dr-mode-mongo-processor branch from 486c8e2 to 8b693e6 Compare September 10, 2026 12:52

delthas requested review from a team, SylvainSenechal and francoisferrand September 11, 2026 07:50

delthas force-pushed the improvement/BB-811/dr-mode-mongo-processor branch from 8b693e6 to e1dd485 Compare September 11, 2026 09:19

delthas requested review from maeldonn and removed request for SylvainSenechal September 11, 2026 15:07

delthas requested review from SylvainSenechal and removed request for maeldonn September 14, 2026 00:32

delthas mentioned this pull request Sep 14, 2026

Extend PRA coverage to deletions, overwrites and non-versioned buckets scality/Zenko#2532

Open

delthas commented Sep 25, 2026

View reviewed changes

Comment thread extensions/mongoProcessor/metadataPolicy/IngestionMetadataPolicy.js

delthas requested a review from francoisferrand September 25, 2026 09:35

francoisferrand reviewed Sep 28, 2026

View reviewed changes

delthas force-pushed the improvement/BB-811/dr-mode-mongo-processor branch from df647fc to 9ed5c73 Compare September 29, 2026 09:21

delthas force-pushed the improvement/BB-811/dr-mode-mongo-processor branch from 9ed5c73 to e923570 Compare September 29, 2026 09:45

claude Bot reviewed Sep 29, 2026

View reviewed changes

scality deleted a comment from bert-e Sep 29, 2026

delthas force-pushed the improvement/BB-811/dr-mode-mongo-processor branch from e923570 to 6d7bcb2 Compare September 29, 2026 10:42

francoisferrand force-pushed the improvement/BB-811/dr-mode-mongo-processor branch from 06718c5 to aab6f69 Compare September 30, 2026 19:15

francoisferrand approved these changes Sep 30, 2026

View reviewed changes

SylvainSenechal approved these changes Sep 30, 2026

View reviewed changes

delthas and others added 8 commits October 1, 2026 00:12

francoisferrand force-pushed the improvement/BB-811/dr-mode-mongo-processor branch from aab6f69 to d558470 Compare September 30, 2026 22:12

scality deleted a comment from bert-e Sep 30, 2026

bert-e merged commit d558470 into development/9.6 Sep 30, 2026
24 checks passed

bert-e deleted the improvement/BB-811/dr-mode-mongo-processor branch September 30, 2026 22:24

Conversation

delthas commented Aug 27, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

bert-e commented Aug 27, 2026

Hello delthas,

Uh oh!

codecov Bot commented Aug 27, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

bert-e commented Aug 27, 2026

Waiting for approval

Uh oh!

bert-e commented Aug 27, 2026

Waiting for approval

Uh oh!

Uh oh!

Uh oh!

Uh oh!

delthas commented Sep 11, 2026

Uh oh!

delthas commented Sep 14, 2026

Uh oh!

Uh oh!

francoisferrand commented Sep 28, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

delthas commented Sep 29, 2026

Uh oh!

bert-e commented Sep 29, 2026

Waiting for approval

Uh oh!

claude Bot Sep 29, 2026

Choose a reason for hiding this comment

Uh oh!

delthas commented Sep 29, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

francoisferrand Sep 30, 2026

Choose a reason for hiding this comment

Uh oh!

bert-e commented Sep 30, 2026

Waiting for approval

Uh oh!

francoisferrand commented Sep 30, 2026

Uh oh!

bert-e commented Sep 30, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

delthas commented Aug 27, 2026 •

edited

Loading

codecov Bot commented Aug 27, 2026 •

edited

Loading