fix(shard): widen partition repair when a view-adopted suffix is missing bodies - #4074
Open
mattp5657 wants to merge 2 commits into
Open
fix(shard): widen partition repair when a view-adopted suffix is missing bodies#4074mattp5657 wants to merge 2 commits into
mattp5657 wants to merge 2 commits into
Conversation
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## master #4074 +/- ##
=============================================
- Coverage 85.55% 44.84% -40.71%
Complexity 1402 1402
=============================================
Files 1241 1240 -1
Lines 186306 155199 -31107
Branches 152594 121489 -31105
=============================================
- Hits 159395 69605 -89790
- Misses 22865 83746 +60881
+ Partials 4046 1848 -2198
🚀 New features to boost your workflow:
|
ethanlin01x
approved these changes
Sep 6, 2026
| // missing bodies sit above `commit_to_op`, not within it -- so the | ||
| // cap only holds when no suffix is missing; otherwise it must widen | ||
| // to `head` to ever reach those bodies. | ||
| let fetch_to_op = if commit_lag && !missing_suffix { |
Contributor
There was a problem hiding this comment.
nit: Could you add a unit test for the commit_lag && missing_suffix case?
spetz
approved these changes
Sep 6, 2026
hubcio
approved these changes
Sep 6, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes
#4075
Rationale
A partition group can wedge permanently and stop committing after a view change lands while a node is mid-restart, hanging every producer to that partition until its read timeout expires.
What changed?
After a view change, a backup that adopts a
StartViewsuffix advances its sequencer to the new head immediately, but repair only fetched bodies up to the commit point when the replica also had a commit lag, which a just-restarted node always does. The adopted op's body was never fetched. Once any later op advanced the sequencer past it, the backup's replication gap check rejected every retransmission of the missing op as a stale duplicate, permanently, and commit could never advance past it.Widens the repair fetch window using
missing_suffix, already computed for this but not consulted in this branch, so the first repair pass covers the adopted suffix before anything else can move the sequencer past it.Fixes
cluster::fast_primary_rejoin::given_a_quic_producer_when_its_first_roster_hop_is_down_should_reach_the_partition_primary, failing ~70-90% of local runs and on all 4 CI attempts.Local Execution
fast_primary_rejoin.rs(8 tests), fullcluster::integration suite (61 tests), andconsensus/partitions/shardunit suites (462 tests): all pass, no regressions.cargo fmt,cargo sort --no-format, andcargo clippywere run manually.AI Usage
Claude was used in the generation of this.