Skip to content

Check application height in post-upgrade checks - #69

Merged
qezz merged 2 commits into
mainfrom
check-app-height-in-post
Sep 11, 2026
Merged

qezz merged 2 commits into
mainfrom
check-app-height-in-post

Conversation

@qezz

@qezz qezz commented Aug 21, 2026

Copy link
Copy Markdown
Contributor

Application height is updated after the block is committed and after it is executed/applied.

We need this height to know whether the node actually managed to apply the new block, or whether it failed to do so.

The behavior we saw was: when rolling out a "wrong" version of the software, the node commits a block at height N, but fails to execute it. Blazar happily reports that the height N has been reached, and marks the upgrade as "successful".

That works fine for the most upgrades, e.g. when a new version is released and new upgrade handlers are expected expected to execute properly. When the version is wrong, the upgrade handlers are not found, the execution fails, but Blazar still considers it successful.

This change should prevent this issue from happening.

@qezz

qezz commented Aug 21, 2026 •

Copy link
Copy Markdown
Contributor Author

interesting, CI reports a race condition data race, but I didn't touch any related stuff. and the golang version seems to be the same as on the previous patch

@qezz

qezz commented Aug 21, 2026 •

Copy link
Copy Markdown
Contributor Author

here we go

it's not an issue in real code, only in tests, which run in parallel, i.e. it triggers the race condition data race only in tests... which is annoying

@qezz

qezz commented Aug 31, 2026

Copy link
Copy Markdown
Contributor Author

tests are passing with -parallel 1

i will remove it before the merge, and follow up once the upstream library is released

@qezz
qezz marked this pull request as ready for review August 31, 2026 14:40
@qezz
qezz requested a review from a team August 31, 2026 14:40

@Szymongib Szymongib left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Makes sense 👍

@qezz

qezz commented Sep 11, 2026

Copy link
Copy Markdown
Contributor Author

the compose-go released 2.15 https://github.com/compose-spec/compose-go/releases/tag/v2.15.0, will bump it

@qezz
qezz force-pushed the check-app-height-in-post branch from 0259aed to 1c25c1c Compare September 11, 2026 14:03
Comment thread internal/pkg/daemon/checks/post.go Outdated
// as "successful".
//
// That works fine for the most upgrades, e.g. when a new version is released
// and new upgrade handlers are expected expected to execute properly.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
// and new upgrade handlers are expected expected to execute properly.
// and new upgrade handlers are expected to execute properly.

oops

Application height is updated after the block is committed and after
it is executed/applied.

We need this height to know whether the node actually managed to apply
the new block, or whether it failed to do so.

The behavior we saw was: when rolling out a "wrong" version of the
software, the node commits a block at height `N`, but fails to execute
it. Blazar happily reports that the height `N` has been reached, and
marks the upgrade as "successful".

That works fine for the most upgrades, e.g. when a new version is
released and new upgrade handlers are expected expected to execute
properly.  When the version is wrong, the upgrade handlers are not
found, the execution fails, but Blazar still considers it successful.

This change should prevent this issue from happening.
@qezz
qezz force-pushed the check-app-height-in-post branch from 1c25c1c to 75c1c94 Compare September 11, 2026 14:05
@qezz
qezz merged commit dcb32fe into main Sep 11, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants