Repository navigation
Conversation
data:retention (cron 00:10 daily) anonymises every qualifying AskToLeave member in one run, with one flush per member. If a deletion policy ever moves thousands of members to AskToLeave at once, that night's run would load the prod DB for hours. Now it cleans at most --limit members per run (default 500, 0 = no limit), oldest id first; cleaned members are renamed retired_<id> and drop out of the query, so the rest follow on the next nights. Cron line unchanged. (BeWelcome/sysadmins-infra#654, plan 8.4)
Neophytis
added
improvement
theme:bw-platform
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Target branch:
main. Part of BeWelcome/sysadmins-infra#693 (sustainable rate for deletes, plan section 8.4).Why
data:retention(cron 00:10 daily) anonymises every qualifying member in one run:AskToLeave, more than a year without login, not yetretired_. It flushes once per member and has no limit.Today that is a handful per night. But if a deletion policy (to be decided at the GA in November) moves thousands of long-suspended members to
AskToLeaveat once (83,740 members carry the 5-year suspension marker), that night's run would load the prod DB for hours, during normal use.Change
--limit(default 500,0= no limit).MemberRepository::loadDataRetentionMembers(int $limit = 0):ORDER BY id, plussetMaxResultswhen a limit is given.retired_<id>, which drops them out of the query, so the rest follow on the next nights. Nobody is processed twice.bin/console data:retention), so the default applies.Tests
php -lclean on both files.bin/console data:retention --limit 2on stage and check that the output stops at 2 with the note, then rerun and check the next members are picked.