Migrate user records to ghost in batch job runs

What does this MR do and why?

It splits Users::DestroyService into a 2 steps workflow:

• keeps the first part of Users::DestroyService
• migrating associated records to the ghost user and deleting records is migrated to Users::MigrateRecordsToGhostUserService

Migration and deletion is now done in batches asynchronously with a limited execution time:

1. Users::MigrateRecordsToGhostUserInBatchesWorker is a cron worker running every minutes
2. The worker executes Users::MigrateRecordsToGhostUserInBatchesService with execution time limit of 30 seconds
3. Users::MigrateRecordsToGhostUserService is executed for each user

Related to https://gitlab.com/gitlab-org/gitlab/-/issues/366655.

Screenshots or screen recordings

These are strongly recommended to assist reviewers and reduce the time to merge your change.

How to set up and validate locally

Numbered steps to set up and validate the change are strongly suggested.

MR acceptance checklist

This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.

• I have evaluated the MR acceptance checklist for this MR.

added bugperformance groupauthentication and authorization [DEPRECATED] typebug labels

assigned to @ifarkas

Suggested Reviewers (beta)

The individuals below may be good candidates to participate in the review based on various factors.

You can use slash commands in comments to quickly assign /assign_reviewer @user1.

Suggested Reviewers
@rspeicher, @rymai, @mayra-cabrera, @nick.thomas, @ahegyi

If you do not believe these suggestions are useful, please apply the label Bad Suggested Reviewer. You can also provide feedback for this feature on this issue: https://gitlab.com/gitlab-org/gitlab/-/issues/357923.

Automatically generated by Suggested Reviewers Bot - an experimental ML-based recommendation engine created by ~"group::applied ml".

added backend label

	9 Warnings
	This merge request is quite big (1267 lines changed), please consider splitting it into multiple merge requests.
	This merge request includes more than 10 commits. Each commit should meet the following criteria: Have a well-written commit message. Has all tests passing when used on its own (e.g. when using git checkout SHA). Can be reverted on its own without also requiring the revert of commit that came before it. Is small enough that it can be reviewed in isolation in under 30 minutes or so. If this merge request contains commits that do not meet this criteria and/or contains intermediate work, please rebase these commits into a smaller number of commits or split this merge request into multiple smaller merge requests.
	3d6a9806: The commit subject must contain at least 3 words. For more information, take a look at our Commit message guidelines.
	3d6a9806: Commits that change 30 or more lines across at least 3 files should describe these changes in the commit body. For more information, take a look at our Commit message guidelines.
	6e6782dd: The commit subject must contain at least 3 words. For more information, take a look at our Commit message guidelines.
	a25733f4: The commit subject must contain at least 3 words. For more information, take a look at our Commit message guidelines.
	067ab899: The commit subject must contain at least 3 words. For more information, take a look at our Commit message guidelines.
	067ab899: Commits that change 30 or more lines across at least 3 files should describe these changes in the commit body. For more information, take a look at our Commit message guidelines.
	This merge request does not refer to an existing milestone.

	2 Messages
	This merge request adds or changes files that require a review from the Database team.
	This merge request adds or changes documentation files. A review from the Technical Writing team before you merge is recommended. Reviews can happen after you merge.

This merge request requires a database review. To make sure these changes are reviewed, take the following steps:

1. Ensure the merge request has database and databasereview pending labels. If the merge request modifies database files, Danger will do this for you.

2. Prepare your MR for database review according to the docs.

3. Assign and mention the database reviewer suggested by Reviewer Roulette.

4. If this is not a Community contribution or from a Fork, kick off the db:gitlabcom-database-testing manual job.

The following files require a review from the Database team:

• db/migrate/20220726171440_create_migrate_to_ghost_user.rb
• db/schema_migrations/20220726171440
• db/structure.sql
• lib/gitlab/database/gitlab_schemas.yml

Documentation review

The following files require a review from a technical writer:

• db/docs/migrate_to_ghost_user.yml

The review does not need to block merging this merge request. See the:

• Metadata for the *.md files that you've changed. The first few lines of each *.md file identify the stage and group most closely associated with your docs change.
• The Technical Writer assigned for that stage and group.
• Documentation workflows for information on when to assign a merge request for review.

Reviewer roulette

Changes that require review have been detected!

Please refer to the table below for assigning reviewers and maintainers suggested by Danger in the specified category:

Category	Reviewer	Maintainer
backend	Michał Wielich (@michold) (UTC+2, same timezone as @ifarkas)	Steve Abrams (@sabrams) (UTC-6, 8 hours behind @ifarkas)
database	Mehmet Emin Inac (@minac) (UTC+2, same timezone as @ifarkas)	Steve Abrams (@sabrams) (UTC-6, 8 hours behind @ifarkas)
~"migration"	No reviewer available	No maintainer available

To spread load more evenly across eligible reviewers, Danger has picked a candidate for each review slot, based on their timezone. Feel free to override these selections if you think someone else would be better-suited or use the GitLab Review Workload Dashboard to find other available reviewers.

To read more on how to use the reviewer roulette, please take a look at the Engineering workflow and code review guidelines. Please consider assigning a reviewer or maintainer who is a domain expert in the area of the merge request.

Once you've decided who will review this merge request, assign them as a reviewer! Danger does not automatically notify them for you.

Sidekiq queue changes

This merge request contains changes to Sidekiq queues. Please follow the documentation on changing a queue's urgency.

These queues were added:

• cronjob:users_migrate_records_to_ghost_user_in_batches

If needed, you can retry the danger-review job that generated this comment.

Generated by Danger

added 1 commit

• 635cccd6 - Destroy user memberships early

Compare with previous version

@ifarkas, please can you answer the question: Should this have a feature flag? to help with code review for the Authentication and Authorization group.

This nudge was added by this triage-ops policy.

added 811 commits

• 635cccd6...f0cb418c - 810 commits from branch master
• 7faa5872 - Destroy user memberships early

Compare with previous version

added 1 commit

• 8787a5e6 - Remove return value from Users::DestroyService

Compare with previous version

added 2 commits

• 960307de - Add deletion_scheduled state to User
• 5d1eeaa7 - Migration for creating migrate_to_ghost_user table

Compare with previous version

added database databasereview pending labels

Allure report

allure-report-publisher generated test report!

review-qa-blocking: test report for 438d8993

expand test summary

+-----------------------------------------------------------------------------------------+
|                                     suites summary                                      |
+------------------------------------+--------+--------+---------+-------+-------+--------+
|                                    | passed | failed | skipped | flaky | total | result |
+------------------------------------+--------+--------+---------+-------+-------+--------+
| Verify                             | 0      | 14     | 1       | 14    | 15    | ❌     |
| Manage                             | 0      | 36     | 0       | 36    | 36    | ❌     |
| Plan                               | 0      | 47     | 1       | 47    | 48    | ❌     |
| Create                             | 0      | 28     | 1       | 28    | 29    | ❌     |
| Configure                          | 0      | 0      | 1       | 0     | 1     | ➖     |
| Feature flag handler sanity checks | 0      | 9      | 0       | 9     | 9     | ❌     |
| Protect                            | 0      | 2      | 0       | 2     | 2     | ❌     |
| Secure                             | 0      | 2      | 0       | 2     | 2     | ❌     |
| Version sanity check               | 0      | 0      | 1       | 0     | 1     | ➖     |
+------------------------------------+--------+--------+---------+-------+-------+--------+
| Total                              | 0      | 138    | 5       | 138   | 143   | ❌     |
+------------------------------------+--------+--------+---------+-------+-------+--------+

Setting label(s) devopsmanage sectiondev based on ~"group::authentication and authorization".

added devopsmanage sectiondev labels

added 2 commits

• ab8c3c64 - Add deletion_scheduled state to User
• 11a9c4cf - Migration for creating migrate_to_ghost_user table

Compare with previous version

added documentation label

added 1 commit

• 438d8993 - Delete personal snippets before migrating records to ghost

Compare with previous version

added 404 commits

• 438d8993...8aa2e4c9 - 400 commits from branch master
• 0ce8b581 - Destroy user memberships early
• 2498d2c6 - Add deletion_scheduled state to User
• 39f050d2 - Migration for creating migrate_to_ghost_user table
• 87818ca6 - Refactor Snippets::BulkDestroyService flag to skip authorization

Compare with previous version

added 3 commits

• 7c2fad84 - Migration for creating migrate_to_ghost_user table
• cf52e608 - Refactor Snippets::BulkDestroyService flag to skip authorization
• 36ce0ef2 - Add Users::MigrateRecordsToGhostUserService

Compare with previous version