Update epic cache counts on project/group deletion

What does this MR do and why?

When issue's count or weight is changed for an epic, we update its cached values. These cached values are sum of both direct epic's issues and also sub-epic's issues.

Because we want to include subepics weights/counts too, when cached value is changed we propagate this change to all epic ancestors. This is achieved by scheduling cache update for epic's parent in epic's after_commit callback.

Related to #356683 (closed)

DB changes:

• queries to get impacted epics when destroying group or project - !90615 (comment 1000841520) and !90615 (comment 1000999580)
• queries to get epic's cached sum/counts - !90615 (comment 1004585335) and !90615 (comment 1004589231)

Screenshots or screen recordings

These are strongly recommended to assist reviewers and reduce the time to merge your change.

How to set up and validate locally

Numbered steps to set up and validate the change are strongly suggested.

MR acceptance checklist

This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.

• I have evaluated the MR acceptance checklist for this MR.

assigned to @jprovaznik

Suggested Reviewers (beta)

The individuals below may be good candidates to participate in the review based on various factors.

You can use slash commands in comments to quickly assign /assign_reviewer @user1.

Suggested Reviewers
@rspeicher, @rymai, @mayra-cabrera, @ntepluhina, @jivanvl

If you do not believe these suggestions are useful, please apply the label Bad Suggested Reviewer. You can also provide feedback for this feature on this issue: https://gitlab.com/gitlab-org/gitlab/-/issues/357923.

Automatically generated by Suggested Reviewers Bot - an experimental ML-based recommendation engine created by ~"group::applied ml".

added featureaddition typefeature labels

	2 Warnings
	This merge request is quite big (609 lines changed), please consider splitting it into multiple merge requests.
	featureaddition and featureenhancement merge requests normally have a documentation change. Consider adding a documentation update or confirming the documentation plan with the Technical Writer counterpart. For more information, see: The Handbook page on merge request types. The definition of done documentation.

	2 Messages
	CHANGELOG missing: If you want to create a changelog entry for GitLab FOSS, add the Changelog trailer to the commit message you want to add to the changelog. If you want to create a changelog entry for GitLab EE, also add the EE: true trailer to your commit message. If this merge request doesn't need a CHANGELOG entry, feel free to ignore this message.
	This merge request adds or changes files that require a review from the Database team.

This merge request requires a database review. To make sure these changes are reviewed, take the following steps:

1. Ensure the merge request has database and databasereview pending labels. If the merge request modifies database files, Danger will do this for you.
2. Prepare your MR for database review according to the docs.
3. Assign and mention the database reviewer suggested by Reviewer Roulette.

If you no longer require a database review, you can remove this suggestion by removing the database label and re-running the danger-review job.

Reviewer roulette

Changes that require review have been detected!

Please refer to the table below for assigning reviewers and maintainers suggested by Danger in the specified category:

Category	Reviewer	Maintainer
backend	Michał Zając (@Quintasan) (UTC+2, same timezone as @jprovaznik)	Kerri Miller (@kerrizor) (UTC-7, 9 hours behind @jprovaznik)
database	Pavel Shutsin (@pshutsin) (UTC+2, same timezone as @jprovaznik)	Krasimir Angelov (@krasio) (UTC+12, 10 hours ahead of @jprovaznik)

To spread load more evenly across eligible reviewers, Danger has picked a candidate for each review slot, based on their timezone. Feel free to override these selections if you think someone else would be better-suited or use the GitLab Review Workload Dashboard to find other available reviewers.

To read more on how to use the reviewer roulette, please take a look at the Engineering workflow and code review guidelines. Please consider assigning a reviewer or maintainer who is a domain expert in the area of the merge request.

Once you've decided who will review this merge request, assign them as a reviewer! Danger does not automatically notify them for you.

Sidekiq queue changes

This merge request contains changes to Sidekiq queues. Please follow the documentation on changing a queue's urgency.

These queues were added:

• epics:epics_update_cached_metadata

If needed, you can retry the danger-review job that generated this comment.

Generated by Danger

added groupproduct planning label

added 1 commit

• 97e80016 - Updated cached issue weight and count for epics

Compare with previous version

changed milestone to %15.2

Allure report

allure-report-publisher generated test report!

review-qa-blocking: test report for 607c8ec6

expand test summary

+-----------------------------------------------------------------------------------------+
|                                     suites summary                                      |
+------------------------------------+--------+--------+---------+-------+-------+--------+
|                                    | passed | failed | skipped | flaky | total | result |
+------------------------------------+--------+--------+---------+-------+-------+--------+
| Protect                            | 2      | 0      | 0       | 2     | 2     | ❗     |
| Manage                             | 38     | 0      | 2       | 40    | 40    | ❗     |
| Create                             | 24     | 0      | 1       | 24    | 25    | ❗     |
| Plan                               | 47     | 0      | 1       | 47    | 48    | ❗     |
| Configure                          | 0      | 0      | 1       | 0     | 1     | ➖     |
| Verify                             | 12     | 0      | 1       | 12    | 13    | ❗     |
| Secure                             | 2      | 0      | 0       | 2     | 2     | ❗     |
| Version sanity check               | 0      | 0      | 1       | 0     | 1     | ➖     |
| Feature flag handler sanity checks | 9      | 0      | 0       | 9     | 9     | ❗     |
+------------------------------------+--------+--------+---------+-------+-------+--------+
| Total                              | 134    | 0      | 7       | 136   | 141   | ❗     |
+------------------------------------+--------+--------+---------+-------+-------+--------+

package-and-qa-ff-disabled: test report for 607c8ec6

expand test summary

+---------------------------------------------------------------------------+
|                              suites summary                               |
+----------------------+--------+--------+---------+-------+-------+--------+
|                      | passed | failed | skipped | flaky | total | result |
+----------------------+--------+--------+---------+-------+-------+--------+
| Release              | 15     | 0      | 0       | 3     | 15    | ❗     |
| Create               | 453    | 0      | 12      | 5     | 465   | ❗     |
| Plan                 | 162    | 0      | 0       | 0     | 162   | ✅     |
| Manage               | 330    | 0      | 9       | 10    | 339   | ❗     |
| Secure               | 63     | 0      | 6       | 9     | 69    | ❗     |
| Configure            | 0      | 0      | 9       | 0     | 9     | ➖     |
| Verify               | 120    | 0      | 9       | 0     | 129   | ✅     |
| Fulfillment          | 6      | 0      | 33      | 0     | 39    | ✅     |
| Package              | 0      | 0      | 9       | 0     | 9     | ➖     |
| Product Intelligence | 6      | 0      | 0       | 0     | 6     | ✅     |
| Version sanity check | 0      | 0      | 3       | 0     | 3     | ➖     |
| Protect              | 6      | 0      | 0       | 0     | 6     | ✅     |
+----------------------+--------+--------+---------+-------+-------+--------+
| Total                | 1161   | 0      | 90      | 27    | 1251  | ❗     |
+----------------------+--------+--------+---------+-------+-------+--------+

package-and-qa-ff-enabled: test report for 607c8ec6

expand test summary

+---------------------------------------------------------------------------+
|                              suites summary                               |
+----------------------+--------+--------+---------+-------+-------+--------+
|                      | passed | failed | skipped | flaky | total | result |
+----------------------+--------+--------+---------+-------+-------+--------+
| Plan                 | 162    | 0      | 0       | 0     | 162   | ✅     |
| Secure               | 63     | 0      | 6       | 8     | 69    | ❗     |
| Verify               | 120    | 0      | 9       | 0     | 129   | ✅     |
| Create               | 453    | 0      | 12      | 3     | 465   | ❗     |
| Package              | 0      | 0      | 9       | 0     | 9     | ➖     |
| Manage               | 330    | 0      | 9       | 6     | 339   | ❗     |
| Fulfillment          | 6      | 0      | 33      | 0     | 39    | ✅     |
| Product Intelligence | 6      | 0      | 0       | 0     | 6     | ✅     |
| Release              | 15     | 0      | 0       | 1     | 15    | ❗     |
| Configure            | 0      | 0      | 9       | 0     | 9     | ➖     |
| Version sanity check | 0      | 0      | 3       | 0     | 3     | ➖     |
| Protect              | 6      | 0      | 0       | 0     | 6     | ✅     |
+----------------------+--------+--------+---------+-------+-------+--------+
| Total                | 1161   | 0      | 90      | 18    | 1251  | ❗     |
+----------------------+--------+--------+---------+-------+-------+--------+

added 1 commit

• c17540aa - Updated cached issue weight and count for epics

Compare with previous version

added 1 commit

• d23711d3 - Updated cached issue weight and count for epics

Compare with previous version

added 1 commit

• 6aa6058c - Updated cached issue weight and count for epics

Compare with previous version

added 1 commit

• 136d2798 - Updated cached issue weight and count for epics

Compare with previous version

added feature flag label

added 1 commit

• b938bffe - Updated cached issue weight and count for epics

Compare with previous version

added 1 commit

• 9a7b4b4e - Updated cached issue weight and count for epics

Compare with previous version

changed the description

added 1 commit

• 5d5eebdb - Updated cached issue weight and count for epics

Compare with previous version

• The gitlab-qa-mirror downstream pipeline for !90615 (b938bffe) passed.
• The gitlab-qa-mirror downstream pipeline for !90615 (b938bffe) passed.
• The gitlab-qa-mirror downstream pipeline for !90615 (9a7b4b4e) failed!
• The gitlab-qa-mirror downstream pipeline for !90615 (9a7b4b4e) failed!
• The gitlab-qa-mirror downstream pipeline for !90615 (5d5eebdb) passed.
• The gitlab-qa-mirror downstream pipeline for !90615 (5d5eebdb) failed!
• The gitlab-qa-mirror downstream pipeline for !90615 (33dbd6fb) failed!
• The gitlab-qa-mirror downstream pipeline for !90615 (33dbd6fb) failed!
• The gitlab-qa-mirror downstream pipeline for !90615 (6899c10e) passed.
• The gitlab-qa-mirror downstream pipeline for !90615 (6899c10e) passed.
• The gitlab-qa-mirror downstream pipeline for !90615 (79b0bad5) passed.
• The gitlab-qa-mirror downstream pipeline for !90615 (79b0bad5) passed.
• The gitlab-qa-mirror downstream pipeline for !90615 (79b0bad5) failed!
• The gitlab-qa-mirror downstream pipeline for !90615 (79b0bad5) passed.
• The gitlab-qa-mirror downstream pipeline for !90615 (08893cea) failed!
• The gitlab-qa-mirror downstream pipeline for !90615 (08893cea) passed.
• The gitlab-qa-mirror downstream pipeline for !90615 (fbd919c0) passed.
• The gitlab-qa-mirror downstream pipeline for !90615 (fbd919c0) failed!
• The gitlab-qa-mirror downstream pipeline for !90615 (b3ec0244) failed!
• The gitlab-qa-mirror downstream pipeline for !90615 (b3ec0244) failed!
• The gitlab-qa-mirror downstream pipeline for !90615 (c8585f90) failed!
• The gitlab-qa-mirror downstream pipeline for !90615 (c8585f90) failed!
• The gitlab-qa-mirror downstream pipeline for !90615 (607c8ec6) passed.
• The gitlab-qa-mirror downstream pipeline for !90615 (607c8ec6) passed.
• The gitlab-qa-mirror downstream pipeline for !90615 (3392fe7f) passed.
• The gitlab-qa-mirror downstream pipeline for !90615 (3392fe7f) passed.
• The gitlab-qa-mirror downstream pipeline for !90615 (488b38ef) passed.
• The gitlab-qa-mirror downstream pipeline for !90615 (488b38ef) passed.
• The gitlab-qa-mirror downstream pipeline for !90615 (37b40441) passed.
• The gitlab-qa-mirror downstream pipeline for !90615 (37b40441) passed.
• The gitlab-qa-mirror downstream pipeline for !90615 (94fc01cd) passed.
• The gitlab-qa-mirror downstream pipeline for !90615 (94fc01cd) passed.
• The gitlab-qa-mirror downstream pipeline for !90615 (764ec17b) passed.
• The gitlab-qa-mirror downstream pipeline for !90615 (764ec17b) passed.
• The gitlab-qa-mirror downstream pipeline for !90615 (9926eaa8) passed.
• The gitlab-qa-mirror downstream pipeline for !90615 (9926eaa8) passed.
• The gitlab-qa-mirror downstream pipeline for !90615 (1efed070) passed.
• The gitlab-qa-mirror downstream pipeline for !90615 (1efed070) passed.
• The gitlab-qa-mirror downstream pipeline for !90615 (fd4f84ed) passed.
• The gitlab-qa-mirror downstream pipeline for !90615 (fd4f84ed) passed.

added 1 commit

• 33dbd6fb - Updated cached issue weight and count for epics

Compare with previous version

added database label

@felipe_artur could you please review BE changes?

requested review from @felipe_artur

added 621 commits

• 33dbd6fb...5b73b8a9 - 620 commits from branch master
• 6899c10e - Updated cached issue weight and count for epics

Compare with previous version

added 55 commits

• 6899c10e...c61666f5 - 54 commits from branch master
• 79b0bad5 - Updated cached issue weight and count for epics

Compare with previous version

changed the description

@huzaifaiftikhar1 could you please do DB review (!90615 (merged))? Project/group destroy queries are quite slow (so may need an optimization), though I'd like to know DB opinion first.

requested review from @huzaifaiftikhar1

approved this merge request

added databasereviewed label

requested review from @mayra-cabrera and removed review request for @huzaifaiftikhar1

@huzaifaiftikhar1, thanks for approving this merge request.

This is the first time the merge request is approved. To ensure full test coverage, a new pipeline has been started.

For more info, please refer to the following links:

• rspec minimal jobs
• jest minimal jobs
• merging a merge request.

approved this merge request

removed review request for @mayra-cabrera

added 1 commit

• 08893cea - Minor changes based on review

Compare with previous version

added 209 commits

• 08893cea...5cf88c17 - 207 commits from branch master
• f07bd733 - Updated cached issue weight and count for epics
• fbd919c0 - Minor changes based on review

Compare with previous version

added 1 commit

• b3ec0244 - Minor changes based on review

Compare with previous version

requested review from @mayra-cabrera

removed review request for @mayra-cabrera

added 344 commits

• b3ec0244...648a9d06 - 342 commits from branch master
• eba96a05 - Updated cached issue weight and count for epics
• c8585f90 - Minor changes based on review

Compare with previous version

requested review from @mayra-cabrera

Setting label(s) devopsplan sectiondev based on groupproduct planning.

added devopsplan sectiondev labels

removed review request for @mayra-cabrera

requested review from @mayra-cabrera

removed review request for @mayra-cabrera

added 1072 commits

• c8585f90...e38baeb8 - 1070 commits from branch master
• 5b0d1052 - Updated cached issue weight and count for epics
• e4e8e6df - Minor changes based on review

Compare with previous version

added backend label

requested review from @mayra-cabrera

removed review request for @mayra-cabrera

changed milestone to %15.3

added missed:15.2 label

added 2405 commits

• e4e8e6df...8f58bc37 - 2402 commits from branch master
• c20c30c3 - Updated cached issue weight and count for epics
• 5418ea22 - Minor changes based on review
• 607c8ec6 - Address DB review feedback

Compare with previous version

requested review from @mayra-cabrera

requested review from @ahegyi

removed review request for @mayra-cabrera

added 2234 commits

• 607c8ec6...19cd1eea - 2230 commits from branch master
• e33a17a2 - Updated cached issue weight and count for epics
• 4695e6bf - Minor changes based on review
• 61554d7d - Address DB review feedback
• 3392fe7f - Use in-operator optimization

Compare with previous version

mentioned in merge request !95048 (merged)

changed title from Update cached issue weight and count for epics to Update cached issue weight and count for epics (project/group deletion)

changed title from Update cached issue weight and count for epics (project/group deletion) to Update epic cache counts on project/group deletion

added 270 commits

• 3392fe7f...0148f2e7 - 268 commits from branch master
• 995f5802 - Updated cached issue weight and count for epics
• 488b38ef - Update epic cache counts on project/group destroy

Compare with previous version

removed review request for @felipe_artur and @ahegyi

changed milestone to %15.4

added missed:15.3 label

added 1661 commits

• 488b38ef...eff9b033 - 1660 commits from branch master
• e7326d2c - Update epic cache counts on project/group destroy

Compare with previous version

	2 Warnings
	11fbd513: Commits that change 30 or more lines across at least 3 files should describe these changes in the commit body. For more information, take a look at our Commit message guidelines.
	featureaddition and featureenhancement merge requests normally have a documentation change. Consider adding a documentation update or confirming the documentation plan with the Technical Writer counterpart. For more information, see: The Handbook page on merge request types. The definition of done documentation.

	1 Message
	CHANGELOG missing: If you want to create a changelog entry for GitLab FOSS, add the Changelog trailer to the commit message you want to add to the changelog. If you want to create a changelog entry for GitLab EE, also add the EE: true trailer to your commit message. If this merge request doesn't need a CHANGELOG entry, feel free to ignore this message.

Reviewer roulette

Changes that require review have been detected!

Please refer to the table below for assigning reviewers and maintainers suggested by Danger in the specified category:

Category	Reviewer	Maintainer
backend	Tarun Vellishetty (@tvellishetty) (UTC+5.5, 3.5 hours ahead of @jprovaznik)	Kamil Trzciński (@ayufan) (UTC+2, same timezone as @jprovaznik)
database	Alexandru Croitor (@acroitor) (UTC+3, 1 hour ahead of @jprovaznik)	Adam Hegyi (@ahegyi) (UTC+2, same timezone as @jprovaznik)
~"codeowners"	Reviewer review is optional for ~"codeowners"	Adam Hegyi (@ahegyi) (UTC+2, same timezone as @jprovaznik)

To spread load more evenly across eligible reviewers, Danger has picked a candidate for each review slot, based on their timezone. Feel free to override these selections if you think someone else would be better-suited or use the GitLab Review Workload Dashboard to find other available reviewers.

To read more on how to use the reviewer roulette, please take a look at the Engineering workflow and code review guidelines. Please consider assigning a reviewer or maintainer who is a domain expert in the area of the merge request.

Once you've decided who will review this merge request, assign them as a reviewer! Danger does not automatically notify them for you.

If needed, you can retry the danger-review job that generated this comment.

Generated by Danger

added 1 commit

• c516087f - Iterate through issues using each_batch

Compare with previous version

added 38 commits

• c516087f...0414f16c - 36 commits from branch master
• c73d4d66 - Update epic cache counts on project/group destroy
• 644f2cc2 - Iterate through issues using each_batch

Compare with previous version

added 1 commit

• 37b40441 - Iterate through issues using each_batch

Compare with previous version

requested review from @ahegyi

added 143 commits

• 37b40441...8b544f7c - 141 commits from branch master
• 249366be - Update epic cache counts on project/group destroy
• 94fc01cd - Iterate through issues using each_batch

Compare with previous version

added 1 commit

• 764ec17b - Iterate through issues using each_batch

Compare with previous version

Thanks, @jprovaznik! I left a few small suggestions.

added 1 commit

• 52092e65 - Iterate through issues using each_batch

Compare with previous version

added 1 commit

• 9926eaa8 - Iterate through issues using each_batch

Compare with previous version

requested review from @felipe_artur

added databaseapproved label and removed databasereviewed label

approved this merge request

Thanks, @jprovaznik! DB LGTM!

added 371 commits

• 9926eaa8...758b9a40 - 369 commits from branch master
• 0a9ed750 - Update epic cache counts on project/group destroy
• 1efed070 - Iterate through issues using each_batch

Compare with previous version

added 1 commit

• fd4f84ed - Iterate through issues using each_batch

Compare with previous version

added 290 commits

• fd4f84ed...2bdb05a6 - 287 commits from branch master
• 1e1fe7fd - Update epic cache counts on project/group destroy
• 31013833 - Iterate through issues using each_batch
• 8b63519c - Address review comments

Compare with previous version

added 235 commits

• 8b63519c...8e9ee5e9 - 234 commits from branch master
• 889d31f2 - Update epic cache counts on project/group destroy

Compare with previous version

requested review from @cablett

Looks great @jprovaznik

I've left some very minor comments. I won't be able to finish this review, unfortunately.

@engwan would you be able to take over this review?

requested review from @engwan

added 219 commits

• 889d31f2...05342339 - 218 commits from branch master
• 4ecdb3e8 - Update epic cache counts on project/group destroy

Compare with previous version

Not related to MR, but just curious: How would cache_issue_sums work when some users don't have access to all issues under the epic?

I guess showing the total count / weight isn't really a problem in the epic boards. It would just be confusing if we use this in the epic tree and the user sees numbers that are inconsistent with the child issues they see in the tree.

Thanks @jprovaznik, left some questions / suggestions.

removed review request for @engwan

added 177 commits

• 4ecdb3e8...67fcef19 - 175 commits from branch master
• 9a2161e4 - Update epic cache counts on project/group destroy
• 11fbd513 - Use bulk_perform_in for scheduling jobs

Compare with previous version

resolved all threads

enabled an automatic merge when the pipeline for 451a8734 succeeds

added pipeline:run-as-if-foss label

requested review from @engwan

removed review request for @cablett

approved this merge request

resolved all threads

merged

@engwan, did you forget to run a pipeline before you merged this work? Based on our code review process, if the latest pipeline finished more than 2 hours ago, you should:

1. Ensure the merge request is not in Draft status.
2. Start a pipeline (especially important for Community contribution merge requests).
3. Set the merge request to merge when pipeline succeeds.

This is a guideline, not a rule. Please consider replying to this comment for transparency.

This message was generated automatically. You're welcome to improve it.

mentioned in commit 48247e7e

added workflowstaging-canary label

added workflowcanary label and removed workflowstaging-canary label

added workflowstaging label and removed workflowcanary label

added workflowproduction label and removed workflowstaging label

added workflowpost-deploy-db-staging label and removed workflowproduction label

added workflowpost-deploy-db-production label and removed workflowpost-deploy-db-staging label

resolved all threads

mentioned in merge request !98077 (merged)

added releasedcandidate label