Fine tune over limit email notification worker

What does this MR do and why?

Fine tune over limit email notification worker as current method times out for sql statements - see https://gitlab.com/gitlab-org/gitlab/-/issues/413547#note_1422454511

As per !118906 (comment 1420459287)

• Remove skip locked since we only use one worker with a 24hr buffer
• Use read replica for reads
• load the namespace_details in a separate query

Database

• select namespaces for the update - https://console.postgres.ai/gitlab/gitlab-production-tunnel-pg12/sessions/19300/commands/63702 (~530ms)
• update those namespaces selected - (ids 102-202 as a simulation of the output from the select) - https://console.postgres.ai/gitlab/gitlab-production-tunnel-pg12/sessions/19300/commands/63705 (~45ms)

MR acceptance checklist

This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.

• I have evaluated the MR acceptance checklist for this MR.

Related to #413547

changed milestone to %16.1

added Next Up bugperformance devopsgrowth groupacquisition priority1 sectiongrowth severity4 typebug workflowverification labels

assigned to @dstull

added backend label

Reviewer roulette

Changes that require review have been detected!

Please refer to the table below for assigning reviewers and maintainers suggested by Danger in the specified category:

Category	Reviewer	Maintainer
backend	Sylvester Chin (@schin1) (UTC+8, 12 hours ahead of @dstull)	Stan Hu (@stanhu) (UTC-7, 3 hours behind @dstull)
database	Gregory Havenga (@ghavenga) (UTC+2, 6 hours ahead of @dstull)	Matt Kasa (@mattkasa) (UTC-7, 3 hours behind @dstull)

To spread load more evenly across eligible reviewers, Danger has picked a candidate for each review slot, based on their timezone. Feel free to override these selections if you think someone else would be better-suited or use the GitLab Review Workload Dashboard to find other available reviewers.

To read more on how to use the reviewer roulette, please take a look at the Engineering workflow and code review guidelines. Please consider assigning a reviewer or maintainer who is a domain expert in the area of the merge request.

Once you've decided who will review this merge request, assign them as a reviewer! Danger does not automatically notify them for you.

If needed, you can retry the danger-review job that generated this comment.

Generated by Danger

added 1 commit

• 8678d77b - Fine tune over limit email notification worker

Compare with previous version

changed the description

Allure report

allure-report-publisher generated test report!

e2e-test-on-gdk: test report for c4a3c017

expand test summary

+-----------------------------------------------------------------------+
|                            suites summary                             |
+------------------+--------+--------+---------+-------+-------+--------+
|                  | passed | failed | skipped | flaky | total | result |
+------------------+--------+--------+---------+-------+-------+--------+
| Create           | 8      | 0      | 1       | 0     | 9     | ✅     |
| Data Stores      | 2      | 0      | 0       | 1     | 2     | ❗     |
| Monitor          | 4      | 0      | 0       | 0     | 4     | ✅     |
| Plan             | 4      | 0      | 0       | 0     | 4     | ✅     |
| Govern           | 2      | 0      | 0       | 0     | 2     | ✅     |
| Manage           | 1      | 0      | 0       | 0     | 1     | ✅     |
| Framework sanity | 0      | 0      | 1       | 0     | 1     | ➖     |
+------------------+--------+--------+---------+-------+-------+--------+
| Total            | 21     | 0      | 2       | 1     | 23    | ❗     |
+------------------+--------+--------+---------+-------+-------+--------+

marked the checklist item I have evaluated the MR acceptance checklist for this MR. as completed

@syarynovskyi can you perform initial backend review?

added workflowin review label and removed workflowverification label

requested review from @syarynovskyi

added database label

added databasereview pending label

added 1 commit

• c7addfc2 - Fine tune over limit email notification worker

Compare with previous version

changed the description

@ck3g can you perform database review on this one due to your knowledge from !118906 (merged)?

Afterwards if you would please pass this onto @ahegyi that would be great!

requested review from @ck3g