Make register job service to be resillient

What does this MR do?

Make scheduler to be resilient to data integrity issues: https://gitlab.com/gitlab-org/gitlab/issues/34897.

This prevents the problem as https://gitlab.com/gitlab-com/gl-infra/infrastructure/issues/8262 https://gitlab.com/gitlab-org/gitlab/issues/34897.

The scheduler failure is considered retry able, and by default it is retried 2 extra times.

Does this MR meet the acceptance criteria?

Conformity

• Changelog entry
• Documentation created/updated or follow-up review issue created
• Code review guidelines
• Merge request performance guidelines
• Style guides
• Database guides
• Separation of EE specific content

changed milestone to %12.6

added Category:Continuous Integration GitLab.com Priority [deprecated] Accepting merge requests backend devopsverify grouppipeline execution internal customer priority2 severity2 typebug labels

@fabiopitino @smcgivern No tests yet, but this is what I have on my mind :) Asking reviewer and maintainer :)

assigned to @fabiopitino and @smcgivern

added 1 commit

• 8acc8834 - Make `jobs/request` to be resillient

Compare with previous version

added 1 commit

• f90565f6 - Make `jobs/request` to be resillient

Compare with previous version

added 1 commit

• 552247c3 - Make `jobs/request` to be resillient

Compare with previous version

added 1 commit

• ee836c09 - Make `jobs/request` to be resillient

Compare with previous version

resolved all threads

Thanks @ayufan! This sounds reasonable, I just want to make sure I understand correctly what is happening.

Also, the current milestone is %12.5

changed milestone to %12.5

unassigned @smcgivern

added 1 commit

• 383c953f - Make `jobs/request` to be resillient

Compare with previous version

Reviewer roulette

Changes that require review have been detected! A merge request is normally reviewed by both a reviewer and a maintainer in its primary category (e.g. frontend or backend), and by a maintainer in all other categories.

To spread load more evenly across eligible reviewers, Danger has randomly picked a candidate for each review slot. Feel free to override this selection if you think someone else would be better-suited, or the chosen person is unavailable.

Once you've decided who will review this merge request, mention them as you normally would! Danger does not (yet?) automatically notify them for you.

Category	Reviewer	Maintainer
backend	Adrien Kohlbecker (@akohlbecker)	Grzegorz Bizon (@grzesiek)

Generated by Danger

added 186 commits

• 383c953f...1d7bcbf3 - 185 commits from branch master
• b81dba0c - Make `jobs/request` to be resillient

Compare with previous version

@ayufan I only have small considerations. Looks good though!

assigned to @ayufan

unassigned @fabiopitino

removed [deprecated] Accepting merge requests label

added 1 commit

• e76c9212 - Make `jobs/request` to be resillient

Compare with previous version

unmarked as a Work In Progress

added 1 commit

• 838164b9 - Make `jobs/request` to be resillient

Compare with previous version

added 1 commit

• 8345d955 - Make `jobs/request` to be resillient

Compare with previous version

resolved all threads

mentioned in issue #35158 (closed)

assigned to @fabiopitino and @smcgivern and unassigned @ayufan

I resolved outstanding discussions with extra issue: #35158 (closed).

added 1 commit

• ec16f95c - Make `jobs/request` to be resillient

Compare with previous version

added 337 commits

• ec16f95c...9a4ca79b - 336 commits from branch master
• 486d0394 - Make `jobs/request` to be resillient

Compare with previous version

Thanks! @fabiopitino over to you.

unassigned @smcgivern

added 1 commit

• f5204bd7 - Make `jobs/request` to be resillient

Compare with previous version

assigned to @smcgivern

LGTM

assigned to @ayufan and unassigned @fabiopitino

added 23 commits

• f5204bd7...168db043 - 22 commits from branch master
• fd24f258 - Make `jobs/request` to be resillient

Compare with previous version