Make register job service to be resillient

What does this MR do?

Make scheduler to be resilient to data integrity issues: https://gitlab.com/gitlab-org/gitlab/issues/34897.

This prevents the problem as https://gitlab.com/gitlab-com/gl-infra/infrastructure/issues/8262 https://gitlab.com/gitlab-org/gitlab/issues/34897.

The scheduler failure is considered retry able, and by default it is retried 2 extra times.

Does this MR meet the acceptance criteria?

Conformity

• Changelog entry
• Documentation created/updated or follow-up review issue created
• Code review guidelines
• Merge request performance guidelines
• Style guides
• Database guides
• Separation of EE specific content

app/models/ci/build.rb

@@ -35,6 +35,10 @@ class Build < CommitStatus
       refspecs: -> (build) { build.merge_request_ref? }
     }.freeze
 
+    DEFAULT_RETRIES = {
+      "scheduler_failure" => 2
+    }.freeze
+
     has_one :deployment, as: :deployable, class_name: 'Deployment'
     has_many :trace_sections, class_name: 'Ci::BuildTraceSection'
     has_many :trace_chunks, class_name: 'Ci::BuildTraceChunk', foreign_key: :build_id
@@ -372,6 +376,14 @@ def retries_count
       pipeline.builds.retried.where(name: self.name).count
     end
 
+    def max_retries_for_reason(reason)
+      if retry_when.include?(reason)
+        retries_max
+      else
+        DEFAULT_RETRIES[reason]
+      end
+    end
+
     def retries_max
       normalized_retry.fetch(:max, 0)
     end
@@ -381,9 +393,12 @@ def retry_when
     end
 
     def retry_failure?
-      return false if retries_max.zero? || retries_count >= retries_max
+      max_allowed_retries =
+        max_retries_for_reason('always') ||
+        max_retries_for_reason(failure_reason.to_s) ||
+        0
 
-      retry_when.include?('always') || retry_when.include?(failure_reason.to_s)
+      max_allowed_retries > 0 && retries_count < max_allowed_retries
     end
 
     def latest?
@@ -831,6 +846,13 @@ def deployment_status
       :creating
     end
 
+    # Consider this object to have a structural integrity problems
+    def doom!
+      update_columns(
+        status: :failed,
+        failure_reason: :data_integrity_failure)
+    end
+
     private
 
     def successful_deployment_status