Skip to content

Draft: Move retry after build failure to be async [RUN ALL RSPEC] [RUN AS-IF-FOSS]

Allison Browne requested to merge 324369-reduce-sync-queries-triggered-by-api-jobs-request-assign_runner into master

What does this MR do?

Right now as part of the state machine code we execute reries on failure. This currently happens synchronously. Here are the queries executed via a retry according to the rails console:

[3] pry(main)> Ci::Build.retry(build, build.user)
  User Load (2.5ms)  SELECT "users".* FROM "users" WHERE "users"."id" = 1 LIMIT 1 /*application:console,line:(pry):3:in `<main>'*/
  Project Load (3.5ms)  SELECT "projects".* FROM "projects" WHERE "projects"."id" = 28 LIMIT 1 /*application:console,line:/app/models/ci/build.rb:223:in `retry'*/
  Namespace Load (1.6ms)  SELECT "namespaces".* FROM "namespaces" WHERE "namespaces"."id" = 1 LIMIT 1 /*application:console,line:/app/models/project.rb:2515:in `root_namespace'*/
  ApplicationSetting Load (2.3ms)  SELECT "application_settings".* FROM "application_settings" ORDER BY "application_settings"."id" DESC LIMIT 1 /*application:console,line:/app/models/concerns/cacheable_attributes.rb:19:in `current_without_cache'*/
  License Load (0.5ms)  SELECT "licenses".* FROM "licenses" ORDER BY "licenses"."id" DESC LIMIT 100 /*application:console,line:/ee/app/models/license.rb:297:in `load_license'*/
  License Load (0.3ms)  SELECT "licenses".* FROM "licenses" ORDER BY "licenses"."id" DESC LIMIT 100 /*application:console,line:/ee/app/models/license.rb:297:in `load_license'*/
  ProjectFeature Load (0.5ms)  SELECT "project_features".* FROM "project_features" WHERE "project_features"."project_id" = 28 LIMIT 1 /*application:console,line:/app/policies/project_policy.rb:719:in `access_allowed_to?'*/
  Group Load (0.7ms)  SELECT "namespaces".* FROM "namespaces" WHERE "namespaces"."type" = 'Group' AND "namespaces"."id" = 1 AND "namespaces"."type" = 'Group' LIMIT 1 /*application:console,line:/ee/app/policies/ee/project_policy.rb:342:in `block (2 levels) in <module:ProjectPolicy>'*/
  Route Load (0.7ms)  SELECT "routes".* FROM "routes" WHERE "routes"."source_id" = 28 AND "routes"."source_type" = 'Project' LIMIT 1 /*application:console,line:/app/models/concerns/routable.rb:103:in `full_path'*/
  ProtectedBranch Load (0.5ms)  SELECT "protected_branches".* FROM "protected_branches" WHERE "protected_branches"."project_id" = 28 /*application:console,line:/app/models/concerns/protected_ref.rb:66:in `matching'*/
  Ci::Pipeline Load (1.5ms)  SELECT "ci_pipelines".* FROM "ci_pipelines" WHERE "ci_pipelines"."id" = 171 LIMIT 1 /*application:console,line:/app/services/ci/retry_build_service.rb:37:in `public_send'*/
  ActsAsTaggableOn::Tagging Load (0.5ms)  SELECT "taggings".* FROM "taggings" WHERE "taggings"."taggable_id" = 1194 AND "taggings"."taggable_type" = 'CommitStatus' /*application:console,line:/app/services/ci/retry_build_service.rb:37:in `public_send'*/
  ActsAsTaggableOn::Tag Load (0.5ms)  SELECT "tags".* FROM "tags" INNER JOIN "taggings" ON "tags"."id" = "taggings"."tag_id" WHERE "taggings"."taggable_id" = 1194 AND "taggings"."taggable_type" = 'CommitStatus' AND (taggings.context = 'tags' AND taggings.tagger_id IS NULL) /*application:console,line:/app/services/ci/retry_build_service.rb:37:in `public_send'*/
  Ci::BuildNeed Load (0.3ms)  SELECT "ci_build_needs".* FROM "ci_build_needs" WHERE "ci_build_needs"."build_id" = 1194 /*application:console,line:/app/models/ci/processable.rb:147:in `map'*/
  Ci::BuildMetadata Load (0.8ms)  SELECT "ci_builds_metadata".* FROM "ci_builds_metadata" WHERE "ci_builds_metadata"."build_id" = 1194 LIMIT 1 /*application:console,line:/ee/app/models/concerns/ee/ci/metadatable.rb:9:in `secrets'*/
   (0.1ms)  BEGIN /*application:console,line:/app/services/ci/retry_build_service.rb:46:in `block in reprocess!'*/
  Ci::Build Update All (5.1ms)  UPDATE "ci_builds" SET "retried" = TRUE, "processed" = TRUE WHERE "ci_builds"."type" = 'Ci::Build' AND "ci_builds"."commit_id" = 171 AND ("ci_builds"."retried" = FALSE OR "ci_builds"."retried" IS NULL) AND "ci_builds"."name" = 'test' /*application:console,line:/app/services/ci/retry_build_service.rb:46:in `block in reprocess!'*/
  Ci::Build Load (1.0ms)  SELECT "ci_builds".* FROM "ci_builds" WHERE "ci_builds"."type" = 'Ci::Build' AND "ci_builds"."token_encrypted" = 'YTLsUZxUV95UiSdG1Usn1dJZ3wyX/IDmMZ+hw0A3a+vAO6H8' LIMIT 1 /*application:console,line:/app/models/concerns/token_authenticatable_strategies/encrypted.rb:91:in `find_by_encrypted_token'*/
  Ci::Build Load (0.5ms)  SELECT "ci_builds".* FROM "ci_builds" WHERE "ci_builds"."type" = 'Ci::Build' AND "ci_builds"."token" = 'eAPLx9thPs9f7VwwzM9x' LIMIT 1 /*application:console,line:/app/models/concerns/token_authenticatable_strategies/insecure.rb:6:in `find_token_authenticatable'*/
  Ci::Build Create (5.6ms)  INSERT INTO "ci_builds" ("status", "created_at", "updated_at", "commit_id", "name", "options", "stage", "stage_idx", "tag", "ref", "user_id", "type", "project_id", "when", "yaml_variables", "stage_id", "protected", "token_encrypted", "processed", "scheduling_type", "lock_version", "retried") VALUES ('created', '2021-03-30 14:44:47.960395', '2021-03-30 14:44:47.960395', 171, 'test', '---
:script:
- exit 1
', 'test', 2, FALSE, 'refs/merge-requests/2/head', 1, 'Ci::Build', 28, 'on_success', '--- []
', 355, FALSE, 'YTLsUZxUV95UiSdG1Usn1dJZ3wyX/IDmMZ+hw0A3a+vAO6H8', FALSE, 0, 0, FALSE) RETURNING "id" /*application:console,line:/lib/gitlab/database.rb:342:in `block in transaction'*/
  Ci::BuildMetadata Create (3.6ms)  INSERT INTO "ci_builds_metadata" ("build_id", "project_id", "has_exposed_artifacts") VALUES (1274, 28, FALSE) RETURNING "id" /*application:console,line:/lib/gitlab/database.rb:342:in `block in transaction'*/
  ActsAsTaggableOn::Tag Load (0.4ms)  SELECT "tags".* FROM "tags" INNER JOIN "taggings" ON "tags"."id" = "taggings"."tag_id" WHERE "taggings"."taggable_id" = 1274 AND "taggings"."taggable_type" = 'CommitStatus' AND (taggings.context = 'tags' AND taggings.tagger_id IS NULL) /*application:console,line:/lib/gitlab/database.rb:342:in `block in transaction'*/
   (1.1ms)  COMMIT /*application:console,line:/lib/gitlab/database.rb:342:in `block in transaction'*/
  License Load (0.2ms)  SELECT "licenses".* FROM "licenses" ORDER BY "licenses"."id" DESC LIMIT 100 /*application:console,line:/ee/app/models/license.rb:297:in `load_license'*/
  Ci::Processable Load (1.0ms)  SELECT "ci_builds".* FROM "ci_builds" WHERE "ci_builds"."type" IN ('Ci::Processable', 'Ci::Bridge', 'Ci::Build') AND "ci_builds"."commit_id" = 171 AND ("ci_builds"."status" IN ('skipped')) AND (stage_idx > 2) ORDER BY "ci_builds"."id" ASC LIMIT 1000 /*application:console,line:/app/services/ci/retry_build_service.rb:70:in `mark_subsequent_stages_as_processable'*/
  Ci::Pipeline Load (2.2ms)  WITH RECURSIVE "base_and_ancestors" AS ((SELECT "ci_pipelines".* FROM "ci_pipelines" WHERE "ci_pipelines"."id" = 171)
UNION
(SELECT "ci_pipelines".* FROM "ci_pipelines", "base_and_ancestors", "ci_sources_pipelines" WHERE "ci_sources_pipelines"."source_pipeline_id" = "ci_pipelines"."id" AND "ci_sources_pipelines"."pipeline_id" = "base_and_ancestors"."id" AND TRUE)) SELECT "ci_pipelines".* FROM "base_and_ancestors" AS "ci_pipelines" /*application:console,line:/app/models/ci/pipeline.rb:1211:in `reset_ancestor_bridges!'*/
  SQL (1.0ms)  SELECT "ci_sources_pipelines"."id" AS t0_r0, "ci_sources_pipelines"."project_id" AS t0_r1, "ci_sources_pipelines"."pipeline_id" AS t0_r2, "ci_sources_pipelines"."source_project_id" AS t0_r3, "ci_sources_pipelines"."source_job_id" AS t0_r4, "ci_sources_pipelines"."source_pipeline_id" AS t0_r5, "ci_builds"."id" AS t1_r0, "ci_builds"."status" AS t1_r1, "ci_builds"."finished_at" AS t1_r2, "ci_builds"."trace" AS t1_r3, "ci_builds"."created_at" AS t1_r4, "ci_builds"."updated_at" AS t1_r5, "ci_builds"."started_at" AS t1_r6, "ci_builds"."runner_id" AS t1_r7, "ci_builds"."coverage" AS t1_r8, "ci_builds"."commit_id" AS t1_r9, "ci_builds"."name" AS t1_r10, "ci_builds"."options" AS t1_r11, "ci_builds"."allow_failure" AS t1_r12, "ci_builds"."stage" AS t1_r13, "ci_builds"."trigger_request_id" AS t1_r14, "ci_builds"."stage_idx" AS t1_r15, "ci_builds"."tag" AS t1_r16, "ci_builds"."ref" AS t1_r17, "ci_builds"."user_id" AS t1_r18, "ci_builds"."type" AS t1_r19, "ci_builds"."target_url" AS t1_r20, "ci_builds"."description" AS t1_r21, "ci_builds"."project_id" AS t1_r22, "ci_builds"."erased_by_id" AS t1_r23, "ci_builds"."erased_at" AS t1_r24, "ci_builds"."artifacts_expire_at" AS t1_r25, "ci_builds"."environment" AS t1_r26, "ci_builds"."when" AS t1_r27, "ci_builds"."yaml_variables" AS t1_r28, "ci_builds"."queued_at" AS t1_r29, "ci_builds"."token" AS t1_r30, "ci_builds"."lock_version" AS t1_r31, "ci_builds"."coverage_regex" AS t1_r32, "ci_builds"."auto_canceled_by_id" AS t1_r33, "ci_builds"."retried" AS t1_r34, "ci_builds"."stage_id" AS t1_r35, "ci_builds"."protected" AS t1_r36, "ci_builds"."failure_reason" AS t1_r37, "ci_builds"."scheduled_at" AS t1_r38, "ci_builds"."token_encrypted" AS t1_r39, "ci_builds"."upstream_pipeline_id" AS t1_r40, "ci_builds"."resource_group_id" AS t1_r41, "ci_builds"."waiting_for_resource_at" AS t1_r42, "ci_builds"."processed" AS t1_r43, "ci_builds"."scheduling_type" AS t1_r44 FROM "ci_sources_pipelines" LEFT OUTER JOIN "ci_builds" ON "ci_builds"."id" = "ci_sources_pipelines"."source_job_id" AND "ci_builds"."type" = 'Ci::Bridge' WHERE "ci_builds"."type" = 'Ci::Bridge' AND "ci_sources_pipelines"."pipeline_id" = 171 /*application:console,line:/app/models/ci/pipeline.rb:1211:in `reset_ancestor_bridges!'*/
   (0.2ms)  BEGIN /*application:console,line:/lib/gitlab/ci/build/prerequisite/kubernetes_namespace.rb:27:in `deployment_cluster'*/
  Deployment Load (1.5ms)  SELECT "deployments".* FROM "deployments" WHERE "deployments"."deployable_id" = 1274 AND "deployments"."deployable_type" = 'CommitStatus' LIMIT 1 /*application:console,line:/lib/gitlab/ci/build/prerequisite/kubernetes_namespace.rb:27:in `deployment_cluster'*/
  Ci::Build Update (1.0ms)  UPDATE "ci_builds" SET "status" = 'pending', "updated_at" = '2021-03-30 14:44:48.073727', "queued_at" = '2021-03-30 14:44:48.072767', "processed" = FALSE, "lock_version" = 1 WHERE "ci_builds"."id" = 1274 AND "ci_builds"."lock_version" = 0 /*application:console,line:/lib/gitlab/database.rb:342:in `block in transaction'*/
  ActsAsTaggableOn::Tag Load (0.5ms)  SELECT "tags".* FROM "tags" INNER JOIN "taggings" ON "tags"."id" = "taggings"."tag_id" WHERE "taggings"."taggable_id" = 1274 AND "taggings"."taggable_type" = 'CommitStatus' AND (taggings.context = 'tags' AND taggings.tagger_id IS NULL) /*application:console,line:/lib/gitlab/database.rb:342:in `block in transaction'*/
   (0.7ms)  COMMIT /*application:console,line:/lib/gitlab/database.rb:342:in `block in transaction'*/
  MergeRequest Load (2.5ms)  SELECT "merge_requests".* FROM "merge_requests" WHERE "merge_requests"."id" = 71 AND ("merge_requests"."state_id" IN (1)) /*application:console,line:/app/services/merge_requests/base_service.rb:166:in `pipeline_merge_requests'*/
  User Load (0.5ms)  SELECT "users".* FROM "users" WHERE "users"."id" = 1 LIMIT 1 /*application:console,line:/app/models/merge_request.rb:684:in `merge_participants'*/
  Project Load (0.8ms)  SELECT "projects".* FROM "projects" WHERE "projects"."id" = 28 LIMIT 1 /*application:console,line:/app/policies/issuable_policy.rb:4:in `block in <class:IssuablePolicy>'*/
  ProjectFeature Load (0.3ms)  SELECT "project_features".* FROM "project_features" WHERE "project_features"."project_id" = 28 LIMIT 1 /*application:console,line:/app/policies/project_policy.rb:719:in `access_allowed_to?'*/
  Group Load (0.4ms)  SELECT "namespaces".* FROM "namespaces" WHERE "namespaces"."type" = 'Group' AND "namespaces"."id" = 1 AND "namespaces"."type" = 'Group' LIMIT 1 /*application:console,line:/ee/app/policies/ee/project_policy.rb:342:in `block (2 levels) in <module:ProjectPolicy>'*/
   (1.0ms)  SELECT "todos"."id" FROM "todos" WHERE "todos"."user_id" = 1 AND ("todos"."state" IN ('pending')) AND "todos"."project_id" = 28 AND "todos"."target_id" = 71 AND "todos"."target_type" = 'MergeRequest' AND "todos"."state" != 'done' /*application:console,line:/app/models/todo.rb:116:in `batch_update'*/
  Todo Update All (0.4ms)  UPDATE "todos" SET "state" = 'done', "resolved_by_action" = 0, "updated_at" = '2021-03-30 14:44:48.176014' WHERE "todos"."user_id" = 1 AND ("todos"."state" IN ('pending')) AND "todos"."project_id" = 28 AND "todos"."target_id" = 71 AND "todos"."target_type" = 'MergeRequest' AND "todos"."state" != 'done' /*application:console,line:/app/models/todo.rb:118:in `batch_update'*/
   (0.4ms)  SELECT COUNT(*) FROM "todos" WHERE "todos"."user_id" = 1 AND ("todos"."state" IN ('done')) /*application:console,line:/app/models/user.rb:1634:in `block in todos_done_count'*/
   (0.3ms)  SELECT COUNT(*) FROM "todos" WHERE "todos"."user_id" = 1 AND ("todos"."state" IN ('pending')) /*application:console,line:/app/models/user.rb:1640:in `block in todos_pending_count'*/

This MR shifts that logic to an async worker to reduce the number of API calls made syncronously.

Feature flag issue: #326776 (closed)

Screenshots (strongly suggested)

Does this MR meet the acceptance criteria?

Conformity

Availability and Testing

Security

If this MR contains changes to processing or storing of credentials or tokens, authorization and authentication methods and other items described in the security review guidelines:

  • Label as security and @ mention @gitlab-com/gl-security/appsec
  • The MR includes necessary changes to maintain consistency between UI, API, email, or other methods
  • Security reports checked/validated by a reviewer from the AppSec team

Related to #324369 (closed) and #326769 (closed)

Edited by Allison Browne

Merge request reports