Skip to content

Update Create pipeline worker

Marcos Rocha requested to merge mc_rocha-update-create-pipeline-worker into master

What does this MR do and why?

A scheduled scan execution policy on a group level for large groups can quickly exhaust resources and slow development, as all scans are executed simultaneously. To avoid this scenario, we added a new worker in MR !147691 (merged) and an application setting to set the concurrency limit in MR !152855 (merged).

This MR adds a pause strategy for the CreatePipelineWorker, considering the concurrency limit.

Related &13997

Database query

SELECT
    COUNT(*)
FROM
    "p_ci_builds"
    INNER JOIN "ci_pipelines" "pipeline" ON "pipeline"."partition_id" IS NOT NULL
        AND "pipeline"."id" = "p_ci_builds"."commit_id"
        AND "pipeline"."partition_id" = "p_ci_builds"."partition_id"
WHERE
    "p_ci_builds"."type" = 'Ci::Build'
    AND "pipeline"."source" = 15
    AND ("p_ci_builds"."status" IN ('preparing', 'pending', 'running', 'waiting_for_callback', 'waiting_for_resource', 'canceling', 'created'))
    AND "p_ci_builds"."created_at" > '2024-06-25 17:35:38.652585'
    AND "p_ci_builds"."updated_at" > '2024-06-25 17:35:38.652722'

https://postgres.ai/console/gitlab/gitlab-production-ci/sessions/29468/commands/91557

MR acceptance checklist

Please evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.

Screenshots or screen recordings

Worker pause

Screen_Recording_2024-06-25_at_3.46.52_PM

Worker resume

Screen_Recording_2024-06-25_at_4.34.32_PM

How to set up and validate locally

  1. Create a new group
  2. Create some projects using the script
user = User.first
namespace_id = Group.last.id

5.times do
  project_params = {
    namespace_id: namespace_id,
    name: "Test-#{FFaker::Lorem.characters(15)}"
  }

  project = ::Projects::CreateService.new(user, project_params).execute
  project.save!

  project.repository.create_file(user, 'Gemfile.lock', '', branch_name: Gitlab::DefaultBranch.value,
    message: 'Add Gemfile.lock file')
  project.repository.create_file(user, 'test.rb', 'puts "hello world"', branch_name: Gitlab::DefaultBranch.value,
    message: 'Add test.rb file')

  5.times do
    branch_name = "branch-#{FFaker::Lorem.characters(15)}"
    ::Branches::CreateService.new(project, user).execute(branch_name, project.default_branch)
  end
end
  1. Go to the Group page
  2. Go to Secure > Policies
  3. Click in new policy
  4. Select Scan Execution Policy
  5. Change to the .yaml mode
  6. Copy the policy content below
type: scan_execution_policy
name: policy
description: ''
enabled: true
policy_scope:
  projects:
    excluding: []
rules:
  - type: schedule
    cadence: 0 0 * * *
    timezone: Etc/UTC
    branch_type: all
actions:
  - scan: secret_detection
  - scan: sast
  - scan: sast_iac
  - scan: container_scanning
  - scan: dependency_scanning

  1. Merge the policy

  2. Enable the feature flags

Feature.enable(:scan_execution_pipeline_worker)
Feature.enable(:scan_execution_pipeline_worker_pause)
  1. Go to the Admin Area
  2. Go to settings > CI/CD > Continuous Integration and Deployment
  3. Update the Security policy scheduled scans maximum concurrency value to 5
  4. Trigger the scheduled scans

Get the schedule id in rails console

rule_schedule_id = Security::OrchestrationPolicyRuleSchedule.last.id

Update the schedule next run_at to a time in the past using the gdk psql

UPDATE security_orchestration_policy_rule_schedules SET next_run_at = '2024-05-28 00:15:00+00' WHERE id = <rule_schedule_id>;

trigger the schedule in the rails console

Security::OrchestrationPolicyRuleScheduleNamespaceWorker.new.perform(rule_schedule_id)
  1. Go to Monitoring > Background Jobs in the Admin Area

<gdk_url>/admin/sidekiq/busy

  1. Verify the job will be paused
  2. Verify the job will be resumed

To make the worker resume faster you can increase the concurrency limit:

Go to the Admin Area Go to settings > CI/CD > Continuous Integration and Deployment Update the Security policy scheduled scans maximum concurrency value to a higher value

You can use the pause strategy query to check the number of current jobs

 ::Ci::Build.with_pipeline_source_type('security_orchestration_policy')
.with_status(*::Ci::HasStatus::ALIVE_STATUSES)
.created_after(1.hour.ago)
.updated_after(1.hour.ago)
.order_by_created_at_asc
.order_by_project_id_asc
Edited by Marcos Rocha

Merge request reports