Add worker to execute project hooks async in sidekiq

What does this MR do and why?

Add worker to execute project hooks async in sidekiq. This is one of many efforts to improve pipeline creation performance and make it fast.

Add feature flag to derisk .com deployments

Feature flag issue - #512832 (closed)

References

Please include cross links to any resources that are relevant to this MR. This will give reviewers and future readers helpful context to give an efficient review of the changes introduced.

MR acceptance checklist

Please evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.

Screenshots or screen recordings

Performance before/after the async execution
Before without async execution of hooks After async execution of hooks
Screenshot_2025-01-20_at_10.51.45_AM Screenshot_2025-01-20_at_10.52.05_AM
execution time is 31.18ms execution time is 9.28ms 🎉

How to set up and validate locally

Setup required for testing

Enable webhooks in local gdk

  1. Allow accepting localhost URLs for webhook URL input. This can be enabled from Outbound requests. Check the box for Allow requests to the local network from webhooks and integrations. This is done by instance Admin user.
  2. Use the sample script to start a webhook server -
Webhook server code
require 'webrick'

server = WEBrick::HTTPServer.new(
  Host: '0.0.0.0',
  Port: 8000,
  Logger: WEBrick::Log.new($stdout),
  AccessLog: []
)

server.mount_proc '/' do |req, res|
  puts "\n=== Received Webhook at #{Time.now} ==="
  puts "Method: #{req.request_method}"
  puts "Path: #{req.path}"
  puts "Headers: #{req.header}"
  puts "Body: #{JSON.parse(req.body)}" rescue puts "Body: #{req.body}"
  puts "==================\n"
  
  res.status = 200
  res.body = 'OK'
end

trap 'INT' do 
  server.shutdown
  puts "\nWebhook server stopped"
end

puts "Starting webhook server on http://localhost:8000"
server.start
  1. Enable webhooks for Job events by going to Settings > Webhooks. Add a new webhook by using the URL that you got from starting the webhook server from the code in step 2. Uncheck the SSL Verification option.
  2. Make sure you have CI/CD setup for running pipelines in local GDK.
  3. Enable the feature flag in rails console with Feature.enable(:ci_async_build_hooks_execution) and run the webhook server from step 2.
  4. Start a new pipeline in local gdk on the project that you enabled webhooks with any sample CI/CD Yaml.
  5. Everything should be run as expected. You should see some logs printed to shell by the webhook server.

Testing for reduced execution time for executing project hooks

  1. We use flamegraph and speedscope to understand the request timings.

  2. First, we disable the feature flag and check execution times for execute_hooks method call.

Curl for starting pipeline with feature flag disabled
curl --request POST \
  --header "PRIVATE-TOKEN: <PAT>" \
  --header "Content-Type: application/json" \
  "http://gdk.test:3000/api/v4/projects/<project_id>/pipeline?ref=master&performance_bar=flamegraph" \
  --output flamegraph_old.html
Curl for starting pipeline with feature flag enabled
curl --request POST \
  --header "PRIVATE-TOKEN: <PAT>" \
  --header "Content-Type: application/json" \
  "http://gdk.test:3000/api/v4/projects/<project_id>/pipeline?ref=master&performance_bar=flamegraph" \
  --output flamegraph_new.html
  1. Extract the information to json using the below script.

#!/usr/bin/env ruby

def extract_flamegraph(html_file, json_file)
  content = File.read(html_file)
  if match = content.match(/var graph = (.*?);/)
    File.write(json_file, match[1])
    puts "Extracted flamegraph data to #{json_file}"
  else
    puts "No flamegraph data found in #{html_file}"
  end
end

# Usage
if ARGV.length != 2
  puts "Usage: ruby extract_flamegraph.rb <input_html> <output_json>"
  exit 1
end

extract_flamegraph(ARGV[0], ARGV[1]) # pass the file paths here for old and new.
  1. Head over to Speedscope and upload both json in 2 different tabs. Search for Ci::Build#execute_hooks and hover over the horizontal line to see the execution time. There should be visible difference in old timings with feature flag disabled and new timings with feature flag enabled.

Related to #499290 (closed)

Edited by Rajendra Kadam

Merge request reports

Loading
Loading