Skip to content

Optimize decryption of CI variables

Stan Hu requested to merge sh-cache-runner-variable-decryption into master

What does this MR do and why?

In pipelines that have many environments and many CI variables, the decryption in Ci::HasVariable#to_runner_variable is a significant bottleneck due to repeated decryption via OpenSSL::PKCS5.pbkdf2_hmac. We can optimize this by caching the value in the request. In one customer's pipeline, this cuts down Ci::CreatePipelineService from 80+ seconds to 30 seconds.

Relates to #326271 (closed)

How to set up and validate locally

  1. Enable Feature.enable(:enable_ci_variable_caching).

  2. Create 170 environments and 100+ CI group variables (see #326271 (comment 812180988)) and profile the creation of the pipeline. For example:

project = Project.find(15)
current_user = User.first
pipeline_options = {}
pipeline_params = {
  before: '3bb7f2ff983cf6ceb62be4408b0e651c2c605c70',
  after: '11ca6d4aa27d17beeb49b63e0286e013fd149655',
  ref: 'refs/heads/master',
  checkout_sha: '11ca6d4aa27d17beeb49b63e0286e013fd149655'

service =, current_user, pipeline_params)

require 'stackprof'
Benchmark.measure do :wall, out: '/tmp/create-pipeline-service-test4.dump', interval: 1000, raw: true) do
    service.execute!(:push, pipeline_options)

MR acceptance checklist

This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.

Edited by Stan Hu

Merge request reports