Log pipeline variables count and largest value size

Oleg Yakovenkorequested to merge
613416-instrument-pipeline-variables-usage into master

What does this MR do and why?

We plan to add size and count limits to CI pipeline variables. We cannot pick safe values without knowing how many variables real pipelines pass today, and how large the largest single value is. This MR adds that measurement. There is no behaviour change.

The data is not available today. Pipeline variables used to live in p_ci_pipeline_variables. They are now written to an object storage backed Ci::PipelineArtifact as JSON. That artifact row stores only the total byte size of the whole set, so the number of variables and the size of any single value cannot be queried with SQL. The old table holds no recent records.

The change:

  • lib/gitlab/ci/pipeline/chain/build/associations.rb observes pipeline_variables_count and pipeline_variables_max_value_bytesize during pipeline creation.
  • It uses the existing Gitlab::Ci::Pipeline::Logger, which already tracks the job count as pipeline_size_count the same way.
  • app/services/ci/create_pipeline_service.rb adds two log_when thresholds: 500 variables (LOG_MAX_PIPELINE_VARIABLES) and 200 KB for a single value (LOG_MAX_PIPELINE_VARIABLE_VALUE_BYTESIZE). Those are the values currently under consideration as limits.

Cost, since pipeline creation is a high throughput path:

  • Logger#observe returns immediately when the logger is disabled, so the cost is two hash writes at most.
  • The logger sits behind the existing ci_pipeline_creation_logger ops feature flag. No new flag is added.
  • A log line is written only when a threshold trips. Volume scales with the tail, so it does not grow with pipeline throughput.
  • The count is Array#size. The largest value is one pass over the variables. String#bytesize is constant time in Ruby, so the work scales with the number of variables and not with their size. Most pipelines pass no variables at all.

What the data can answer: whether any real pipeline exceeds the values under consideration, and which project and root namespace it belongs to, so those users can be contacted before a limit ships. It does not produce a full distribution below the thresholds. The distribution of total size per pipeline is already available from the artifact size column, so this MR does not duplicate it.

No changelog entry is needed. The change is internal observability with no user facing effect.

References

Screenshots or screen recordings

Not applicable. There is no UI change.

How to set up and validate locally

  1. Enable the ops flag: Feature.enable(:ci_pipeline_creation_logger).
  2. Create a pipeline that passes more than 500 variables, or one variable whose value is over 200 KB. Use the run pipeline UI or POST /api/v4/projects/:id/pipeline.
  3. Check log/application_json.log for a Gitlab::Ci::Pipeline::Logger entry carrying pipeline_variables_count and pipeline_variables_max_value_bytesize.

Verified on a local GDK. A pipeline with a 250,000 byte value logged:

{
  "class": "Gitlab::Ci::Pipeline::Logger",
  "pipeline_variables_count": 2,
  "pipeline_variables_max_value_bytesize": 250000,
  "meta.project": "root/tesrails",
  "meta.root_namespace": "root"
}

A pipeline under both thresholds collected the same observations and wrote no log line.

MR acceptance checklist

Evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.

Edited by Oleg Yakovenko

Merge request reports