Add histograms for pipeline variable count and size
What does this MR do and why?
Adds two Prometheus histograms next to the log observations from !253383 (merged), in Chain::Build::Associations:
gitlab_ci_pipeline_variables_count, unlabelledgitlab_ci_pipeline_variables_max_value_bytes, labelled byvariable_type
The logger only writes a line when one of its thresholds trips, so it gives a dependable maximum and no distribution. Percentiles taken from it describe the pipelines that tripped a threshold, not the fleet. That was not enough to size the limits in !253433, where the observed peak kept rising as the measurement window grew. Histograms observe every pipeline.
They are also not feature flag gated, unlike the logger, which is off by default. GitLab Dedicated cannot set feature flags, so it produces no logger data today and would start reporting these as soon as the release lands.
Labels and buckets
variable_type on the size histogram, because env_var and file have separate limits (200 KiB and 1 MiB on GitLab.com, 600 KiB and 3 MiB by default). A combined maximum cannot show which limit the largest values are near, which is the open question. Only types present in a pipeline are observed, so each series _count means "pipelines that passed a variable of this type".
The count histogram is unlabelled, since a pipeline carries a mix of types.
Bucket boundaries map to questions rather than being evenly spaced. For the count, [0, 1, 10, 100, 500, 1000, 1200, 3600]: 0 is the denominator, 500 matches LOG_MAX_PIPELINE_VARIABLES so the two sources agree, 1200 is the GitLab.com limit in !253433, and 3600 is the default limit for GitLab Self-Managed and GitLab Dedicated, so +Inf is the set the default would reject. For the size, [0, 1 KiB, 8 KiB, 64 KiB, 128 KiB, 200 KiB, 512 KiB, 600 KiB, 1 MiB, 3 MiB]: 200 KiB and 1 MiB are the GitLab.com limits, and 600 KiB and 3 MiB are the defaults.
Cost
Two observations per pipeline creation, in a method that already walks the variables array for the logger. No queries, no flag, about 32 series.
References
- https://gitlab.com/gitlab-org/gitlab/-/issues/613416 (confidential)
- !253383 (merged), merged, added the log observations these sit beside
- !253433, the limits these are intended to inform
Screenshots or screen recordings
Not applicable, no UI change.
How to set up and validate locally
Create a pipeline passing a few variables, at least one with variable_type: file, then check curl localhost:3000/-/metrics | grep gitlab_ci_pipeline_variables.
No changelog entry: instrumentation only, no user facing effect.
MR acceptance checklist
Evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.