Skip to content

Instrument CI template usage across projects

Hordur Freyr Yngvason requested to merge instrument-ci-template-usage-data into master

What does this MR do?

Uses Redis HLL counters to track the number of distinct projects that run pipelines with include:template for a few templates of interest.

The tracking is behind the common feature flag track_ci_templates_unique_projects so that it can all be turned off at once if there are performance concerns.

See #219457 (closed)

Redis performance considerations

Quoting @cmiskell #219457 (comment 453068217):

[...] if we had to touch 22 template counters each time (would that actually be the case? Surely it would only touch the counter if the relevant template was actually included?), that'd be ~100/s, which is non-trivial but probably ok (pfadd is supposed to be fairly light/efficient).

So a generous upper bound is 100 pfadd/s. Additionally, based on the existing usage data of Auto DevOps, we fairly are confident the templates being measured are only used by a small fraction of pipelines, so the actual load should be much smaller.

Manual QA

  1. Created a pipeline in my GDK with one of the tracked templates
  2. Observed the correct counters incremented in redis

Does this MR meet the acceptance criteria?

Conformity

Availability and Testing

Security

If this MR contains changes to processing or storing of credentials or tokens, authorization and authentication methods and other items described in the security review guidelines:

  • [-] Label as security and @ mention @gitlab-com/gl-security/appsec
  • [-] The MR includes necessary changes to maintain consistency between UI, API, email, or other methods
  • [-] Security reports checked/validated by a reviewer from the AppSec team
Edited by Hordur Freyr Yngvason

Merge request reports