Instrument CI template usage across projects
What does this MR do?
Uses Redis HLL counters to track the number of distinct projects that
run pipelines with include:template
for a few templates of interest.
The tracking is behind the common feature flag track_ci_templates_unique_projects
so that it can all be turned off at once if there are performance concerns.
See #219457 (closed)
Redis performance considerations
Quoting @cmiskell
#219457 (comment 453068217):
[...] if we had to touch 22 template counters each time (would that actually be the case? Surely it would only touch the counter if the relevant template was actually included?), that'd be ~100/s, which is non-trivial but probably ok (
pfadd
is supposed to be fairly light/efficient).
So a generous upper bound is 100 pfadd/s
. Additionally, based on the existing usage data of Auto DevOps, we fairly are confident the templates being measured are only used by a small fraction of pipelines, so the actual load should be much smaller.
Manual QA
- Created a pipeline in my GDK with one of the tracked templates
- Observed the correct counters incremented in redis
Does this MR meet the acceptance criteria?
Conformity
-
Changelog entry - [-] Documentation (if required)
-
Code review guidelines -
Merge request performance guidelines -
Style guides - [-] Database guides
- [-] Separation of EE specific content
Availability and Testing
-
Review and add/update tests for this feature/bug. Consider all test levels. See the Test Planning Process. - [-] Tested in all supported browsers
- [-] Informed Infrastructure department of a default or new setting change, if applicable per definition of done
Security
If this MR contains changes to processing or storing of credentials or tokens, authorization and authentication methods and other items described in the security review guidelines:
- [-] Label as security and @ mention
@gitlab-com/gl-security/appsec
- [-] The MR includes necessary changes to maintain consistency between UI, API, email, or other methods
- [-] Security reports checked/validated by a reviewer from the AppSec team