Track agent versions metrics for Gitlab.com users
Follow-up issue for Export a metric with connected agents versions (#505 - closed).
We've added an API that returns agent versions and number of connections for each version. But, we need to find a way to expose the agent versions metrics without high cardinality problem. From Mikhail's comment:
if we naively expose this data as a prometheus metric, we are running a risk of introducing a high cardinality metric. This is because we don't control what versions clients send us. A rogue client can flood us with bogus version numbers, causing an incident. So, ideally we'd use a tool that can handle high cardinality data. Maybe something from the analytics team?
Proposal
For Gitlab.com, I propose to create a new event that will be emitted when KAS registers the agent. We can add additional properties to the event.
- Define additional properties.
We can use two string fields, that areWe will track two properties –label
andproperty
. Addlabel
:agent_version:v17.0.0-rc1|k8s_version:1.30.1
(=<AGENT_VERSION>|<K8S_VERSION>
) andproperty
:os:debian-10|architecture:x86_64
(=<OS>|<ARCH>
).agent_version
(added tolabel
) andarchitecture
(added toproperty
). (We selected only two most important attributes because we will be able to add more attributes after gitlab-org/gitlab#462378 (closed) is merged. See gitlab-org/gitlab!154883 (comment 1942463320) for more information.) - Update KAS to send the new event with additional properties information.
- Update Rails monolith to receive them from KAS, and emit the new event with additional properties.
We will need a different approach for self-managed instances. Here's the issue for self-managed: Track agentk usage in self-managed instances (#547)
Click to expand for previous proposal
Previous Proposal
Use ServicePing to send agent versions metrics.
- Clarify how we can query the data. If the data is not good to use them in Tableau, we need to change.
- Define object schema for ServicePing. A json with schema described like this.
- Add instrumentation class to Gitlab monolith. This is a weekly process. It will fetch data from KAS and calculates the value for ServicePing. This will be available in Tableau.
Proposed schema:
{
"type": "array",
"items": {
"type": [
{
"type": "object",
"properties": {
"agent_version": {
"type": "string"
},
"connected_agents": {
"type": "integer"
}
}
}
]
}
}
Data example:
// Gitlab Agent Versions
[
{
"agent_version": "v16.9.0",
"connected_agents": 1234
},
{
"agent_version": "v16.9.1",
"connected_agents": 2345
}
]