Track agent versions metrics for Gitlab.com users
Follow-up issue for Export a metric with connected agents versions (#505 - closed).
We've added an API that returns agent versions and number of connections for each version. But, we need to find a way to expose the agent versions metrics without high cardinality problem. From Mikhail's comment:
if we naively expose this data as a prometheus metric, we are running a risk of introducing a high cardinality metric. This is because we don't control what versions clients send us. A rogue client can flood us with bogus version numbers, causing an incident. So, ideally we'd use a tool that can handle high cardinality data. Maybe something from the analytics team?
Proposal
For Gitlab.com, I propose to leverage existing internal event tracking by adding additional properties. We can add maximum of three properties to the events. Those properties will be able to queried in Tableau.
- Define three additional properties. It should be
agent_version
,kubernetes_version
, andos-architecture
. (We combineos
andarchitecture
not to lose information under the limitation of max. three properties.) - Update KAS to send events with additional properties information.
- Update Rails monolith to send additional properties to Internal tracking after receiving them from KAS.
We will need a different approach for self-managed instances. Here's the issue for self-managed: Track agentk usage in self-managed instances (#547)
Click to expand for previous proposal
Previous Proposal
Use ServicePing to send agent versions metrics.
- Clarify how we can query the data. If the data is not good to use them in Tableau, we need to change.
- Define object schema for ServicePing. A json with schema described like this.
- Add instrumentation class to Gitlab monolith. This is a weekly process. It will fetch data from KAS and calculates the value for ServicePing. This will be available in Tableau.
Proposed schema:
{
"type": "array",
"items": {
"type": [
{
"type": "object",
"properties": {
"agent_version": {
"type": "string"
},
"connected_agents": {
"type": "integer"
}
}
}
]
}
}
Data example:
// Gitlab Agent Versions
[
{
"agent_version": "v16.9.0",
"connected_agents": 1234
},
{
"agent_version": "v16.9.1",
"connected_agents": 2345
}
]