Track agent versions metrics for Gitlab.com users

Follow-up issue for Export a metric with connected agents versions (#505 - closed).

We've added an API that returns agent versions and number of connections for each version. But, we need to find a way to expose the agent versions metrics without high cardinality problem. From Mikhail's comment:

if we naively expose this data as a prometheus metric, we are running a risk of introducing a high cardinality metric. This is because we don't control what versions clients send us. A rogue client can flood us with bogus version numbers, causing an incident. So, ideally we'd use a tool that can handle high cardinality data. Maybe something from the analytics team?

Proposal

For Gitlab.com, I propose to create a new event that will be emitted when KAS registers the agent. We can add additional properties to the event.

1. Define additional properties. We can use two string fields, that are label and property. Add label: agent_version:v17.0.0-rc1|k8s_version:1.30.1 (= <AGENT_VERSION>|<K8S_VERSION>) and property: os:debian-10|architecture:x86_64 (=<OS>|<ARCH>). We will track two properties – agent_version (added to label) and architecture (added to property). (We selected only two most important attributes because we will be able to add more attributes after gitlab-org/gitlab#462378 (closed) is merged. See gitlab-org/gitlab!154883 (comment 1942463320) for more information.)
2. Update KAS to send the new event with additional properties information.
3. Update Rails monolith to receive them from KAS, and emit the new event with additional properties.

We will need a different approach for self-managed instances. Here's the issue for self-managed: Track agentk usage in self-managed instances (#547 - closed)

Click to expand for previous proposal

Previous Proposal

Use ServicePing to send agent versions metrics.

1. Clarify how we can query the data. If the data is not good to use them in Tableau, we need to change.
2. Define object schema for ServicePing. A json with schema described like this.
3. Add instrumentation class to Gitlab monolith. This is a weekly process. It will fetch data from KAS and calculates the value for ServicePing. This will be available in Tableau.

Proposed schema:

{
    "type": "array",
    "items": {
      "type": [
        {
          "type": "object",
          "properties": {
            "agent_version": {
              "type": "string"
            },
            "connected_agents": {
              "type": "integer"
            }
          }
        }
      ]
    }
}

Data example:

// Gitlab Agent Versions
[
    {
        "agent_version": "v16.9.0",
        "connected_agents": 1234
    },
    {
        "agent_version": "v16.9.1",
        "connected_agents": 2345
    }
]