Process webhook notifs for container registry events to track usage

What does this MR do and why?

Closes #385497 (closed)

This MR utilizes the existing webhook notifications and processes them to push data to Gitlab::UsageDataCounters::HLLRedisCounter and adding new events to the Service Ping.

New events:

• i_container_registry_delete_tag_user
• i_container_registry_push_tag_user
• i_container_registry_create_repository_user
• i_container_registry_delete_repository_user
• i_container_registry_push_repository_user

For groupcontainer registry, we want to have weekly and monthly aggregations of these events thus we have both monthly and weekly metric definitions. The events that are considered within this MR are the ones from a User.

To push unique user data, an id is needed and we get this from the User with the username in the actor["name"] of the event hash. We make sure that the event is from a User by looking into the actor["user_type"]. The User related ones are:

• personal_access_token
• build
• gitlab_or_ldap

See below for the event structure (reference:

{
   "events": [
      {
         "id": "a582a0f3-e620-43e6-8e98-ff850fc9d984",
         "timestamp": "2023-01-25T14:45:54.17327+11:00",
         "action": "push",
         "target": {
            "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
            "size": 528,
            "digest": "sha256:af06af3514c44a964d3b905b498cf6493db8f1cde7c10e078213a89c87308ba0",
            "length": 528,
            "repository": "root/test",
            "url": "http://registry.test:5000/v2/root/test/manifests/sha256:af06af3514c44a964d3b905b498cf6493db8f1cde7c10e078213a89c87308ba0",
            "tag": "more-auth"
         },
         "request": {
            "id": "8ef8d69b-f957-45ab-ba2c-9153291f47b6",
            "addr": "172.16.123.1:56969",
            "host": "registry.test:5000",
            "method": "PUT",
            "useragent": "docker/20.10.18 go/go1.18.6 git-commit/e42327a6d3c55ceda3bd5475be7aae6036d02db3 kernel/5.15.68-0-virt os/linux arch/arm64 UpstreamClient(Docker-Client/20.10.22 \\(darwin\\))"
         },
         "actor": {
            "name": "root",
            "user_type": "personal_access_token"
         },
         "source": {
            "addr": "Jaimes-MBP:5000",
            "instanceID": "45681f21-a006-42f2-ab7c-4cc37d8906b4"
         }
      }
   ]

Validating Locally

To validate locally, setup the Service Ping locally with this guide.

1.Verifying the metric definitions appear on usage data

curl http://gdk.test:3000/api/v4/usage_data/metric_definitions | grep  -C 5 'i_container_registry'

This would return the metric definitions.

2.Verify that our new counters are part of the redis_hll_counters

Open the rails console:

bundle exec rails c

And then query the following:

usage = GitlabServicePingWorker.new.usage_data;nil
usage.dig('redis_hll_counters', 'user_container_registry').keys

3.Verify that the counts are increased

The counters are incremented whenever the above events are emitted. They are emitted whenever:

• a tag gets deleted
• a tag gets pushed
• a repository gets deleted
• a repository gets created
• a repository gets pushed

To be able to do this, we will have to setup the Container Registry standalone on fresh master (outside the GDK) and enable notifications through the instructions here.

An alternative is to call the endpoint with the ContainerRegistry event payload format as documented in container-registry!1205 (merged):

To be able to curl without any tokens,

• open api/container_registry_event.rb and
• comment out the line before { authenticate_registry_notification! }

curl -d '{
    "events": [
      {
        "id": "a582a0f3-e620-43e6-8e98-ff850fc9d984",
        "timestamp": "2023-01-25T14:45:54.17327+11:00",
        "action": "delete",
        "target": {
          "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
          "size": 528,
          "digest": "sha256:af06af3514c44a964d3b905b498cf6493db8f1cde7c10e078213a89c87308ba0",
          "length": 528,
          "repository": "root/test",
          "url": "http://registry.test:5000/v2/root/test/manifests/sha256:af06af3514c44a964d3b905b498cf6493db8f1cde7c10e078213a89c87308ba0",
          "tag": "latest"
        },
        "request": {
          "id": "8ef8d69b-f957-45ab-ba2c-9153291f47b6",
          "addr": "172.16.123.1:56969",
          "host": "registry.test:5000",
          "method": "PUT",
          "useragent": "docker/20.10.18 go/go1.18.6 git-commit/e42327a6d3c55ceda3bd5475be7aae6036d02db3 kernel/5.15.68-0-virt os/linux arch/arm64 UpstreamClient(Docker-Client/20.10.22 \\(darwin\\))"
        },
        "actor": {
          "name": "root",
          "user_type": "personal_access_token"
        },
        "source": {
          "addr": "127.0.0.1:5000",
          "instanceID": "45681f21-a006-42f2-ab7c-4cc37d8906b4"
        }
      }
    ]
  }' -H "Content-Type: application/vnd.docker.distribution.events.v1+json"  -X POST  'http://localhost:3000/api/v4/container_registry_event/events'
 

And then to verify that the metric was counted, do a:

Gitlab::UsageDataCounters::HLLRedisCounter.unique_events(event_names: 'i_container_registry_delete_tag_user',
  start_date: Date.current.beginning_of_week, end_date: Date.current.next_week)

Replace i_container_registry_delete_tag_user with the event that you tried.

Notes:

1. Feel free to replace name: "root" to other users in your local machine.
2. Feel free to replace "action": "delete" with "action": "push" for a push event
3. Removing the line "tag": "latest" and the preceding comma will make it a repository event.

MR acceptance checklist

This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.

• I have evaluated the MR acceptance checklist for this MR.

Related to #385497 (closed)