Evaluate strategy to include container registry read events in usage metrics
Context
Right now the container registry is delivery async webhook notifications for write (and write only) operations to Rails. Rails then uses this to (among other) increment internal counters, which in turn are used for usage metrics.
Problem
Right now the container registry usage metrics are lacking contribution from read operations, which are the vast majority of the registry usage (~95% on GitLab.com, so we're missing the vast majority of user interactions).
We can easily enable webhook notifications for pull events through the registry settings (docs), but this would lead to an extreme increase of outgoing events, which could pose scalability/performance challenges to both registry (publisher) and rails (consumer).
Aside from this, we also need to process these new events on the Rails side and feed them into the usage metrics.
Task
- Determine current request rate of outbound and inbound events
- Estimate % increase for the above from enabling events for read operations
- Consider temporary registry change to allow a gradual enablement of pull events (currently it's either all or no pull events, ideally we'd be able to enable pull events for e.g. manifest downloads and only then for blob downloads) to soften the curve
- Identify what's missing on the Rails side to process the upcoming pull events and increment usage metrics
- Create issues for all that needs to be changed on both ends