The regional registry cluster is still required for the Sidekiq internal API
While executing the change for production#3095 (closed) I realized that there is a dependency on Sidekiq to use the internal endpoint for registry. Luckily we halted the change and reverted the removal of the registry service from the regional cluster gitlab-com/gl-infra/k8s-workloads/gitlab-com!552 (merged) before it caused any issue.
Sidekiq connects to the Registry API when containers are deleted to remove tags. The internal API URL default in the Chart is configured as api_url: http://gitlab-registry.gitlab.svc:5000
. Once the registry service is removed from the regional cluster, Sidekiq will not have a way to connect to registry over the internal API.
We don't have a very good way currently to connect to the registry using an internal IP and I'm not sure if we want to use the external registry.gitlab.com address for this. One option is to leave the registry service running in the regional cluster but with a smaller pod count by lowering the min pods. This might be the best option since we need to maintain the registry node pool in the regional cluster anyway.
Given that we never had an internal API endpoint for Registry on VMs, maybe it would be better for us to switch our registry endpoint to registry[.staging].gitlab.com
for consistency, and then we can address https://gitlab.com/gitlab-com/gl-infra/infrastructure/-/issues/12029 separately.
Resolution
The registry deployment for the regional cluster is now removed after reconfiguring sidekiq to use the external registry endpoint with gitlab-com/gl-infra/k8s-workloads/gitlab-com!554 (merged)
This will be necessary until we have an internal endpoint https://gitlab.com/gitlab-com/gl-infra/infrastructure/-/issues/12029.