Investigate removing the token gitlab_admin_api_token

Problem

While investigating a problem in CDot stg where the GitLab admin token had expired, it was noted that this token seems to only be used in one place in CDot code to check if GitLab is alive. As noted in #13032 (comment 2507204731), the token is used in about 10 places in Mechanizer code used by Support. This should be included in consideration for removing the token, as long as Mechanizer is still in use.

There is a lot of overhead in managing and rotating these tokens every year for each environment. If we can avoid this token, it would save time and effort.

Proposal

Let's investigate if we can use something other than the admin token to check if the GitLab API is alive. Perhaps we could use JWT or check a different endpoint that doesn't require authentication at all.

If the token is not needed, we can do some cleanup in various places:

  • Remove the gitlab_admin_api_token references in CDot code, including all credentials files
  • Update documentation to no longer mention this token
  • Close any issues mentioning rotating this token in the future
  • Delete any Slack reminders about rotating this token. Reminders may be referenced in the rotation issues.

If the token is still needed, we should be mindful that IT support considers PAT are not the best practice. They have suggested to create a service account with admin privileges that we can impersonate into and create the token. Let's consider this as an option if we keep the token.

Note: In https://gitlab.com/gitlab-org/customers-gitlab-com/-/issues/9772+, this issue involves removing the usage of the Admin's PAT for the health check service similar to this, but goes further into changing the health check service to work within the Cells architecture. This issue could be a step towards that work. Whatever we end up doing here, let's make a note in that issue as well.

Edited by Tyler Amos