Manual validation of root user is required before running QA for a Cell through HTTP router

Summary

When QA tests are run, the QA user specified using the environment variables GITLAB_USERNAME and GITLAB_PASSWORD must have the ability to create top-level groups. This ability is available only to users who have a validated credit card.

Immediately after the creation of a Cell, the root user is used to run QA. This user is not validated by default. This causes QA to fail with the following error:

Fabrication of QA::Resource::Sandbox using the API failed (400) with `{"message":"Failed to save group {:identity_verification=>[\"You have reached the group limit until you verify your account.\"]}"}`.

Source: https://ops.gitlab.net/gitlab-com/gl-infra/cells/tissue/-/jobs/20287056#L1136

In order to get QA to work on the AWS cell, @tkhandelwal3 had to manually validate the user by logging in as the admin user (which also happens to be the root user) and editing the user in the Web UI:

image

image

After validation, the QA job was able to create a top-level group, and proceed with tests appropriately: https://ops.gitlab.net/gitlab-com/gl-infra/cells/tissue/-/jobs/20291256#L654

We don't want to perform this validation manually, on each cell. This is hard to discover and we have had to do it twice (once on GCP and once on AWS)

Slack conversation screenshots

Screenshots for Posterity

Link: https://gitlab.slack.com/archives/C07URAK4J59/p1757989914193379?thread_ts=1757985711.371429&cid=C07URAK4J59 (available until 2025-12-16)

image

image

Exit criteria

  • Discuss: Is it appropriate to use the root user as the QA user?
  • Discuss: Should a user be created only for QA during Cell bootstrap?
  • How can we create a specific user for QA and validate it during Cell creation? Possibly Instrumentor changes.
  • Implement changes in Instrumentor
  • Update the qa.jsonnet file to emit the QA user's username and password (instead of the Root user's username and password)
  • Verify that QA works through the HTTP router using the new user (QA works = QA does not fail due to the inability to create a top-level group)