Run Secrets Manager QA specs against the paid experience

Erick Bajaorequested to merge
eb-sm-qa-specs-paid-experience into master

What does this MR do and why?

This MR is split out of the larger draft MR !254806, which removes every Secrets Manager feature flag. After the go/no-go call, the team decided not to merge that MR until the end-to-end tests give confidence.

This MR carries only the QA and CI pieces from that larger MR. It makes the cng-secrets-manager e2e job run the Secrets Manager specs against the paid experience, with all flags still in place on master. No application code changes. No database changes. Only files under qa/ and .gitlab/ci/test-on-cng/main.gitlab-ci.yml are touched.

How the QA job is set up

  • The cng-secrets-manager job now enables secrets_manager_paid_experience through QA_FEATURE_FLAGS, in addition to the flags it already enabled.
  • With the paid experience on, the "GitLab Secrets Manager" settings toggle no longer exists in project or group settings. The QA specs and page objects now provision and deprovision through GraphQL mutations (projectSecretsManagerInitialize, groupSecretsManagerInitialize, and the matching deprovision mutations) instead of clicking the toggle.
  • The two feature_provision specs now check the "User permissions" section in settings, and the Owner role in the Roles tab, instead of the toggle.
  • The unused EE::Flow::Project#enable_secrets_manager_feature flow is removed.
  • The group feature_provision spec stays on Settings > General. QA groups are subgroups under the sandbox, and Settings > Secure only exists on top-level groups.
  • A small page object fix: go_back_to_secrets_list keeps a non-default port such as :3000 on GDK and drops a default one, by changing only the path of the parsed URL.

License handling

With the paid experience on, self-managed resolves the entitlement offline and needs a secrets_manager add-on purchase from the license. The shared e2e license has no such add-on.

  • The job now reads the license from the new CI variable QA_EE_LICENSE_SECRETS_MANAGER when it is set, and falls back to QA_EE_LICENSE otherwise.
  • The variable is set on the project today, but see the next section for why the job is still allowed to fail.
  • On CNG the orchestrator writes the license into a Kubernetes Secret and the chart loads it through GITLAB_LICENSE_FILE. That path, like POST /api/v4/license, does not run the self-managed add-on provisioners, so the add-on purchase would only appear after the daily OfflineCloudLicenseProvisionWorker runs.
  • To make the add-on available before the specs run, the Secrets Manager QA helper adds a second copy of the same license through the API, deletes the superseded ones, then polls the secretsManagerInstanceEntitlement GraphQL query until it reports a paid state. Licenses::DestroyService re-runs the add-on provisioners against the current license, and the poll confirms the outcome, since the delete endpoint hides the service result and GET /license does not expose add_on_products.
  • This is QA-only code. It is skipped when no license is configured, which is the case for local runs that uploaded a license by hand, and when the entitlement is already paid because another QA process got there first. It runs once per QA process. A failed attempt is remembered, so later examples in that process fail fast with the same error instead of re-adding the license and waiting again. The error names the likely cause, a license without the secrets_manager add-on, and points at QA_EE_LICENSE_SECRETS_MANAGER.
  • Because this setup replaces instance licenses, every Secrets Manager spec carries the :requires_admin tag, and the helper logs the license ids it removes.
  • Provisioning through GraphQL runs as the project or group Owner, so the specs keep asserting that role can provision. Project authorizations refresh asynchronously after a member is added, so the helper retries that one denial for up to 30 seconds.

Why the job has allow_failure: true

This is temporary and deliberate. It is not because the specs are flaky.

  • The job still depends on a temporary masked CI/CD variable, QA_EE_LICENSE_SECRETS_MANAGER, that was added by hand in the project settings. The other e2e secrets live in Google Secret Manager and are applied by Terraform. A variable that Terraform does not know about can be erased the next time it runs.
  • If that happens, the job falls back to the shared QA_EE_LICENSE, which has no secrets_manager add-on. The entitlement then resolves as blocked and every spec fails with "Secrets Manager access is restricted for this namespace". That is what every run of the parent MR looked like before the dedicated license existed.
  • cng-secrets-manager runs on every merge request pipeline in this project, so that failure would block merges for everyone, with no code change to explain it.
  • The permanent QA license setup, with the license in Google Secret Manager and the variable managed by Terraform, is tracked in https://gitlab.com/gitlab-org/gitlab/-/work_items/628661. Removing allow_failure is part of that issue.

What this means in practice: the job still runs on every pipeline and its result stays visible. A red cng-secrets-manager job does not block a merge, so until the follow-up lands, the team checks the job result on Secrets Manager merge requests by hand.

Verification

On the parent MR, with the same QA and CI changes, the cng-secrets-manager job passed all 39 examples: https://gitlab.com/gitlab-org/gitlab/-/jobs/16444227733. That run had the flags removed in code.

On this MR, with the flags enabled through QA_FEATURE_FLAGS, the cng-secrets-manager job passed on the first head (https://gitlab.com/gitlab-org/gitlab/-/jobs/16449015399) and again on the current head after the review changes (https://gitlab.com/gitlab-org/gitlab/-/jobs/16452762133). After the second review round it passed a third time with no retries at all: https://gitlab.com/gitlab-org/gitlab/-/jobs/16454351347. All 39 examples passed on both runs. On the first run two examples hit a transient connection error, and on the second run one group provisioning wait hit its 120 second timeout. All of them passed on the automatic retry. The log on the current head also shows the entitlement check working: the first processes saw BLOCKED, re-added the license, then every process saw OFFLINE_PAID.

References

MR acceptance checklist

This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.

Edited by Erick Bajao

Merge request reports