Draft: Remove Secrets Manager feature flags (!254806) · Merge requests · GitLab.org / GitLab

What does this MR do and why?

Draft / exploratory MR. Do not merge as is.

The goal of this MR is to see how the CI pipeline behaves, especially the end-to-end QA specs and the rspec suites, once every Secrets Manager feature flag is rolled out and removed. This matches the plan for GitLab 19.4. The team still needs to agree on the rollout order before any real MR follows this pattern.

This MR removes 11 feature flags and their YAML definitions, treating each as permanently enabled, and flips one flag to default on:

secrets_manager (beta, default on): project-level Secrets Manager gate.
group_secrets_manager (beta, default on): group-level Secrets Manager gate.
secrets_manager_instance_enrollment (beta, default on): admin instance enrollment toggle.
secrets_manager_namespace_enrollment (wip, default off): SaaS top-level group enrollment toggle.
secrets_manager_paid_experience (wip, default off): CustomersDot-backed entitlement, trial UI, sidebar gating, entitlement GraphQL field.
end_secrets_manager_beta_program (beta): kept, flipped to default_enabled: true. The SaaS beta cutoff on 2026-09-21 still relies on this flag, so it stays until then. Default on means self-managed 19.4 installs treat the beta program as ended, so trial-eligible namespaces no longer get direct secret reads from CI or the access-token API.
secrets_manager_denial_telemetry (beta, default off): always emits the secrets_manager_access_denied internal event and log line.
secrets_manager_api_access (beta, default off): the non-CI/CD access-token REST endpoint no longer returns 404 behind a flag.
secrets_manager_async_audit_logs (gitlab_com_derisk, default off): OpenBao audit logs are always processed in SecretsManagement::AuditLogWorker. The synchronous fallback path is removed.
secrets_manager_emit_secret_read_events and secrets_manager_emit_secret_stored_events (gitlab_com_derisk, default off): billable usage events are always emitted.
billing_event_tracking (development, default off, owned by group::analytics instrumentation): Gitlab::BillingEvents::Client#track_billing_event dropped every billing event while this was off, including the Secrets Manager secrets_read and secrets_stored events above. Removing it lets those events reach the billing destination. The two Secrets Manager emitters are the only callers of the client today, so no other product starts emitting. With the emitter now live, SecretsReadEmitter also mirrors the stored emitter's enrollment check and skips reads without an OpenBao request id, so a retried audit job cannot bill twice. billing_events_oidc_auth was removed on master by !254395 (merged) and came in with the rebase.

Behaviour notes for reviewers:

Availability now depends only on license plus enrollment: namespace enrollment on GitLab.com, instance enrollment on self-managed. On GitLab.com, root groups get the paid-experience implicit grant before enrolling, unless they explicitly opted out.
Entitlement checks (policies, GraphQL write concern, service gate, CI runner payload, job scheduling) always run now.
Enrollment always writes beta: false (namespace) and secrets_manager_instance_beta_enrolled: false (instance). Unenroll on SaaS always soft-disables the enrollment record instead of deleting it.
The "New" nav badge no longer takes a root namespace argument, since the flag check was its only use of it.
The two QA specs project_secret_api_access_spec.rb and group_secret_api_access_spec.rb, under qa/qa/specs/features/ee/browser_ui/10_software_supply_chain_security/secrets_management/, no longer carry feature_flag: metadata. They now run in the normal QA pipeline.
Frontend: glFeatures.secretsManagerPaidExperience checks are removed. Components behave as if the flag is always on.
Docs: history entries added in doc/api/secrets_manager.md and doc/ci/secrets/secrets_manager/non_cicd_access.md noting the secrets_manager_api_access flag removal in 19.4. The GraphQL reference is updated for the secretsManagerEntitlement field description.
Specs for the "flag disabled" paths were deleted. Specs for the enabled paths were kept.
QA: the settings toggle only existed in the beta experience, so the QA specs now provision and deprovision through the GraphQL mutations. The two feature_provision specs check the permissions section in settings instead of the toggle. Both stay on Settings > General, because QA groups are subgroups under the sandbox and Settings > Secure only exists on top-level groups. The stale QA_FEATURE_FLAGS line was removed from the cng-secrets-manager job.
Deprovision UI: the toggle was the only frontend caller of the deprovision mutations, so the unreachable disable config, the two .graphql mutation files and the ACTION_* constants are removed. Deprovisioning still works through GraphQL. The user docs now describe provisioning through New secret and deprovisioning through the projectSecretsManagerDeprovision and groupSecretsManagerDeprovision mutations. Whether a deprovision UI should return is a product decision for the team.
The CI/CD variables nudge only resolves the CustomersDot entitlement when SecretsManagement::Availability.for_group? is true for the root namespace, so unlicensed or opted-out namespaces make no CustomersDot call.
The secrets_manager_namespace_enrollment factory defaults to beta: false, matching the only value production writes now. The column default flip is a follow-up migration.

Self-managed at GA. With secrets_manager_paid_experience gone, a self-managed instance whose license has no secrets_manager add-on resolves to blocked and loses writes and CI reads. This is the intended GA behaviour: self-managed has no grace period. Online-cloud-licensed instances can still start a trial through CustomersDot. Offline instances need the add-on on their license. The instance beta marker and the 19.4 backfill only affect online-license instances, since the offline path does not read them.

Known blocker for the QA job. On the e2e instance the entitlement resolves offline and finds no Secrets Manager add-on purchase, so it comes back blocked and every write and CI read is denied. The e2e license (QA_EE_LICENSE, an offline cloud license from Zuora staging subscription A-S00158811) needs the Secrets Manager rate plan added and the license regenerated. Until then the cng-secrets-manager job is expected to fail, so it is temporarily set to allow_failure: true. The job now reads the license from QA_EE_LICENSE_SECRETS_MANAGER when that CI variable exists and falls back to the shared QA_EE_LICENSE otherwise. The variable is set, but allow_failure stays until the dedicated license is provisioned through GSM and the tracking issue is closed, because a hand-added variable can be removed by the infrastructure pipeline.

The QA job is green. With the dedicated license variable and the API-based add-on provisioning, cng-secrets-manager passed all 39 examples for ba5d54db (https://gitlab.com/gitlab-org/gitlab/-/jobs/16444227733). The run before it passed 38 of 39, with the one failure being the group feature_provision spec that has since been fixed.

Why the QA specs add a copy of the license through the API. The QA bootstrap adds the e2e license through POST /api/v4/license. That endpoint saves the license but does not run the self-managed add-on provisioners, so the secrets_manager add-on in the license never becomes an AddOnPurchase until the daily OfflineCloudLicenseProvisionWorker. The entitlement then resolves as blocked for the whole run. The Secrets Manager QA helper now adds a second copy of the same license through the API and removes the superseded one, once per process. Licenses::DestroyService re-runs the add-on provisioners against the current license, so the purchase exists before the specs run. A first attempt re-uploaded through the Admin UI, but that was not reliable with four parallel QA processes and admin mode. This is QA-only. The same gap exists for any offline license added through the REST API or the chart, and is worth a separate look by the Fulfillment team. Tracking issue for the dedicated license: https://gitlab.com/gitlab-org/gitlab/-/work_items/628661

No database migrations are included. Suggested commit trailer: Changelog: other with EE: true.

References

Screenshots or screen recordings

Not applicable. This MR does not intend any visible UI change beyond flag-gated paths becoming always on.

How to set up and validate locally

How the Secrets Manager QA specs were tested locally

The cng-secrets-manager job in the MR pipeline cannot pass until the e2e license carries the Secrets Manager add-on. This is tracked in https://gitlab.com/gitlab-org/gitlab/-/work_items/628661. Until then, the QA specs were run against a local GDK with a temporary license that has the add-on. This section lists every setup step so anyone can repeat it.

Check out the branch and restart Rails

The branch was checked out in a worktree, so use a detached checkout in the main GDK checkout instead:

git checkout --detach claude/secrets-manager-flags-removal-fc220e

Then restart the app so Puma and Sidekiq pick up the new code:

gdk restart rails-web rails-background-jobs

If your GDK is behind master, run gdk update first so pending migrations are applied.

Sidekiq must be running because provisioning and audit logs both run in background workers.

Get a license with the Secrets Manager add-on

With the feature flags removed, the entitlement check always runs. On self-managed it looks for an active secrets_manager add-on purchase that comes from the license. A normal Ultimate license, legacy or offline, has no such add-on. Without it, everything resolves as blocked and both writes and CI reads are denied.

A temporary offline license from CustomersDot staging, with the add-on included, is in the GitLab QA 1Password vault for now. Save it to a file outside the repo, for example ~/gitlab-development-kit/tmp/qa-sm.gitlab-license. Do not commit it or paste it in any comment.

Upload the license and check the entitlement

Run this in bundle exec rails runner or a Rails console:

result = GitlabSubscriptions::UploadLicenseService.new(
  { data: File.read('/path/to/qa-sm.gitlab-license').strip }, 'http://gdk.test:3000'
).execute
result.success? # => true

SecretsManagement::Entitlement::Resolver.clear_cache(nil)
License.current.offline_cloud_license?                                                # => true
License.current.license.restrictions.deep_symbolize_keys[:add_on_products].keys      # => [:secrets_manager]
GitlabSubscriptions::AddOnPurchase.for_secrets_manager.for_self_managed.active.exists? # => true
SecretsManagement::Entitlement.for(nil).state                                        # => :offline_paid

Uploading through the service is the same as going to Admin > Subscription. It also runs the add-on provisioners, which create the add-on purchase row.

If the entitlement still says blocked, check that GDK is not in simulate SaaS mode (GITLAB_SIMULATE_SAAS should be unset). SaaS mode routes the check to CustomersDot instead of reading the local license.

Optional shortcut without a license

For spec work only, not as proof of the license path, the add-on purchase can be seeded by hand:

add_on = GitlabSubscriptions::AddOn.find_or_create_by_name(:secrets_manager)
GitlabSubscriptions::AddOnPurchase.create!(
  add_on: add_on, namespace: nil, organization: Organizations::Organization.default_organization,
  quantity: 1, started_at: Date.current, expires_on: 1.year.from_now.to_date, purchase_xid: 'local-dev-secrets-manager'
)

Remove it before testing a real license:

GitlabSubscriptions::AddOnPurchase.for_secrets_manager.for_self_managed.destroy_all

Make sure the instance is enrolled

Secrets Manager also needs instance enrollment on self-managed. The QA specs enroll the instance through GraphQL themselves, so there is nothing to do by hand unless testing outside the specs. To check:

Gitlab::CurrentSettings.secrets_manager_instance_enrolled # => true

Let Docker containers reach GDK

This step is needed for the CI access specs. They start their own gitlab-runner Docker container, and the CI job reads the secret from OpenBao. Inside containers, gdk.test normally resolves to 127.0.0.1, which is the container itself. That means the runner can never register, and the specs time out.

With colima, fix this with the internal resolver. In ~/.colima/default/colima.yaml, change dnsHosts: {} under the network section to:

  dnsHosts:
    gdk.test: 192.168.5.2
    registry.test: 192.168.5.2

192.168.5.2 is colima's host gateway, also known as host.lima.internal. Then restart colima:

colima stop && colima start

This restarts every container in colima, so expect other local services to bounce too. Check it worked:

docker run --rm alpine:3 sh -c 'getent hosts gdk.test; wget -S -qO- http://gdk.test:3000/api/v4/version 2>&1 | head -1'
# expect 192.168.5.2 and HTTP/1.1 401 Unauthorized

No runner needs to be registered in GDK ahead of time, and a host shell runner does not help, because the specs create their own tagged runner.

Turn on OpenBao audit streaming to Rails

This step is needed for the audit event specs. The two *_read_audit_event_streaming_spec.rb specs check that a secret read shows up at an audit streaming destination. GDK's OpenBao only has a file audit device by default, so it never posts events to Rails.

Follow the GDK howto at https://gitlab-org.gitlab.io/gitlab-development-kit/howto/openbao/#configure-http-audit-log-streaming-to-rails:

cd /path/to/your/gdk
echo "gdk_test_auth_token" > gitlab/.gitlab_openbao_authentication_token_secret

Add this to openbao/config.hcl:

audit "http" "debug" {
  options {
    uri = "http://gdk.test:3000/api/v4/internal/secrets_manager/audit_logs"
    headers = "{\"Content-Type\":[\"application/json\"], \"Accept\":[\"application/json\"], \"Gitlab-Openbao-Auth-Token\":[\"gdk_test_auth_token\"]}"
  }
}

Then reload OpenBao:

kill -SIGHUP $(pgrep -x bao)

Check gdk tail openbao for enabled audit backend: path=debug/ type=http. Rails should log a 202 on /api/v4/internal/secrets_manager/audit_logs. The token file is gitignored, so it will not show up in git status.

Run the specs

From the qa/ directory, after bundle install:

QA_GITLAB_URL="http://gdk.test:3000" WEBDRIVER_HEADLESS=true \
  bundle exec rspec qa/specs/features/ee/browser_ui/10_software_supply_chain_security/secrets_management

No feature flags need to be set. Running the whole directory takes about an hour. Running a single file takes one to three minutes. Specs that run a CI job take longer, because they wait for a pipeline to finish.

Results

Spec paths are relative to qa/qa/specs/features/ee/browser_ui/10_software_supply_chain_security/secrets_management/. All ran with the temporary license against a GDK on this branch before it was rebased onto the current master. The two audit_events_spec.rb runs used the QA page-object port fix that is now in the branch.

Spec	Examples	Result
project_secrets_manager/feature_provision_spec.rb	1	Passed
project_secret_api_access_spec.rb	1	Passed (mints a token, reads the value straight from OpenBao)
group_secrets_manager/group_secret_api_access_spec.rb	1	Passed
project_secret_ci_access_success_spec.rb	1	Passed
project_secret_ci_access_failure_spec.rb	1	Passed
group_secrets_manager/group_secret_ci_access_success_spec.rb	1	Passed
group_secrets_manager/group_secret_ci_access_failure_spec.rb	1	Passed
project_secret_read_audit_event_streaming_spec.rb	1	Passed (after enabling the HTTP audit device)
group_secret_read_audit_event_streaming_spec.rb	1	Passed (after enabling the HTTP audit device)
project_secrets_manager/project_secret/create_spec.rb	2	Passed
project_secrets_manager/project_secret/read_spec.rb	2	Passed
project_secrets_manager/permissions/create_spec.rb	3	Passed
group_secrets_manager/group_secret/delete_spec.rb	2	Passed
group_secrets_manager/permissions/create_spec.rb	3	Passed
group_secrets_manager/permissions/delete_spec.rb	1	Passed

The remaining 11 spec files (group secret create, read, update, audit events, permission access control and group sharing, project secret update, delete, audit events, project permissions access control and delete) ran last: 19 examples, all passed. The two audit_events_spec.rb files first failed because the QA page object dropped the :3000 port when going back to the secrets list. That is fixed in this MR and both pass now. That covers 26 of the 27 spec files in the directory. The missing one is group_secrets_manager/feature_provision_spec.rb. It was moved back to Settings > General after review and has not passed locally yet. The one local attempt failed while waiting for provisioning because the GDK Sidekiq raised an OpenBao URL error, which is a local setup problem and not the spec. It will be re-run on a GDK checked out on this branch.

Good to know

Two failures seen on the way were flakes: a QA sign-in step timing out before the Secrets Manager page loaded, and a permissions case that passed on rerun.
The audit streaming specs also prove the new always-asynchronous audit path. OpenBao posts to the internal API, SecretsManagement::AuditLogWorker processes it, and the event reaches the streaming destination. Secret read events are only visible at a streaming destination, not in the audit events table.
The temporary license expires 2027-09-10 and is not tied to any instance. Treat it like a password, and delete the local copy when done. GitLab always uses the newest license in its date range, so remove it from Admin > Subscription to go back to a previous license.
The cng-secrets-manager job stays allow_failure: true until the dedicated license variable exists. See https://gitlab.com/gitlab-org/gitlab/-/work_items/628661.

MR acceptance checklist

Evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.

Edited Sep 11, 2026 by Erick Bajao

Draft: Remove Secrets Manager feature flags