[FF] root_group_organization_backfill -- Automatic organization creation for root groups

Summary

This issue is to roll out the feature on production, that is currently behind the root_group_organization_backfill feature flag.

Owners

  • Most appropriate Slack channel to reach out to: #g_tenant-scale
  • Best individual to reach out to: @dblessing

Expectations

What are we expecting to happen?

When the feature flag is enabled for a specific root group (actor):

  • A new organization is created matching the root group's name/path
  • The root group is transferred from the default organization to the new organization
  • The new organization is created in an "unconfirmed" state

When the feature flag is disabled for an actor:

  • The root group is transferred back to the default organization (if in an unconfirmed org)
  • The unconfirmed organization is deleted if it has no remaining root groups

What can go wrong and how would we detect it?

  • Failed organization creation: Would result in an error being raised and logged. Monitor Kibana for errors from Organizations::RootGroupOrganizationBackfillWorker.
  • Failed group transfer: Would result in an error being raised and logged. The group would remain in its original organization.
  • Unexpected data state: Groups could end up in an inconsistent state if the worker fails mid-operation. The worker is idempotent so retries should resolve this.

Relevant dashboards:

  • Sidekiq error rates
  • organizations_root_group_organization_backfill queue metrics

Rollout Steps

Note: Please make sure to run the chatops commands in the Slack channel that gets impacted by the command.

Rollout on non-production environments

  • Verify the MR with the feature flag is merged to master and has been deployed to non-production environments with /chatops gitlab run auto_deploy status <merge-commit-of-your-feature>
  • Enable the feature for a specific test group on staging with /chatops gitlab run feature set root_group_organization_backfill true --group=<test-group-path> --staging
  • Verify that an organization was created and the group was transferred
  • Disable the feature for the test group and verify the group is transferred back

Before production rollout

  • Ensure that documentation exists for the feature, and the version history text has been updated.
  • Notify the #g_tenant-scale Slack channel.

Specific rollout on production

For visibility, all /chatops commands that target production must be executed in the #production Slack channel and cross-posted (with the command results) to the responsible team's Slack channel.

  • Enable for a specific test group: /chatops gitlab run feature set root_group_organization_backfill true --group=<test-group-path>
  • Verify the organization was created and group transferred correctly
  • Incrementally enable for additional groups as needed

Global rollout on production

Note: Global enable/disable operations are explicitly not supported by this feature flag. The backfill process is designed to be enabled on a per-group basis to allow for controlled rollout.

Release the feature

  • Create a merge request to remove the root_group_organization_backfill feature flag
  • Clean up the feature flag from all environments: /chatops gitlab run feature delete root_group_organization_backfill --dev --pre --staging --staging-ref --production
  • Close this rollout issue

Rollback Steps

  • Disable the feature flag for affected groups:
/chatops gitlab run feature set root_group_organization_backfill false --group=<affected-group-path>

This will trigger the revert process, transferring groups back to the default organization.