[Feature flag] Rollout of recursive_namespace_lookup_as_inner_join

What

Rollout of recursive_namespace_lookup_as_inner_join feature flag in !56078 (merged).

Owners

  • Team: groupdatabase
  • Most appropriate slack channel to reach out to: #database
  • Best individual to reach out to: @ahegyi

Expectations

What are we expecting to happen?

We expect to see query plans for projects within a group to stay sane and avoid gitlab-com/gl-infra/production#3875 (closed) from happening again.

What might happen if this goes wrong?

Error 500s due to database statement timeouts and PgBouncer connection saturation.

What can we monitor to detect problems with this?

  1. PostgreSQL Overview: https://dashboards.gitlab.net/d/000000144/postgresql-overview?orgId=1. Check for errors, processes, and locks.
  2. PgBouncer Overview: https://dashboards.gitlab.net/d/pgbouncer-main/pgbouncer-overview?orgId=1. Check for connection saturation.
  3. PostgreSQL slow logs: https://log.gprd.gitlab.net/goto/50a306efe11337da81c25913f66b5f63
  4. For one marquee customer, the Maven package route timings are several seconds for a HEAD request: https://log.gprd.gitlab.net/goto/bb2f42f013a6273ed9597044f1546776. That dashboard should show fewer hits when this is rolled out. Note the 404s are normal here.

Beta groups/projects

If applicable, any groups/projects that are happy to have this feature turned on early. Some organizations may wish to test big changes they are interested in with a small subset of users ahead of time for example.

  • gitlab-org/gitlab-com groups
  • ...

Roll Out Steps

  • Enable on staging (/chatops run feature set recursive_namespace_lookup_as_inner_join true --staging)
  • Test on staging
  • Ensure that documentation has been updated
  • Enable on GitLab.com for individual groups/projects listed above and verify behaviour (/chatops run feature set --group=gitlab-org recursive_namespace_lookup_as_inner_join true)
  • Coordinate a time to enable the flag with the SRE oncall and release managers
    • In #production mention @sre-oncall and @release-managers. Once an SRE on call and Release Manager on call confirm, you can proceed with the rollout
  • Announce on the issue an estimated time this will be enabled on GitLab.com
  • Enable on GitLab.com by running chatops command in #production (/chatops run feature set recursive_namespace_lookup_as_inner_join true --ignore-incidents)
  • Cross post chatops Slack command to #support_gitlab-com (more guidance when this is necessary in the dev docs) and in your team channel
  • Announce on the issue that the flag has been enabled
  • Remove feature flag and add changelog entry. Ensure that the feature flag definition YAML file has been removed in the same MR that is removing the feature flag from the code
  • After the flag removal is deployed, clean up the feature flag by running chatops command in #production channel

Rollback Steps

  • This feature can be disabled by running the following Chatops command:
/chatops run feature set recursive_namespace_lookup_as_inner_join false
Edited by Stan Hu