Draft: Use CI mirror tables for data isolation on CI database

Rutger Wesselsrequested to merge
627588-ci-join-mirror-tables into master

What does this MR do and why?

This MR makes the data isolation scoping work for tables on the ci database.

The scoping rewrote queries on CI tables into subqueries on the projects and namespaces tables, which do not exist on the ci database. Now the gitlab-database-data_isolation gem passes the table name to current_sharding_key_value, so the initializer can scope gitlab_ci tables through ci_project_mirrors and ci_namespace_mirrors instead:

-- before (broken on a decomposed ci database):
WHERE "p_ci_builds"."project_id" IN (SELECT "projects"."id" FROM "projects" WHERE ...)
-- after:
WHERE "p_ci_builds"."project_id" IN (SELECT "ci_project_mirrors"."project_id" FROM "ci_project_mirrors" WHERE "ci_project_mirrors"."organization_id" = ...)

There is no recursion when the subquery table is itself in the sharding key map, because the gem builds sharding key value subqueries inside Context.without_data_isolation.

This MR is part 4 of a series:

  1. !253978 (merged) - add the organization_id columns
  2. !254205 (merged) - sync organization_id on changes
  3. !254209 (merged) - backfill organization_id for existing rows
  4. This MR - use the mirror tables in the data isolation logic

References

How to set up and validate locally

  1. Make sure the mirror tables are populated (see the earlier MRs in the series).
  2. Enable isolation for an organization and set it as the current organization.
  3. In a Rails console, Ci::Pipeline.all.to_sql (or any CI model sharded by project_id/namespace_id) should show an IN (SELECT ... FROM ci_project_mirrors ...) condition instead of a subquery on projects.

MR acceptance checklist

Evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.

Merge request reports