Use CI mirror tables for data isolation on CI database
What does this MR do and why?
This MR makes the data isolation scoping work for tables on the ci database.
The scoping rewrote queries on CI tables into subqueries on the projects and namespaces tables, which do not exist on the ci database. The gitlab-database-data_isolation gem is unchanged: the initializer now maps the project_id and namespace_id sharding keys of gitlab_ci tables to the types :ci_project_mirrors and :ci_namespace_mirrors, which current_sharding_key_value resolves through the mirror tables:
-- before (broken on a decomposed ci database):
WHERE "p_ci_builds"."project_id" IN (SELECT "projects"."id" FROM "projects" WHERE ...)
-- after:
WHERE "p_ci_builds"."project_id" IN (SELECT "ci_project_mirrors"."project_id" FROM "ci_project_mirrors" WHERE "ci_project_mirrors"."organization_id" = ...)There is no recursion when the subquery table is itself in the sharding key map, because the gem builds sharding key value subqueries inside Context.without_data_isolation.
This MR is part 4 of a series:
- !253978 (merged) - add the
organization_idcolumns - !254205 (merged) - sync
organization_idon changes - !254209 (merged) - backfill
organization_idfor existing rows - This MR - use the mirror tables in the data isolation logic
Query plans
The CI database has no projects or namespaces tables, so the previous subquery cannot run there. Plans for the new subquery on gitlab-production-ci, with the mirror lookup done by index (ci_project_mirrors.project_id, ci_namespace_mirrors.namespace_id, or (organization_id, project_id) when the outer query has no project_id filter):
| Query | Plan | Time (cold cache) | Buffers |
|---|---|---|---|
Ci::Pipeline.where(project_id: ?).order(id: :desc).limit(20) |
#174041 | 314 ms (275 ms I/O) | 1 hit, 137 reads |
Ci::RunnerNamespace.where(namespace_id: ?) |
#174042 | 64 ms | 39 reads |
Ci::Pipeline.where(id: ?) |
#174043 | 169 ms (130 ms I/O) | 3 hits, 66 reads |
SELECT "p_ci_pipelines".* FROM "p_ci_pipelines"
WHERE "p_ci_pipelines"."project_id" = ?
AND "p_ci_pipelines"."project_id" IN (
SELECT "ci_project_mirrors"."project_id" FROM "ci_project_mirrors"
WHERE "ci_project_mirrors"."organization_id" = ?)
ORDER BY "p_ci_pipelines"."id" DESC LIMIT 20Only the mirror lookup is new (5 buffers in each plan); the rest of each plan is what the query does without isolation.
References
- Related to #627588
How to set up and validate locally
- Make sure the mirror tables are populated (see the earlier MRs in the series).
- Enable isolation for an organization and set it as the current organization.
- In a Rails console,
Ci::Pipeline.all.to_sql(or any CI model sharded byproject_id/namespace_id) should show anIN (SELECT ... FROM ci_project_mirrors ...)condition instead of a subquery onprojects.
MR acceptance checklist
Evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.