Support data isolation on sec databases

Data isolation is now based on injecting an organization_id sub query, something like:

SELECT * FROM "issues" 
WHERE "issues"."namespace_id" IN (
  SELECT "namespaces"."id" FROM "namespaces" WHERE "namespaces"."organization_id" = 1000
)

These queries don't work on sec if the sharding key is project_id or namespace_id: there is no mirror of namespaces and projects

Boring solution:

Extend the existing syncing CI solution so it will work with sec database.

  • Create sec_project_mirrors and sec_namespace_mirrors tables
  • Update these tables in the same process as ci_project_mirrors and ci_namespace_mirrors
  • Backfill existing project and namespace data
  • Duplicate or refactor CiNamespaceMirrorsConsistencyCheckWorker and Database::CiProjectMirrorsConsistencyCheckWorker so it will work with sec database as well
  • After backfill is done: refactor Data Isolation work so it supports this table

Data retention

Since this data is related to projects and namespaces, the data retention policy is 'infinite'

Edited by Rutger Wessels