Skip to content

Implement syncing ci_project_mirrors and ci_namespace_mirrors tables

Furkan Ayhan requested to merge ci-namespace-project-mirrors-1 into master

What does this MR do and why?

This MR is an updated and simplified version of !74899 (closed). In this MR, we only create tables and start syncing. The next MR will be about backfilling the mirror tables.

These tables will be shadow tables from the main DB to CI DB. In this MR, we

  • create ci_project_mirrors and ci_namespace_mirrors tables => !75621 (merged)
  • create projects_sync_events and namespaces_sync_events tables
  • create PG triggers to populate events when changing namespace or project
  • introduce a feature flag ci_namespace_project_mirrors #346786.

Syncing the deletion is covered by loose foreign keys: !75559 (merged) and !75621 (merged)

Triggers

namespaces

When;

  • AFTER INSERT of namespaces
  • AFTER UPDATE of namespaces with changing of parent_id

Then;

INSERT INTO namespaces_sync_events

projects

When;

  • AFTER INSERT of projects
  • AFTER UPDATE of projects with changing of namespace_id

Then;

INSERT INTO projects_sync_events

DB

UP

== 20211011140932 CreateNamespacesSyncEvents: migrating =======================
-- create_table(:namespaces_sync_events, {})
   -> 0.0090s
== 20211011140932 CreateNamespacesSyncEvents: migrated (0.0090s) ==============

== 20211011141239 CreateProjectsSyncEvents: migrating =========================
-- create_table(:projects_sync_events, {})
   -> 0.0080s
== 20211011141239 CreateProjectsSyncEvents: migrated (0.0081s) ================

== 20211011141242 CreateNamespacesSyncTrigger: migrating ======================
-- execute("CREATE OR REPLACE FUNCTION insert_namespaces_sync_event()\nRETURNS TRIGGER AS\n$$\nBEGIN\nINSERT INTO namespaces_sync_events (namespace_id)\nVALUES(COALESCE(NEW.id, OLD.id));\nRETURN NULL;\n\nEND\n$$ LANGUAGE PLPGSQL\n")
   -> 0.0037s
-- execute("CREATE TRIGGER trigger_namespaces_parent_id_on_insert\nAFTER INSERT ON namespaces\nFOR EACH ROW\n\nEXECUTE FUNCTION insert_namespaces_sync_event()\n")
   -> 0.0015s
-- execute("CREATE TRIGGER trigger_namespaces_parent_id_on_update\nAFTER UPDATE ON namespaces\nFOR EACH ROW\nWHEN (OLD.parent_id IS DISTINCT FROM NEW.parent_id)\n\nEXECUTE FUNCTION insert_namespaces_sync_event()\n")
   -> 0.0009s
== 20211011141242 CreateNamespacesSyncTrigger: migrated (0.0063s) =============

== 20211011141243 CreateProjectsSyncTrigger: migrating ========================
-- execute("CREATE OR REPLACE FUNCTION insert_projects_sync_event()\nRETURNS TRIGGER AS\n$$\nBEGIN\nINSERT INTO projects_sync_events (project_id)\nVALUES(COALESCE(NEW.id, OLD.id));\nRETURN NULL;\n\nEND\n$$ LANGUAGE PLPGSQL\n")
   -> 0.0016s
-- execute("CREATE TRIGGER trigger_projects_parent_id_on_insert\nAFTER INSERT ON projects\nFOR EACH ROW\n\nEXECUTE FUNCTION insert_projects_sync_event()\n")
   -> 0.0013s
-- execute("CREATE TRIGGER trigger_projects_parent_id_on_update\nAFTER UPDATE ON projects\nFOR EACH ROW\nWHEN (OLD.namespace_id IS DISTINCT FROM NEW.namespace_id)\n\nEXECUTE FUNCTION insert_projects_sync_event()\n")
   -> 0.0011s
== 20211011141243 CreateProjectsSyncTrigger: migrated (0.0041s) ===============

DOWN

== 20211011141243 CreateProjectsSyncTrigger: reverting ========================
-- execute("DROP TRIGGER IF EXISTS trigger_projects_parent_id_on_insert ON projects")
   -> 0.0022s
-- execute("DROP TRIGGER IF EXISTS trigger_projects_parent_id_on_update ON projects")
   -> 0.0009s
   -> 0.0009s
-- execute("DROP FUNCTION IF EXISTS insert_projects_sync_event()")
   -> 0.0007s
== 20211011141243 CreateProjectsSyncTrigger: reverted (0.0049s) ===============

== 20211011141242 CreateNamespacesSyncTrigger: reverting ======================
-- execute("DROP TRIGGER IF EXISTS trigger_namespaces_parent_id_on_insert ON namespaces")
   -> 0.0022s
-- execute("DROP TRIGGER IF EXISTS trigger_namespaces_parent_id_on_update ON namespaces")
   -> 0.0008s
   -> 0.0008s
-- execute("DROP FUNCTION IF EXISTS insert_namespaces_sync_event()")
   -> 0.0007s
== 20211011141242 CreateNamespacesSyncTrigger: reverted (0.0047s) =============

== 20211011141239 CreateProjectsSyncEvents: reverting =========================
-- drop_table(:projects_sync_events, {})
   -> 0.0038s
== 20211011141239 CreateProjectsSyncEvents: reverted (0.0058s) ================

== 20211011140932 CreateNamespacesSyncEvents: reverting =======================
-- drop_table(:namespaces_sync_events, {})
   -> 0.0036s
== 20211011140932 CreateNamespacesSyncEvents: reverted (0.0055s) ==============

MR acceptance checklist

This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.

Edited by Furkan Ayhan

Merge request reports