Skip to content

Add Geo pipeline ref created event to sync pipeline refs to secondary

Catalin Irimie requested to merge cat-geo-pipeline-ref-created-event into master

What does this MR do and why?

This adds a Geo event for when we create a persistent pipeline ref, to replicate this creation to all Geo secondary sites, instead of waiting for the next repository sync.

📰 Context

With !81989 (merged), in %14.9, we've changed the refspec that runners use, from:

+#{pipeline.sha}:refs/pipelines/#{pipeline.id}

To:

+refs/pipelines/#{pipeline.id}:refs/pipelines/#{pipeline.id}

This resulted in runners pulling from a secondary not working anymore, as a secondary doesn't get the new ref when the pipeline is created, rather on the next sync (which may not happen for a while if the repo is not particularly active)

Related to #360816 (closed)

🐘 Migration outputs

$ scripts/db_tasks db:migrate
main: == 20220601130717 CreateGeoPipelineRefCreatedEvents: migrating ================
main: -- create_table(:geo_pipeline_ref_created_events, {:id=>:bigserial})
main:    -> 0.0059s
main: -- add_column(:geo_event_log, :pipeline_ref_created_event_id, :integer, {:limit=>8})
main:    -> 0.0016s
main: == 20220601130717 CreateGeoPipelineRefCreatedEvents: migrated (0.0088s) =======
main: == 20220601132057 AddIndexToGeoEventLogPipelineRefCreatedEventId: migrating ===
main: -- transaction_open?()
main:    -> 0.0000s
main: -- index_exists?(:geo_event_log, :pipeline_ref_created_event_id, {:name=>"index_geo_event_log_on_pipeline_ref_created_event_id", :algorithm=>:concurrently})
main:    -> 0.0161s
main: -- execute("SET statement_timeout TO 0")
main:    -> 0.0007s
main: -- add_index(:geo_event_log, :pipeline_ref_created_event_id, {:name=>"index_geo_event_log_on_pipeline_ref_created_event_id", :algorithm=>:concurrently})
main:    -> 0.0022s
main: -- execute("RESET statement_timeout")
main:    -> 0.0007s
main: -- transaction_open?()
main:    -> 0.0000s
main: -- foreign_keys(:geo_event_log)
main:    -> 0.0054s
main: -- transaction_open?()
main:    -> 0.0000s
main: -- execute("ALTER TABLE geo_event_log\nADD CONSTRAINT fk_4614ed001f\nFOREIGN KEY (pipeline_ref_created_event_id)\nREFERENCES geo_pipeline_ref_created_events (id)\nON DELETE CASCADE\nNOT VALID;\n")
main:    -> 0.0015s
main: -- execute("ALTER TABLE geo_event_log VALIDATE CONSTRAINT fk_4614ed001f;")
main:    -> 0.0017s
main: == 20220601132057 AddIndexToGeoEventLogPipelineRefCreatedEventId: migrated (0.0517s)
main: == 20220606015514 AddIndexToGeoPipelineRefCreatedEventsPipelineId: migrating ==
main: -- transaction_open?()
main:    -> 0.0000s
main: -- index_exists?(:geo_pipeline_ref_created_events, :pipeline_id, {:name=>"index_geo_pipeline_ref_created_events_on_pipeline_id", :algorithm=>:concurrently})
main:    -> 0.0024s
main: -- add_index(:geo_pipeline_ref_created_events, :pipeline_id, {:name=>"index_geo_pipeline_ref_created_events_on_pipeline_id", :algorithm=>:concurrently})
main:    -> 0.0016s
main: == 20220606015514 AddIndexToGeoPipelineRefCreatedEventsPipelineId: migrated (0.0095s)
$ scripts/db_tasks db:migrate:down VERSION=20220606015514
main: == 20220606015514 AddIndexToGeoPipelineRefCreatedEventsPipelineId: reverting ==
main: -- transaction_open?()
main:    -> 0.0000s
main: -- index_exists?(:geo_pipeline_ref_created_events, :pipeline_id, {:name=>"index_geo_pipeline_ref_created_events_on_pipeline_id", :algorithm=>:concurrently})
main:    -> 0.0169s
main: -- execute("SET statement_timeout TO 0")
main:    -> 0.0007s
main: -- remove_index(:geo_pipeline_ref_created_events, {:name=>"index_geo_pipeline_ref_created_events_on_pipeline_id", :algorithm=>:concurrently, :column=>:pipeline_id})
main:    -> 0.0046s
main: -- execute("RESET statement_timeout")
main:    -> 0.0007s
main: == 20220606015514 AddIndexToGeoPipelineRefCreatedEventsPipelineId: reverted (0.0357s)
$ scripts/db_tasks db:migrate:down VERSION=20220601132057
main: == 20220601132057 AddIndexToGeoEventLogPipelineRefCreatedEventId: reverting ===
main: -- transaction_open?()
main:    -> 0.0000s
main: -- remove_foreign_key(:geo_event_log, {:column=>:pipeline_ref_created_event_id})
main:    -> 0.0105s
main: -- transaction_open?()
main:    -> 0.0000s
main: -- indexes(:geo_event_log)
main:    -> 0.0117s
main: -- execute("SET statement_timeout TO 0")
main:    -> 0.0005s
main: -- remove_index(:geo_event_log, {:algorithm=>:concurrently, :name=>"index_geo_event_log_on_pipeline_ref_created_event_id"})
main:    -> 0.0013s
main: -- execute("RESET statement_timeout")
main:    -> 0.0005s
main: == 20220601132057 AddIndexToGeoEventLogPipelineRefCreatedEventId: reverted (0.0452s) 
$ scripts/db_tasks db:migrate:down VERSION=20220601130717
main: == 20220601130717 CreateGeoPipelineRefCreatedEvents: reverting ================
main: -- remove_column(:geo_event_log, :pipeline_ref_created_event_id, :integer, {:limit=>8})
main:    -> 0.0032s
main: -- drop_table(:geo_pipeline_ref_created_events, {:id=>:bigserial})
main:    -> 0.0021s
main: == 20220601130717 CreateGeoPipelineRefCreatedEvents: reverted (0.0101s) =======

How to set up and validate locally

It's probably easiest to run the ./qa/specs/features/browser_ui/6_release/pages/pages_pipeline_spec.rb:34 test - it would fail before this branch, it should succeed on this branch. (I've personally tested this on a GET env)

MR acceptance checklist

This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.

Edited by Catalin Irimie

Merge request reports