Eventual namespace path consistency worker for CA
What does this MR do and why?
This MR implements eventual consistency for the events
table in ClickHouse. When the namespace hierarchy changes for some reason, this worker will apply the changes to the ClickHouse table.
- Namespace (including project) is deleted: delete
events
records from CH - Namespace (including project) is moved: update the
path
column (same as traversal ids but represented as string)
The worker re-uses the same logic as we implemented in EventAuthorsConsistencyCronWorker
:
- Batch over the
events
table. - For each batch, look up
Namespace
records from PostgreSQL. - Compare PostgreSQL
traversal_ids
with ClickHousepath
value. - If there is a a difference, record the change in memory.
- Once time limit or record limit is reached, apply the change to ClickHouse DB by invoking
UPDATE
orDELETE
queries.
Database
Loading a batch of namespaces
records by id: https://console.postgres.ai/gitlab/gitlab-production-tunnel-pg12/sessions/25219/commands/80112
MR acceptance checklist
Please evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.
Screenshots or screen recordings
Screenshots are required for UI changes, and strongly recommended for all other merge requests.
Before | After |
---|---|
How to set up and validate locally
- Ensure that you're on premium plan
- Ensure CH is configured: https://docs.gitlab.com/ee/development/database/clickhouse/clickhouse_within_gitlab.html#gdk-setup
- Enable the sync feature flag:
Feature.enabled(:event_sync_worker_for_click_house)
- Invoke the snippet that updates a namespace path and deletes another one (created via factory bot)
Snippet for "testing":
event1 = FactoryBot.create(:closed_issue_event)
event2 = FactoryBot.create(:closed_issue_event)
ClickHouse::EventsSyncWorker.new.perform
namespace_id = event1.project.project_namespace.id
path_to_delete = Namespace.find(namespace_id).traversal_ids.join('/') + '/'
puts "to delete: #{path_to_delete}"
Namespace.where(id: namespace_id).delete_all
ch_namespace_path = event2.project.project_namespace.traversal_ids.join('/') + '/'
# prepend a 0 to make it outdated
ClickHouse::Client.execute("alter table events update path='0/#{ch_namespace_path}' where path='#{ch_namespace_path}'", :main)
ClickHouse::Client.execute("alter table event_namespace_paths update path='0/#{ch_namespace_path}' where path='#{ch_namespace_path}'", :main)
puts "Test 1: Record is still visible"
puts ClickHouse::Client.select("select * from events final where path ='#{path_to_delete}'", :main).inspect
puts "Test2: Record cannot be found"
puts ClickHouse::Client.select("select * from events final where path = '#{ch_namespace_path}'", :main).inspect
# Invoke the worker to fix the data
ClickHouse::EventPathsConsistencyCronWorker.new.perform
puts "Consistency worker ran"
puts "Test 1: Record should be gone:"
puts ClickHouse::Client.select("select * from events final where path ='#{path_to_delete}'", :main).inspect
puts "Test2: Record should be found"
puts ClickHouse::Client.select("select * from events final where path = '#{ch_namespace_path}'", :main).inspect
Example output:
Test 1: Record is still visible
[{"id"=>188, "path"=>"189/190/", "author_id"=>121, "target_id"=>644, "target_type"=>"Issue", "action"=>3, "deleted"=>0, "created_at"=>Mon, 15 Jan 2024 16:24:29.212483000 UTC +00:00, "updated_at"=>Mon, 15 Jan 2024 16:24:29.212483000 UTC +00:00}]
Test2: Record cannot be found
[]
Namespace Load (0.5ms) SELECT "namespaces"."id", "namespaces"."traversal_ids" FROM "namespaces" WHERE "namespaces"."id" IN (193, 109, 92, 116, 125, 86, 187, 190, 23, 25, 26, 28, 30, 32, 34, 95, 36, 90, 93, 87, 91, 89, 88, 94, 85, 97, 98) /*application:console,db_config_name:main,console_hostname:dell-precision-gitlab,console_username:ahegyi,line:/app/workers/click_house/event_paths_consistency_cron_worker.rb:30:in `collect_values'*/
Consistency worker ran
Test 1: Record should be gone:
[]
Test2: Record should be found
[{"id"=>189, "path"=>"192/193/", "author_id"=>123, "target_id"=>645, "target_type"=>"Issue", "action"=>3, "deleted"=>0, "created_at"=>Mon, 15 Jan 2024 16:24:29.590786000 UTC +00:00, "updated_at"=>Mon, 15 Jan 2024 16:24:29.590786000 UTC +00:00}]
Related to #428260 (closed)