Eventual namespace path consistency worker for CA

What does this MR do and why?

This MR implements eventual consistency for the events table in ClickHouse. When the namespace hierarchy changes for some reason, this worker will apply the changes to the ClickHouse table.

• Namespace (including project) is deleted: delete events records from CH
• Namespace (including project) is moved: update the path column (same as traversal ids but represented as string)

The worker re-uses the same logic as we implemented in EventAuthorsConsistencyCronWorker:

1. Batch over the events table.
2. For each batch, look up Namespace records from PostgreSQL.
3. Compare PostgreSQL traversal_ids with ClickHouse path value.
4. If there is a a difference, record the change in memory.
5. Once time limit or record limit is reached, apply the change to ClickHouse DB by invoking UPDATE or DELETE queries.

Database

Loading a batch of namespaces records by id: https://console.postgres.ai/gitlab/gitlab-production-tunnel-pg12/sessions/25219/commands/80112

MR acceptance checklist

Please evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.

Screenshots or screen recordings

Screenshots are required for UI changes, and strongly recommended for all other merge requests.

Before	After

How to set up and validate locally

1. Ensure that you're on premium plan
2. Ensure CH is configured: https://docs.gitlab.com/ee/development/database/clickhouse/clickhouse_within_gitlab.html#gdk-setup
3. Enable the sync feature flag: Feature.enabled(:event_sync_worker_for_click_house)
4. Invoke the snippet that updates a namespace path and deletes another one (created via factory bot)

Snippet for "testing":

event1 = FactoryBot.create(:closed_issue_event)
event2 = FactoryBot.create(:closed_issue_event)

ClickHouse::EventsSyncWorker.new.perform

namespace_id = event1.project.project_namespace.id
path_to_delete = Namespace.find(namespace_id).traversal_ids.join('/') + '/'
puts "to delete: #{path_to_delete}"

Namespace.where(id: namespace_id).delete_all

ch_namespace_path = event2.project.project_namespace.traversal_ids.join('/') + '/'
# prepend a 0 to make it outdated
ClickHouse::Client.execute("alter table events update path='0/#{ch_namespace_path}' where path='#{ch_namespace_path}'", :main)
ClickHouse::Client.execute("alter table event_namespace_paths update path='0/#{ch_namespace_path}' where path='#{ch_namespace_path}'", :main)


puts "Test 1: Record is still visible"
puts ClickHouse::Client.select("select * from events final where path ='#{path_to_delete}'", :main).inspect

puts "Test2: Record cannot be found"
puts ClickHouse::Client.select("select * from events final where path = '#{ch_namespace_path}'", :main).inspect

# Invoke the worker to fix the data
ClickHouse::EventPathsConsistencyCronWorker.new.perform

puts "Consistency worker ran"

puts "Test 1: Record should be gone:"
puts ClickHouse::Client.select("select * from events final where path ='#{path_to_delete}'", :main).inspect

puts "Test2: Record should be found"
puts ClickHouse::Client.select("select * from events final where path = '#{ch_namespace_path}'", :main).inspect

Example output:

Test 1: Record is still visible
[{"id"=>188, "path"=>"189/190/", "author_id"=>121, "target_id"=>644, "target_type"=>"Issue", "action"=>3, "deleted"=>0, "created_at"=>Mon, 15 Jan 2024 16:24:29.212483000 UTC +00:00, "updated_at"=>Mon, 15 Jan 2024 16:24:29.212483000 UTC +00:00}]
Test2: Record cannot be found
[]
  Namespace Load (0.5ms)  SELECT "namespaces"."id", "namespaces"."traversal_ids" FROM "namespaces" WHERE "namespaces"."id" IN (193, 109, 92, 116, 125, 86, 187, 190, 23, 25, 26, 28, 30, 32, 34, 95, 36, 90, 93, 87, 91, 89, 88, 94, 85, 97, 98) /*application:console,db_config_name:main,console_hostname:dell-precision-gitlab,console_username:ahegyi,line:/app/workers/click_house/event_paths_consistency_cron_worker.rb:30:in `collect_values'*/
Consistency worker ran
Test 1: Record should be gone:
[]
Test2: Record should be found
[{"id"=>189, "path"=>"192/193/", "author_id"=>123, "target_id"=>645, "target_type"=>"Issue", "action"=>3, "deleted"=>0, "created_at"=>Mon, 15 Jan 2024 16:24:29.590786000 UTC +00:00, "updated_at"=>Mon, 15 Jan 2024 16:24:29.590786000 UTC +00:00}]

Related to #428260 (closed)