Geo: Delay in syncing new project repositories in 16.3 and 16.4
The following discussion from !130565 (merged) should be addressed:
-
@mkozono started a discussion: (+4 comments) For Project Repos, #427200 (closed) may be important to large customers, since
RegistryConsistencyWorker
may not reach new projects quickly. This bug will be fixed by !133239 (merged) in 16.5+.WDYT about opening a bug issue "Geo: Delay in syncing new project repositories in 16.3 and 16.4" to discuss possibly adding an explicit create event on project creation?
Impact
In the worst case, the delay is 1 hr 40 min per 1,000,000 projects. I would guess that the average delay will be more like 30 minutes per 1M projects because:
- when a project gets updated after creation, then Geo will sync it
- when an MR or pipeline gets created, then Geo will sync the project
- the background worker will run more often than the worst case, on average
- the background worker will not be starting from the farthest point away from the new project, on average
Workaround
Run a script in the background on a Sidekiq node which enqueues Geo::Secondary::RegistryConsistencyWorker
continuously (safe because it exits early if another one is running, and it doesn't enqueue if another one is enqueued). If the worker takes 1-5s to run, then this brings the delay down to 2-9 minutes. Unfortunately, this workaround increases the chance of deleted projects leaving behind project repos on disk.
Potential fix for 16.3 and 16.4
(This bug will be fixed in 16.5.0 by !133239 (merged).)
Insert above this line https://gitlab.com/gitlab-org/gitlab/-/blob/631fb0cd8550ba25344a5e14efae8e5d06c5aac6/ee/app/services/ee/projects/create_service.rb#L67:
return project&.geo_handle_after_update if ::Geo::ProjectRepositoryReplicator.enabled?