Add cron to run catalog resources ProcessSyncEventsWorker every minute
What does this MR do and why?
In !137238 (merged), a background syncing process was implemented to sync the denormalized columns of catalog_resources
with projects
. This process involves a PG trigger which adds sync events to a queue table; then the sync events are processed by a worker. The worker is enqueued by a Project model callback when name
, description
, or visibility_level
is updated.
Since there could be direct/bulk database updates to the projects
columns that don't use the Project AR model, we could miss enqueuing the worker. So in this MR, we add a cron job to enqueue the worker Ci::Catalog::Resources::ProcessSyncEventsWorker
every 1 minute.
Note: As most updates will be done using Project AR, we are still keeping the model callback to process SyncEvents as quickly as possible. (See !137238 (comment 1678850707) for details.)
The existing Feature Flag ci_process_catalog_resource_sync_events
is used here to allow us to turn off the worker action if needed.
FF Roll-out issue: #432963 (closed)
Resolves the last step (Step 4) of #429376 (closed).
How to set up and validate locally
- Enable the feature flag:
Feature.enable(:ci_process_catalog_resource_sync_events)
. - Create a catalog resource.
- First observe that the catalog resource attributes match the project's.
project = Project.find(<YOUR-PROJECT-ID>)
project.slice(:name, :description, :visibility_level)
project.catalog_resource.reload.slice(:name, :description, :visibility_level)
- Let's ensure that the Project model callback still enqueues the worker immediately as before. Update any of one of
name
,description
, orvisibility_level
on the project with.update!
.
project.update!(name: 'My new name')
project.catalog_resource.reload.name # We see that the name matches. Typically the worker processes the sync event almost instantaneously since our local queue is so small.
- Update any of one of
name
,description
, orvisibility_level
on the project. Useupdate_column
to ensure the model's callbacks aren't triggered; that way we can verify that the cron schedule works as expected.
project.update_column(:visibility_level, 20)
project.catalog_resource.reload.visibility_level # The value is not updated yet
Ci::Catalog::Resources::SyncEvent.last # We have a new sync event in pending status
# Wait up to 1 minute. You may repeat the last command until the status updates to "processed".
project.catalog_resource.reload.visibility_level # Now the value matches
MR acceptance checklist
This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.
-
I have evaluated the MR acceptance checklist for this MR.
Related to #429376 (closed)