Allow scheduling jobs in batch with contexts (!23519) · Merge requests · GitLab.org / GitLab

Bob Van Landuyt requested to merge reprazent/gitlab-ee:bvl-context-for-batch-scheduling into master Jan 22, 2020

What does this MR do?

This adds helper methods for the bulk_perform_in and bulk_perform_async methods that allow specifying a context per job.

Scheduling can be done by creating a hash that maps worker arguments to the context to be used for those arguments:

   { [<arguments for worker>] => { user: user, project: project} }

These will be stored on the worker class when scheduling the job, so the sidekiq middleware can fetch the right context based on those arguments and apply them to the job when scheduling it.

This also implements this method for the UpdateAllMirrorsWorker which schedules ProjectImportScheduleWorker in batch. When adding this, the ProjectImportScheduleWorker-jobs will have a context.

Which means that job now shows up in the logs as follows:

2020-01-28_12:33:35.46230 rails-background-jobs : {"severity":"INFO","time":"2020-01-28T12:33:35.461Z","class":"ProjectImportScheduleWorker","args":[13],"retry":false,"queue":"project_import_schedule","backtrace":true,"jid":"535f30fd3bde0bcd8a74811d","created_at":"2020-01-28T12:33:27.754Z","enqueued_at":"2020-01-28T12:33:28.312Z","meta.project":"gitlab-org/gitlab","meta.root_namespace":"gitlab-org","meta.subscription_plan":"default","meta.caller_id":"UpdateAllMirrorsWorker","correlation_id":"PJHJKnbQdL1","pid":50331,"message":"ProjectImportScheduleWorker JID-535f30fd3bde0bcd8a74811d: done: 1.294023 sec","job_status":"done","scheduling_latency_s":5.855765,"duration":1.294023,"cpu_s":0.27924,"completed_at":"2020-01-28T12:33:35.461Z","db_duration":7.35100032761693,"db_duration_s":0.00735100032761693}

gitlab-com/gl-infra/scalability#112 (closed)

Some parts are duplicated with !23444 (merged)