Backend: Bulk insert Sidekiq jobs when creating a pipeline
Note
Creating a pipeline results in many Redis calls; https://log.gprd.gitlab.net/app/r/s/C3VDQ
Summary
Looking at the logs for Redis in a pipeline with ~ 500 jobs, we can see json.redis_queues_write_bytes: 799,016
and the full writes are 920967
which is getting close to a megabyte. Maybe we should look into bulk inserting these?
Related issue: #342637 (comment 776533664)
Proposal
Use bulk_perform_async_with_contexts
to enqueue all of the builds after the database transaction for pipeline creation completes.
Edited by Avielle Wolfe