Skip to content

Backend: Bulk insert Sidekiq jobs when creating a pipeline

Note

Creating a pipeline results in many Redis calls; https://log.gprd.gitlab.net/app/r/s/C3VDQ

Summary

Looking at the logs for Redis in a pipeline with ~ 500 jobs, we can see json.redis_queues_write_bytes: 799,016 and the full writes are 920967 which is getting close to a megabyte. Maybe we should look into bulk inserting these?

image

Related issue: #342637 (comment 776533664)

Proposal

Use bulk_perform_async_with_contexts to enqueue all of the builds after the database transaction for pipeline creation completes.

Edited by Avielle Wolfe