Avoid idling in transaction while saving CI build logs in object storage
This is related to a production incident in gitlab-com/gl-infra/production#5158 (closed) and #285597 (closed).
We saw that if HTTP transfers to object storage start to get slow, Sidekiq jobs ArchiveTraceWorker
and BuildTraceChunkFlushWorker
would get stuck in idle in transaction while CarrierWave finalizes the chunks into object storage. This caused saturation on PgBouncer, which in turn slowed other database activity.
By default, CarrierWave saves the file in object storage in the transaction. In !63350 (merged), we overrode this behavior by saving the file in after_commit
, which happens outside of the transaction.
We may want to consider doing this as well.
/cc: @grzesiek
Edited by Stan Hu