PipelineProcessWorker shows spikes in database duration
From https://dashboards.gitlab.net/d/sidekiq-queue-detail/sidekiq-queue-detail?orgId=1&var-PROMETHEUS_DS=Global&var-environment=gprd&var-stage=main&var-queue=pipeline_processing:pipeline_process&from=now-7d&to=now&fullscreen&panelId=3, we can see that this worker sees spikes in overall duration:
In general, this worker's performance is very good. The median DB time is 0.064 seconds, and the P95 is 0.356 seconds. However, we do see spikes at the tail (up to over 30 seconds), and they appear to be related to which pipeline is being processed: https://log.gitlab.net/goto/99996c87fba6ca30bf6f8bb5bdf1350e
Pipeline IDs 112983097 and 112674504 (among others) appear a bunch of times in those results, which suggests the spikes might be due to something intrinsic about the pipeline. We should try to get this under control if possible as the worker is marked as latency sensitive.