Profile and optimize PipelineProcessWorker
This started as a production incident: gitlab-com/gl-infra/production#1014 (closed)
From https://dashboards.gitlab.net/d/000000124/sidekiq-workers?orgId=1&var-worker=PipelineProcessWorker%23perform&var-database=influxdb-01-inf-gprd&from=1564582438010&to=1564593714112, we observed a number of PipelineProcessWorker
jobs making over 2,000 SQL queries:
Possibly due to PgBouncer contention, this led to SQL timings of minutes at times:
It's quite possible Group#ci_variables_for
may be contributing to this: