defaults of ci_unlock_pipelines feature gate are too high for self-managed

Summary

We recently updated from gitlab 16.4 to 16.6 (with an short intermediate step of 16.5) and ran into the issue that the now default feature gate ci_unlock_pipelines has values which are far too high for probably many self managed instances using omnibus default values.

feature addition !124112 (merged) defaults were increased in !133459 (merged)

We run an instance on a 32 core (+SMT) machine and ssd disks using the omnibus package. This setup with default sidekiq settings has one sidekiq process and 20 threads. After the update we had 450.000 enqueued Ci::UnlockPipelineRequest from Ci::UnlockPipelinesInQueueWorker each taking around 1-2s to process.

This used all available resources making any other action on gitlab very unresponsive, the default queue latency went up to 16 minutes and 10.000+ entries in the queue.

We fixed this by first disabling the featuregate and later decreasing the defaults. It seems we need around 8 concurrency alone to work through our incoming rate of requests (in addition to the existing 450k queue), so maybe something needs to be optimized either in this process or our setup, so far we are using defaults for sidekiq and postgres instances.

Possible fixes

The default values should by default probably be at most half the number of sidekiq workers the instance has available.