Provide visibility that the runner has hit the maxium concurrency set for the runner on Kubernetes
Problem
- Customer spent days investigating why their pipelines were showing high latency. This included writing GitLab API scripts.
- In this case, the customer never saw, or was provided any indication the the runner on GKE, had hit its concurrency maximum.
- Only by observing that the maximum number of concurrent jobs was a suspiciously round number did the customer eventually figure out the issue.
Proposal
- The concurrent setting should be required to appear in values.yaml. Anyone using k8s is expecting lots of concurrency to be possible, and if there is a maximum, users should be forced to decide what setting is right for them, or at least see a default setting in values.yaml.
- When pipelines are being delayed as a result of a low concurrency setting, there should be an indication in the GitLab web UI for pipelines.