Display count of active jobs in PE Grafana dashboard

Problem

The current application limit job_activity_limit is only enabled on Gitlab.com for projects on Free tier and the limit is set to 500 active jobs. This limit is not enabled for paid tiers.

Calculating a safe threshold for job_activity_limit to apply to paid tiers is very difficult because the number of active jobs is calculated at a specific point in time when the pipeline is being created.

Solution

Tracking of this data was added in !64623 (merged), and that histogram should now be added to the grouppipeline execution Grafana dashboard.

---

In order to come up with safe thresholds for paid tiers we could log a Prometheus metric on how many active jobs are present when a pipeline is created. We don't need to be accurate and using a histogram would be sufficient. If possible we should try to label the metrics with the namespace's plan name so we could observe how the job activity changes per plan. This would help us to:

• understand which possible limit is most effective.
• understand whether there needs to be a per-plan differentiation.

Consider logging a Prometheus histogram as such:

module Gitlab
  module Ci
    module Pipeline
      class Metrics
        def pipeline_size_histogram
          strong_memoize(:active_jobs_histogram) do
            name = :gitlab_ci_active_jobs
            comment = 'Active jobs in pipeline'
            labels = { plan: nil }
            buckets = [0, 200, 500, 1_000, 2_000, 5_000, 10_000]

            ::Gitlab::Metrics.histogram(name, comment, labels, buckets)
          end
        end
      end
    end
  end
end