FY20-Q4 OKR Get CI queue times (p95) under 1 min

This issue is to track development optimization of CI queue times until they are reliably under 1 min, and infrastructure is comfortable alerting on breach of that SLA:

• Based on what I’ve found, the initial concern here was related to large spikes in CI queue time every hour due to scheduled jobs all being enqueued at the same time: https://gitlab.com/gitlab-com/gl-infra/infrastructure/issues/7814
• Two changes are currently being tested to mitigate this problem: gitlab-com/gl-infra/production#1136 (closed)
• Test 1: Adjust warm pool for shared runner managers - https://ops.gitlab.net/gitlab-cookbooks/chef-repo/merge_requests/1784
• Test 2: Adjust cron schedule for pipeline_schedule_worker from ‘19 * * * *’ - https://ops.gitlab.net/gitlab-cookbooks/chef-repo/merge_requests/1806 The early results of these changes are promising, but the tests are still in progress and there will be some additional evaluations still to be done gitlab-com/gl-infra/production#1136 (comment 218753472)

Update 2019-09-30 to include a graph with metric that will be used to measure this outcome.

CC @clefelhocz1 @ansdval @glopezfernandez

Additional context from product scaling agenda at https://docs.google.com/document/d/1nMJzrDfG7C14WP5v7P226oPFuXkwqIk7bdIT8ai0DNU/edit?ts=5d84fb07&skip_itp2_check=true&pli=1#bookmark=id.acbz08dge98p

added Engineering Management + 1 deleted label

added Infrastructure Department label and removed 1 deleted label

added 1 deleted label and removed Infrastructure Department label

assigned to @darbyfrey

changed the description

To refine this a bit it would be good to define the target perc for the < 1 min queue time goal. Should this be max, p99, p95, etc. It seems like a good starting point would be p95 and we could iterate from there.

added 1 deleted label

unassigned @darbyfrey

assigned to @darbyfrey

@darbyfrey CC @ansdval

Based on https://gitlab.com/gitlab-com/gl-infra/infrastructure/issues/7814#note_221666878 it looks like the spikiness is almost completely gone.

@darbyfrey I agree with you on starting with a percentile. Although, as far as actual expectation for runner times, we may need to be even more specific and alert on timing based on the job being performed. I called out in the Google Doc that project exporter times run long. We might miss out on noisy alerting by using a percentile, but 1 min is completely unreasonable for large projects.

@ansdval, could you link to the Google Doc you referenced here so we're all looking at the same thing?

Also, perhaps we need to clarify the specific queue time we are referring to. My understanding is that we're talking the time a CI jobs sits in the queued state before it kicks off, so specifically the Ci::BuildPrepare worker, which wouldn't include project exporters, importers, etc. Does that sound right?

Ah, sorry, I misspoke. I linked to https://gitlab.com/gitlab-com/gl-infra/infrastructure/issues/7967 in a comment in the Performance & Availability meeting agenda (Google Doc).

Here's a relevant snippet from the above issue's description:

https://dashboards.gitlab.net/d/sidekiq-main/sidekiq-overview?orgId=1 shows the metrics in detail. It's been raised that certain–maybe even all?–background processing jobs should adhere to a 1 min latency average. Though, at first glance, this may not be possible for all jobs–project exports take substantially longer than others.

I can't comment about whether or not that sounds right, because I don't know enough about the workers. The fact that you can identify a worker by name indicates to me that you have a much better understanding of the architecture and workflow than me.

The initial issue that was referenced was focused on the job_queue_duration_seconds_bucket metric in prometheus. So I'd like to propose we focus on the p95 for that metric for this issue.

Sample Chart

If we intend to alert on > 60s, then the target here should be closer to < 30s.

Does that seem reasonable?

/cc @erushton

I think that makes sense.

FYI @dawsmith this is relevant to your interests

Yup - makes sense - like tracking to either p99 or p95. I'd also ask that for infra, we have some test investigations into spikes we currently see. It seems like we have a daily jump around 0400UTC and the last 2 days were not that great: https://dashboards.gitlab.net/d/000000159/ci?orgId=1&refresh=5m&fullscreen&panelId=89&from=now-2d&to=now

and

https://dashboards.gitlab.net/d/000000159/ci?orgId=1&refresh=5m&fullscreen&panelId=89&from=now-7d&to=now

After looking at the 7d graph, I wouldn't be comfortable alerting until we:

1. understand the 0400 daily spike
2. get past the point of certain daily alerts - it will just create fatigue.

added gitlab.com label

changed title from Get CI queue times under 1 min to Get CI queue times (p95) under 1 min

changed the description

Updated the issue description to include a chart for tracking: https://dashboards.gitlab.net/d/000000159/ci?orgId=1&from=now-12h&to=now&fullscreen&panelId=89&refresh=5m&var-runner_type=All&var-runner_managers=All&var-gitlab_env=gprd&var-gl_monitor_fqdn=All&var-has_minutes=yes&var-runner_job_failure_reason=All&var-jobs_running_for_project=0&var-runner_request_endpoint_status=All

Also, a change was deployed on 2019-09-30 to change the cron for scheduled jobs to run every 10 minutes instead of every 20 minutes. This is expected to improve the spikes in queue times as well. Will monitor the results.

gitlab-com/gl-infra/production#1136 (comment 224149949)

@andrewn, is there anything additional we need to consider here: https://gitlab.com/gitlab-com/gl-infra/infrastructure/issues/7967. Or now that the metric exists on the dashboard, is this is straightforward as adding the PromQL definition and alerting threshold?

@ansdval I have not been focused on CI job queue times, but it's important to understand that CI queue times (what this issue is about) and Sidekiq queue times (what I've been focused on) have no correlation. Certainly, you can't use sidekiq queue latencies to measure CI job queue latencies.

FWIW (and not strictly related to this request), the p95 scheduling time for all CI Sidekiq jobs over the past 7 days is 120 milliseconds.

Going forward, what I would recommend is that we build a Apdex key metric for the ci-runners service for the CI latency, using the metrics from this chart: https://dashboards.gitlab.net/d/000000159/ci?orgId=1&from=now-12h&to=now&fullscreen&panelId=89&refresh=5m&var-runner_type=All&var-runner_managers=All&var-gitlab_env=gprd&var-gl_monitor_fqdn=All&var-has_minutes=yes&var-runner_job_failure_reason=All&var-jobs_running_for_project=0&var-runner_request_endpoint_status=All

Once we have that we can define an SLO for that key metric.

@ansdval this would be a good apdex measurement to use:

https://thanos-query.ops.gitlab.net/graph?g0.range_input=1w&g0.expr=sum(job_queue_duration_seconds_bucket%7Bshared_runner%3D%22true%22%2C%20le%3D%2260%22%7D)%20by%20(environment)%0A%2F%0Asum(job_queue_duration_seconds_bucket%7Bshared_runner%3D%22true%22%2C%20le%3D%22%2BInf%22%7D)%20by%20(environment)%0A&g0.tab=0

According to this, during week days, about 95% of CI jobs start within 1 minute, on the weekend it's only about 75%.

What's really strange about this graph is that it's the only graph I've ever seen at GitLab where the performance during the week is better than the performance on the weekends. With most of our other metrics, we see better performance on the weekends on account of the relative overprovisioning during this time.

Obviously with shared runners we have autoscaling in place, leading to few machines being available during the weekend, and slower CI startup times as machines need to be provisioned before jobs can run.

mentioned in merge request runbooks!1508 (merged)

I've added a key-metric for apdex on the CI-runners service: runbooks!1508 (merged)

There's not much data yet, but here's how it looks:

The dotted red line shows us the initial SLO, which I've set initially to 80% until we can improve the service.

This metric gives us a long-term SLO for the CI-runners service queuing time, which will be tracked in the SLO dashboard and will be included in the headline GitLab.com availability figure generated at https://dashboards.gitlab.net/d/general-slas/general-slas?orgId=1

Quick update on these changes: since we setup an SLO for the ci-runners service yesterday, we had a production incident around the time of the daily 4AM UTC CI spike: gitlab-com/gl-infra/production#1232 (moved).

FYI: @dawsmith, during the Infradev call, it was discussed that a good first step in reducing this latency would be to experiment with the autoscaling parameters that we use for GitLab CI.

cc @aamarsanaa as EOC for the incident.

cc @tmaczukin FYI for expertise in autoscaling GitLab CI.

mentioned in issue gitlab-com/gl-infra/production#1232 (moved)

marked this issue as related to gitlab-com/gl-infra/production#1232 (moved)

marked this issue as related to gitlab-com/gl-infra/scalability#41

All,

After discussion with @tmaczukin we've adjusted the SLO to include what's called the fair-usage policy. See gitlab-com/gl-infra/scalability#41 + runbooks!1522 (merged) and comprehensively explained in gitlab-com/gl-infra/production#1232 (comment 228609563).

The SLO now works as follows:

If a user has between 0 and 4 jobs currently running on our shared CI runners (aka fair usage), the p95 for the CI queue time should be less than 1 minute.

• This relies on built in functionality in the CI scheduler which progressively deprioritises jobs for projects that have more more jobs currently executing.
• The reason we chose 4 is that from an observability point of view, everything beyond 4 jobs is currently thrown into a single bucket. If we want to make this number bigger than 4, we need to make some (small) application changes.
• This fair-usage strategy protects users running a reasonable number of jobs from users issuing 10000 simultaneous CI jobs, which would be very expensive to provision within a short period.

I've also opened https://gitlab.com/gitlab-com/gl-infra/infrastructure/issues/8130 for the CI/CD infrastructure team to adjust the off-peak (weekend) base load of CI runners, which is set to 15 nodes, several years ago, and which explains the poor SLI that we see over weekends.

mentioned in merge request runbooks!1531 (merged)

Note that we've lowered the alerting threshold for this SLO to 50% (down from 80%) as that proved to be too ambitious for now: runbooks!1531 (merged)

assigned to @erushton

Note that we've lowered the alerting threshold for this SLO to 50% (down from 80%) as that proved to be too ambitious for now: runbooks!1531 (merged)

Since bringing the target down from 80% to 50%, the SRE team are still receiving too many non-actionable alerts for this service. @msmiley opened up an issue to discuss it here: https://gitlab.com/gitlab-com/gl-infra/infrastructure/issues/8215

@darbyfrey @erushton is there a list of backstage items that needs to be scheduled to make this reality? I'd be happy to schedule them (or possibly @DarrenEastman / @joshlambert depending on if the changes land in their area.)

mentioned in issue #5649 (closed)

We had a call yesterday to discuss this and will be having a recurring call each week until this is resolved.

Outcomes of yesterday's call:

• @andrewn is tuning the alerts so that they're not overly noisy and need muting but that means they'll not quite be as tight as we want them to be for the time being.
• We're further investigating adjusting the auto scale idle count size that is in place on weekends as it is out of date
• There's a LOT of issues on the go in both the gitlab-com and gitlab-org groups - I'll be making an epic for each to help track them
• We also discussed that using an apdex measurement as opposed to a raw queue time measurement from prometheus makes more sense.

marked this issue as related to #5665 (closed)

added to epic &187 (closed)

assigned to @jlenny

assigned to @DarrenEastman

@DarrenEastman as runner PM I'm transfering this item to you. Please follow along, and as part of managing your product ensure that timely runner shared fleet availability is good. Please keep me posted and let me know when this is closed out for good, and feel free to add to our 1:1 agenda if you want to discuss further.

unassigned @jlenny

changed title from Get CI queue times (p95) under 1 min to FY20-Q4 OKR Get CI queue times (p95) under 1 min

added CI/CD Sub-Department label

mentioned in merge request runbooks!1563 (merged)

Another issue appears to be that we are bumping up against the limit of our shared runner manager infrastructure.

runbooks!1563 (merged) adds a saturation metric to the capacity planning framework to allow us to track this saturation point and plan ahead.

removed Engineering Management label

I've created an issue to scale up our shared runner fleet to reduce the contention we are currently seeing there, which is leading to increased CI job latency: https://gitlab.com/gitlab-com/gl-infra/infrastructure/issues/8302

@dawsmith & @ansdval I've provisionally labelled this the CI/CD & Enablement team, but @ansdval is directly mentioned in this issue so feel free to reassign if this its incorrectly labelled.

We are now monitoring the runner saturation and will get alerts on this https://dashboards.gitlab.net/d/alerts-saturation_component/alerts-saturation-component-alert?orgId=1&from=now-3h&to=now&var-PROMETHEUS_DS=Global&var-environment=gprd&var-type=ci-runners&var-stage=main&var-component=shared_runners

added wg_CIQueueStability label

mentioned in commit 80208a17

mentioned in merge request !34171 (merged)

mentioned in commit 74df6fcd

marked this issue as related to gitlab-org/gitlab#33098 (closed)

Adding a quick status update, discussion is happening in this issue to address the 4am UTC spikes gitlab-org/gitlab#33098 (closed)

@darbyfrey how are we doing wrapping up this Q4 goal by the end of this month? Is there an easy way to see status?

@jyavorska, things have improved quite a bit, and I think we're on track to wrap this up soon. You can see the current status in the Where we’re at now section of this doc: https://docs.google.com/document/d/1YC098dO9_zm1jjNlx8tXTY__l_BniQDV10ejoG_Q_GU/edit#heading=h.f5246k2nvssn

Oh, that's great info @darbyfrey. Any chance we could share it here in this issue where it's more discoverable?

I'll add basically the contents of the above doc to a new comment thread below.

mentioned in issue gitlab-org/gitlab#196732

Summary Update as of Jan 15, 2020

Short Summary

After the work we've done over the last few months we now have very stable, and very predictable queue times for the CI jobs running on shared runners on GitLab.com. There is one predictable spike of about 4-minutes at 4am UTC every day, other than that jobs are spending less than a few seconds in the queue on average.

Deeper Examination

Let's start with a couple of quick graphs:

Here's what an average day looked like before we started this initiative. The graph spans a 24-hour window from 10:00 UTC to 9:59 UTC. There are spikes every hour or so and most them peaking to 13-14 minutes.

Here's what an average day looks like now. The graph shows a 24-hour span of the same time range as the one above. There is one spike at 4am UTC peaking at about 4-minutes. The other jittery little spikes are in the 2-3 second range.

The dashboard where this and other graphs are available is here

What we did to get here

• Changed the GitLab.com scheduled job frequency from once an hour on the 19th of the hour to every 10 minutes - effectively dividing the scheduled job load by 6
• Increasing our “offPeak” idle count to match our regular idle count. Idle count is the number of “warm” VMs we keep on hand to assign CI jobs to. Due to the ~5-minute lag time between making an API call to create a new VM and the time it’s ready, we must pre-provision them. The number of VM’s we aim to have pre-provisioned and ready to go is the idlecount. On weekends we switched to an off-peak lower number, which hadn’t been adjusted for growth in recent years. This value now matches regular peak values
• Drastically increase the (regular) idlecount. We now have a lot more idle machines ready to go at any given moment. This gives us an immediate buffer to help us absorb any thunder herds or other large influxes of CI jobs.

What we can do to improve further

Short Term

• Add a feature to make increasing the idle count scheduledable ahead of known peak times (4am UTC) gitlab-org/gitlab-runner#5069 (closed)
  • This won’t improve the situation from where it is now but make it more cost effective
• Don’t add new scheduled jobs at 4am gitlab-org/gitlab#33098 (closed)
  • Again, this won’t actually solve the situation from where we are now but make it so new scheduled jobs don’t pile on and make it worse

Longer Term

• Explore options around sharding how we’re queuing jobs - separate queues for scheduled jobs, separate queues for paying vs free, etc.

The goals of this issue have been accomplished by the CI Queue Time Stabilization Working Group. That WG is in the process of closing down so I will close this issue as well !39195 (merged).

closed

changed the description

added CEO Interest label and removed 1 deleted label

added DepartmentDevelopment label