2023-04: Error Budget and Infradev Monthly Report

Budget Spend Information

As per the error budget handbook page. You can see all previous error budget reports as child items in our Monthly Error Budget Reports epic.

Please see #20 (closed) for the previous month's spend.

Generated from data on 2023-04-01 00:00:00 UTC

Recent Changes and Work In Progress

GitLab.com-wide Incidents

These incidents affected all of GitLab.com:

  1. 2023-03-14: PatroniServiceRailsPrimarySqlApdexS... (gitlab-com/gl-infra/production#8541 - closed).
  2. 2023-03-29: Degraded performance on Gitlab.com (gitlab-com/gl-infra/production#8619 - closed).
  3. https://gitlab.com/gitlab-com/gl-infra/reliability-reports/-/issues/156+ has more details.

Individual Stage Groups

  1. The devopsfulfillment exception has expired. We will update the handbook with this shortly.
  2. There is an open exception request for grouptenant scale in Add error budget exception for Tenant Scale group (gitlab-com/www-gitlab-com!121190 - merged), and we'll review this report's data as part of that.
    1. grouptenant scale also absorbed the former Organization group, so they have two rows in the below table.
  3. While groupcode review is over budget, if we take the 28 days up to 2023-03-29 (when the Sidekiq incident above was on GitLab.com), they were at 99.9519%, and so we're considering them within budget for this month.
  4. Both groupproject management and ~"group::integrations" are over budget for the whole month. We'll add more details in the comments below.

Report Updates

  1. The work done in Improvements to error budget for stage group's ... (gitlab-com/gl-infra&918 - closed) means that the issues with inaccurate metrics are now resolved: these numbers are trustworthy, as far as we can tell.
  2. We have added the count of overdue infradev issues for each team. See gitlab-com/gl-infra/scalability#2253 for more details.
  3. We will be updating the handbook with these changes later this month.
  4. Our backlog for error budget reporting is available here: gitlab-com/gl-infra/scalability#1871.
  5. We round these numbers to 2 decimal places up and down. If your stage is over budget, but showing 99.95%, it is likely that your value has been rounded up from < 99.95%. If you'd like this to be done differently or have some other feedback, please drop a comment on 💬 Internal Feedback - Error Budgets (#21 - closed).

Budget Spend by Stage Group

🔴 Over budget (under 99.95% availability)

Product Stage Stage Group Error Budget (28d) Error Budget (7d) Traffic Share (28d) Error Budget Issues Infradev Issues Incident Issues Notes
plan project_management 99.94% 99.94% 4.70% 0 5 0
manage organization 99.88% 99.88% 2.21% 0 23 0 2 0 Moved into grouptenant scale
data_stores tenant_scale 99.87% 99.87% 0.02% 23 4 (with 2 overdue) 2 (S2, S2) Exception request: gitlab-com/www-gitlab-com!121190 (merged)
manage integrations 99.78% 99.90% 0.69% 2 1 0
not_owned not_owned 99.37% 99.41% 1.29% 0 0 0

🔶 Active exceptions

Product Stage Stage Group Error Budget (28d) Error Budget (7d) Traffic Share (28d) Error Budget Issues Infradev Issues Incident Issues Notes
data_stores global_search 99.86% 99.88% 0.16% 1 0 0
govern threat_insights 99.95% 99.95% 0.07% 9 3 0

✅ Within budget (over 99.95% availability)

Product Stage Stage Group Error Budget (28d) Error Budget (7d) Traffic Share (28d) Error Budget Issues Infradev Issues Incident Issues Notes
verify runner 100.00% 100.00% 4.88% 0 0 0
growth acquisition 100.00% 100.00% 0.03% 0 0 0
systems gitaly_cluster 100.00% 100.00% 24.71% 0 0 0
govern compliance 100.00% 100.00% 0.38% 2 0 0
secure composition_analysis 99.99% 99.99% 0.11% 0 0 0
secure static_analysis 99.99% 99.99% 0.06% 1 0 0
configure configure 99.99% 99.99% 0.58% 0 0 0
manage foundations 99.99% 100.00% 0.22% 0 0 0
package container_registry 99.99% 100.00% 12.69% 0 0 0
manage authentication_and_authorization 99.99% 99.99% 2.61% 1 4 (with 1 overdue) 0
plan knowledge 99.98% 99.98% 0.06% 6 1 1 (S2)
package package_registry 99.98% 99.99% 1.22% 3 0 0
verify pipeline_security 99.98% 99.99% 0.30% 2 1 0
plan optimize 99.98% 99.98% 0.01% 0 0 0
manage import 99.98% 99.98% 0.03% 1 2 0
create editor 99.98% 99.98% 5.11% 1 0 0
analytics product_intelligence 99.98% 99.97% 0.03% 0 0 0
verify pipeline_authoring 99.97% 99.97% 0.06% 7 0 0
create source_code 99.96% 99.97% 19.29% 0 1 0
plan product_planning 99.96% 99.96% 0.06% 0 0 0
verify pipeline_execution 99.96% 99.95% 10.46% 6 2 (with 1 overdue) 1 (S3)
deploy environments 99.96% 99.96% 0.04% 2 1 0
release release 99.95% 99.96% 4.33% 0 0 0
create code_review 99.95% 99.93% 3.53% 0 5 (with 3 overdue) 0 Moved (see notes above)

🔷 Below traffic share threshold (under 0.01%)

Product Stage Stage Group Error Budget (28d) Error Budget (7d) Traffic Share (28d) Error Budget Issues Infradev Issues Incident Issues Notes
secure dynamic_analysis 100.00% 99.97% 0.00% 0 0 0
5-min-app 5-min-app 100.00% 100.00% 0.00% 0 0 0
plan certify 100.00% 100.00% 0.01% 0 0 0
deploy 5-min-app 100.00% 100.00% 0.00% 0 0 0
systems geo 100.00% 100.00% 0.00% 0 0 0
data_stores pods 99.99% 99.99% 0.00% 0 0 0
monitor respond 99.99% 100.00% 0.01% 1 0 0
growth activation 99.97% 99.94% 0.00% 0 0 0
modelops mlops 99.97% 0.00% 0.00% 0 0 0
anti-abuse anti-abuse 99.95% 99.94% 0.00% 0 0 0
modelops ai_assisted 99.95% 99.94% 0.00% 0 0 0
data_stores database 99.84% 99.65% 0.00% 0 2 (with 1 overdue) 1 (S3)
analytics product_analytics 99.83% 99.33% 0.00% 0 0 0
monitor observability 99.83% 99.78% 0.00% 0 1 0
mobile mobile_devops 99.76% 99.66% 0.00% 0 0 0
fulfillment utilization 99.75% 99.79% 0.01% 1 0 0
fulfillment purchase 99.74% 99.67% 0.00% 0 0 0
fulfillment fulfillment_platform 97.59% 97.80% 0.00% 0 0 0
fulfillment billing_and_subscription_management 95.83% 97.85% 0.00% 0 0 0
govern security_policies 86.48% 79.46% 0.00% 2 1 0
fulfillment provision 82.93% 82.97% 0.00% 5 0 0
no_code no_code_automation 0.00% 0.00% 0.00% 0 0 0
Edited by Crystal Poole