2022-11: Error Budget Monthly Report

Budget Spend Information

As per the error budget handbook page.

Please see #13 (closed) for the previous month's spend.

Generated from data on 2022-11-02 00:00:00 UTC

Recent Changes and Work In Progress

We will be consolidation this report with the Infradev report over the next few months. We are also working on bringing all of the data that the Scalability team generates into a single location for ease of reference. The first iteration is being worked on here: gitlab-com/gl-infra&790 (closed)

Budget Spend by Stage Group

🔴 Over budget

Product Stage Stage Group Error Budget DRI Notes
analytics product_analytics 0.00
mobile moble_devops 0.00
modelops mlops 0.00
fulfillment commerce_integrations 90.34 @isandin There is currently an issue with how error reporting data is collected which is hampering in depth analysis. The availability issues are likely from saleforce issues being addressed elsewhere, next month should let us know if that was the right theory. For the apdex issues they're consistently bad enough to suggest we might just have our expectations calibrated poorly, once data issues are sorted out we should be able to confirm or deny that theory.
modelops applied_ml 91.40 From: @a_akgun @mray2020 @tle_gitlab. Suggested reviewer is a progressive feature and is integrated to a new tech stack running on GCP - We don't have customer facing endpoints or controllers which deplete the error budget. We are in Open Beta. We let errors surface in the background jobs to be able to see. Soon we shall eliminate all the errors.
monitor observability 93.75
fulfillment billing_and_subscription_management 94.64
govern security_policies 96.79 @mparuszewski The 7-day budget is at 99.98%, planned background migration has finished recently and we see less errors. In scope of &9102 (closed) we are working on improvements to the Scan Result Policies that should positively affect our error budget. Especially we are focused on gitlab#381889 (closed) to work on potential problems that affects error budget.
data_stores database 98.49
manage import 99.11
fulfillment fulfillment_platform 99.62 @jameslopez customers-gitlab-com#5119 (closed)
not_owned not_owned 99.69
plan certify 99.74
fulfillment purchase 99.85 @shreyasagarwal We have 1 issue open to investigate the performance customers-gitlab-com#5080 (closed). The error budget went up from 99.77 last month to 99.85
manage integrations 99.88 @arturoherrero @.luke Impacted by one issue that we resolved gitlab#378081 (closed) and by a new scalability issue gitlab-com/gl-infra/scalability#1980
deploy 5-min-app 99.92
create editor 99.94

🔶 Active Exceptions

Product Stage Stage Group Error Budget DRI Notes
fulfillment provision 78.97 @mlunoe Provision is still over the error budget and we will be until the issues below are closed. We did see improvement from the original issue, but after the change landed we discovered three additional issues that we are currently looking into:

1. workflowin dev: customers-gitlab-com#4884 (closed)
1. Scheduled for %15.7: customers-gitlab-com#4876 (closed)
1. Scheduled for %15.7: customers-gitlab-com#4875 (closed)
data_stores global_search 99.26 @changzhengliu The exception was expired. This MR removes it. The correction is tracked by https://gitlab.com/gitlab-org/search-team/team-tasks/-/issues/120
manage workspace 99.86
govern threat_insights 99.94 @quintasan Exception for 99.5%.

Within budget

Product Stage Stage Group Error Budget DRI Notes
plan product_planning 99.95
package package_registry 99.95
create source_code 99.95
plan optimize 99.96
create code_review 99.96
plan project_management 99.96
growth acquisition 99.96
analytics product_intelligence 99.96
verify pipeline_insights 99.96
systems geo 99.96
manage optimize 99.96
verify pipeline_authoring 99.97
secure composition_analysis 99.97
verify pipeline_execution 99.97
ecosystem integrations 99.98
secure dynamic_analysis 99.98
ecosystem foundations 99.98
package container_registry 99.98
fulfillment utilization 99.98
manage foundations 99.99
release release 99.99
data_stores pods 99.99
manage authentication_and_authorization 99.99
package package 99.99
configure configure 99.99
secure static_analysis 99.99
govern compliance 100.00
growth activation 100.00
systems gitaly_cluster 100.00
systems gitaly 100.00
verify runner 100.00
anti-abuse anti-abuse 100.00
monitor respond 100.00
Edited by Isabel Sandin