Skip to content

Include dev master issues on deployemnt blockers report

Mayra Cabrera requested to merge include-dev-failures-on-deployment-blockers-report into master

What does this MR do and why?

Adjust the deployment blockers report with a dedicated section for dev master failures. This is a way for release managers to be aware that there are pending dev failures that have not been addressed. The new section is split in two subsections:

  • A section to list the pending issues (with state = 'opened')
  • Another section listing the ones that are closed (for historic purposes)

The dev master issues no longer appeared as "Deployment blockers".

Additional steps were added into the "Instructions" section for guidance about how to deal with these issues.

Example:

Click to expand

Deployment Blockers - Week: 2023-09-11-2023-09-17

Overview

Start date End date Production deployments blocked for # of blockers
2023-09-11 2023-09-17 25.5 12

Weekly overview

Issue Blocker type gstg gprd
Kubernetes deployment to staging is stuck (gitlab-org/release/tasks#6522 - closed) 2.0 0.0
GitLab Deployment Health Status: no data (gitlab-org/release/tasks#6521 - closed) 0.0 0.0
QA failure: qa/specs/features/api/9_data_stores... (gitlab-org/release/tasks#6514 - closed) 1.0 1.0
QA failures on gstg-cny (gitlab-org/release/tasks#6513 - closed) ~"RootCause::Flaky-Test" 0.5 0.5
2023-09-12: gstg-cny QA failed (gitlab-org/release/tasks#6481 - closed) ~"RootCause::Flaky-Test" 0.5 0.5
QA failure: qa/specs/features/browser_ui/9_data... (gitlab-org/release/tasks#6474 - closed) ~"RootCause::Flaky-Test" 0.5 0.5
Multiple failures on gstg-cny (gitlab-org/release/tasks#6472 - closed) ~"RootCause::Config-Change" 3.0 3.0
2023-09-14: Issues and comments not loading cor... (gitlab-com/gl-infra/production#16366 - closed) ~"RootCause::Feature-Flag" 1.0 1.0
2023-09-13: Increase in errors and slow respons... (gitlab-com/gl-infra/production#16359 - closed) ~"RootCause::Needed" 0.0 4.0
2023-09-12: ApiServiceLoadbalancerErrorSLOViola... (gitlab-com/gl-infra/production#16355 - closed) ~"RootCause::Config-Change" 2.0 2.0
2023-09-12: Increase in sidekiq traffic causing... (gitlab-com/gl-infra/production#16348 - closed) ~"RootCause::Saturation" 0.0 3.0
2023-09-11: Some project sites looks are broken (gitlab-com/gl-infra/production#16345 - closed) ~"RootCause::Known-Software-Issue" 10.0 10.0
Total 20.5 25.5

Additional incidents

Below is a list of production incidents created last week.

Click to expand
Issue
2023-09-16: WebServiceLoadbalancerErrorSLOViola... (gitlab-com/gl-infra/production#16381 - closed)
2023-09-16: PostgreSQL queries dominating total... (gitlab-com/gl-infra/production#16380 - closed)
2023-09-15: Chef client failures on prometheus ... (gitlab-com/gl-infra/production#16379 - closed)
2023-09-15: GitalyServiceGoserverTrafficAbsentS... (gitlab-com/gl-infra/production#16378 - closed)
2023-09-14: [TEST] delivery hot patching practi... (gitlab-com/gl-infra/production#16371 - closed)
2023-09-14: Pipeline creation keeps timing out (gitlab-com/gl-infra/production#16364 - closed)
2023-09-13: [TEST] delivery hot patching practice (gitlab-com/gl-infra/production#16360 - closed)
2023-09-12: LoggingServiceFluentdLogOutputError... (gitlab-com/gl-infra/production#16357 - closed)
2023-09-11: PraefectServiceProxyErrorSLOViolation (gitlab-com/gl-infra/production#16347 - closed)

Dev incidents

Below is a list of pending broken master issues on the GitLab dev project.

Issue
Friday 2023-09-15 10:46 UTC - `gitlab/gitlab-ee... (gitlab-org/release/tasks#6529 - closed)

Closed incidents

Click to expand

List of closed broken master issues on the GitLab dev project for historic purposes.

Issue
Friday 2023-09-15 06:31 UTC - `gitlab/gitlab-ee... (gitlab-org/release/tasks#6526 - closed)
Thursday 2023-09-14 20:47 UTC - `gitlab/gitlab-... (gitlab-org/release/tasks#6519 - closed)
Thursday 2023-09-14 18:26 UTC - `gitlab/gitlab-... (gitlab-org/release/tasks#6517 - closed)
Thursday 2023-09-14 16:39 UTC - `gitlab/gitlab-... (gitlab-org/release/tasks#6515 - closed)
Thursday 2023-09-14 16:13 UTC - `gitlab/gitlab-... (gitlab-org/release/tasks#6512 - closed)
Thursday 2023-09-14 15:38 UTC - `gitlab/gitlab-... (gitlab-org/release/tasks#6510 - closed)
Thursday 2023-09-14 14:08 UTC - `gitlab/gitlab-... (gitlab-org/release/tasks#6509 - closed)
Thursday 2023-09-14 13:57 UTC - `gitlab/gitlab-... (gitlab-org/release/tasks#6508 - closed)

Instructions

  • Review the "Additional incidents" list and add the Deploys-blocked label if required.
  • Update the "weekly overview" table of this issue to also include:
    • any blocking CRs
    • the Root Cause type for each blocker (if this one is missing)
  • Update the Deployments metric review epic.
    • Add a new row to the Overview section: Copy and paste the information from the Overview section in this issue and link to this issue in the Breakdown of blockers column.
    • Update the Graph: Update the data on the spreadsheet and then update the graph on the Deployments metric review epic.
  • Review if there is any pending master:broken issues on the "Dev incidents" section.
    • Dev issues can be closed if the latest master on GitLab dev is green.

Author Check-list

  • [-] Has documentation been updated?
Edited by Mayra Cabrera

Merge request reports