Managing regressions and taxonomy for tracking affected version of a regression
Introduction
We want to use the regression label to be consistent as what it actually means. We also want a better mechanism to allow us to track regressions in each release and also not affecting the existing release process.
Problem statement
- The way we are using the label regression is not consistent to what a regression actually means.
- The regression label is only applied to regressions that impacted the last monthly release and the RCs of the current release.
- The regression label is also used for bugs for new features for that release.
- This causes inconsistency on how we are categorizing and prioritizing defects.
- The regression label does not capture regressed functionality that are older than 2 months. We have regressed functionality in the system that is not correctly represented.
- The regression label currently looses its meaning overtime and is only used in the context of the current milestone.
- There are regressions that are 6 to 12+ months old, if we are going by all the process that we have laid down all of these bugs can technically skip the exception request which is not accurate.
- CE open regressions https://gitlab.com/gitlab-org/gitlab-ce/issues?label_name=regression
- EE open regressions https://gitlab.com/gitlab-org/gitlab-ee/issues?label_name=regression
- There are discussions around "live" regressions vs "stale" regressions these are ways to prioritize the regressions but it does not speak to what a regression truely means.
- We cannot accurately capture the big picture of overall quality of our product.
Examples
- https://gitlab.com/gitlab-org/gitlab-ce/issues/36907 is a regression from 9.5 which worked in 9.3, however is currently not catagorized as a regression.
- Some of us also leave the regression label assigned since it worked before and was fixed earlier at some point in time. https://gitlab.com/gitlab-org/gitlab-ce/issues/37408#note_57121510
Current definition of a regression
A regression for a particular monthly release is a bug that exists in that release, but wasn't present in the release before. This includes bugs in features that were only added in that monthly release. Every regression must have the milestone of the release it was introduced in - if a regression doesn't have a milestone, it might be 'just' a bug!
Proposal
Use regression to label ~bug that caused a regress in functionality. A regression label tells us a functionality which worked previously but no longer works currently. Introduce a new label taxonomy regression:xx.x
to flag which version the regression was founded in or impacted. For example:
- In 11.0 we released a new
feature X
that is verified as working. Then in release 11.1 the feature no longer works, this is regression for 11.1. The issue should have theregression:11.1
label. - In 11.1-RC3 we shipped a new feature which has been verified as working. Then in 11.1-RC5 the feature no longer works, this is regression for 11.1. The issue should have the
regression:11.1
label.
In addition we will have an automated grooming process that runs every night to ensure that we have label hygiene and data can be extracted with confidence.
-
Manual & Process change Use the
regression
label inconjunction withbug
for definition consistency. ADd theregression:xx.x
to indicate which release the regression affected.- When a regression is found we add the regression label in addition to ~bug .
- Add
Regression:xx.x
when the affected version is determined. - If the affected version
xx.x
inregression:xx.x
is the Current release, it will be scheduled for the current milestone and it can bypass the release exception request. - If the affected version
xx.x
inregression:xx.x
is older than Current release schedule for the current milestone only if it is a ~S1.
-
Automated Use gitlab-bot to automate continous grooming.
- If a
bug
has theregression:xx.x
label, automatically add the regression label - If a
bug
only has theregression
label, automatically add the ~bug label - If a
regression
has noRegression:xx.x
label, post a message to remind the author or EM to set aRegression:xx.x
label.
- If a
-
Taxonomy Introduce a new label
Regression:xx.x
to build granular taxonomy around specific milestones for regressed functionality. This will also be the main mechanism for release managers to pick a fix that does not require an exception request after the 7th.- For 11.0 and onwards we will create this label for every release. e.g.
regression:11.0
regression:11.1
regression:11.2
regression:11.3
and etc - For 10.x and releases prior, we will track at a high level. e.g.
Regression:10.x
regression:9.x
regression:8.x
- For 11.0 and onwards we will create this label for every release. e.g.
Data slicing with labels
This will also allow us to slice data further on missed deliverables.
Data slice | Labels to use |
---|---|
All bugs | ~bug |
All regressions | regression |
Comparing ratio of regressions vs bugs | Total ~bug vs total regression |
Regression for a given release | Regression:xx.x |
All regressions for a team |
Team label + regression
|
Regressions on a release for the team |
Team label + Regression:xx.x
|
Follow up task
-
@godfat Create labels. -
@meks Update process -
@meks Update Release documentation. -
@godfat Implement gitlab-triage logic. -
@meks Announce change.
Related stakeholders
- EMs: @DouweM @smcgivern @plafoucriere @ayufan @bjk-gitlab @andr3 @ClemMakesApps @DylanGriffith @stanhu @erushton
- RMs: @marin @rspeicher @jameslopez (ex RMs @filipa @mayra-cabrera)
- PMs: TBD
- Quality: @markglenfletcher @rymai @ddavison @godfat @meks