Skip to content

Managing regressions and taxonomy for tracking affected version of a regression

Introduction

We want to use the regression label to be consistent as what it actually means. We also want a better mechanism to allow us to track regressions in each release and also not affecting the existing release process.

Problem statement

  • The way we are using the label regression is not consistent to what a regression actually means.
    • The regression label is only applied to regressions that impacted the last monthly release and the RCs of the current release.
    • The regression label is also used for bugs for new features for that release.
  • This causes inconsistency on how we are categorizing and prioritizing defects.
    1. The regression label does not capture regressed functionality that are older than 2 months. We have regressed functionality in the system that is not correctly represented.
    • The regression label currently looses its meaning overtime and is only used in the context of the current milestone.
    1. There are regressions that are 6 to 12+ months old, if we are going by all the process that we have laid down all of these bugs can technically skip the exception request which is not accurate.
    2. We cannot accurately capture the big picture of overall quality of our product.

Examples

Current definition of a regression

A regression for a particular monthly release is a bug that exists in that release, but wasn't present in the release before. This includes bugs in features that were only added in that monthly release. Every regression must have the milestone of the release it was introduced in - if a regression doesn't have a milestone, it might be 'just' a bug!

Proposal

Use regression to label ~bug that caused a regress in functionality. A regression label tells us a functionality which worked previously but no longer works currently. Introduce a new label taxonomy regression:xx.x to flag which version the regression was founded in or impacted. For example:

  • In 11.0 we released a new feature X that is verified as working. Then in release 11.1 the feature no longer works, this is regression for 11.1. The issue should have the regression:11.1 label.
  • In 11.1-RC3 we shipped a new feature which has been verified as working. Then in 11.1-RC5 the feature no longer works, this is regression for 11.1. The issue should have the regression:11.1 label.

In addition we will have an automated grooming process that runs every night to ensure that we have label hygiene and data can be extracted with confidence.

  • Manual & Process change Use the regression label inconjunction with bug for definition consistency. ADd the regression:xx.x to indicate which release the regression affected.
    • When a regression is found we add the regression label in addition to ~bug .
    • Add Regression:xx.x when the affected version is determined.
    • If the affected version xx.x in regression:xx.x is the Current release, it will be scheduled for the current milestone and it can bypass the release exception request.
    • If the affected version xx.x in regression:xx.x is older than Current release schedule for the current milestone only if it is a ~S1.
  • Automated Use gitlab-bot to automate continous grooming.
    • If a bug has the regression:xx.x label, automatically add the regression label
    • If a bug only has the regression label, automatically add the ~bug label
    • If a regression has no Regression:xx.x label, post a message to remind the author or EM to set a Regression:xx.x label.
  • Taxonomy Introduce a new label Regression:xx.x to build granular taxonomy around specific milestones for regressed functionality. This will also be the main mechanism for release managers to pick a fix that does not require an exception request after the 7th.
    • For 11.0 and onwards we will create this label for every release. e.g. regression:11.0 regression:11.1 regression:11.2 regression:11.3 and etc
    • For 10.x and releases prior, we will track at a high level. e.g. Regression:10.x regression:9.x regression:8.x

Data slicing with labels

This will also allow us to slice data further on missed deliverables.

Data slice Labels to use
All bugs ~bug
All regressions regression
Comparing ratio of regressions vs bugs Total ~bug vs total regression
Regression for a given release Regression:xx.x
All regressions for a team Team label + regression
Regressions on a release for the team Team label + Regression:xx.x

Follow up task

Related stakeholders

Edited by Mek Stittri