Skip to content

Sitespeed Metrics threshold configurable

Problem to solve

This is a follow up to https://gitlab.com/gitlab-org/gitlab-ce/issues/57881. We would definitely not want to alert developers for minor drop in metrics. Is there a way to control the sensitivity of the alerts - maybe defining a threshold/range of values for each of the metric would be helpful to avoid any unnecessary alarms.

Could you please provide some thoughts on this? /cc @joshlambert

Intended users

Further details

  • When Sasha makes a change that results in overall performance score, transfer size, speed index or requests to degrade by a small amount, they do not want to be bothered by alarms in the Performance Metrics Widget so they can ship valuable features. ** This is addressed.
  • When Web Performance has decreased a significant amount over a set number of builds, Sasha wants to see alarms in the Performance Metrics Widget, so that customers do not stop using the website/app because of degraded performance. ** This use case will not be addressed by this change.

Proposal

  • Add to the variables a keyword to specify the degradation threshold for display of the widget.
  • This threshold should trigger off of the total_score so any degradation greater or equal to that value triggers display of the widget in the MR. If the degradation is less than the threshold do not display the widget.
  • This will be an optional variable and if the value is not set the widget will always display.
  • The pipeline passing/failing will not be impacted by the results of the widget displaying / the threshold being exceeded. This is informational only for the user.

Sample usage:

stages:
  - performance

include:
  template: Verify/Browser-Performance.gitlab-ci.yml

performance:
  variables:
    URL: https://example.com
    DEGRADATION_THRESHOLD: 5

Outcomes

  • When Sasha sets up the web performance job, they want to specify for the various metric types how big of a degradation will trigger an alert in the MR Widget, so that they can ship valuable features.

Permissions and Security

Documentation

Testing

What does success look like, and how can we measure that?

  • Web Performance can be configured by a user through the gitlab-ci.yml
  • Users trust that web performance will not slowly degrade over time without alarms eg; a boiled frog.

What is the type of buyer?

The buyer and user of this feature is the developer who will set and tweak the setting.

Links / references

Edited by James Heimbuck