Sitespeed Metrics threshold configurable
Problem to solve
This is a follow up to https://gitlab.com/gitlab-org/gitlab-ce/issues/57881. We would definitely not want to alert developers for minor drop in metrics. Is there a way to control the sensitivity of the alerts - maybe defining a threshold/range of values for each of the metric would be helpful to avoid any unnecessary alarms.
Could you please provide some thoughts on this? /cc @joshlambert
- When Sasha makes a change that results in overall performance score, transfer size, speed index or requests to degrade by a small amount, they do not want to be bothered by alarms in the Performance Metrics Widget so they can ship valuable features. ** This is addressed.
- When Web Performance has decreased a significant amount over a set number of builds, Sasha wants to see alarms in the Performance Metrics Widget, so that customers do not stop using the website/app because of degraded performance. ** This use case will not be addressed by this change.
- Add to the variables a keyword to specify the degradation threshold for display of the widget.
- This threshold should trigger off of the
total_scoreso any degradation greater or equal to that value triggers display of the widget in the MR. If the degradation is less than the threshold do not display the widget.
- This will be an optional variable and if the value is not set the widget will always display.
- The pipeline passing/failing will not be impacted by the results of the widget displaying / the threshold being exceeded. This is informational only for the user.
stages: - performance include: template: Verify/Browser-Performance.gitlab-ci.yml performance: variables: URL: https://example.com DEGRADATION_THRESHOLD: 5
- When Sasha sets up the web performance job, they want to specify for the various metric types how big of a degradation will trigger an alert in the MR Widget, so that they can ship valuable features.
Permissions and Security
- The feature documentation needs updated as the issue called out has been completed.
What does success look like, and how can we measure that?
- Web Performance can be configured by a user through the gitlab-ci.yml
- Users trust that web performance will not slowly degrade over time without alarms eg; a boiled frog.
What is the type of buyer?
The buyer and user of this feature is the developer who will set and tweak the setting.