Performance widget in MR incorrectly reports changes
Summary
The performance widget incorrectly classifies metrics. Any desiredSize
that is larger
will always be marked as an improvement. Any desiredSize
that is smaller
will always be marked as a regression.
The MR and change that resulted in this bug, as far as I can tell, is here: https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/6685#note_92971338
Basically, after the MR, metrics will be sorted only based on desiredSize
, and the delta
between the base and HEAD does no longer affect whether it's a improvement or regression.
Steps to reproduce
- Create a new empty Gitlab project
- Add the following JSON to a file called
performance.json
[
{
"subject": "https://example.com/foo",
"metrics": [
{ "name": "Total Score", "value": 80, "desiredSize": "larger" },
{ "name": "Total Score (test)", "value": 80, "desiredSize": "larger" },
{ "name": "Requests", "value": 4, "desiredSize": "smaller" },
{ "name": "Transfer Size (KB)", "value": 453.2, "desiredSize": "smaller" },
{ "name": "Speed Index", "value": 400, "desiredSize": "smaller" }
]
}
]
- Create a
.gitlab-ci.yml
file with the content:
performance:
stage: test
image: node:latest
script: echo Do nothing
artifacts:
name: performance
expire_in: 1 week
paths:
- performance.json
reports:
performance: performance.json
- Create a branch + MR, and change the metric values:
[
{
"subject": "https://example.com/foo",
"metrics": [
{ "name": "Total Score", "value": 85, "desiredSize": "larger" },
{ "name": "Total Score (test)", "value": 20, "desiredSize": "larger" },
{ "name": "Requests", "value": 3, "desiredSize": "smaller" },
{ "name": "Transfer Size (KB)", "value": 401.6, "desiredSize": "smaller" },
{ "name": "Speed Index", "value": 350, "desiredSize": "smaller" }
]
}
]
- The performance report will not care about the actual changes
(How one can reproduce the issue - this is very important)
Example Project
https://gitlab.com/csvn/reproduction-performance-widget-issue/merge_requests/1
EDIT: Removed the project above, since issue can no longer be reproduced there
What is the current bug behavior?
If a metric has desiredSize: 'smaller'
, it will be
If a metric has desiredSize: 'larger'
, it will be
What is the expected correct behavior?
If a metric has desiredSize: 'smaller'
, a lower value will be
If a metric has desiredSize: 'larger'
, a lower value will be
Relevant logs and/or screenshots
(Paste any relevant logs - please use code blocks (```) to format console output, logs, and code as it's very hard to read otherwise.)
Output of checks
This bug happens on Gitlab.com
- GitLab Enterprise Edition 11.8.0-rc8-ee
Possible fixes
if (metricData.delta * metricDirection > 0) {
improved.push(metricData);
} else {
degraded.push(metricData);
}