Performance widget in MR incorrectly reports changes

Summary

The performance widget incorrectly classifies metrics. Any desiredSize that is larger will always be marked as an improvement. Any desiredSize that is smaller will always be marked as a regression.

The MR and change that resulted in this bug, as far as I can tell, is here: https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/6685#note_92971338

Basically, after the MR, metrics will be sorted only based on desiredSize, and the delta between the base and HEAD does no longer affect whether it's a improvement or regression.

Steps to reproduce

  1. Create a new empty Gitlab project
  2. Add the following JSON to a file called performance.json
[
  {
    "subject": "https://example.com/foo",
    "metrics": [
      { "name": "Total Score", "value": 80, "desiredSize": "larger" },
      { "name": "Total Score (test)", "value": 80, "desiredSize": "larger" },
      { "name": "Requests", "value": 4, "desiredSize": "smaller" },
      { "name": "Transfer Size (KB)", "value": 453.2, "desiredSize": "smaller" },
      { "name": "Speed Index", "value": 400, "desiredSize": "smaller" }
    ]
  }
]
  1. Create a .gitlab-ci.yml file with the content:
performance:
  stage: test
  image: node:latest
  script: echo Do nothing
  artifacts:
    name: performance
    expire_in: 1 week
    paths:
      - performance.json
    reports:
      performance: performance.json
  1. Create a branch + MR, and change the metric values:
[
  {
    "subject": "https://example.com/foo",
    "metrics": [
      { "name": "Total Score", "value": 85, "desiredSize": "larger" },
      { "name": "Total Score (test)", "value": 20, "desiredSize": "larger" },
      { "name": "Requests", "value": 3, "desiredSize": "smaller" },
      { "name": "Transfer Size (KB)", "value": 401.6, "desiredSize": "smaller" },
      { "name": "Speed Index", "value": 350, "desiredSize": "smaller" }
    ]
  }
]
  1. The performance report will not care about the actual changes

image (How one can reproduce the issue - this is very important)

Example Project

https://gitlab.com/csvn/reproduction-performance-widget-issue/merge_requests/1

EDIT: Removed the project above, since issue can no longer be reproduced there

What is the current bug behavior?

If a metric has desiredSize: 'smaller', it will be ❌
If a metric has desiredSize: 'larger', it will be ✔

What is the expected correct behavior?

If a metric has desiredSize: 'smaller', a lower value will be ✔, a higher value ❌
If a metric has desiredSize: 'larger', a lower value will be ❌, a higher value ✔

Relevant logs and/or screenshots

(Paste any relevant logs - please use code blocks (```) to format console output, logs, and code as it's very hard to read otherwise.)

Output of checks

This bug happens on Gitlab.com

  • GitLab Enterprise Edition 11.8.0-rc8-ee

Possible fixes

https://gitlab.com/gitlab-org/gitlab-ee/blob/master/ee/app/assets/javascripts/vue_merge_request_widget/stores/mr_widget_store.js#L122-126

if (metricData.delta * metricDirection > 0) {
  improved.push(metricData);
} else {
  degraded.push(metricData);
}
Edited Apr 08, 2019 by Christian Svensson
Assignee Loading
Time tracking Loading