Record latencies for Sidekiq failures

Previously, the sidekiq metrics middleware would not record latency
and cpu duration metrics for failed sidekiq jobs.

This change does two things:
1. Ensures that Sidekiq jobs that fail are also included in latency
   histograms.
1. Adds a `job_status` label to Sidekiq latency metrics so that failed
   job latencies can be differentiated from successful jobs.
174 jobs for !18909 with an-sidekiq-records-failure-durations in 111 minutes and 16 seconds (queued for 7 seconds)
detached
Status Job ID Name Coverage
  Prepare
passed #329528236
docker gitlab-org
build-qa-image

00:05:40

passed #329528228
gitlab-org
compile-assets pull-cache

00:06:33

passed #329528231
gitlab-org
compile-assets pull-cache foss

00:08:04

passed #329528241
retrieve-tests-metadata

00:02:13

passed #329528238
gitlab-org
review-cleanup-failed-deployment

00:00:22

passed #329528234
gitlab-org
setup-test-env

00:09:16

 
  Test
passed #329528275
allowed to fail
code_quality

00:11:26

passed #329528448
gitlab-org
danger-review

00:02:35

passed #329528443
gitlab-org
db:check-schema

00:02:54

passed #329528444
gitlab-org
db:migrate-from-v11.11.0

00:09:37

passed #329528442
gitlab-org
db:migrate:reset

00:03:17

passed #329528445
gitlab-org
db:rollback

00:03:38

passed #329528447
gitlab-org
db:rollback geo

00:03:08

passed #329528277
allowed to fail
dependency_scanning

00:10:21

passed #329528441
gitlab-org
downtime_check

00:02:36

passed #329528251
docker gitlab-org
gitlab:assets:compile pull-cache

00:20:28

passed #329528446
gitlab-org
gitlab:setup

00:03:46

passed #329528449
gitlab-org
gitlab_git_test

00:00:33

passed #329528254
gitlab-org
jest

00:21:00

passed #329528256
gitlab-org
jest-foss

00:17:15

passed #329528252
gitlab-org
karma

00:23:27

67.73%
passed #329528253
gitlab-org
karma-foss

00:13:57

passed #329528450
gitlab-org
lint-ci-gitlab

00:00:33

passed #329528268
gitlab-org
memory-on-boot

00:24:30

passed #329528266
gitlab-org
memory-static

00:07:20

passed #329528259
gitlab-org
qa-frontend-node:10

00:16:27

passed #329528257
gitlab-org
qa-frontend-node:8

00:24:05

passed #329528261
gitlab-org allowed to fail
qa-frontend-node:latest

00:21:31

passed #329528271
gitlab-org
qa:internal

00:01:41

passed #329528273
gitlab-org
qa:selectors

00:01:51

passed #329528274
gitlab-org
qa:selectors-foss

00:01:43

passed #329528439
gitlab-org
rspec fast_spec_helper

00:03:54

passed #329528335
gitlab-org
rspec integration pg9 1/6

00:19:45

passed #329528336
gitlab-org
rspec integration pg9 2/6

00:16:05

passed #329528337
gitlab-org
rspec integration pg9 3/6

00:18:52

passed #329528339
gitlab-org
rspec integration pg9 4/6

00:15:24

passed #329528340
gitlab-org
rspec integration pg9 5/6

00:22:32

passed #329528341
gitlab-org
rspec integration pg9 6/6

00:19:26

passed #329528342
gitlab-org
rspec integration pg9-foss 1/6

00:16:43

passed #329528343
gitlab-org
rspec integration pg9-foss 2/6

00:13:49

passed #329528344
gitlab-org
rspec integration pg9-foss 3/6

00:17:15

passed #329528345
gitlab-org
rspec integration pg9-foss 4/6

00:15:47

passed #329528346
gitlab-org
rspec integration pg9-foss 5/6

00:17:12

passed #329528347
gitlab-org
rspec integration pg9-foss 6/6

00:16:12

passed #329528348
gitlab-org
rspec system pg9 1/24

00:15:25

passed #329528358
gitlab-org
rspec system pg9 10/24

00:17:06

passed #329528359
gitlab-org
rspec system pg9 11/24

00:11:12

passed #329528360
gitlab-org
rspec system pg9 12/24

00:15:03

passed #329528361
gitlab-org
rspec system pg9 13/24

00:15:28

passed #329528362
gitlab-org
rspec system pg9 14/24

00:13:36

passed #329528363
gitlab-org
rspec system pg9 15/24

00:14:22

passed #329528364
gitlab-org
rspec system pg9 16/24

00:13:07

passed #329528365
gitlab-org
rspec system pg9 17/24

00:15:20

passed #329528366
gitlab-org
rspec system pg9 18/24

00:15:46

passed #329528367
gitlab-org
rspec system pg9 19/24

00:16:56

passed #329528349
gitlab-org
rspec system pg9 2/24

00:15:22

passed #329528369
gitlab-org
rspec system pg9 20/24

00:11:47

passed #329528371
gitlab-org
rspec system pg9 21/24

00:16:06

passed #329528373
gitlab-org
rspec system pg9 22/24

00:12:57

passed #329528375
gitlab-org
rspec system pg9 23/24

00:13:46

passed #329528377
gitlab-org
rspec system pg9 24/24

00:14:33

passed #329528350
gitlab-org
rspec system pg9 3/24

00:13:59

passed #329528352
gitlab-org
rspec system pg9 4/24

00:13:43

passed #329528353
gitlab-org
rspec system pg9 5/24

00:14:07

passed #329528354
gitlab-org
rspec system pg9 6/24

00:13:14

passed #329528355
gitlab-org
rspec system pg9 7/24

00:13:39

passed #329528356
gitlab-org
rspec system pg9 8/24

00:14:48

passed #329528357
gitlab-org
rspec system pg9 9/24

00:13:37

passed #329528379
gitlab-org
rspec system pg9-foss 1/24

00:14:08

passed #329528398
gitlab-org
rspec system pg9-foss 10/24

00:14:28

passed #329528399
gitlab-org
rspec system pg9-foss 11/24

00:11:05

passed #329528400
gitlab-org
rspec system pg9-foss 12/24

00:12:59

passed #329528401
gitlab-org
rspec system pg9-foss 13/24

00:12:25

passed #329528402
gitlab-org
rspec system pg9-foss 14/24

00:13:11

passed #329528403
gitlab-org
rspec system pg9-foss 15/24

00:13:14

passed #329528405
gitlab-org
rspec system pg9-foss 16/24

00:13:26

passed #329528406
gitlab-org
rspec system pg9-foss 17/24

00:12:37

passed #329528407
gitlab-org
rspec system pg9-foss 18/24

00:12:19

passed #329528408
gitlab-org
rspec system pg9-foss 19/24

00:13:41

passed #329528381
gitlab-org
rspec system pg9-foss 2/24

00:11:54

passed #329528409
gitlab-org
rspec system pg9-foss 20/24

00:13:08

passed #329528410
gitlab-org
rspec system pg9-foss 21/24

00:13:28

passed #329528411
gitlab-org
rspec system pg9-foss 22/24

00:12:01

passed #329528412
gitlab-org
rspec system pg9-foss 23/24

00:13:09

passed #329528413
gitlab-org
rspec system pg9-foss 24/24

00:14:04

passed #329528383
gitlab-org
rspec system pg9-foss 3/24

00:14:14

passed #329528385
gitlab-org
rspec system pg9-foss 4/24

00:12:27

passed #329528387
gitlab-org
rspec system pg9-foss 5/24

00:11:40

passed #329528390
gitlab-org
rspec system pg9-foss 6/24

00:11:48

passed #329528393
gitlab-org
rspec system pg9-foss 7/24

00:12:35

passed #329528395
gitlab-org
rspec system pg9-foss 8/24

00:11:49

passed #329528397
gitlab-org
rspec system pg9-foss 9/24

00:14:12

passed #329528278
gitlab-org
rspec unit pg9 1/20

00:13:54

passed #329528287
gitlab-org
rspec unit pg9 10/20

00:12:59

passed #329528288
gitlab-org
rspec unit pg9 11/20

00:15:15

passed #329528289
gitlab-org
rspec unit pg9 12/20

00:25:07

passed #329528290
gitlab-org
rspec unit pg9 13/20

00:18:52

passed #329528292
gitlab-org
rspec unit pg9 14/20

00:17:43

passed #329528295
gitlab-org
rspec unit pg9 15/20

00:24:15

passed #329528296
gitlab-org
rspec unit pg9 16/20

00:16:29

passed #329528297
gitlab-org
rspec unit pg9 17/20

00:19:53

passed #329528301
gitlab-org
rspec unit pg9 18/20

00:16:49

passed #329528305
gitlab-org
rspec unit pg9 19/20

00:20:04

passed #329528279
gitlab-org
rspec unit pg9 2/20

00:16:02

passed #329528306
gitlab-org
rspec unit pg9 20/20

00:15:24

passed #329528280
gitlab-org
rspec unit pg9 3/20

00:15:30

passed #329528281
gitlab-org
rspec unit pg9 4/20

00:12:44

passed #329528282
gitlab-org
rspec unit pg9 5/20

00:13:56

passed #329528283
gitlab-org
rspec unit pg9 6/20

00:15:50

passed #329528284
gitlab-org
rspec unit pg9 7/20

00:12:02

passed #329528285
gitlab-org
rspec unit pg9 8/20

00:17:05

passed #329528286
gitlab-org
rspec unit pg9 9/20

00:15:30

passed #329528307
gitlab-org
rspec unit pg9-foss 1/20

00:17:52

passed #329528317
gitlab-org
rspec unit pg9-foss 10/20

00:17:11

passed #329528320
gitlab-org
rspec unit pg9-foss 11/20

00:15:27

passed #329528323
gitlab-org
rspec unit pg9-foss 12/20

00:17:06

passed #329528324
gitlab-org
rspec unit pg9-foss 13/20

00:15:45

passed #329528325
gitlab-org
rspec unit pg9-foss 14/20

00:15:35

passed #329528326
gitlab-org
rspec unit pg9-foss 15/20

00:15:08

passed #329528329
gitlab-org
rspec unit pg9-foss 16/20

00:13:10

passed #329528330
gitlab-org
rspec unit pg9-foss 17/20

00:16:42

passed #329528331
gitlab-org
rspec unit pg9-foss 18/20

00:15:32

passed #329528332
gitlab-org
rspec unit pg9-foss 19/20

00:15:14

passed #329528308
gitlab-org
rspec unit pg9-foss 2/20

00:12:09

passed #329528334
gitlab-org
rspec unit pg9-foss 20/20

00:13:47

passed #329528309
gitlab-org
rspec unit pg9-foss 3/20

00:14:45

passed #329528310
gitlab-org
rspec unit pg9-foss 4/20

00:13:07

passed #329528311
gitlab-org
rspec unit pg9-foss 5/20

00:15:25

passed #329528312
gitlab-org
rspec unit pg9-foss 6/20

00:12:21

passed #329528313
gitlab-org
rspec unit pg9-foss 7/20

00:14:45

passed #329528314
gitlab-org
rspec unit pg9-foss 8/20

00:14:56

passed #329528315
gitlab-org
rspec unit pg9-foss 9/20

00:15:49

passed #329528437
gitlab-org
rspec-ee integration pg10 geo

00:06:42

passed #329528421
gitlab-org
rspec-ee integration pg9 1/3

00:14:59

passed #329528422
gitlab-org
rspec-ee integration pg9 2/3

00:14:12

passed #329528424
gitlab-org
rspec-ee integration pg9 3/3

00:16:33

passed #329528433
gitlab-org
rspec-ee integration pg9 geo

00:06:36

passed #329528438
gitlab-org
rspec-ee system pg10 geo

00:05:59

passed #329528425
gitlab-org
rspec-ee system pg9 1/5

00:15:45

passed #329528426
gitlab-org
rspec-ee system pg9 2/5

00:19:42

passed #329528427
gitlab-org
rspec-ee system pg9 3/5

00:16:21

passed #329528428
gitlab-org
rspec-ee system pg9 4/5

00:15:08

passed #329528429
gitlab-org
rspec-ee system pg9 5/5

00:15:27

passed #329528434
gitlab-org
rspec-ee system pg9 geo

00:05:48

passed #329528435
gitlab-org
rspec-ee unit pg10 geo 1/2

00:44:43

passed #329528436
gitlab-org
rspec-ee unit pg10 geo 2/2

00:11:33

passed #329528414
gitlab-org
rspec-ee unit pg9 1/7

00:15:05

passed #329528415
gitlab-org
rspec-ee unit pg9 2/7

00:14:20

passed #329528416
gitlab-org
rspec-ee unit pg9 3/7

00:15:16

passed #329528417
gitlab-org
rspec-ee unit pg9 4/7

00:15:52

passed #329528418
gitlab-org
rspec-ee unit pg9 5/7

00:15:41

passed #329528419
gitlab-org
rspec-ee unit pg9 6/7

00:17:20

passed #329528420
gitlab-org
rspec-ee unit pg9 7/7

00:15:14

passed #329528430
gitlab-org
rspec-ee unit pg9 geo 1/2

00:13:35

passed #329528432
gitlab-org
rspec-ee unit pg9 geo 2/2

00:13:03

passed #329528276
allowed to fail
sast

00:10:32

passed #329528440
gitlab-org
static-analysis

00:17:20

passed #329528263
gitlab-org
webpack-dev-server

00:04:29

 
  Review Prepare
passed #329528451
review-build-cng

00:32:50

 
  Review
failed #329528452
gitlab-org allowed to fail
review-deploy

00:27:05

manual #329528453
gitlab-org allowed to fail manual
review-stop
 
  Qa
passed #329528455
allowed to fail
dast

00:07:28

manual #329528454
allowed to fail manual
package-and-qa-manual
failed #329739101
gitlab-org docker allowed to fail
review-performance

00:06:57

manual #329528457
docker gitlab-org allowed to fail manual
review-qa-all 1/5
manual #329528458
docker gitlab-org allowed to fail manual
review-qa-all 2/5
manual #329528459
docker gitlab-org allowed to fail manual
review-qa-all 3/5
manual #329528460
docker gitlab-org allowed to fail manual
review-qa-all 4/5
manual #329528461
docker gitlab-org allowed to fail manual
review-qa-all 5/5
failed #329528456
docker gitlab-org allowed to fail
review-qa-smoke

00:08:02

failed #329528462
docker gitlab-org allowed to fail
review-performance

00:04:34

 
  Post Test
passed #329528463
gitlab-org
coverage

00:06:27

97.26%
passed #329528465
gitlab-org allowed to fail
flaky-examples-check

00:03:45

manual #329528464
gitlab-org allowed to fail manual
parallel-spec-reports
 
Name Stage Failure
failed
review-performance Qa
    at processImmediate [as _immediateCallback] (timers.js:722:5)
[2019-10-23 10:06:24] INFO: HTML stored in /sitespeed.io/sitespeed-results
[2019-10-23 10:06:24] INFO: Finished analysing https://gitlab-review-an-sidekiq-5mv21w.gitlab-review.app
Running after script...
$ mv sitespeed-results/data/performance.json performance.json
Uploading artifacts...
performance.json: found 1 matching files
Uploading artifacts to coordinator... ok
id=329739101 responseStatus=201 Created token=viQSEyzU
ERROR: Job failed: exit code 1
failed
review-qa-smoke Qa
	 1: from /usr/local/lib/ruby/2.6.0/net/protocol.rb:191:in `readuntil'
/usr/local/lib/ruby/2.6.0/net/protocol.rb:217:in `rbuf_fill':
Net::ReadTimeout with #<TCPSocket:(closed)> (Net::ReadTimeout)
Running after script...
$ date
Wed Oct 23 08:18:46 UTC 2019
Uploading artifacts...
./qa/gitlab-qa-run-*: found 2 matching files
Uploading artifacts to coordinator... ok
id=329528456 responseStatus=201 Created token=-4WbpeN_
ERROR: Job failed: exit code 1
failed
review-deploy Review
    helm upgrade --install     --wait     --timeout 900     --set ci.branch="an-sidekiq-records-failure-durations"     --set ci.commit.sha="c17922ea"     --set ci.job.url="https://gitlab.com/gitlab-org/gitlab/-/jobs/329528452"     --set ci.pipeline.url="https://gitlab.com/gitlab-org/gitlab/pipelines/90788780"     --set releaseOverride="review-an-sidekiq-5mv21w"     --set global.hosts.hostSuffix="review-an-sidekiq-5mv21w"     --set global.hosts.domain="gitlab-review.app"     --set gitlab.migrations.image.repository="registry.gitlab.com/gitlab-org/build/cng-mirror/gitlab-rails-ee"     --set gitlab.migrations.image.tag="an-sidekiq-records-failure-durations"     --set gitlab.gitaly.image.repository="registry.gitlab.com/gitlab-org/build/cng-mirror/gitaly"     --set gitlab.gitaly.image.tag="v1.70.0"     --set gitlab.gitlab-shell.image.repository="registry.gitlab.com/gitlab-org/build/cng-mirror/gitlab-shell"     --set gitlab.gitlab-shell.image.tag="v10.2.0"     --set gitlab.sidekiq.image.repository="registry.gitlab.com/gitlab-org/build/cng-mirror/gitlab-sidekiq-ee"     --set gitlab.sidekiq.image.tag="an-sidekiq-records-failure-durations"     --set gitlab.unicorn.image.repository="registry.gitlab.com/gitlab-org/build/cng-mirror/gitlab-unicorn-ee"     --set gitlab.unicorn.image.tag="an-sidekiq-records-failure-durations"     --set gitlab.unicorn.workhorse.image="registry.gitlab.com/gitlab-org/build/cng-mirror/gitlab-workhorse-ee"     --set gitlab.unicorn.workhorse.tag="an-sidekiq-records-failure-durations"     --set gitlab.task-runner.image.repository="registry.gitlab.com/gitlab-org/build/cng-mirror/gitlab-task-runner-ee"     --set gitlab.task-runner.image.tag="an-sidekiq-records-failure-durations"   --namespace="review-apps-ee"   --version="90788780-329528452"   -f "../scripts/review_apps/base-config.yaml"   "review-an-sidekiq-5mv21w" .
Release "review-an-sidekiq-5mv21w" does not exist. Installing it now.
Running after script...
$ date
Wed Oct 23 08:10:40 UTC 2019
Uploading artifacts...
review_app_url.txt: found 1 matching files
Uploading artifacts to coordinator... ok
id=329528452 responseStatus=201 Created token=7W2rtpwi
ERROR: Job failed: exit code 1