Migrate diffs immediately when an MR is closed or merged

It seems difficult to perform full table scans on merge_request_diffs,
so consider an approach that allows us to queue migrations to external
storage in an event-based manner instead.

This necessitates dropping the 7-days-old cutoff for merged and closed
MRs, but that doesn't seem like a huge problem to me.
139 jobs for 54670-external-diffs-when-outdated-immediate in 57 minutes and 22 seconds (queued for 36 seconds)
latest
Status Job ID Name Coverage
  Prepare
passed #182620246
gitlab-org
compile-assets

00:07:15

passed #182620245
gitlab-org
retrieve-tests-metadata

00:01:53

passed #182620247
gitlab-org
setup-test-env

00:07:04

 
  Test
passed #182620251
allowed to fail
code_quality

00:10:02

passed #182620649
gitlab-org
db:check-schema-pg

00:03:25

passed #182620647
gitlab-org
db:migrate:reset-mysql

00:05:15

passed #182620646
gitlab-org
db:migrate:reset-pg

00:03:56

passed #182620661
gitlab-org
db:rollback-mysql

00:04:51

passed #182620658
gitlab-org
db:rollback-pg

00:05:35

passed #182620704
allowed to fail
dependency_scanning

00:04:28

passed #182620640
gitlab-org
docs lint

00:02:34

passed #182620642
gitlab-org
downtime_check

00:02:53

failed #182620645
gitlab-org
ee_compat_check

00:04:11

passed #182620688
gitlab-org
gitlab:setup-mysql

00:07:52

passed #182620669
gitlab-org
gitlab:setup-pg

00:06:59

passed #182620693
gitlab-org allowed to fail
gitlab:ui:visual

00:05:47

passed #182620711
gitlab-org
gitlab_git_test

00:00:30

passed #182620700
gitlab-org
jest

00:10:01

passed #182620696
gitlab-org
karma

00:13:25

58.74%
passed #182620651
gitlab-org
migration:path-mysql

00:14:13

passed #182620650
gitlab-org
migration:path-pg

00:14:00

passed #182620709
gitlab-org
qa-frontend-node:10

00:06:22

passed #182620708
gitlab-org
qa-frontend-node:8

00:07:04

passed #182620710
gitlab-org allowed to fail
qa-frontend-node:latest

00:06:44

passed #182620705
gitlab-org
qa:internal

00:04:19

passed #182620706
gitlab-org
qa:selectors

00:04:05

passed #182620364
gitlab-org
rspec-mysql 1/50

00:16:12

passed #182620569
gitlab-org
rspec-mysql 10/50

00:19:31

passed #182620587
gitlab-org
rspec-mysql 11/50

00:16:44

passed #182620588
gitlab-org
rspec-mysql 12/50

00:18:35

passed #182620589
gitlab-org
rspec-mysql 13/50

00:19:47

passed #182620590
gitlab-org
rspec-mysql 14/50

00:19:23

passed #182620591
gitlab-org
rspec-mysql 15/50

00:18:09

passed #182620593
gitlab-org
rspec-mysql 16/50

00:19:18

passed #182620594
gitlab-org
rspec-mysql 17/50

00:14:27

passed #182620595
gitlab-org
rspec-mysql 18/50

00:19:51

passed #182620596
gitlab-org
rspec-mysql 19/50

00:20:23

passed #182620372
gitlab-org
rspec-mysql 2/50

00:17:46

passed #182620597
gitlab-org
rspec-mysql 20/50

00:19:58

passed #182620599
gitlab-org
rspec-mysql 21/50

00:19:47

passed #182620600
gitlab-org
rspec-mysql 22/50

00:23:22

passed #182620601
gitlab-org
rspec-mysql 23/50

00:20:40

passed #182620602
gitlab-org
rspec-mysql 24/50

00:19:33

passed #182620603
gitlab-org
rspec-mysql 25/50

00:21:39

passed #182620604
gitlab-org
rspec-mysql 26/50

00:21:24

passed #182620605
gitlab-org
rspec-mysql 27/50

00:18:23

passed #182620606
gitlab-org
rspec-mysql 28/50

00:14:33

passed #182620607
gitlab-org
rspec-mysql 29/50

00:19:08

passed #182620550
gitlab-org
rspec-mysql 3/50

00:18:51

passed #182620609
gitlab-org
rspec-mysql 30/50

00:25:27

passed #182620610
gitlab-org
rspec-mysql 31/50

00:18:26

passed #182620611
gitlab-org
rspec-mysql 32/50

00:21:27

passed #182620612
gitlab-org
rspec-mysql 33/50

00:20:39

passed #182620614
gitlab-org
rspec-mysql 34/50

00:20:39

passed #182620617
gitlab-org
rspec-mysql 35/50

00:19:01

failed #182642285
gitlab-org
rspec-mysql 36/50

00:17:12

passed #182620622
gitlab-org
rspec-mysql 37/50

00:21:45

passed #182620623
gitlab-org
rspec-mysql 38/50

00:20:05

passed #182620624
gitlab-org
rspec-mysql 39/50

00:23:02

passed #182620559
gitlab-org
rspec-mysql 4/50

00:21:56

failed #182641120
gitlab-org
rspec-mysql 40/50

00:18:27

passed #182620626
gitlab-org
rspec-mysql 41/50

00:19:37

passed #182620627
gitlab-org
rspec-mysql 42/50

00:20:08

passed #182620628
gitlab-org
rspec-mysql 43/50

00:23:39

passed #182620629
gitlab-org
rspec-mysql 44/50

00:16:56

passed #182620630
gitlab-org
rspec-mysql 45/50

00:19:04

passed #182620631
gitlab-org
rspec-mysql 46/50

00:22:03

passed #182620632
gitlab-org
rspec-mysql 47/50

00:18:25

passed #182620633
gitlab-org
rspec-mysql 48/50

00:17:52

passed #182620634
gitlab-org
rspec-mysql 49/50

00:24:00

passed #182620561
gitlab-org
rspec-mysql 5/50

00:16:33

passed #182620635
gitlab-org
rspec-mysql 50/50

00:20:04

passed #182620563
gitlab-org
rspec-mysql 6/50

00:22:43

passed #182620565
gitlab-org
rspec-mysql 7/50

00:15:46

passed #182620567
gitlab-org
rspec-mysql 8/50

00:17:20

passed #182620568
gitlab-org
rspec-mysql 9/50

00:20:38

passed #182620638
gitlab-org allowed to fail
rspec-mysql-quarantine

00:11:15

passed #182620253
gitlab-org
rspec-pg 1/50

00:16:25

passed #182620267
gitlab-org
rspec-pg 10/50

00:16:05

passed #182620268
gitlab-org
rspec-pg 11/50

00:17:03

passed #182620270
gitlab-org
rspec-pg 12/50

00:19:07

passed #182620271
gitlab-org
rspec-pg 13/50

00:16:56

passed #182620272
gitlab-org
rspec-pg 14/50

00:19:02

passed #182620273
gitlab-org
rspec-pg 15/50

00:18:38

passed #182620274
gitlab-org
rspec-pg 16/50

00:17:41

passed #182620275
gitlab-org
rspec-pg 17/50

00:15:41

passed #182620277
gitlab-org
rspec-pg 18/50

00:19:21

passed #182620278
gitlab-org
rspec-pg 19/50

00:19:49

passed #182620256
gitlab-org
rspec-pg 2/50

00:19:11

passed #182620279
gitlab-org
rspec-pg 20/50

00:20:41

passed #182620281
gitlab-org
rspec-pg 21/50

00:18:44

passed #182620284
gitlab-org
rspec-pg 22/50

00:21:56

passed #182620285
gitlab-org
rspec-pg 23/50

00:18:01

passed #182620287
gitlab-org
rspec-pg 24/50

00:17:23

passed #182620288
gitlab-org
rspec-pg 25/50

00:21:20

passed #182620290
gitlab-org
rspec-pg 26/50

00:19:33

passed #182620292
gitlab-org
rspec-pg 27/50

00:16:49

passed #182620295
gitlab-org
rspec-pg 28/50

00:16:40

passed #182620297
gitlab-org
rspec-pg 29/50

00:14:36

passed #182620258
gitlab-org
rspec-pg 3/50

00:15:48

passed #182620299
gitlab-org
rspec-pg 30/50

00:25:07

passed #182620301
gitlab-org
rspec-pg 31/50

00:19:12

passed #182620302
gitlab-org
rspec-pg 32/50

00:19:27

passed #182620306
gitlab-org
rspec-pg 33/50

00:17:53

passed #182620310
gitlab-org
rspec-pg 34/50

00:19:32

passed #182620313
gitlab-org
rspec-pg 35/50

00:19:21

failed #182642698
gitlab-org
rspec-pg 36/50

00:18:47

passed #182620319
gitlab-org
rspec-pg 37/50

00:16:58

passed #182620323
gitlab-org
rspec-pg 38/50

00:22:21

passed #182620327
gitlab-org
rspec-pg 39/50

00:20:34

passed #182620259
gitlab-org
rspec-pg 4/50

00:22:38

failed #182642716
gitlab-org
rspec-pg 40/50

00:17:55

passed #182620332
gitlab-org
rspec-pg 41/50

00:18:02

passed #182620336
gitlab-org
rspec-pg 42/50

00:18:36

passed #182620340
gitlab-org
rspec-pg 43/50

00:20:20

passed #182620343
gitlab-org
rspec-pg 44/50

00:18:51

passed #182620345
gitlab-org
rspec-pg 45/50

00:19:36

passed #182620349
gitlab-org
rspec-pg 46/50

00:18:06

passed #182620351
gitlab-org
rspec-pg 47/50

00:23:50

passed #182620354
gitlab-org
rspec-pg 48/50

00:18:05

passed #182620357
gitlab-org
rspec-pg 49/50

00:17:26

passed #182620260
gitlab-org
rspec-pg 5/50

00:20:21

passed #182620360
gitlab-org
rspec-pg 50/50

00:23:36

passed #182620261
gitlab-org
rspec-pg 6/50

00:19:28

passed #182620262
gitlab-org
rspec-pg 7/50

00:17:45

passed #182620264
gitlab-org
rspec-pg 8/50

00:16:22

passed #182620266
gitlab-org
rspec-pg 9/50

00:27:06

passed #182620636
gitlab-org allowed to fail
rspec-pg-quarantine

00:10:37

failed #182645545
allowed to fail
sast

00:25:01

passed #182620639
gitlab-org
static-analysis

00:13:53

failed #182620619
gitlab-org
rspec-mysql 36/50

00:18:16

failed #182620625
gitlab-org
rspec-mysql 40/50

00:17:19

failed #182620316
gitlab-org
rspec-pg 36/50

00:19:15

failed #182620329
gitlab-org
rspec-pg 40/50

00:19:15

failed #182620702
allowed to fail
sast

00:25:03

 
  Post Test
skipped #182620713
gitlab-org
coverage
skipped #182620712
gitlab-org allowed to fail
flaky-examples-check
skipped #182620715
gitlab-org
jsdoc
skipped #182620714
gitlab-org
lint:javascript:report
 
Name Stage Failure
failed
ee_compat_check Test
    

Running after script...
$ date
Fri Mar 22 16:21:18 UTC 2019
Uploading artifacts...
ee_compat_check/patches/*.patch: found 1 matching files
Uploading artifacts to coordinator... ok id=182620645 responseStatus=201 Created token=3v1HczSf
ERROR: Job failed: exit code 1
failed
rspec-mysql 40/50 Test
knapsack/: found 4 matching files                  
rspec_flaky/: found 4 matching files
rspec_profiling/: found 1 matching files
WARNING: tmp/capybara/: no matching files
Uploading artifacts to coordinator... ok id=182641120 responseStatus=201 Created token=aqi8jph5
Uploading artifacts...
junit_rspec.xml: found 1 matching files
Uploading artifacts to coordinator... ok id=182641120 responseStatus=201 Created token=aqi8jph5
ERROR: Job failed: exit code 1
failed
rspec-mysql 36/50 Test
knapsack/: found 4 matching files                  
rspec_flaky/: found 4 matching files
rspec_profiling/: found 1 matching files
WARNING: tmp/capybara/: no matching files
Uploading artifacts to coordinator... ok id=182642285 responseStatus=201 Created token=b2eM8s3E
Uploading artifacts...
junit_rspec.xml: found 1 matching files
Uploading artifacts to coordinator... ok id=182642285 responseStatus=201 Created token=b2eM8s3E
ERROR: Job failed: exit code 1
failed
rspec-pg 40/50 Test
knapsack/: found 4 matching files                  
rspec_flaky/: found 4 matching files
rspec_profiling/: found 1 matching files
WARNING: tmp/capybara/: no matching files
Uploading artifacts to coordinator... ok id=182642716 responseStatus=201 Created token=QK4kFviX
Uploading artifacts...
junit_rspec.xml: found 1 matching files
Uploading artifacts to coordinator... ok id=182642716 responseStatus=201 Created token=QK4kFviX
ERROR: Job failed: exit code 1
failed
rspec-pg 36/50 Test
knapsack/: found 4 matching files                  
rspec_flaky/: found 4 matching files
rspec_profiling/: found 1 matching files
WARNING: tmp/capybara/: no matching files
Uploading artifacts to coordinator... ok id=182642698 responseStatus=201 Created token=CBbudSMu
Uploading artifacts...
junit_rspec.xml: found 1 matching files
Uploading artifacts to coordinator... ok id=182642698 responseStatus=201 Created token=CBbudSMu
ERROR: Job failed: exit code 1
failed
sast Test
2019/03/22 17:07:11 Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
2019/03/22 17:07:11 Timeout when running the analyzer
Running after script...
$ date
Fri Mar 22 17:07:13 UTC 2019
Uploading artifacts...
WARNING: gl-sast-report.json: no matching files
ERROR: No files to upload
ERROR: Job failed: exit code 1