Gitlab standalone gives ERR_TOO_MANY_REDIRECTS error after upgrade
Summary
I have Gitlab installed in LXD container on my Ubuntu Server. I have Apache running in another container and I have a website in Apache set up to perform redirects with Gitlab instance (with HTTPS setup). Everything was working just fine for a few years, but after update (12.7.5-ee.0) over (12.6.2-ee.0) it doest't load webiste and throws ERR_TOO_MANY_REDIRECTS error in browser.
I wasn't able to find any info on how to resolve that and decided to restore whole LXD container from backup image (I have 1 image from the previous day and another one from sunday). This was successfull, Gitlab started and loaded, but gives me 503 on any attempt to load a full list of projects, a task, etc. After running reconfigure it goes back to throwing ERR_TOO_MANY_REDIRECTS.
Steps to reproduce (not exactly, just some info about my installation)
- Install Gitlab 12.6.2-ee.0 in LXD container (latest Ubuntu LTS)
- Configure external Apache site as a reverse proxy for Gitlab installation, with HTTPS (Let's Encrypt cert)
HTTPS config:
<IfModule mod_ssl.c>
<VirtualHost *:443>
ProxyPreserveHost On
ProxyPass / http://192.168.0.5/
ProxyPassReverse / http://192.168.0.5/
ServerName redacted
ServerAlias redacted
SSLCertificateFile /etc/letsencrypt/live/api.redacted/fullchain.pem
SSLCertificateKeyFile /etc/letsencrypt/live/api.redacted/privkey.pem
Include /etc/letsencrypt/options-ssl-apache.conf
</VirtualHost>
</IfModule>Regular config:
<VirtualHost *:80>
ProxyPreserveHost On
ProxyPass / http://192.168.0.5/
ProxyPassReverse / http://192.168.0.5/
ServerName redacted
ServerAlias redacted
RewriteEngine on
RewriteCond %{SERVER_NAME} =redacted
RewriteRule ^ https://%{SERVER_NAME}%{REQUEST_URI} [END,NE,R=permanent]
</VirtualHost>- backup LXD container into an image, export image to store externally
sudo lxc snapshot <container_name> tmpsnap
echo "Create image"
sudo lxc publish <container_name>/tmpsnap --alias tmpimg
echo "Exporting image"
sudo lxc image export tmpimg /media/storage1/backup/<backup-name>
sudo lxc delete <container_name>/tmpsnap
sudo lxc image delete tmpimg- update Gitlab to (12.7.5-ee.0) or run reconfigure and see ERR_TOO_MANY_REDIRECTS
- delete lxd container, restore from backup
lxc image import /media/storage1/backup/long-gitlab.tar.gz --alias gitlab-back
sudo lxc launch gitlab-back gitlab- now we have it running, but throwing 503 on each attempt to access projects or issues
- if we run update or reconfigure now, we'll be back to ERR_TOO_MANY_REDIRECTS issue
What is the current bug behavior?
Gitlab throws ERR_TOO_MANY_REDIRECTS after update. Gitlab throws 503 on trying to access projects/issues/etc when I restore it from backup (which was created while Gitlab was working perfectly fine, nothing had changed in system since backup was created.
Relevant logs and/or screenshots
Expand for output from tail
==> /var/log/gitlab/postgres-exporter/state <====> /var/log/gitlab/postgres-exporter/current <== 2020-02-06_10:22:29.80813 time="2020-02-06T10:22:29Z" level=error msg="Error opening connection to database (user=gitlab-psql%20host=/var/opt/gitlab/postgresql%20database=postgres): dial unix /var/opt/gitlab/postgresql/.s.PGSQL.5432: connect: no such file or directory" source="postgres_exporter.go:1336" 2020-02-06_10:22:29.80815 time="2020-02-06T10:22:29Z" level=info msg="Starting Server: localhost:9187" source="postgres_exporter.go:1437" 2020-02-06_10:22:50.34338 time="2020-02-06T10:22:50Z" level=info msg="Established new database connection to "/var/opt/gitlab/postgresql:5432"." source="postgres_exporter.go:777" 2020-02-06_10:22:50.36501 time="2020-02-06T10:22:50Z" level=info msg="Semantic Version Changed on "/var/opt/gitlab/postgresql:5432": 0.0.0 -> 10.9.0" source="postgres_exporter.go:1229" 2020-02-06_10:31:27.95990 time="2020-02-06T10:31:27Z" level=info msg="Established new database connection to "/var/opt/gitlab/postgresql:5432"." source="postgres_exporter.go:777" 2020-02-06_10:31:27.97250 time="2020-02-06T10:31:27Z" level=info msg="Semantic Version Changed on "/var/opt/gitlab/postgresql:5432": 0.0.0 -> 10.9.0" source="postgres_exporter.go:1229" 2020-02-06_10:31:28.03414 time="2020-02-06T10:31:28Z" level=info msg="Starting Server: localhost:9187" source="postgres_exporter.go:1437" 2020-02-06_10:49:25.59505 time="2020-02-06T10:49:25Z" level=info msg="Established new database connection to "/var/opt/gitlab/postgresql:5432"." source="postgres_exporter.go:777" 2020-02-06_10:49:25.59982 time="2020-02-06T10:49:25Z" level=info msg="Semantic Version Changed on "/var/opt/gitlab/postgresql:5432": 0.0.0 -> 10.9.0" source="postgres_exporter.go:1229" 2020-02-06_10:49:25.64626 time="2020-02-06T10:49:25Z" level=info msg="Starting Server: localhost:9187" source="postgres_exporter.go:1437"
==> /var/log/gitlab/sidekiq/current <== {"severity":"INFO","time":"2020-02-06T18:00:12.186Z","queue":"cronjob:stuck_ci_jobs","class":"StuckCiJobsWorker","retry":false,"queue_namespace":"cronjob","jid":"ae4bd1179eca415cc98873f7","created_at":"2020-02-06T18:00:12.184Z","correlation_id":"a221228e26ce78101ded8b3955dfa5b0","enqueued_at":"2020-02-06T18:00:12.184Z","pid":26165,"message":"StuckCiJobsWorker JID-ae4bd1179eca415cc98873f7: start","job_status":"start","scheduling_latency_s":0.00209} {"severity":"INFO","time":"2020-02-06T18:00:12.201Z","queue":"cronjob:stuck_merge_jobs","class":"StuckMergeJobsWorker","retry":false,"queue_namespace":"cronjob","jid":"912a8b6d17391190e67207b2","created_at":"2020-02-06T18:00:12.191Z","correlation_id":"4659987889ccf907a2ac2acecd46164f","enqueued_at":"2020-02-06T18:00:12.191Z","pid":26165,"message":"StuckMergeJobsWorker JID-912a8b6d17391190e67207b2: start","job_status":"start","scheduling_latency_s":0.00999} {"severity":"INFO","time":"2020-02-06T18:00:12.205Z","queue":"cronjob:geo_sidekiq_cron_config","class":"Geo::SidekiqCronConfigWorker","retry":false,"queue_namespace":"cronjob","jid":"88c4f8c0a9171e67dc1698d8","created_at":"2020-02-06T18:00:12.200Z","correlation_id":"da60912be4ff6996ece15c4fc2cfbb0e","enqueued_at":"2020-02-06T18:00:12.200Z","pid":26165,"message":"Geo::SidekiqCronConfigWorker JID-88c4f8c0a9171e67dc1698d8: start","job_status":"start","scheduling_latency_s":0.004625} {"severity":"INFO","time":"2020-02-06T18:00:12.218Z","queue":"cronjob:update_all_mirrors","class":"UpdateAllMirrorsWorker","retry":false,"queue_namespace":"cronjob","jid":"3195aa5c9d50ce03991e2b5c","created_at":"2020-02-06T18:00:12.127Z","correlation_id":"c85adc175675227a11e37ae772858f38","enqueued_at":"2020-02-06T18:00:12.129Z","pid":26165,"message":"UpdateAllMirrorsWorker JID-3195aa5c9d50ce03991e2b5c: done: 0.088389 sec","job_status":"done","scheduling_latency_s":0.000526,"duration":0.088389,"cpu_s":0.014986,"completed_at":"2020-02-06T18:00:12.218Z","db_duration":37.815504023456015,"db_duration_s":0.037815504023456015} {"severity":"INFO","time":"2020-02-06T18:00:12.227Z","queue":"cronjob:prune_web_hook_logs","class":"PruneWebHookLogsWorker","retry":false,"queue_namespace":"cronjob","jid":"d349e008d0db801520e505da","created_at":"2020-02-06T18:00:12.149Z","correlation_id":"5f22f11ce5805a40d6ddc477312b6e4b","enqueued_at":"2020-02-06T18:00:12.149Z","pid":26165,"message":"PruneWebHookLogsWorker JID-d349e008d0db801520e505da: done: 0.075594 sec","job_status":"done","scheduling_latency_s":0.00291,"duration":0.075594,"cpu_s":0.005269,"completed_at":"2020-02-06T18:00:12.227Z","db_duration":35.93571799865458,"db_duration_s":0.03593571799865458} {"severity":"INFO","time":"2020-02-06T18:00:12.235Z","queue":"cronjob:import_export_project_cleanup","class":"ImportExportProjectCleanupWorker","retry":false,"queue_namespace":"cronjob","jid":"ea052c223370e06de56c86f6","created_at":"2020-02-06T18:00:12.171Z","correlation_id":"6fe5a9b1eab6b800b770600bdd92a188","enqueued_at":"2020-02-06T18:00:12.172Z","pid":26165,"message":"ImportExportProjectCleanupWorker JID-ea052c223370e06de56c86f6: done: 0.060982 sec","job_status":"done","scheduling_latency_s":0.002476,"duration":0.060982,"cpu_s":0.005682,"completed_at":"2020-02-06T18:00:12.235Z","db_duration":20.278171985410154,"db_duration_s":0.020278171985410154} {"severity":"INFO","time":"2020-02-06T18:00:12.238Z","queue":"cronjob:pages_domain_ssl_renewal_cron","class":"PagesDomainSslRenewalCronWorker","retry":false,"queue_namespace":"cronjob","jid":"66f242b128b2ab4b1e5cf2f0","created_at":"2020-02-06T18:00:12.177Z","correlation_id":"4387775794c398ba4f391138dd7360cf","enqueued_at":"2020-02-06T18:00:12.179Z","pid":26165,"message":"PagesDomainSslRenewalCronWorker JID-66f242b128b2ab4b1e5cf2f0: done: 0.058483 sec","job_status":"done","scheduling_latency_s":0.000496,"duration":0.058483,"cpu_s":0.014642,"completed_at":"2020-02-06T18:00:12.238Z","db_duration":20.45236699632369,"db_duration_s":0.02045236699632369} {"severity":"INFO","time":"2020-02-06T18:00:12.254Z","queue":"cronjob:stuck_merge_jobs","class":"StuckMergeJobsWorker","retry":false,"queue_namespace":"cronjob","jid":"912a8b6d17391190e67207b2","created_at":"2020-02-06T18:00:12.191Z","correlation_id":"4659987889ccf907a2ac2acecd46164f","enqueued_at":"2020-02-06T18:00:12.191Z","pid":26165,"message":"StuckMergeJobsWorker JID-912a8b6d17391190e67207b2: done: 0.052623 sec","job_status":"done","scheduling_latency_s":0.00999,"duration":0.052623,"cpu_s":0.006036,"completed_at":"2020-02-06T18:00:12.254Z","db_duration":24.748683004872873,"db_duration_s":0.024748683004872873} {"severity":"INFO","time":"2020-02-06T18:00:12.262Z","queue":"cronjob:stuck_ci_jobs","class":"StuckCiJobsWorker","retry":false,"queue_namespace":"cronjob","jid":"ae4bd1179eca415cc98873f7","created_at":"2020-02-06T18:00:12.184Z","correlation_id":"a221228e26ce78101ded8b3955dfa5b0","enqueued_at":"2020-02-06T18:00:12.184Z","pid":26165,"message":"StuckCiJobsWorker JID-ae4bd1179eca415cc98873f7: done: 0.075314 sec","job_status":"done","scheduling_latency_s":0.00209,"duration":0.075314,"cpu_s":0.009185,"completed_at":"2020-02-06T18:00:12.262Z","db_duration":33.95069499674719,"db_duration_s":0.03395069499674719} {"severity":"INFO","time":"2020-02-06T18:00:12.264Z","queue":"cronjob:geo_sidekiq_cron_config","class":"Geo::SidekiqCronConfigWorker","retry":false,"queue_namespace":"cronjob","jid":"88c4f8c0a9171e67dc1698d8","created_at":"2020-02-06T18:00:12.200Z","correlation_id":"da60912be4ff6996ece15c4fc2cfbb0e","enqueued_at":"2020-02-06T18:00:12.200Z","pid":26165,"message":"Geo::SidekiqCronConfigWorker JID-88c4f8c0a9171e67dc1698d8: done: 0.059689 sec","job_status":"done","scheduling_latency_s":0.004625,"duration":0.059689,"cpu_s":0.015371,"completed_at":"2020-02-06T18:00:12.264Z","db_duration":25.947274989448488,"db_duration_s":0.025947274989448488}
==> /var/log/gitlab/sidekiq/state <==
==> /var/log/gitlab/grafana/current <== 2020-02-06_10:49:23.40248 t=2020-02-06T10:49:23+0000 lvl=info msg="Initializing SearchService" logger=server 2020-02-06_10:49:23.40249 t=2020-02-06T10:49:23+0000 lvl=info msg="Initializing ServerLockService" logger=server 2020-02-06_10:49:23.40250 t=2020-02-06T10:49:23+0000 lvl=info msg="Initializing TracingService" logger=server 2020-02-06_10:49:23.40252 t=2020-02-06T10:49:23+0000 lvl=info msg="Initializing UsageStatsService" logger=server 2020-02-06_10:49:23.40255 t=2020-02-06T10:49:23+0000 lvl=info msg="Initializing UserAuthTokenService" logger=server 2020-02-06_10:49:23.40259 t=2020-02-06T10:49:23+0000 lvl=info msg="Initializing CleanUpService" logger=server 2020-02-06_10:49:23.40260 t=2020-02-06T10:49:23+0000 lvl=info msg="Initializing NotificationService" logger=server 2020-02-06_10:49:23.40897 t=2020-02-06T10:49:23+0000 lvl=info msg="Initializing provisioningServiceImpl" logger=server 2020-02-06_10:49:23.46615 t=2020-02-06T10:49:23+0000 lvl=info msg="Initializing Stream Manager" 2020-02-06_10:49:23.46624 t=2020-02-06T10:49:23+0000 lvl=info msg="HTTP Server Listen" logger=http.server address=localhost:3000 protocol=http subUrl=/-/grafana socket=
==> /var/log/gitlab/grafana/state <==
==> /var/log/gitlab/gitlab-exporter/state <==
==> /var/log/gitlab/gitlab-exporter/current <== 2020-02-06_10:48:18.79932 - Goodbye! 2020-02-06_10:48:18.79945 == Sinatra has ended his set (crowd applauds) 2020-02-06_10:49:23.36025 == Sinatra (v2.0.8.1) has taken the stage on 9168 for development with backup from Puma 2020-02-06_10:49:23.36029 Puma starting in single mode... 2020-02-06_10:49:23.36032 * Version 4.3.1.gitlab.2 (ruby 2.6.5-p114), codename: Mysterious Traveller 2020-02-06_10:49:23.36034 * Min threads: 0, max threads: 16 2020-02-06_10:49:23.36035 * Environment: development 2020-02-06_10:49:23.36036 * Listening on tcp://127.0.0.1:9168 2020-02-06_10:49:23.36038 * Listening on tcp://[::1]:9168 2020-02-06_10:49:23.36039 Use Ctrl-C to stop
==> /var/log/gitlab/nginx/current <==
==> /var/log/gitlab/nginx/gitlab_access.log <== 192.168.0.3 - - [06/Feb/2020:17:46:01 +0000] "GET / HTTP/1.1" 301 162 "" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/80.0.3987.87 Safari/537.36" 192.168.0.3 - - [06/Feb/2020:17:46:01 +0000] "GET / HTTP/1.1" 301 162 "" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/80.0.3987.87 Safari/537.36" 192.168.0.3 - - [06/Feb/2020:17:46:01 +0000] "GET / HTTP/1.1" 301 162 "" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/80.0.3987.87 Safari/537.36" 192.168.0.3 - - [06/Feb/2020:17:46:01 +0000] "GET / HTTP/1.1" 301 162 "" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/80.0.3987.87 Safari/537.36" 192.168.0.3 - - [06/Feb/2020:17:46:01 +0000] "GET / HTTP/1.1" 301 162 "" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/80.0.3987.87 Safari/537.36" 192.168.0.3 - - [06/Feb/2020:17:46:01 +0000] "GET / HTTP/1.1" 301 162 "" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/80.0.3987.87 Safari/537.36" 192.168.0.3 - - [06/Feb/2020:17:46:01 +0000] "GET / HTTP/1.1" 301 162 "" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/80.0.3987.87 Safari/537.36" 192.168.0.3 - - [06/Feb/2020:17:46:01 +0000] "GET / HTTP/1.1" 301 162 "" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/80.0.3987.87 Safari/537.36" 192.168.0.3 - - [06/Feb/2020:17:46:02 +0000] "GET / HTTP/1.1" 301 162 "" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/80.0.3987.87 Safari/537.36" 192.168.0.3 - - [06/Feb/2020:17:46:02 +0000] "GET / HTTP/1.1" 301 162 "" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/80.0.3987.87 Safari/537.36"
==> /var/log/gitlab/nginx/error.log <==
==> /var/log/gitlab/nginx/access.log <==
==> /var/log/gitlab/nginx/gitlab_error.log <==
==> /var/log/gitlab/nginx/gitlab_registry_access.log <==
==> /var/log/gitlab/nginx/gitlab_registry_error.log <==
==> /var/log/gitlab/node-exporter/current <== 2020-02-06_10:49:25.17486 time="2020-02-06T10:49:25Z" level=info msg=" - sockstat" source="node_exporter.go:104" 2020-02-06_10:49:25.17488 time="2020-02-06T10:49:25Z" level=info msg=" - stat" source="node_exporter.go:104" 2020-02-06_10:49:25.17489 time="2020-02-06T10:49:25Z" level=info msg=" - textfile" source="node_exporter.go:104" 2020-02-06_10:49:25.17491 time="2020-02-06T10:49:25Z" level=info msg=" - time" source="node_exporter.go:104" 2020-02-06_10:49:25.17493 time="2020-02-06T10:49:25Z" level=info msg=" - timex" source="node_exporter.go:104" 2020-02-06_10:49:25.17494 time="2020-02-06T10:49:25Z" level=info msg=" - uname" source="node_exporter.go:104" 2020-02-06_10:49:25.17496 time="2020-02-06T10:49:25Z" level=info msg=" - vmstat" source="node_exporter.go:104" 2020-02-06_10:49:25.17497 time="2020-02-06T10:49:25Z" level=info msg=" - xfs" source="node_exporter.go:104" 2020-02-06_10:49:25.17500 time="2020-02-06T10:49:25Z" level=info msg=" - zfs" source="node_exporter.go:104" 2020-02-06_10:49:25.17508 time="2020-02-06T10:49:25Z" level=info msg="Listening on localhost:9100" source="node_exporter.go:170" ==> /var/log/gitlab/node-exporter/state <==
==> /var/log/gitlab/redis/state <==
==> /var/log/gitlab/redis/current <== 2020-02-06_17:52:57.03399 12284:M 06 Feb 17:52:57.033 * 10 changes in 300 seconds. Saving... 2020-02-06_17:52:57.03464 12284:M 06 Feb 17:52:57.034 * Background saving started by pid 12434 2020-02-06_17:52:57.05300 12434:C 06 Feb 17:52:57.052 * DB saved on disk 2020-02-06_17:52:57.05404 12434:C 06 Feb 17:52:57.053 * RDB: 0 MB of memory used by copy-on-write 2020-02-06_17:52:57.13528 12284:M 06 Feb 17:52:57.135 * Background saving terminated with success 2020-02-06_17:57:58.00854 12284:M 06 Feb 17:57:58.008 * 10 changes in 300 seconds. Saving... 2020-02-06_17:57:58.00938 12284:M 06 Feb 17:57:58.009 * Background saving started by pid 13015 2020-02-06_17:57:58.02482 13015:C 06 Feb 17:57:58.024 * DB saved on disk 2020-02-06_17:57:58.02546 13015:C 06 Feb 17:57:58.025 * RDB: 0 MB of memory used by copy-on-write 2020-02-06_17:57:58.11108 12284:M 06 Feb 17:57:58.110 * Background saving terminated with success
==> /var/log/gitlab/prometheus/current <== 2020-02-06_13:00:00.76399 level=info ts=2020-02-06T13:00:00.763Z caller=compact.go:495 component=tsdb msg="write block" mint=1580983200000 maxt=1580990400000 ulid=01E0D9BXRQVV8ZSHGGHA9YMQJD duration=100.753839ms 2020-02-06_13:00:00.79236 level=info ts=2020-02-06T13:00:00.792Z caller=head.go:593 component=tsdb msg="head GC completed" duration=7.008242ms 2020-02-06_13:00:00.84154 level=info ts=2020-02-06T13:00:00.841Z caller=head.go:663 component=tsdb msg="WAL checkpoint complete" first=1462 last=1463 duration=49.151024ms 2020-02-06_15:00:00.79758 level=info ts=2020-02-06T15:00:00.797Z caller=compact.go:495 component=tsdb msg="write block" mint=1580990400000 maxt=1580997600000 ulid=01E0DG7N0Q65QKNWKMEH1PVCBP duration=134.328205ms 2020-02-06_15:00:00.82753 level=info ts=2020-02-06T15:00:00.827Z caller=head.go:593 component=tsdb msg="head GC completed" duration=6.975482ms 2020-02-06_15:00:00.95414 level=info ts=2020-02-06T15:00:00.954Z caller=compact.go:440 component=tsdb msg="compact blocks" count=2 mint=1580493600000 maxt=1580508000000 ulid=01E0DG7N6GB328TZZYM5AZBE3M sources="[01DZYPEGRQX7KWKHT4KJ51SCHA 01E0D0BSXGHYR5MRR70R82EM8C]" duration=105.727447ms 2020-02-06_15:00:01.15903 level=info ts=2020-02-06T15:00:01.158Z caller=compact.go:440 component=tsdb msg="compact blocks" count=2 mint=1580472000000 maxt=1580508000000 ulid=01E0DG7NB2KG7ZHFAG4HAFV0H6 sources="[01DZYPEGWPHCA8BH7D7GM5KK2F 01E0DG7N6GB328TZZYM5AZBE3M]" duration=164.026055ms 2020-02-06_17:00:00.77022 level=info ts=2020-02-06T17:00:00.770Z caller=compact.go:495 component=tsdb msg="write block" mint=1580997600000 maxt=1581004800000 ulid=01E0DQ3C8QWZ6HEA5HQ3E4656F duration=106.894492ms 2020-02-06_17:00:00.79499 level=info ts=2020-02-06T17:00:00.794Z caller=head.go:593 component=tsdb msg="head GC completed" duration=5.88491ms 2020-02-06_17:00:00.82512 level=info ts=2020-02-06T17:00:00.825Z caller=head.go:663 component=tsdb msg="WAL checkpoint complete" first=1464 last=1465 duration=30.098635ms
==> /var/log/gitlab/prometheus/state <==
==> /var/log/gitlab/logrotate/state <==
==> /var/log/gitlab/logrotate/current <== 2020-02-06_10:31:26.66160 Received TERM from runit, sending to process group (-PID) 2020-02-06_10:48:20.07879 Received TERM from runit, sending to process group (-PID)
==> /var/log/gitlab/postgresql/state <==
==> /var/log/gitlab/postgresql/current <== 2020-02-06_10:31:28.36966 LOG: aborting any active transactions 2020-02-06_10:31:28.37306 FATAL: terminating connection due to administrator command 2020-02-06_10:31:28.37316 FATAL: terminating connection due to administrator command 2020-02-06_10:31:28.37703 LOG: worker process: logical replication launcher (PID 579) exited with exit code 1 2020-02-06_10:31:28.37742 FATAL: terminating connection due to administrator command 2020-02-06_10:31:28.38760 LOG: shutting down 2020-02-06_10:31:28.44203 LOG: database system is shut down 2020-02-06_10:31:28.47167 LOG: listening on Unix socket "/var/opt/gitlab/postgresql/.s.PGSQL.5432" 2020-02-06_10:31:28.56292 LOG: database system was shut down at 2020-02-06 10:31:28 GMT 2020-02-06_10:31:28.56790 LOG: database system is ready to accept connections
==> /var/log/gitlab/unicorn/state <==
==> /var/log/gitlab/unicorn/unicorn_stdout.log <==
==> /var/log/gitlab/unicorn/unicorn_stderr.log <== I, [2020-02-06T10:50:10.041338 #26200] INFO -- : unlinking existing socket=/var/opt/gitlab/gitlab-rails/sockets/gitlab.socket I, [2020-02-06T10:50:10.041496 #26200] INFO -- : listening on addr=/var/opt/gitlab/gitlab-rails/sockets/gitlab.socket fd=15 I, [2020-02-06T10:50:10.082072 #27376] INFO -- : worker=0 ready I, [2020-02-06T10:50:10.092036 #27378] INFO -- : worker=1 ready I, [2020-02-06T10:50:10.102539 #27380] INFO -- : worker=2 ready I, [2020-02-06T10:50:10.112842 #27382] INFO -- : worker=3 ready I, [2020-02-06T10:50:10.123682 #27384] INFO -- : worker=4 ready I, [2020-02-06T10:50:10.134878 #27386] INFO -- : worker=5 ready I, [2020-02-06T10:50:10.143259 #26200] INFO -- : master process ready I, [2020-02-06T10:50:10.145780 #27388] INFO -- : worker=6 ready ==> /var/log/gitlab/unicorn/current <== 2020-02-06_10:32:20.39397 adopted new unicorn master 12349 2020-02-06_10:48:26.00924 Received TERM from runit, sending to process group (-PID) 2020-02-06_10:48:26.01327 Terminated 2020-02-06_10:48:26.01338 forwarding TERM to unicorn master 12349 2020-02-06_10:48:26.01609 Waiting for unicorn to die... 2020-02-06_10:48:27.01964 forwarding CONT to unicorn master 12349 2020-02-06_10:48:27.01974 /opt/gitlab/embedded/bin/gitlab-unicorn-wrapper: line 132: kill: (12349) - No such process 2020-02-06_10:48:27.01995 wrapper for unicorn master 12349 exiting 2020-02-06_10:49:27.30167 starting new unicorn master 2020-02-06_10:50:11.16368 adopted new unicorn master 26200
==> /var/log/gitlab/gitlab-monitor/state <==
==> /var/log/gitlab/gitlab-monitor/current <== 2019-12-25_20:55:55.85467 Puma starting in single mode... 2019-12-25_20:55:55.85472 * Version 3.12.0 (ruby 2.6.3-p62), codename: Llamas in Pajamas 2019-12-25_20:55:55.85477 * Min threads: 0, max threads: 16 2019-12-25_20:55:55.85482 * Environment: development 2019-12-25_20:55:55.85495 * Listening on tcp://localhost:9168 2019-12-25_20:55:55.85541 Use Ctrl-C to stop 2019-12-25_21:00:28.26891 - Gracefully stopping, waiting for requests to finish 2019-12-25_21:00:28.27030 === puma shutdown: 2019-12-25 21:00:28 +0000 === 2019-12-25_21:00:28.27040 - Goodbye! 2019-12-25_21:00:28.27047 == Sinatra has ended his set (crowd applauds)
==> /var/log/gitlab/gitaly/state <== ==> /var/log/gitlab/gitaly/gitlab-shell.log <== {"method":"POST","url":"http://127.0.0.1:8080/api/v4/internal/post_receive","duration":0.064269214,"gitaly_embedded":true,"pid":29560,"level":"info","msg":"finished HTTP request","time":"2020-01-16T16:06:47+00:00"} {"method":"POST","url":"http://127.0.0.1:8080/api/v4/internal/allowed","duration":0.084610053,"gitaly_embedded":true,"pid":7056,"level":"info","msg":"finished HTTP request","time":"2020-01-19T13:23:33+00:00"} {"method":"POST","url":"http://127.0.0.1:8080/api/v4/internal/pre_receive","duration":0.021827921,"gitaly_embedded":true,"pid":7056,"level":"info","msg":"finished HTTP request","time":"2020-01-19T13:23:34+00:00"} {"method":"POST","url":"http://127.0.0.1:8080/api/v4/internal/post_receive","duration":0.033410076,"gitaly_embedded":true,"pid":7083,"level":"info","msg":"finished HTTP request","time":"2020-01-19T13:23:34+00:00"} {"method":"POST","url":"http://127.0.0.1:8080/api/v4/internal/allowed","duration":0.048128896,"gitaly_embedded":true,"pid":8339,"level":"info","msg":"finished HTTP request","time":"2020-01-19T17:45:21+00:00"} {"method":"POST","url":"http://127.0.0.1:8080/api/v4/internal/pre_receive","duration":0.013079952,"gitaly_embedded":true,"pid":8339,"level":"info","msg":"finished HTTP request","time":"2020-01-19T17:45:21+00:00"} {"method":"POST","url":"http://127.0.0.1:8080/api/v4/internal/post_receive","duration":0.02448864,"gitaly_embedded":true,"pid":8363,"level":"info","msg":"finished HTTP request","time":"2020-01-19T17:45:21+00:00"} {"method":"POST","url":"http://127.0.0.1:8080/api/v4/internal/allowed","duration":0.110594309,"gitaly_embedded":true,"pid":30989,"level":"info","msg":"finished HTTP request","time":"2020-01-26T15:17:55+00:00"} {"method":"POST","url":"http://127.0.0.1:8080/api/v4/internal/pre_receive","duration":0.033015992,"gitaly_embedded":true,"pid":30989,"level":"info","msg":"finished HTTP request","time":"2020-01-26T15:17:55+00:00"} {"method":"POST","url":"http://127.0.0.1:8080/api/v4/internal/post_receive","duration":0.038900104,"gitaly_embedded":true,"pid":31021,"level":"info","msg":"finished HTTP request","time":"2020-01-26T15:17:55+00:00"}
==> /var/log/gitlab/gitaly/gitaly_hooks.log <==
==> /var/log/gitlab/gitaly/current <== {"gitaly":12055,"level":"warning","msg":"forwarding signal","signal":17,"time":"2020-02-06T10:49:22Z","wrapper":12048} {"error":"os: process already finished","gitaly":12055,"level":"error","msg":"can't forward the signal","signal":17,"time":"2020-02-06T10:49:22Z","wrapper":12048} {"level":"info","msg":"PID 25962 BUNDLE_GEMFILE=/opt/gitlab/embedded/service/gitaly-ruby/Gemfile","supervisor.args":["bundle","exec","bin/ruby-cd","/var/opt/gitlab/gitaly","/opt/gitlab/embedded/service/gitaly-ruby/bin/gitaly-ruby","25936","/var/opt/gitlab/gitaly/internal_sockets/ruby.1"],"supervisor.name":"gitaly-ruby.1","time":"2020-02-06T10:49:22Z"} {"level":"info","msg":"PID 25961 BUNDLE_GEMFILE=/opt/gitlab/embedded/service/gitaly-ruby/Gemfile","supervisor.args":["bundle","exec","bin/ruby-cd","/var/opt/gitlab/gitaly","/opt/gitlab/embedded/service/gitaly-ruby/bin/gitaly-ruby","25936","/var/opt/gitlab/gitaly/internal_sockets/ruby.0"],"supervisor.name":"gitaly-ruby.0","time":"2020-02-06T10:49:23Z"} {"gitaly":12055,"level":"error","msg":"wrapper for gitaly shutting down","time":"2020-02-06T10:49:23Z","wrapper":12048} {"level":"info","msg":"Wrapper started","time":"2020-02-06T10:49:23Z","wrapper":25972} {"level":"info","msg":"finding gitaly","pid_file":"/var/opt/gitlab/gitaly/gitaly.pid","time":"2020-02-06T10:49:23Z","wrapper":25972} {"level":"info","msg":"adopting a process","time":"2020-02-06T10:49:23Z","wrapper":25972} {"gitaly":25936,"level":"info","msg":"monitoring gitaly","time":"2020-02-06T10:49:23Z","wrapper":25972} {"grpc.code":"OK","grpc.meta.auth_version":"v2","grpc.meta.client_name":"gitlab-web","grpc.method":"Check","grpc.request.deadline":"2020-02-06T11:01:22Z","grpc.request.fullMethod":"/grpc.health.v1.Health/Check","grpc.service":"grpc.health.v1.Health","grpc.start_time":"2020-02-06T11:01:12Z","grpc.time_ms":0.542,"level":"info","msg":"finished unary call with code OK","peer.address":"@","pid":25936,"span.kind":"server","system":"grpc","time":"2020-02-06T11:01:12Z"}
==> /var/log/gitlab/gitlab-rails/gitlab-rails-db-migrate-2018-04-23-22-08-35.log <==
==> /var/log/gitlab/gitlab-rails/exceptions_json.log <==
==> /var/log/gitlab/gitlab-rails/importer.log <==
==> /var/log/gitlab/gitlab-rails/audit_json.log <== ==> /var/log/gitlab/gitlab-rails/gitlab-rails-db-migrate-2018-08-15-14-36-24.log <==
==> /var/log/gitlab/gitlab-rails/gitlab-rails-db-migrate-2017-12-10-16-47-25.log <==
==> /var/log/gitlab/gitlab-rails/gitlab-rails-db-migrate-2019-06-19-23-44-28.log <==
==> /var/log/gitlab/gitlab-rails/gitlab-rails-db-migrate-2018-03-17-11-20-28.log <==
==> /var/log/gitlab/gitlab-rails/gitlab-rails-db-migrate-2018-07-25-19-10-08.log <==
==> /var/log/gitlab/gitlab-rails/gitlab-rails-db-migrate-2019-05-08-23-26-36.log <==
==> /var/log/gitlab/gitlab-rails/gitlab-rails-db-migrate-2018-06-12-19-31-52.log <==
==> /var/log/gitlab/gitlab-rails/sidekiq.log <== {"severity":"INFO","time":"2020-02-06T18:00:12.186Z","queue":"cronjob:stuck_ci_jobs","class":"StuckCiJobsWorker","retry":false,"queue_namespace":"cronjob","jid":"ae4bd1179eca415cc98873f7","created_at":"2020-02-06T18:00:12.184Z","correlation_id":"a221228e26ce78101ded8b3955dfa5b0","enqueued_at":"2020-02-06T18:00:12.184Z","pid":26165,"message":"StuckCiJobsWorker JID-ae4bd1179eca415cc98873f7: start","job_status":"start","scheduling_latency_s":0.00209} {"severity":"INFO","time":"2020-02-06T18:00:12.201Z","queue":"cronjob:stuck_merge_jobs","class":"StuckMergeJobsWorker","retry":false,"queue_namespace":"cronjob","jid":"912a8b6d17391190e67207b2","created_at":"2020-02-06T18:00:12.191Z","correlation_id":"4659987889ccf907a2ac2acecd46164f","enqueued_at":"2020-02-06T18:00:12.191Z","pid":26165,"message":"StuckMergeJobsWorker JID-912a8b6d17391190e67207b2: start","job_status":"start","scheduling_latency_s":0.00999} {"severity":"INFO","time":"2020-02-06T18:00:12.205Z","queue":"cronjob:geo_sidekiq_cron_config","class":"Geo::SidekiqCronConfigWorker","retry":false,"queue_namespace":"cronjob","jid":"88c4f8c0a9171e67dc1698d8","created_at":"2020-02-06T18:00:12.200Z","correlation_id":"da60912be4ff6996ece15c4fc2cfbb0e","enqueued_at":"2020-02-06T18:00:12.200Z","pid":26165,"message":"Geo::SidekiqCronConfigWorker JID-88c4f8c0a9171e67dc1698d8: start","job_status":"start","scheduling_latency_s":0.004625} {"severity":"INFO","time":"2020-02-06T18:00:12.218Z","queue":"cronjob:update_all_mirrors","class":"UpdateAllMirrorsWorker","retry":false,"queue_namespace":"cronjob","jid":"3195aa5c9d50ce03991e2b5c","created_at":"2020-02-06T18:00:12.127Z","correlation_id":"c85adc175675227a11e37ae772858f38","enqueued_at":"2020-02-06T18:00:12.129Z","pid":26165,"message":"UpdateAllMirrorsWorker JID-3195aa5c9d50ce03991e2b5c: done: 0.088389 sec","job_status":"done","scheduling_latency_s":0.000526,"duration":0.088389,"cpu_s":0.014986,"completed_at":"2020-02-06T18:00:12.218Z","db_duration":37.815504023456015,"db_duration_s":0.037815504023456015} {"severity":"INFO","time":"2020-02-06T18:00:12.227Z","queue":"cronjob:prune_web_hook_logs","class":"PruneWebHookLogsWorker","retry":false,"queue_namespace":"cronjob","jid":"d349e008d0db801520e505da","created_at":"2020-02-06T18:00:12.149Z","correlation_id":"5f22f11ce5805a40d6ddc477312b6e4b","enqueued_at":"2020-02-06T18:00:12.149Z","pid":26165,"message":"PruneWebHookLogsWorker JID-d349e008d0db801520e505da: done: 0.075594 sec","job_status":"done","scheduling_latency_s":0.00291,"duration":0.075594,"cpu_s":0.005269,"completed_at":"2020-02-06T18:00:12.227Z","db_duration":35.93571799865458,"db_duration_s":0.03593571799865458} {"severity":"INFO","time":"2020-02-06T18:00:12.235Z","queue":"cronjob:import_export_project_cleanup","class":"ImportExportProjectCleanupWorker","retry":false,"queue_namespace":"cronjob","jid":"ea052c223370e06de56c86f6","created_at":"2020-02-06T18:00:12.171Z","correlation_id":"6fe5a9b1eab6b800b770600bdd92a188","enqueued_at":"2020-02-06T18:00:12.172Z","pid":26165,"message":"ImportExportProjectCleanupWorker JID-ea052c223370e06de56c86f6: done: 0.060982 sec","job_status":"done","scheduling_latency_s":0.002476,"duration":0.060982,"cpu_s":0.005682,"completed_at":"2020-02-06T18:00:12.235Z","db_duration":20.278171985410154,"db_duration_s":0.020278171985410154} {"severity":"INFO","time":"2020-02-06T18:00:12.238Z","queue":"cronjob:pages_domain_ssl_renewal_cron","class":"PagesDomainSslRenewalCronWorker","retry":false,"queue_namespace":"cronjob","jid":"66f242b128b2ab4b1e5cf2f0","created_at":"2020-02-06T18:00:12.177Z","correlation_id":"4387775794c398ba4f391138dd7360cf","enqueued_at":"2020-02-06T18:00:12.179Z","pid":26165,"message":"PagesDomainSslRenewalCronWorker JID-66f242b128b2ab4b1e5cf2f0: done: 0.058483 sec","job_status":"done","scheduling_latency_s":0.000496,"duration":0.058483,"cpu_s":0.014642,"completed_at":"2020-02-06T18:00:12.238Z","db_duration":20.45236699632369,"db_duration_s":0.02045236699632369} {"severity":"INFO","time":"2020-02-06T18:00:12.254Z","queue":"cronjob:stuck_merge_jobs","class":"StuckMergeJobsWorker","retry":false,"queue_namespace":"cronjob","jid":"912a8b6d17391190e67207b2","created_at":"2020-02-06T18:00:12.191Z","correlation_id":"4659987889ccf907a2ac2acecd46164f","enqueued_at":"2020-02-06T18:00:12.191Z","pid":26165,"message":"StuckMergeJobsWorker JID-912a8b6d17391190e67207b2: done: 0.052623 sec","job_status":"done","scheduling_latency_s":0.00999,"duration":0.052623,"cpu_s":0.006036,"completed_at":"2020-02-06T18:00:12.254Z","db_duration":24.748683004872873,"db_duration_s":0.024748683004872873} {"severity":"INFO","time":"2020-02-06T18:00:12.262Z","queue":"cronjob:stuck_ci_jobs","class":"StuckCiJobsWorker","retry":false,"queue_namespace":"cronjob","jid":"ae4bd1179eca415cc98873f7","created_at":"2020-02-06T18:00:12.184Z","correlation_id":"a221228e26ce78101ded8b3955dfa5b0","enqueued_at":"2020-02-06T18:00:12.184Z","pid":26165,"message":"StuckCiJobsWorker JID-ae4bd1179eca415cc98873f7: done: 0.075314 sec","job_status":"done","scheduling_latency_s":0.00209,"duration":0.075314,"cpu_s":0.009185,"completed_at":"2020-02-06T18:00:12.262Z","db_duration":33.95069499674719,"db_duration_s":0.03395069499674719} {"severity":"INFO","time":"2020-02-06T18:00:12.264Z","queue":"cronjob:geo_sidekiq_cron_config","class":"Geo::SidekiqCronConfigWorker","retry":false,"queue_namespace":"cronjob","jid":"88c4f8c0a9171e67dc1698d8","created_at":"2020-02-06T18:00:12.200Z","correlation_id":"da60912be4ff6996ece15c4fc2cfbb0e","enqueued_at":"2020-02-06T18:00:12.200Z","pid":26165,"message":"Geo::SidekiqCronConfigWorker JID-88c4f8c0a9171e67dc1698d8: done: 0.059689 sec","job_status":"done","scheduling_latency_s":0.004625,"duration":0.059689,"cpu_s":0.015371,"completed_at":"2020-02-06T18:00:12.264Z","db_duration":25.947274989448488,"db_duration_s":0.025947274989448488}
==> /var/log/gitlab/gitlab-rails/gitlab-rails-db-migrate-2018-11-20-22-02-32.log <==
==> /var/log/gitlab/gitlab-rails/githost.log <==
==> /var/log/gitlab/gitlab-rails/gitlab-rails-db-migrate-2019-03-22-17-06-19.log <==
==> /var/log/gitlab/gitlab-rails/migrations.log <==
==> /var/log/gitlab/gitlab-rails/sidekiq_exporter.log <== [2020-02-06T17:58:04.261+0000] 127.0.0.1 - - [06/Feb/2020:17:58:04 UTC] "GET /metrics HTTP/1.1" 200 25 "-" "Prometheus/2.12.0" [2020-02-06T17:58:19.259+0000] 127.0.0.1 - - [06/Feb/2020:17:58:19 UTC] "GET /metrics HTTP/1.1" 200 25 "-" "Prometheus/2.12.0" [2020-02-06T17:58:34.259+0000] 127.0.0.1 - - [06/Feb/2020:17:58:34 UTC] "GET /metrics HTTP/1.1" 200 25 "-" "Prometheus/2.12.0" [2020-02-06T17:58:49.258+0000] 127.0.0.1 - - [06/Feb/2020:17:58:49 UTC] "GET /metrics HTTP/1.1" 200 25 "-" "Prometheus/2.12.0" [2020-02-06T17:59:04.259+0000] 127.0.0.1 - - [06/Feb/2020:17:59:04 UTC] "GET /metrics HTTP/1.1" 200 25 "-" "Prometheus/2.12.0" [2020-02-06T17:59:19.258+0000] 127.0.0.1 - - [06/Feb/2020:17:59:19 UTC] "GET /metrics HTTP/1.1" 200 25 "-" "Prometheus/2.12.0" [2020-02-06T17:59:34.258+0000] 127.0.0.1 - - [06/Feb/2020:17:59:34 UTC] "GET /metrics HTTP/1.1" 200 25 "-" "Prometheus/2.12.0" [2020-02-06T17:59:49.258+0000] 127.0.0.1 - - [06/Feb/2020:17:59:49 UTC] "GET /metrics HTTP/1.1" 200 25 "-" "Prometheus/2.12.0" [2020-02-06T18:00:04.258+0000] 127.0.0.1 - - [06/Feb/2020:18:00:04 UTC] "GET /metrics HTTP/1.1" 200 25 "-" "Prometheus/2.12.0" [2020-02-06T18:00:19.259+0000] 127.0.0.1 - - [06/Feb/2020:18:00:19 UTC] "GET /metrics HTTP/1.1" 200 25 "-" "Prometheus/2.12.0"
==> /var/log/gitlab/gitlab-rails/production.log <== Started GET "/-/metrics" for 127.0.0.1 at 2020-02-06 17:59:58 +0000 Processing by MetricsController#index as HTML Completed 200 OK in 1ms (Views: 0.2ms | ActiveRecord: 0.0ms | Elasticsearch: 0.0ms) StuckCiJobsWorker: Cleaning stuck builds Started GET "/-/metrics" for 127.0.0.1 at 2020-02-06 18:00:13 +0000 Processing by MetricsController#index as HTML Completed 200 OK in 2ms (Views: 0.5ms | ActiveRecord: 0.0ms | Elasticsearch: 0.0ms) Started GET "/-/metrics" for 127.0.0.1 at 2020-02-06 18:00:28 +0000 Processing by MetricsController#index as HTML Completed 200 OK in 2ms (Views: 0.5ms | ActiveRecord: 0.0ms | Elasticsearch: 0.0ms)
==> /var/log/gitlab/gitlab-rails/application.log <== 2020-02-06T10:49:39.071Z: DB connection pool size: 25 (increased from 1 to match thread count)
==> /var/log/gitlab/gitlab-rails/gitlab-rails-db-migrate-2018-05-21-13-20-26.log <== ==> /var/log/gitlab/gitlab-rails/application_json.log <==
Logfile created on 2020-02-06 10:49:39 +0000 by logger.rb/66358
{"severity":"DEBUG","time":"2020-02-06T10:49:39.072Z","correlation_id":null,"message":"DB connection pool size: 25 (increased from 1 to match thread count)"}
==> /var/log/gitlab/gitlab-rails/api_json.log <== {"time":"2020-02-06T11:01:12.669Z","severity":"INFO","duration":23.11,"db":0.63,"view":22.48,"status":200,"method":"GET","path":"/api/v4/internal/check","params":[],"host":"127.0.0.1","remote_ip":"127.0.0.1","ua":"Go-http-client/1.1","route":"/api/:version/internal/check","correlation_id":"2349a451-5aa9-4295-8224-0e57866e6e24"}
==> /var/log/gitlab/gitlab-rails/gitlab-rails-db-migrate-2018-03-30-19-44-34.log <==
==> /var/log/gitlab/gitlab-rails/grpc.log <==
==> /var/log/gitlab/gitlab-rails/gitlab-rails-db-migrate-2018-09-26-10-49-46.log <==
==> /var/log/gitlab/gitlab-rails/gitlab-rails-db-migrate-2019-02-04-21-29-23.log <==
==> /var/log/gitlab/gitlab-rails/gitlab-rails-db-migrate-2018-01-19-16-29-53.log <==
==> /var/log/gitlab/gitlab-rails/gitlab-rails-db-migrate-2019-04-30-22-40-36.log <==
==> /var/log/gitlab/gitlab-rails/gitlab-rails-db-migrate-2018-12-26-08-20-32.log <==
==> /var/log/gitlab/gitlab-rails/gitlab-rails-db-migrate-2019-12-25-21-00-51.log <==
==> /var/log/gitlab/gitlab-rails/gitlab-rails-db-migrate-2019-03-04-08-01-02.log <==
==> /var/log/gitlab/gitlab-rails/gitlab-rails-db-migrate-2018-04-30-23-38-52.log <==
==> /var/log/gitlab/gitlab-rails/gitlab-rails-db-migrate-2018-08-22-15-15-18.log <==
==> /var/log/gitlab/gitlab-rails/graphql_json.log <==
==> /var/log/gitlab/gitlab-rails/gitlab-rails-db-migrate-2020-01-08-17-18-17.log <==
==> /var/log/gitlab/gitlab-rails/gitlab-rails-db-migrate-2020-02-06-10-48-35.log <== == 20200123155929 RemoveInvalidJiraData: migrating ============================ -- execute("DELETE FROM jira_tracker_data WHERE (length(encrypted_api_url) > 0 AND encrypted_api_url_iv IS NULL) OR (length(encrypted_url) > 0 AND encrypted_url_iv IS NULL) OR (length(encrypted_username) > 0 AND encrypted_username_iv IS NULL) OR (length(encrypted_password) > 0 AND encrypted_password_iv IS NULL)") -> 0.0008s == 20200123155929 RemoveInvalidJiraData: migrated (0.0008s) ===================
== 20200127090233 RemoveInvalidIssueTrackerData: migrating ==================== -- execute("DELETE FROM issue_tracker_data WHERE (length(encrypted_issues_url) > 0 AND encrypted_issues_url_iv IS NULL) OR (length(encrypted_new_issue_url) > 0 AND encrypted_new_issue_url_iv IS NULL) OR (length(encrypted_project_url) > 0 AND encrypted_project_url_iv IS NULL)") -> 0.0005s == 20200127090233 RemoveInvalidIssueTrackerData: migrated (0.0006s) ===========
==> /var/log/gitlab/gitlab-rails/gitlab-rails-db-migrate-2018-02-12-16-45-56.log <==
==> /var/log/gitlab/gitlab-rails/gitlab-rails-db-migrate-2019-07-28-11-05-54.log <==
==> /var/log/gitlab/gitlab-rails/production_json.log <== {"method":"GET","path":"/-/metrics","format":"html","controller":"MetricsController","action":"index","status":200,"duration":0.83,"view":0.22,"db":0.0,"time":"2020-02-06T17:58:13.302Z","params":[],"remote_ip":null,"user_id":null,"username":null,"ua":null,"queue_duration":null,"correlation_id":"79f3b9b1-11bd-415b-a2a0-f2bd6dac4374","cpu_s":0.04645745999999917} {"method":"GET","path":"/-/metrics","format":"html","controller":"MetricsController","action":"index","status":200,"duration":2.35,"view":0.63,"db":0.0,"time":"2020-02-06T17:58:28.265Z","params":[],"remote_ip":null,"user_id":null,"username":null,"ua":null,"queue_duration":null,"correlation_id":"4b915970-cb90-4b31-a140-5cc0b4e1505c","cpu_s":0.015591989999999} {"method":"GET","path":"/-/metrics","format":"html","controller":"MetricsController","action":"index","status":200,"duration":1.9,"view":0.51,"db":0.0,"time":"2020-02-06T17:58:43.314Z","params":[],"remote_ip":null,"user_id":null,"username":null,"ua":null,"queue_duration":null,"correlation_id":"3a2d81ed-ab1b-4715-8d16-2ccbfc612793","cpu_s":0.05514779299999972} {"method":"GET","path":"/-/metrics","format":"html","controller":"MetricsController","action":"index","status":200,"duration":0.73,"view":0.21,"db":0.0,"time":"2020-02-06T17:58:58.294Z","params":[],"remote_ip":null,"user_id":null,"username":null,"ua":null,"queue_duration":null,"correlation_id":"3a7d3933-f3aa-4816-a0f1-9877aed26b41","cpu_s":0.03981610300000149} {"method":"GET","path":"/-/metrics","format":"html","controller":"MetricsController","action":"index","status":200,"duration":1.79,"view":0.49,"db":0.0,"time":"2020-02-06T17:59:13.308Z","params":[],"remote_ip":null,"user_id":null,"username":null,"ua":null,"queue_duration":null,"correlation_id":"0d02cede-eaf8-4cfd-acb9-b969a05ec21a","cpu_s":0.051967027000001664} {"method":"GET","path":"/-/metrics","format":"html","controller":"MetricsController","action":"index","status":200,"duration":0.97,"view":0.27,"db":0.0,"time":"2020-02-06T17:59:28.254Z","params":[],"remote_ip":null,"user_id":null,"username":null,"ua":null,"queue_duration":null,"correlation_id":"ed426c2f-8fa3-425c-a427-9abc1bfa8db1","cpu_s":0.0060748809999999764} {"method":"GET","path":"/-/metrics","format":"html","controller":"MetricsController","action":"index","status":200,"duration":1.0,"view":0.27,"db":0.0,"time":"2020-02-06T17:59:43.300Z","params":[],"remote_ip":null,"user_id":null,"username":null,"ua":null,"queue_duration":null,"correlation_id":"95665d7b-adab-47c6-a51e-b37fdb264925","cpu_s":0.04374724499999871} {"method":"GET","path":"/-/metrics","format":"html","controller":"MetricsController","action":"index","status":200,"duration":0.86,"view":0.23,"db":0.0,"time":"2020-02-06T17:59:58.297Z","params":[],"remote_ip":null,"user_id":null,"username":null,"ua":null,"queue_duration":null,"correlation_id":"bbbb984a-2f6c-4130-8ded-9f648a7c0f56","cpu_s":0.04161411599999987} {"method":"GET","path":"/-/metrics","format":"html","controller":"MetricsController","action":"index","status":200,"duration":1.79,"view":0.49,"db":0.0,"time":"2020-02-06T18:00:13.260Z","params":[],"remote_ip":null,"user_id":null,"username":null,"ua":null,"queue_duration":null,"correlation_id":"08985d96-1277-490e-af42-e6ff432ffcc3","cpu_s":0.011305856999999975} {"method":"GET","path":"/-/metrics","format":"html","controller":"MetricsController","action":"index","status":200,"duration":1.84,"view":0.45,"db":0.0,"time":"2020-02-06T18:00:28.260Z","params":[],"remote_ip":null,"user_id":null,"username":null,"ua":null,"queue_duration":null,"correlation_id":"a0b35f11-a466-4bdf-a158-348e94adee86","cpu_s":0.010905215999999385}
==> /var/log/gitlab/gitlab-rails/gitlab-rails-db-migrate-2019-06-30-19-42-15.log <==
==> /var/log/gitlab/gitlab-workhorse/current <== {"build_time":"20191219.182121","level":"info","msg":"Starting","time":"2020-02-06T10:31:25Z","version":"v8.18.0"} {"level":"info","msg":"keywatcher: starting process loop","time":"2020-02-06T10:31:25Z"} {"address":"/var/opt/gitlab/redis/redis.socket","level":"info","msg":"redis: dialing","network":"unix","time":"2020-02-06T10:31:25Z"} {"error":"keywatcher: pubsub receive: EOF","level":"error","msg":"unknown error","time":"2020-02-06T10:31:29Z"} {"address":"/var/opt/gitlab/redis/redis.socket","level":"info","msg":"redis: dialing","network":"unix","time":"2020-02-06T10:31:29Z"} {"error":"keywatcher: dial unix /var/opt/gitlab/redis/redis.socket: connect: no such file or directory","level":"error","msg":"unknown error","time":"2020-02-06T10:31:29Z"} {"address":"/var/opt/gitlab/redis/redis.socket","level":"info","msg":"redis: dialing","network":"unix","time":"2020-02-06T10:31:29Z"} {"build_time":"20200130.155325","level":"info","msg":"Starting","time":"2020-02-06T10:49:22Z","version":"v8.20.0"} {"level":"info","msg":"keywatcher: starting process loop","time":"2020-02-06T10:49:22Z"} {"address":"/var/opt/gitlab/redis/redis.socket","level":"info","msg":"redis: dialing","network":"unix","time":"2020-02-06T10:49:22Z"} ==> /var/log/gitlab/gitlab-workhorse/state <==
==> /var/log/gitlab/crond/current <== 2020-02-06_10:31:24.53921 go-crond: Terminated 2020-02-06_10:31:24.55525 go-crond: Starting go-crond version 0.6.1 2020-02-06_10:31:24.55742 go-crond: Add cron job spec:'6 0 */4 * *' usr:root cmd:'/opt/gitlab/bin/gitlab-ctl renew-le-certs' 2020-02-06_10:31:24.55774 go-crond: Start runner with 1 jobs 2020-02-06_10:48:18.37243 go-crond: Got signal: terminated 2020-02-06_10:48:18.37280 go-crond: Stop runner 2020-02-06_10:48:18.37294 go-crond: Terminated 2020-02-06_10:49:23.87056 go-crond: Starting go-crond version 0.6.1 2020-02-06_10:49:23.87128 go-crond: Add cron job spec:'6 0 */4 * *' usr:root cmd:'/opt/gitlab/bin/gitlab-ctl renew-le-certs' 2020-02-06_10:49:23.87176 go-crond: Start runner with 1 jobs
==> /var/log/gitlab/registry/current <== 2020-02-06_10:31:30.07765 time="2020-02-06T10:31:30.077495053Z" level=info msg="redis not configured" environment=production go.version=go1.12.13 instance.id=69f52969-bfa6-4c38-a7ac-be878269bebf service=registry version=v2.7.4-gitlab 2020-02-06_10:31:30.08594 time="2020-02-06T10:31:30.085908171Z" level=info msg="using inmemory blob descriptor cache" environment=production go.version=go1.12.13 instance.id=69f52969-bfa6-4c38-a7ac-be878269bebf service=registry version=v2.7.4-gitlab 2020-02-06_10:31:30.08643 time="2020-02-06T10:31:30.086412029Z" level=info msg="listening on 127.0.0.1:5000" environment=production go.version=go1.12.13 instance.id=69f52969-bfa6-4c38-a7ac-be878269bebf service=registry version=v2.7.4-gitlab 2020-02-06_10:49:26.88054 time="2020-02-06T10:49:26.880074625Z" level=info msg="redis not configured" environment=production go.version=go1.12.13 instance.id=bc5a9c8e-088f-41cf-b60f-faef163baf18 service=registry version=v2.7.4-gitlab 2020-02-06_10:49:26.88064 time="2020-02-06T10:49:26.880415119Z" level=info msg="Starting upload purge in 23m0s" environment=production go.version=go1.12.13 instance.id=bc5a9c8e-088f-41cf-b60f-faef163baf18 service=registry version=v2.7.4-gitlab 2020-02-06_10:49:26.89061 time="2020-02-06T10:49:26.890566297Z" level=info msg="using inmemory blob descriptor cache" environment=production go.version=go1.12.13 instance.id=bc5a9c8e-088f-41cf-b60f-faef163baf18 service=registry version=v2.7.4-gitlab 2020-02-06_10:49:26.89287 time="2020-02-06T10:49:26.892850161Z" level=info msg="listening on 127.0.0.1:5000" environment=production go.version=go1.12.13 instance.id=bc5a9c8e-088f-41cf-b60f-faef163baf18 service=registry version=v2.7.4-gitlab 2020-02-06_11:12:26.88144 time="2020-02-06T11:12:26.881355021Z" level=info msg="PurgeUploads starting: olderThan=2020-01-30 11:12:26.880748915 +0000 UTC m=-603419.984312235, actuallyDelete=true" 2020-02-06_11:12:26.88175 time="2020-02-06T11:12:26.88169542Z" level=info msg="Purge uploads finished. Num deleted=0, num errors=1" 2020-02-06_11:12:26.88185 time="2020-02-06T11:12:26.881730395Z" level=info msg="Starting upload purge in 24h0m0s" environment=production go.version=go1.12.13 instance.id=bc5a9c8e-088f-41cf-b60f-faef163baf18 service=registry version=v2.7.4-gitlab
==> /var/log/gitlab/gitlab-shell/gitlab-shell.log <==
==> /var/log/gitlab/alertmanager/current <== 2020-02-06_10:48:17.94543 level=info ts=2020-02-06T10:48:17.945Z caller=main.go:468 msg="Received SIGTERM, exiting gracefully..." 2020-02-06_10:49:23.56184 level=info ts=2020-02-06T10:49:23.561Z caller=main.go:197 msg="Starting Alertmanager" version="(version=0.18.0, branch=master, revision=)" 2020-02-06_10:49:23.56190 level=info ts=2020-02-06T10:49:23.561Z caller=main.go:198 build_context="(go=go1.12.13, user=GitLab-Omnibus, date=)" 2020-02-06_10:49:23.56297 level=info ts=2020-02-06T10:49:23.562Z caller=cluster.go:161 component=cluster msg="setting advertise address explicitly" addr=192.168.0.5 port=9094 2020-02-06_10:49:23.56517 level=info ts=2020-02-06T10:49:23.565Z caller=cluster.go:623 component=cluster msg="Waiting for gossip to settle..." interval=2s 2020-02-06_10:49:23.59640 level=info ts=2020-02-06T10:49:23.596Z caller=coordinator.go:119 component=configuration msg="Loading configuration file" file=/var/opt/gitlab/alertmanager/alertmanager.yml 2020-02-06_10:49:23.59688 level=info ts=2020-02-06T10:49:23.596Z caller=coordinator.go:131 component=configuration msg="Completed loading of configuration file" file=/var/opt/gitlab/alertmanager/alertmanager.yml 2020-02-06_10:49:23.59949 level=info ts=2020-02-06T10:49:23.599Z caller=main.go:429 msg=Listening address=localhost:9093 2020-02-06_10:49:25.56559 level=info ts=2020-02-06T10:49:25.565Z caller=cluster.go:648 component=cluster msg="gossip not settled" polls=0 before=0 now=1 elapsed=2.000169851s 2020-02-06_10:49:33.56832 level=info ts=2020-02-06T10:49:33.568Z caller=cluster.go:640 component=cluster msg="gossip settled; proceeding" elapsed=10.003047067s
==> /var/log/gitlab/alertmanager/state <==
==> /var/log/gitlab/redis-exporter/state <==
==> /var/log/gitlab/redis-exporter/current <== 2020-02-06_10:22:29.89042 time="2020-02-06T10:22:29Z" level=info msg="Redis Metrics Exporter 1.1.0 build date: sha1: Go: go1.12.13" 2020-02-06_10:22:29.89111 time="2020-02-06T10:22:29Z" level=info msg="Providing metrics at localhost:9121/metrics" 2020-02-06_10:31:29.66018 time="2020-02-06T10:31:29Z" level=info msg="Redis Metrics Exporter 1.1.0 build date: sha1: Go: go1.12.13" 2020-02-06_10:31:29.66197 time="2020-02-06T10:31:29Z" level=info msg="Providing metrics at localhost:9121/metrics" 2020-02-06_10:49:26.50062 time="2020-02-06T10:49:26Z" level=info msg="Redis Metrics Exporter 1.3.4 build date: sha1: Go: go1.12.13 GOOS: linux GOARCH: amd64" 2020-02-06_10:49:26.50194 time="2020-02-06T10:49:26Z" level=info msg="Providing metrics at localhost:9121/metrics"
==> /var/log/gitlab/gitlab-rails/sidekiq_exporter.log <== [2020-02-06T18:00:34.258+0000] 127.0.0.1 - - [06/Feb/2020:18:00:34 UTC] "GET /metrics HTTP/1.1" 200 25 "-" "Prometheus/2.12.0"
==> /var/log/gitlab/gitlab-rails/production.log <== Started GET "/-/metrics" for 127.0.0.1 at 2020-02-06 18:00:43 +0000 Processing by MetricsController#index as HTML Completed 200 OK in 2ms (Views: 0.6ms | ActiveRecord: 0.0ms | Elasticsearch: 0.0ms)
==> /var/log/gitlab/gitlab-rails/production_json.log <== {"method":"GET","path":"/-/metrics","format":"html","controller":"MetricsController","action":"index","status":200,"duration":1.85,"view":0.55,"db":0.0,"time":"2020-02-06T18:00:43.261Z","params":[],"remote_ip":null,"user_id":null,"username":null,"ua":null,"queue_duration":null,"correlation_id":"0c16ac07-6d07-466f-b03b-88e2ee1d3ba7","cpu_s":0.01173764899999874}
==> /var/log/gitlab/gitlab-rails/sidekiq_exporter.log <== [2020-02-06T18:00:49.258+0000] 127.0.0.1 - - [06/Feb/2020:18:00:49 UTC] "GET /metrics HTTP/1.1" 200 25 "-" "Prometheus/2.12.0"
==> /var/log/gitlab/gitlab-rails/production.log <== Started GET "/-/metrics" for 127.0.0.1 at 2020-02-06 18:00:58 +0000 Processing by MetricsController#index as HTML Completed 200 OK in 1ms (Views: 0.3ms | ActiveRecord: 0.0ms | Elasticsearch: 0.0ms)
==> /var/log/gitlab/gitlab-rails/production_json.log <== {"method":"GET","path":"/-/metrics","format":"html","controller":"MetricsController","action":"index","status":200,"duration":1.0,"view":0.27,"db":0.0,"time":"2020-02-06T18:00:58.303Z","params":[],"remote_ip":null,"user_id":null,"username":null,"ua":null,"queue_duration":null,"correlation_id":"a900225c-0251-45f0-b736-762cab0fbfc9","cpu_s":0.04622025700000165}
==> /var/log/gitlab/gitlab-rails/sidekiq_exporter.log <== [2020-02-06T18:01:04.258+0000] 127.0.0.1 - - [06/Feb/2020:18:01:04 UTC] "GET /metrics HTTP/1.1" 200 25 "-" "Prometheus/2.12.0"
==> /var/log/gitlab/gitlab-rails/production.log <== Started GET "/-/metrics" for 127.0.0.1 at 2020-02-06 18:01:13 +0000 Processing by MetricsController#index as HTML Completed 200 OK in 1ms (Views: 0.3ms | ActiveRecord: 0.0ms | Elasticsearch: 0.0ms)
==> /var/log/gitlab/gitlab-rails/production_json.log <== {"method":"GET","path":"/-/metrics","format":"html","controller":"MetricsController","action":"index","status":200,"duration":1.07,"view":0.27,"db":0.0,"time":"2020-02-06T18:01:13.293Z","params":[],"remote_ip":null,"user_id":null,"username":null,"ua":null,"queue_duration":null,"correlation_id":"87b16b89-af53-4e9a-8bcf-b93c682e9dbe","cpu_s":0.038430024000000174}
==> /var/log/gitlab/gitlab-rails/sidekiq_exporter.log <== [2020-02-06T18:01:19.323+0000] 127.0.0.1 - - [06/Feb/2020:18:01:19 UTC] "GET /metrics HTTP/1.1" 200 25 "-" "Prometheus/2.12.0" ==> /var/log/gitlab/sidekiq/current <== {"severity":"INFO","time":"2020-02-06T18:01:24.683Z","queue":"cronjob:update_all_mirrors","class":"UpdateAllMirrorsWorker","retry":false,"queue_namespace":"cronjob","jid":"bdd7dd78a3165a5ee2bd0d3e","created_at":"2020-02-06T18:01:24.682Z","correlation_id":"72b1869255b44e4aa4636280461f3b35","enqueued_at":"2020-02-06T18:01:24.682Z","pid":26165,"message":"UpdateAllMirrorsWorker JID-bdd7dd78a3165a5ee2bd0d3e: start","job_status":"start","scheduling_latency_s":0.000753} {"severity":"INFO","time":"2020-02-06T18:01:24.712Z","queue":"cronjob:geo_sidekiq_cron_config","class":"Geo::SidekiqCronConfigWorker","retry":false,"queue_namespace":"cronjob","jid":"ae34bd68b132fe14ebb437a8","created_at":"2020-02-06T18:01:24.710Z","correlation_id":"9c07d85e609a5470e9df6b911e327112","enqueued_at":"2020-02-06T18:01:24.710Z","pid":26165,"message":"Geo::SidekiqCronConfigWorker JID-ae34bd68b132fe14ebb437a8: start","job_status":"start","scheduling_latency_s":0.001215} {"severity":"INFO","time":"2020-02-06T18:01:24.737Z","queue":"cronjob:geo_sidekiq_cron_config","class":"Geo::SidekiqCronConfigWorker","retry":false,"queue_namespace":"cronjob","jid":"ae34bd68b132fe14ebb437a8","created_at":"2020-02-06T18:01:24.710Z","correlation_id":"9c07d85e609a5470e9df6b911e327112","enqueued_at":"2020-02-06T18:01:24.710Z","pid":26165,"message":"Geo::SidekiqCronConfigWorker JID-ae34bd68b132fe14ebb437a8: done: 0.025321 sec","job_status":"done","scheduling_latency_s":0.001215,"duration":0.025321,"cpu_s":0.011174,"completed_at":"2020-02-06T18:01:24.737Z","db_duration":1.1162119917571545,"db_duration_s":0.0011162119917571545} {"severity":"INFO","time":"2020-02-06T18:01:24.747Z","queue":"cronjob:update_all_mirrors","class":"UpdateAllMirrorsWorker","retry":false,"queue_namespace":"cronjob","jid":"bdd7dd78a3165a5ee2bd0d3e","created_at":"2020-02-06T18:01:24.682Z","correlation_id":"72b1869255b44e4aa4636280461f3b35","enqueued_at":"2020-02-06T18:01:24.682Z","pid":26165,"message":"UpdateAllMirrorsWorker JID-bdd7dd78a3165a5ee2bd0d3e: done: 0.064015 sec","job_status":"done","scheduling_latency_s":0.000753,"duration":0.064015,"cpu_s":0.015022,"completed_at":"2020-02-06T18:01:24.747Z","db_duration":8.833198007778265,"db_duration_s":0.008833198007778265}
==> /var/log/gitlab/gitlab-rails/sidekiq.log <== {"severity":"INFO","time":"2020-02-06T18:01:24.683Z","queue":"cronjob:update_all_mirrors","class":"UpdateAllMirrorsWorker","retry":false,"queue_namespace":"cronjob","jid":"bdd7dd78a3165a5ee2bd0d3e","created_at":"2020-02-06T18:01:24.682Z","correlation_id":"72b1869255b44e4aa4636280461f3b35","enqueued_at":"2020-02-06T18:01:24.682Z","pid":26165,"message":"UpdateAllMirrorsWorker JID-bdd7dd78a3165a5ee2bd0d3e: start","job_status":"start","scheduling_latency_s":0.000753} {"severity":"INFO","time":"2020-02-06T18:01:24.712Z","queue":"cronjob:geo_sidekiq_cron_config","class":"Geo::SidekiqCronConfigWorker","retry":false,"queue_namespace":"cronjob","jid":"ae34bd68b132fe14ebb437a8","created_at":"2020-02-06T18:01:24.710Z","correlation_id":"9c07d85e609a5470e9df6b911e327112","enqueued_at":"2020-02-06T18:01:24.710Z","pid":26165,"message":"Geo::SidekiqCronConfigWorker JID-ae34bd68b132fe14ebb437a8: start","job_status":"start","scheduling_latency_s":0.001215} {"severity":"INFO","time":"2020-02-06T18:01:24.737Z","queue":"cronjob:geo_sidekiq_cron_config","class":"Geo::SidekiqCronConfigWorker","retry":false,"queue_namespace":"cronjob","jid":"ae34bd68b132fe14ebb437a8","created_at":"2020-02-06T18:01:24.710Z","correlation_id":"9c07d85e609a5470e9df6b911e327112","enqueued_at":"2020-02-06T18:01:24.710Z","pid":26165,"message":"Geo::SidekiqCronConfigWorker JID-ae34bd68b132fe14ebb437a8: done: 0.025321 sec","job_status":"done","scheduling_latency_s":0.001215,"duration":0.025321,"cpu_s":0.011174,"completed_at":"2020-02-06T18:01:24.737Z","db_duration":1.1162119917571545,"db_duration_s":0.0011162119917571545} {"severity":"INFO","time":"2020-02-06T18:01:24.747Z","queue":"cronjob:update_all_mirrors","class":"UpdateAllMirrorsWorker","retry":false,"queue_namespace":"cronjob","jid":"bdd7dd78a3165a5ee2bd0d3e","created_at":"2020-02-06T18:01:24.682Z","correlation_id":"72b1869255b44e4aa4636280461f3b35","enqueued_at":"2020-02-06T18:01:24.682Z","pid":26165,"message":"UpdateAllMirrorsWorker JID-bdd7dd78a3165a5ee2bd0d3e: done: 0.064015 sec","job_status":"done","scheduling_latency_s":0.000753,"duration":0.064015,"cpu_s":0.015022,"completed_at":"2020-02-06T18:01:24.747Z","db_duration":8.833198007778265,"db_duration_s":0.008833198007778265}
==> /var/log/gitlab/gitlab-rails/production.log <== Started GET "/-/metrics" for 127.0.0.1 at 2020-02-06 18:01:28 +0000 Processing by MetricsController#index as HTML Completed 200 OK in 2ms (Views: 0.5ms | ActiveRecord: 0.0ms | Elasticsearch: 0.0ms)
==> /var/log/gitlab/gitlab-rails/production_json.log <== {"method":"GET","path":"/-/metrics","format":"html","controller":"MetricsController","action":"index","status":200,"duration":1.86,"view":0.5,"db":0.0,"time":"2020-02-06T18:01:28.263Z","params":[],"remote_ip":null,"user_id":null,"username":null,"ua":null,"queue_duration":null,"correlation_id":"c5e994b5-69d9-4d7e-9bcd-992fe44464cc","cpu_s":0.0130565279999999}
==> /var/log/gitlab/gitlab-rails/sidekiq_exporter.log <== [2020-02-06T18:01:34.259+0000] 127.0.0.1 - - [06/Feb/2020:18:01:34 UTC] "GET /metrics HTTP/1.1" 200 25 "-" "Prometheus/2.12.0"
==> /var/log/gitlab/gitlab-rails/production.log <== Started GET "/-/metrics" for 127.0.0.1 at 2020-02-06 18:01:43 +0000 Processing by MetricsController#index as HTML Completed 200 OK in 2ms (Views: 0.5ms | ActiveRecord: 0.0ms | Elasticsearch: 0.0ms)
==> /var/log/gitlab/gitlab-rails/production_json.log <== {"method":"GET","path":"/-/metrics","format":"html","controller":"MetricsController","action":"index","status":200,"duration":1.87,"view":0.48,"db":0.0,"time":"2020-02-06T18:01:43.261Z","params":[],"remote_ip":null,"user_id":null,"username":null,"ua":null,"queue_duration":null,"correlation_id":"004892b0-cc0c-499a-93f1-3f37dec4b438","cpu_s":0.01171426199999992}
==> /var/log/gitlab/gitlab-rails/sidekiq_exporter.log <== [2020-02-06T18:01:49.259+0000] 127.0.0.1 - - [06/Feb/2020:18:01:49 UTC] "GET /metrics HTTP/1.1" 200 25 "-" "Prometheus/2.12.0"
==> /var/log/gitlab/gitlab-rails/production.log <== Started GET "/-/metrics" for 127.0.0.1 at 2020-02-06 18:01:58 +0000 Processing by MetricsController#index as HTML Completed 200 OK in 3ms (Views: 0.8ms | ActiveRecord: 0.0ms | Elasticsearch: 0.0ms)
==> /var/log/gitlab/gitlab-rails/production_json.log <== {"method":"GET","path":"/-/metrics","format":"html","controller":"MetricsController","action":"index","status":200,"duration":3.01,"view":0.84,"db":0.0,"time":"2020-02-06T18:01:58.320Z","params":[],"remote_ip":null,"user_id":null,"username":null,"ua":null,"queue_duration":null,"correlation_id":"2e01449c-8836-4ef2-87f5-757f97fca017","cpu_s":0.06151422900000014}
==> /var/log/gitlab/gitlab-rails/sidekiq_exporter.log <== [2020-02-06T18:02:04.259+0000] 127.0.0.1 - - [06/Feb/2020:18:02:04 UTC] "GET /metrics HTTP/1.1" 200 25 "-" "Prometheus/2.12.0"
==> /var/log/gitlab/sidekiq/current <== {"severity":"INFO","time":"2020-02-06T18:02:10.710Z","queue":"cronjob:update_all_mirrors","class":"UpdateAllMirrorsWorker","retry":false,"queue_namespace":"cronjob","jid":"b9bf1caa5bbec3ec95de606c","created_at":"2020-02-06T18:02:10.709Z","correlation_id":"331e7623066e3cf0692aa21a82c918fe","enqueued_at":"2020-02-06T18:02:10.709Z","pid":26165,"message":"UpdateAllMirrorsWorker JID-b9bf1caa5bbec3ec95de606c: start","job_status":"start","scheduling_latency_s":0.000763} {"severity":"INFO","time":"2020-02-06T18:02:10.739Z","queue":"cronjob:geo_sidekiq_cron_config","class":"Geo::SidekiqCronConfigWorker","retry":false,"queue_namespace":"cronjob","jid":"4594a279e1e187801a40cf49","created_at":"2020-02-06T18:02:10.738Z","correlation_id":"cf8563f2650e23b39e117fb4ce4a9679","enqueued_at":"2020-02-06T18:02:10.738Z","pid":26165,"message":"Geo::SidekiqCronConfigWorker JID-4594a279e1e187801a40cf49: start","job_status":"start","scheduling_latency_s":0.001194} {"severity":"INFO","time":"2020-02-06T18:02:10.768Z","queue":"cronjob:geo_sidekiq_cron_config","class":"Geo::SidekiqCronConfigWorker","retry":false,"queue_namespace":"cronjob","jid":"4594a279e1e187801a40cf49","created_at":"2020-02-06T18:02:10.738Z","correlation_id":"cf8563f2650e23b39e117fb4ce4a9679","enqueued_at":"2020-02-06T18:02:10.738Z","pid":26165,"message":"Geo::SidekiqCronConfigWorker JID-4594a279e1e187801a40cf49: done: 0.029161 sec","job_status":"done","scheduling_latency_s":0.001194,"duration":0.029161,"cpu_s":0.012223,"completed_at":"2020-02-06T18:02:10.768Z","db_duration":0,"db_duration_s":0} {"severity":"INFO","time":"2020-02-06T18:02:10.771Z","queue":"cronjob:update_all_mirrors","class":"UpdateAllMirrorsWorker","retry":false,"queue_namespace":"cronjob","jid":"b9bf1caa5bbec3ec95de606c","created_at":"2020-02-06T18:02:10.709Z","correlation_id":"331e7623066e3cf0692aa21a82c918fe","enqueued_at":"2020-02-06T18:02:10.709Z","pid":26165,"message":"UpdateAllMirrorsWorker JID-b9bf1caa5bbec3ec95de606c: done: 0.061659 sec","job_status":"done","scheduling_latency_s":0.000763,"duration":0.061659,"cpu_s":0.013696,"completed_at":"2020-02-06T18:02:10.771Z","db_duration":13.349421002203599,"db_duration_s":0.013349421002203599}
==> /var/log/gitlab/gitlab-rails/sidekiq.log <== {"severity":"INFO","time":"2020-02-06T18:02:10.710Z","queue":"cronjob:update_all_mirrors","class":"UpdateAllMirrorsWorker","retry":false,"queue_namespace":"cronjob","jid":"b9bf1caa5bbec3ec95de606c","created_at":"2020-02-06T18:02:10.709Z","correlation_id":"331e7623066e3cf0692aa21a82c918fe","enqueued_at":"2020-02-06T18:02:10.709Z","pid":26165,"message":"UpdateAllMirrorsWorker JID-b9bf1caa5bbec3ec95de606c: start","job_status":"start","scheduling_latency_s":0.000763} {"severity":"INFO","time":"2020-02-06T18:02:10.739Z","queue":"cronjob:geo_sidekiq_cron_config","class":"Geo::SidekiqCronConfigWorker","retry":false,"queue_namespace":"cronjob","jid":"4594a279e1e187801a40cf49","created_at":"2020-02-06T18:02:10.738Z","correlation_id":"cf8563f2650e23b39e117fb4ce4a9679","enqueued_at":"2020-02-06T18:02:10.738Z","pid":26165,"message":"Geo::SidekiqCronConfigWorker JID-4594a279e1e187801a40cf49: start","job_status":"start","scheduling_latency_s":0.001194} {"severity":"INFO","time":"2020-02-06T18:02:10.768Z","queue":"cronjob:geo_sidekiq_cron_config","class":"Geo::SidekiqCronConfigWorker","retry":false,"queue_namespace":"cronjob","jid":"4594a279e1e187801a40cf49","created_at":"2020-02-06T18:02:10.738Z","correlation_id":"cf8563f2650e23b39e117fb4ce4a9679","enqueued_at":"2020-02-06T18:02:10.738Z","pid":26165,"message":"Geo::SidekiqCronConfigWorker JID-4594a279e1e187801a40cf49: done: 0.029161 sec","job_status":"done","scheduling_latency_s":0.001194,"duration":0.029161,"cpu_s":0.012223,"completed_at":"2020-02-06T18:02:10.768Z","db_duration":0,"db_duration_s":0} {"severity":"INFO","time":"2020-02-06T18:02:10.771Z","queue":"cronjob:update_all_mirrors","class":"UpdateAllMirrorsWorker","retry":false,"queue_namespace":"cronjob","jid":"b9bf1caa5bbec3ec95de606c","created_at":"2020-02-06T18:02:10.709Z","correlation_id":"331e7623066e3cf0692aa21a82c918fe","enqueued_at":"2020-02-06T18:02:10.709Z","pid":26165,"message":"UpdateAllMirrorsWorker JID-b9bf1caa5bbec3ec95de606c: done: 0.061659 sec","job_status":"done","scheduling_latency_s":0.000763,"duration":0.061659,"cpu_s":0.013696,"completed_at":"2020-02-06T18:02:10.771Z","db_duration":13.349421002203599,"db_duration_s":0.013349421002203599}
==> /var/log/gitlab/gitlab-rails/production.log <== Started GET "/-/metrics" for 127.0.0.1 at 2020-02-06 18:02:13 +0000 Processing by MetricsController#index as HTML Completed 200 OK in 2ms (Views: 0.5ms | ActiveRecord: 0.0ms | Elasticsearch: 0.0ms)
==> /var/log/gitlab/gitlab-rails/production_json.log <== {"method":"GET","path":"/-/metrics","format":"html","controller":"MetricsController","action":"index","status":200,"duration":1.79,"view":0.46,"db":0.0,"time":"2020-02-06T18:02:13.320Z","params":[],"remote_ip":null,"user_id":null,"username":null,"ua":null,"queue_duration":null,"correlation_id":"a39775e1-62c7-49a1-8660-377cd9de5ec0","cpu_s":0.059554224999999406}
==> /var/log/gitlab/gitlab-rails/sidekiq_exporter.log <== [2020-02-06T18:02:19.258+0000] 127.0.0.1 - - [06/Feb/2020:18:02:19 UTC] "GET /metrics HTTP/1.1" 200 25 "-" "Prometheus/2.12.0"
==> /var/log/gitlab/gitlab-rails/production.log <== Started GET "/-/metrics" for 127.0.0.1 at 2020-02-06 18:02:28 +0000 Processing by MetricsController#index as HTML Completed 200 OK in 2ms (Views: 0.7ms | ActiveRecord: 0.0ms | Elasticsearch: 0.0ms)
==> /var/log/gitlab/gitlab-rails/production_json.log <== {"method":"GET","path":"/-/metrics","format":"html","controller":"MetricsController","action":"index","status":200,"duration":2.84,"view":0.74,"db":0.0,"time":"2020-02-06T18:02:28.266Z","params":[],"remote_ip":null,"user_id":null,"username":null,"ua":null,"queue_duration":null,"correlation_id":"ae00d671-9005-48d8-825e-34c9c3005989","cpu_s":0.017342434999999767}
==> /var/log/gitlab/gitlab-rails/sidekiq_exporter.log <== [2020-02-06T18:02:34.258+0000] 127.0.0.1 - - [06/Feb/2020:18:02:34 UTC] "GET /metrics HTTP/1.1" 200 25 "-" "Prometheus/2.12.0"
==> /var/log/gitlab/gitlab-rails/production.log <== Started GET "/-/metrics" for 127.0.0.1 at 2020-02-06 18:02:43 +0000 Processing by MetricsController#index as HTML Completed 200 OK in 2ms (Views: 0.5ms | ActiveRecord: 0.0ms | Elasticsearch: 0.0ms)
==> /var/log/gitlab/gitlab-rails/production_json.log <== {"method":"GET","path":"/-/metrics","format":"html","controller":"MetricsController","action":"index","status":200,"duration":1.95,"view":0.53,"db":0.0,"time":"2020-02-06T18:02:43.287Z","params":[],"remote_ip":null,"user_id":null,"username":null,"ua":null,"queue_duration":null,"correlation_id":"117c7da8-02eb-4cf4-b109-56d34628f3f2","cpu_s":0.03509389400000096}
==> /var/log/gitlab/gitlab-rails/sidekiq_exporter.log <== [2020-02-06T18:02:49.259+0000] 127.0.0.1 - - [06/Feb/2020:18:02:49 UTC] "GET /metrics HTTP/1.1" 200 25 "-" "Prometheus/2.12.0"
==> /var/log/gitlab/redis/current <== 2020-02-06_18:02:59.06218 12284:M 06 Feb 18:02:59.061 * 10 changes in 300 seconds. Saving... 2020-02-06_18:02:59.06279 12284:M 06 Feb 18:02:59.062 * Background saving started by pid 13635 2020-02-06_18:02:59.08574 13635:C 06 Feb 18:02:59.085 * DB saved on disk 2020-02-06_18:02:59.08672 13635:C 06 Feb 18:02:59.086 * RDB: 1 MB of memory used by copy-on-write
==> /var/log/gitlab/gitlab-rails/production.log <== Started GET "/-/metrics" for 127.0.0.1 at 2020-02-06 18:02:58 +0000 Processing by MetricsController#index as HTML Completed 200 OK in 2ms (Views: 0.5ms | ActiveRecord: 0.0ms | Elasticsearch: 0.0ms) ==> /var/log/gitlab/gitlab-rails/production_json.log <== {"method":"GET","path":"/-/metrics","format":"html","controller":"MetricsController","action":"index","status":200,"duration":1.84,"view":0.5,"db":0.0,"time":"2020-02-06T18:02:58.261Z","params":[],"remote_ip":null,"user_id":null,"username":null,"ua":null,"queue_duration":null,"correlation_id":"c4d4382a-a076-4711-a82e-058d183b104c","cpu_s":0.011891419999999542}
==> /var/log/gitlab/redis/current <== 2020-02-06_18:02:59.16432 12284:M 06 Feb 18:02:59.164 * Background saving terminated with success
==> /var/log/gitlab/gitlab-rails/sidekiq_exporter.log <== [2020-02-06T18:03:04.258+0000] 127.0.0.1 - - [06/Feb/2020:18:03:04 UTC] "GET /metrics HTTP/1.1" 200 25 "-" "Prometheus/2.12.0"
==> /var/log/gitlab/gitlab-rails/production.log <== Started GET "/-/metrics" for 127.0.0.1 at 2020-02-06 18:03:13 +0000 Processing by MetricsController#index as HTML Completed 200 OK in 1ms (Views: 0.3ms | ActiveRecord: 0.0ms | Elasticsearch: 0.0ms)
==> /var/log/gitlab/gitlab-rails/production_json.log <== {"method":"GET","path":"/-/metrics","format":"html","controller":"MetricsController","action":"index","status":200,"duration":0.99,"view":0.27,"db":0.0,"time":"2020-02-06T18:03:13.302Z","params":[],"remote_ip":null,"user_id":null,"username":null,"ua":null,"queue_duration":null,"correlation_id":"6d3b4b27-c575-48b2-9451-0a6e685f4ba7","cpu_s":0.04504374199999894}
==> /var/log/gitlab/gitlab-rails/sidekiq_exporter.log <== [2020-02-06T18:03:19.258+0000] 127.0.0.1 - - [06/Feb/2020:18:03:19 UTC] "GET /metrics HTTP/1.1" 200 25 "-" "Prometheus/2.12.0"
==> /var/log/gitlab/gitlab-rails/production.log <== Started GET "/-/metrics" for 127.0.0.1 at 2020-02-06 18:03:28 +0000 Processing by MetricsController#index as HTML Completed 200 OK in 1ms (Views: 0.3ms | ActiveRecord: 0.0ms | Elasticsearch: 0.0ms)
==> /var/log/gitlab/gitlab-rails/production_json.log <== {"method":"GET","path":"/-/metrics","format":"html","controller":"MetricsController","action":"index","status":200,"duration":1.29,"view":0.35,"db":0.0,"time":"2020-02-06T18:03:28.321Z","params":[],"remote_ip":null,"user_id":null,"username":null,"ua":null,"queue_duration":null,"correlation_id":"3f91ca9d-b0d7-48cc-a797-3c0cb2b77cda","cpu_s":0.060173081000000295}
==> /var/log/gitlab/sidekiq/current <== {"severity":"INFO","time":"2020-02-06T18:03:32.542Z","queue":"cronjob:update_all_mirrors","class":"UpdateAllMirrorsWorker","retry":false,"queue_namespace":"cronjob","jid":"fa04c529a594a96ce3e98900","created_at":"2020-02-06T18:03:32.541Z","correlation_id":"4f04fc3b2436eabb44ec9443a231a24c","enqueued_at":"2020-02-06T18:03:32.541Z","pid":26165,"message":"UpdateAllMirrorsWorker JID-fa04c529a594a96ce3e98900: start","job_status":"start","scheduling_latency_s":0.000713} {"severity":"INFO","time":"2020-02-06T18:03:32.572Z","queue":"cronjob:geo_sidekiq_cron_config","class":"Geo::SidekiqCronConfigWorker","retry":false,"queue_namespace":"cronjob","jid":"67b122db30e07c4560001604","created_at":"2020-02-06T18:03:32.571Z","correlation_id":"44fea2b4c2c8a41ae0ab19cf8134d4c4","enqueued_at":"2020-02-06T18:03:32.571Z","pid":26165,"message":"Geo::SidekiqCronConfigWorker JID-67b122db30e07c4560001604: start","job_status":"start","scheduling_latency_s":0.000875} {"severity":"INFO","time":"2020-02-06T18:03:32.599Z","queue":"cronjob:geo_sidekiq_cron_config","class":"Geo::SidekiqCronConfigWorker","retry":false,"queue_namespace":"cronjob","jid":"67b122db30e07c4560001604","created_at":"2020-02-06T18:03:32.571Z","correlation_id":"44fea2b4c2c8a41ae0ab19cf8134d4c4","enqueued_at":"2020-02-06T18:03:32.571Z","pid":26165,"message":"Geo::SidekiqCronConfigWorker JID-67b122db30e07c4560001604: done: 0.027207 sec","job_status":"done","scheduling_latency_s":0.000875,"duration":0.027207,"cpu_s":0.012171,"completed_at":"2020-02-06T18:03:32.599Z","db_duration":0.5825340049341321,"db_duration_s":0.0005825340049341321} {"severity":"INFO","time":"2020-02-06T18:03:32.606Z","queue":"cronjob:update_all_mirrors","class":"UpdateAllMirrorsWorker","retry":false,"queue_namespace":"cronjob","jid":"fa04c529a594a96ce3e98900","created_at":"2020-02-06T18:03:32.541Z","correlation_id":"4f04fc3b2436eabb44ec9443a231a24c","enqueued_at":"2020-02-06T18:03:32.541Z","pid":26165,"message":"UpdateAllMirrorsWorker JID-fa04c529a594a96ce3e98900: done: 0.064461 sec","job_status":"done","scheduling_latency_s":0.000713,"duration":0.064461,"cpu_s":0.016181,"completed_at":"2020-02-06T18:03:32.606Z","db_duration":5.536102020414546,"db_duration_s":0.005536102020414546} ==> /var/log/gitlab/gitlab-rails/sidekiq.log <== {"severity":"INFO","time":"2020-02-06T18:03:32.542Z","queue":"cronjob:update_all_mirrors","class":"UpdateAllMirrorsWorker","retry":false,"queue_namespace":"cronjob","jid":"fa04c529a594a96ce3e98900","created_at":"2020-02-06T18:03:32.541Z","correlation_id":"4f04fc3b2436eabb44ec9443a231a24c","enqueued_at":"2020-02-06T18:03:32.541Z","pid":26165,"message":"UpdateAllMirrorsWorker JID-fa04c529a594a96ce3e98900: start","job_status":"start","scheduling_latency_s":0.000713} {"severity":"INFO","time":"2020-02-06T18:03:32.572Z","queue":"cronjob:geo_sidekiq_cron_config","class":"Geo::SidekiqCronConfigWorker","retry":false,"queue_namespace":"cronjob","jid":"67b122db30e07c4560001604","created_at":"2020-02-06T18:03:32.571Z","correlation_id":"44fea2b4c2c8a41ae0ab19cf8134d4c4","enqueued_at":"2020-02-06T18:03:32.571Z","pid":26165,"message":"Geo::SidekiqCronConfigWorker JID-67b122db30e07c4560001604: start","job_status":"start","scheduling_latency_s":0.000875} {"severity":"INFO","time":"2020-02-06T18:03:32.599Z","queue":"cronjob:geo_sidekiq_cron_config","class":"Geo::SidekiqCronConfigWorker","retry":false,"queue_namespace":"cronjob","jid":"67b122db30e07c4560001604","created_at":"2020-02-06T18:03:32.571Z","correlation_id":"44fea2b4c2c8a41ae0ab19cf8134d4c4","enqueued_at":"2020-02-06T18:03:32.571Z","pid":26165,"message":"Geo::SidekiqCronConfigWorker JID-67b122db30e07c4560001604: done: 0.027207 sec","job_status":"done","scheduling_latency_s":0.000875,"duration":0.027207,"cpu_s":0.012171,"completed_at":"2020-02-06T18:03:32.599Z","db_duration":0.5825340049341321,"db_duration_s":0.0005825340049341321} {"severity":"INFO","time":"2020-02-06T18:03:32.606Z","queue":"cronjob:update_all_mirrors","class":"UpdateAllMirrorsWorker","retry":false,"queue_namespace":"cronjob","jid":"fa04c529a594a96ce3e98900","created_at":"2020-02-06T18:03:32.541Z","correlation_id":"4f04fc3b2436eabb44ec9443a231a24c","enqueued_at":"2020-02-06T18:03:32.541Z","pid":26165,"message":"UpdateAllMirrorsWorker JID-fa04c529a594a96ce3e98900: done: 0.064461 sec","job_status":"done","scheduling_latency_s":0.000713,"duration":0.064461,"cpu_s":0.016181,"completed_at":"2020-02-06T18:03:32.606Z","db_duration":5.536102020414546,"db_duration_s":0.005536102020414546}
==> /var/log/gitlab/gitlab-rails/sidekiq_exporter.log <== [2020-02-06T18:03:34.258+0000] 127.0.0.1 - - [06/Feb/2020:18:03:34 UTC] "GET /metrics HTTP/1.1" 200 25 "-" "Prometheus/2.12.0"
==> /var/log/gitlab/gitlab-rails/production.log <== Started GET "/-/metrics" for 127.0.0.1 at 2020-02-06 18:03:43 +0000 Processing by MetricsController#index as HTML Completed 200 OK in 1ms (Views: 0.2ms | ActiveRecord: 0.0ms | Elasticsearch: 0.0ms)
==> /var/log/gitlab/gitlab-rails/production_json.log <== {"method":"GET","path":"/-/metrics","format":"html","controller":"MetricsController","action":"index","status":200,"duration":0.83,"view":0.22,"db":0.0,"time":"2020-02-06T18:03:43.294Z","params":[],"remote_ip":null,"user_id":null,"username":null,"ua":null,"queue_duration":null,"correlation_id":"72c16474-872d-4e29-b7a4-cafc2cdbf239","cpu_s":0.039551922999999434}
==> /var/log/gitlab/gitlab-rails/sidekiq_exporter.log <== [2020-02-06T18:03:49.259+0000] 127.0.0.1 - - [06/Feb/2020:18:03:49 UTC] "GET /metrics HTTP/1.1" 200 25 "-" "Prometheus/2.12.0"
==> /var/log/gitlab/gitlab-rails/production.log <== Started GET "/-/metrics" for 127.0.0.1 at 2020-02-06 18:03:58 +0000 Processing by MetricsController#index as HTML Completed 200 OK in 2ms (Views: 0.5ms | ActiveRecord: 0.0ms | Elasticsearch: 0.0ms)
==> /var/log/gitlab/gitlab-rails/production_json.log <== {"method":"GET","path":"/-/metrics","format":"html","controller":"MetricsController","action":"index","status":200,"duration":1.83,"view":0.52,"db":0.0,"time":"2020-02-06T18:03:58.321Z","params":[],"remote_ip":null,"user_id":null,"username":null,"ua":null,"queue_duration":null,"correlation_id":"4697d567-ab6b-4317-83c6-7781b3beed68","cpu_s":0.06344727400000139}
==> /var/log/gitlab/gitlab-rails/sidekiq_exporter.log <== [2020-02-06T18:04:04.257+0000] 127.0.0.1 - - [06/Feb/2020:18:04:04 UTC] "GET /metrics HTTP/1.1" 200 25 "-" "Prometheus/2.12.0"
==> /var/log/gitlab/gitlab-rails/production.log <== Started GET "/-/metrics" for 127.0.0.1 at 2020-02-06 18:04:13 +0000 Processing by MetricsController#index as HTML Completed 200 OK in 1ms (Views: 0.4ms | ActiveRecord: 0.0ms | Elasticsearch: 0.0ms)
==> /var/log/gitlab/gitlab-rails/production_json.log <== {"method":"GET","path":"/-/metrics","format":"html","controller":"MetricsController","action":"index","status":200,"duration":1.36,"view":0.37,"db":0.0,"time":"2020-02-06T18:04:13.313Z","params":[],"remote_ip":null,"user_id":null,"username":null,"ua":null,"queue_duration":null,"correlation_id":"6a413378-5ab4-4f43-b6d6-21ef10bedfa8","cpu_s":0.055055516999999554}
==> /var/log/gitlab/sidekiq/current <== {"severity":"INFO","time":"2020-02-06T18:04:16.619Z","queue":"cronjob:update_all_mirrors","class":"UpdateAllMirrorsWorker","retry":false,"queue_namespace":"cronjob","jid":"334511164738c95f7fd13caa","created_at":"2020-02-06T18:04:16.618Z","correlation_id":"9fe498056cfe1f0ba174154ce35e6b57","enqueued_at":"2020-02-06T18:04:16.618Z","pid":26165,"message":"UpdateAllMirrorsWorker JID-334511164738c95f7fd13caa: start","job_status":"start","scheduling_latency_s":0.000726} {"severity":"INFO","time":"2020-02-06T18:04:16.653Z","queue":"cronjob:geo_sidekiq_cron_config","class":"Geo::SidekiqCronConfigWorker","retry":false,"queue_namespace":"cronjob","jid":"c4ef54a32e750ef4b4d44d63","created_at":"2020-02-06T18:04:16.650Z","correlation_id":"219563d5969fc6bf4fd695518c5a7da8","enqueued_at":"2020-02-06T18:04:16.651Z","pid":26165,"message":"Geo::SidekiqCronConfigWorker JID-c4ef54a32e750ef4b4d44d63: start","job_status":"start","scheduling_latency_s":0.001667} {"severity":"INFO","time":"2020-02-06T18:04:16.683Z","queue":"cronjob:geo_sidekiq_cron_config","class":"Geo::SidekiqCronConfigWorker","retry":false,"queue_namespace":"cronjob","jid":"c4ef54a32e750ef4b4d44d63","created_at":"2020-02-06T18:04:16.650Z","correlation_id":"219563d5969fc6bf4fd695518c5a7da8","enqueued_at":"2020-02-06T18:04:16.651Z","pid":26165,"message":"Geo::SidekiqCronConfigWorker JID-c4ef54a32e750ef4b4d44d63: done: 0.030674 sec","job_status":"done","scheduling_latency_s":0.001667,"duration":0.030674,"cpu_s":0.013199,"completed_at":"2020-02-06T18:04:16.683Z","db_duration":0,"db_duration_s":0} {"severity":"INFO","time":"2020-02-06T18:04:16.685Z","queue":"cronjob:update_all_mirrors","class":"UpdateAllMirrorsWorker","retry":false,"queue_namespace":"cronjob","jid":"334511164738c95f7fd13caa","created_at":"2020-02-06T18:04:16.618Z","correlation_id":"9fe498056cfe1f0ba174154ce35e6b57","enqueued_at":"2020-02-06T18:04:16.618Z","pid":26165,"message":"UpdateAllMirrorsWorker JID-334511164738c95f7fd13caa: done: 0.066093 sec","job_status":"done","scheduling_latency_s":0.000726,"duration":0.066093,"cpu_s":0.01568,"completed_at":"2020-02-06T18:04:16.685Z","db_duration":13.063973005046137,"db_duration_s":0.013063973005046137}
==> /var/log/gitlab/gitlab-rails/sidekiq.log <== {"severity":"INFO","time":"2020-02-06T18:04:16.619Z","queue":"cronjob:update_all_mirrors","class":"UpdateAllMirrorsWorker","retry":false,"queue_namespace":"cronjob","jid":"334511164738c95f7fd13caa","created_at":"2020-02-06T18:04:16.618Z","correlation_id":"9fe498056cfe1f0ba174154ce35e6b57","enqueued_at":"2020-02-06T18:04:16.618Z","pid":26165,"message":"UpdateAllMirrorsWorker JID-334511164738c95f7fd13caa: start","job_status":"start","scheduling_latency_s":0.000726} {"severity":"INFO","time":"2020-02-06T18:04:16.653Z","queue":"cronjob:geo_sidekiq_cron_config","class":"Geo::SidekiqCronConfigWorker","retry":false,"queue_namespace":"cronjob","jid":"c4ef54a32e750ef4b4d44d63","created_at":"2020-02-06T18:04:16.650Z","correlation_id":"219563d5969fc6bf4fd695518c5a7da8","enqueued_at":"2020-02-06T18:04:16.651Z","pid":26165,"message":"Geo::SidekiqCronConfigWorker JID-c4ef54a32e750ef4b4d44d63: start","job_status":"start","scheduling_latency_s":0.001667} {"severity":"INFO","time":"2020-02-06T18:04:16.683Z","queue":"cronjob:geo_sidekiq_cron_config","class":"Geo::SidekiqCronConfigWorker","retry":false,"queue_namespace":"cronjob","jid":"c4ef54a32e750ef4b4d44d63","created_at":"2020-02-06T18:04:16.650Z","correlation_id":"219563d5969fc6bf4fd695518c5a7da8","enqueued_at":"2020-02-06T18:04:16.651Z","pid":26165,"message":"Geo::SidekiqCronConfigWorker JID-c4ef54a32e750ef4b4d44d63: done: 0.030674 sec","job_status":"done","scheduling_latency_s":0.001667,"duration":0.030674,"cpu_s":0.013199,"completed_at":"2020-02-06T18:04:16.683Z","db_duration":0,"db_duration_s":0} {"severity":"INFO","time":"2020-02-06T18:04:16.685Z","queue":"cronjob:update_all_mirrors","class":"UpdateAllMirrorsWorker","retry":false,"queue_namespace":"cronjob","jid":"334511164738c95f7fd13caa","created_at":"2020-02-06T18:04:16.618Z","correlation_id":"9fe498056cfe1f0ba174154ce35e6b57","enqueued_at":"2020-02-06T18:04:16.618Z","pid":26165,"message":"UpdateAllMirrorsWorker JID-334511164738c95f7fd13caa: done: 0.066093 sec","job_status":"done","scheduling_latency_s":0.000726,"duration":0.066093,"cpu_s":0.01568,"completed_at":"2020-02-06T18:04:16.685Z","db_duration":13.063973005046137,"db_duration_s":0.013063973005046137}
==> /var/log/gitlab/gitlab-rails/sidekiq_exporter.log <== [2020-02-06T18:04:19.258+0000] 127.0.0.1 - - [06/Feb/2020:18:04:19 UTC] "GET /metrics HTTP/1.1" 200 25 "-" "Prometheus/2.12.0"
==> /var/log/gitlab/gitlab-rails/production.log <== Started GET "/-/metrics" for 127.0.0.1 at 2020-02-06 18:04:28 +0000 Processing by MetricsController#index as HTML Completed 200 OK in 2ms (Views: 0.7ms | ActiveRecord: 0.0ms | Elasticsearch: 0.0ms) ==> /var/log/gitlab/gitlab-rails/production_json.log <== {"method":"GET","path":"/-/metrics","format":"html","controller":"MetricsController","action":"index","status":200,"duration":2.0,"view":0.66,"db":0.0,"time":"2020-02-06T18:04:28.261Z","params":[],"remote_ip":null,"user_id":null,"username":null,"ua":null,"queue_duration":null,"correlation_id":"8dd559d3-6b30-4d64-8406-906f051785c1","cpu_s":0.012113080000000664}
==> /var/log/gitlab/gitlab-rails/sidekiq_exporter.log <== [2020-02-06T18:04:34.259+0000] 127.0.0.1 - - [06/Feb/2020:18:04:34 UTC] "GET /metrics HTTP/1.1" 200 25 "-" "Prometheus/2.12.0"
==> /var/log/gitlab/gitlab-rails/production.log <== Started GET "/-/metrics" for 127.0.0.1 at 2020-02-06 18:04:43 +0000 Processing by MetricsController#index as HTML Completed 200 OK in 2ms (Views: 0.6ms | ActiveRecord: 0.0ms | Elasticsearch: 0.0ms)
==> /var/log/gitlab/gitlab-rails/production_json.log <== {"method":"GET","path":"/-/metrics","format":"html","controller":"MetricsController","action":"index","status":200,"duration":2.19,"view":0.62,"db":0.0,"time":"2020-02-06T18:04:43.320Z","params":[],"remote_ip":null,"user_id":null,"username":null,"ua":null,"queue_duration":null,"correlation_id":"e92ab209-aaae-46dd-8698-2ff0db8f0749","cpu_s":0.061313401999999684}
==> /var/log/gitlab/gitlab-rails/sidekiq_exporter.log <== [2020-02-06T18:04:49.262+0000] 127.0.0.1 - - [06/Feb/2020:18:04:49 UTC] "GET /metrics HTTP/1.1" 200 25 "-" "Prometheus/2.12.0"
==> /var/log/gitlab/gitlab-rails/production.log <== Started GET "/-/metrics" for 127.0.0.1 at 2020-02-06 18:04:58 +0000 Processing by MetricsController#index as HTML Completed 200 OK in 2ms (Views: 0.5ms | ActiveRecord: 0.0ms | Elasticsearch: 0.0ms)
==> /var/log/gitlab/gitlab-rails/production_json.log <== {"method":"GET","path":"/-/metrics","format":"html","controller":"MetricsController","action":"index","status":200,"duration":1.8,"view":0.47,"db":0.0,"time":"2020-02-06T18:04:58.264Z","params":[],"remote_ip":null,"user_id":null,"username":null,"ua":null,"queue_duration":null,"correlation_id":"9fe0593c-2bef-4356-a157-70bbf1ee51b5","cpu_s":0.01430363100000065}
==> /var/log/gitlab/gitlab-rails/sidekiq_exporter.log <== [2020-02-06T18:05:04.260+0000] 127.0.0.1 - - [06/Feb/2020:18:05:04 UTC] "GET /metrics HTTP/1.1" 200 25 "-" "Prometheus/2.12.0"
==> /var/log/gitlab/sidekiq/current <== {"severity":"INFO","time":"2020-02-06T18:05:12.250Z","queue":"cronjob:update_all_mirrors","class":"UpdateAllMirrorsWorker","retry":false,"queue_namespace":"cronjob","jid":"c2e575e05ce1a06a105378d7","created_at":"2020-02-06T18:05:12.249Z","correlation_id":"687c2ff9746dae27a73e7799d7495687","enqueued_at":"2020-02-06T18:05:12.249Z","pid":26165,"message":"UpdateAllMirrorsWorker JID-c2e575e05ce1a06a105378d7: start","job_status":"start","scheduling_latency_s":0.00079} {"severity":"INFO","time":"2020-02-06T18:05:12.280Z","queue":"cronjob:geo_sidekiq_cron_config","class":"Geo::SidekiqCronConfigWorker","retry":false,"queue_namespace":"cronjob","jid":"166a0434904a2d2e38b45be5","created_at":"2020-02-06T18:05:12.278Z","correlation_id":"e64976188fe3314730ca317196b3b422","enqueued_at":"2020-02-06T18:05:12.279Z","pid":26165,"message":"Geo::SidekiqCronConfigWorker JID-166a0434904a2d2e38b45be5: start","job_status":"start","scheduling_latency_s":0.000885}
==> /var/log/gitlab/gitlab-rails/sidekiq.log <== {"severity":"INFO","time":"2020-02-06T18:05:12.250Z","queue":"cronjob:update_all_mirrors","class":"UpdateAllMirrorsWorker","retry":false,"queue_namespace":"cronjob","jid":"c2e575e05ce1a06a105378d7","created_at":"2020-02-06T18:05:12.249Z","correlation_id":"687c2ff9746dae27a73e7799d7495687","enqueued_at":"2020-02-06T18:05:12.249Z","pid":26165,"message":"UpdateAllMirrorsWorker JID-c2e575e05ce1a06a105378d7: start","job_status":"start","scheduling_latency_s":0.00079} {"severity":"INFO","time":"2020-02-06T18:05:12.280Z","queue":"cronjob:geo_sidekiq_cron_config","class":"Geo::SidekiqCronConfigWorker","retry":false,"queue_namespace":"cronjob","jid":"166a0434904a2d2e38b45be5","created_at":"2020-02-06T18:05:12.278Z","correlation_id":"e64976188fe3314730ca317196b3b422","enqueued_at":"2020-02-06T18:05:12.279Z","pid":26165,"message":"Geo::SidekiqCronConfigWorker JID-166a0434904a2d2e38b45be5: start","job_status":"start","scheduling_latency_s":0.000885} ==> /var/log/gitlab/sidekiq/current <== {"severity":"INFO","time":"2020-02-06T18:05:12.304Z","queue":"cronjob:geo_sidekiq_cron_config","class":"Geo::SidekiqCronConfigWorker","retry":false,"queue_namespace":"cronjob","jid":"166a0434904a2d2e38b45be5","created_at":"2020-02-06T18:05:12.278Z","correlation_id":"e64976188fe3314730ca317196b3b422","enqueued_at":"2020-02-06T18:05:12.279Z","pid":26165,"message":"Geo::SidekiqCronConfigWorker JID-166a0434904a2d2e38b45be5: done: 0.02442 sec","job_status":"done","scheduling_latency_s":0.000885,"duration":0.02442,"cpu_s":0.011648,"completed_at":"2020-02-06T18:05:12.304Z","db_duration":0,"db_duration_s":0} {"severity":"INFO","time":"2020-02-06T18:05:12.310Z","queue":"cronjob:update_all_mirrors","class":"UpdateAllMirrorsWorker","retry":false,"queue_namespace":"cronjob","jid":"c2e575e05ce1a06a105378d7","created_at":"2020-02-06T18:05:12.249Z","correlation_id":"687c2ff9746dae27a73e7799d7495687","enqueued_at":"2020-02-06T18:05:12.249Z","pid":26165,"message":"UpdateAllMirrorsWorker JID-c2e575e05ce1a06a105378d7: done: 0.059675 sec","job_status":"done","scheduling_latency_s":0.00079,"duration":0.059675,"cpu_s":0.014204,"completed_at":"2020-02-06T18:05:12.310Z","db_duration":3.9028590108500794,"db_duration_s":0.0039028590108500794}
==> /var/log/gitlab/gitlab-rails/sidekiq.log <== {"severity":"INFO","time":"2020-02-06T18:05:12.304Z","queue":"cronjob:geo_sidekiq_cron_config","class":"Geo::SidekiqCronConfigWorker","retry":false,"queue_namespace":"cronjob","jid":"166a0434904a2d2e38b45be5","created_at":"2020-02-06T18:05:12.278Z","correlation_id":"e64976188fe3314730ca317196b3b422","enqueued_at":"2020-02-06T18:05:12.279Z","pid":26165,"message":"Geo::SidekiqCronConfigWorker JID-166a0434904a2d2e38b45be5: done: 0.02442 sec","job_status":"done","scheduling_latency_s":0.000885,"duration":0.02442,"cpu_s":0.011648,"completed_at":"2020-02-06T18:05:12.304Z","db_duration":0,"db_duration_s":0} {"severity":"INFO","time":"2020-02-06T18:05:12.310Z","queue":"cronjob:update_all_mirrors","class":"UpdateAllMirrorsWorker","retry":false,"queue_namespace":"cronjob","jid":"c2e575e05ce1a06a105378d7","created_at":"2020-02-06T18:05:12.249Z","correlation_id":"687c2ff9746dae27a73e7799d7495687","enqueued_at":"2020-02-06T18:05:12.249Z","pid":26165,"message":"UpdateAllMirrorsWorker JID-c2e575e05ce1a06a105378d7: done: 0.059675 sec","job_status":"done","scheduling_latency_s":0.00079,"duration":0.059675,"cpu_s":0.014204,"completed_at":"2020-02-06T18:05:12.310Z","db_duration":3.9028590108500794,"db_duration_s":0.0039028590108500794}
==> /var/log/gitlab/gitlab-rails/production.log <== Started GET "/-/metrics" for 127.0.0.1 at 2020-02-06 18:05:13 +0000 Processing by MetricsController#index as HTML Completed 200 OK in 1ms (Views: 0.3ms | ActiveRecord: 0.0ms | Elasticsearch: 0.0ms)
==> /var/log/gitlab/gitlab-rails/production_json.log <== {"method":"GET","path":"/-/metrics","format":"html","controller":"MetricsController","action":"index","status":200,"duration":0.92,"view":0.28,"db":0.0,"time":"2020-02-06T18:05:13.366Z","params":[],"remote_ip":null,"user_id":null,"username":null,"ua":null,"queue_duration":null,"correlation_id":"62990c47-b648-487f-8b30-950bcbb6dfea","cpu_s":0.0671569390000002}
==> /var/log/gitlab/gitlab-rails/sidekiq_exporter.log <== [2020-02-06T18:05:19.258+0000] 127.0.0.1 - - [06/Feb/2020:18:05:19 UTC] "GET /metrics HTTP/1.1" 200 25 "-" "Prometheus/2.12.0"
==> /var/log/gitlab/gitlab-rails/production.log <== Started GET "/-/metrics" for 127.0.0.1 at 2020-02-06 18:05:28 +0000 Processing by MetricsController#index as HTML Completed 200 OK in 1ms (Views: 0.2ms | ActiveRecord: 0.0ms | Elasticsearch: 0.0ms)
==> /var/log/gitlab/gitlab-rails/production_json.log <== {"method":"GET","path":"/-/metrics","format":"html","controller":"MetricsController","action":"index","status":200,"duration":0.73,"view":0.21,"db":0.0,"time":"2020-02-06T18:05:28.293Z","params":[],"remote_ip":null,"user_id":null,"username":null,"ua":null,"queue_duration":null,"correlation_id":"7f096786-3df2-44b5-bccd-5371b1c6004e","cpu_s":0.03832903899999884}
==> /var/log/gitlab/gitlab-rails/sidekiq_exporter.log <== [2020-02-06T18:05:34.259+0000] 127.0.0.1 - - [06/Feb/2020:18:05:34 UTC] "GET /metrics HTTP/1.1" 200 25 "-" "Prometheus/2.12.0"
==> /var/log/gitlab/gitlab-rails/production.log <== Started GET "/-/metrics" for 127.0.0.1 at 2020-02-06 18:05:43 +0000 Processing by MetricsController#index as HTML Completed 200 OK in 2ms (Views: 0.5ms | ActiveRecord: 0.0ms | Elasticsearch: 0.0ms)
==> /var/log/gitlab/gitlab-rails/production_json.log <== {"method":"GET","path":"/-/metrics","format":"html","controller":"MetricsController","action":"index","status":200,"duration":2.03,"view":0.54,"db":0.0,"time":"2020-02-06T18:05:43.261Z","params":[],"remote_ip":null,"user_id":null,"username":null,"ua":null,"queue_duration":null,"correlation_id":"11fa705e-c70e-4f01-b66f-b9ea029c1a0a","cpu_s":0.012161144000000235}
==> /var/log/gitlab/gitlab-rails/sidekiq_exporter.log <== [2020-02-06T18:05:49.258+0000] 127.0.0.1 - - [06/Feb/2020:18:05:49 UTC] "GET /metrics HTTP/1.1" 200 25 "-" "Prometheus/2.12.0"
==> /var/log/gitlab/gitlab-rails/production.log <== Started GET "/-/metrics" for 127.0.0.1 at 2020-02-06 18:05:58 +0000 Processing by MetricsController#index as HTML Completed 200 OK in 2ms (Views: 0.5ms | ActiveRecord: 0.0ms | Elasticsearch: 0.0ms)
==> /var/log/gitlab/gitlab-rails/production_json.log <== {"method":"GET","path":"/-/metrics","format":"html","controller":"MetricsController","action":"index","status":200,"duration":2.08,"view":0.51,"db":0.0,"time":"2020-02-06T18:05:58.260Z","params":[],"remote_ip":null,"user_id":null,"username":null,"ua":null,"queue_duration":null,"correlation_id":"7ef58aaf-eb8e-472d-b8d7-7238f1cc3183","cpu_s":0.011615052999999875}
==> /var/log/gitlab/sidekiq/current <== {"severity":"INFO","time":"2020-02-06T18:06:04.361Z","queue":"cronjob:update_all_mirrors","class":"UpdateAllMirrorsWorker","retry":false,"queue_namespace":"cronjob","jid":"97291ce7c09b27601f95f13e","created_at":"2020-02-06T18:06:04.358Z","correlation_id":"1598c20d9d05619e22283b3f6fdd28c3","enqueued_at":"2020-02-06T18:06:04.358Z","pid":26165,"message":"UpdateAllMirrorsWorker JID-97291ce7c09b27601f95f13e: start","job_status":"start","scheduling_latency_s":0.00288}
==> /var/log/gitlab/gitlab-rails/sidekiq.log <== {"severity":"INFO","time":"2020-02-06T18:06:04.361Z","queue":"cronjob:update_all_mirrors","class":"UpdateAllMirrorsWorker","retry":false,"queue_namespace":"cronjob","jid":"97291ce7c09b27601f95f13e","created_at":"2020-02-06T18:06:04.358Z","correlation_id":"1598c20d9d05619e22283b3f6fdd28c3","enqueued_at":"2020-02-06T18:06:04.358Z","pid":26165,"message":"UpdateAllMirrorsWorker JID-97291ce7c09b27601f95f13e: start","job_status":"start","scheduling_latency_s":0.00288}
==> /var/log/gitlab/sidekiq/current <== {"severity":"INFO","time":"2020-02-06T18:06:04.394Z","queue":"cronjob:geo_sidekiq_cron_config","class":"Geo::SidekiqCronConfigWorker","retry":false,"queue_namespace":"cronjob","jid":"ef9c45f915e96b41187a1157","created_at":"2020-02-06T18:06:04.392Z","correlation_id":"e3339397185fe81397c673e5858e330d","enqueued_at":"2020-02-06T18:06:04.393Z","pid":26165,"message":"Geo::SidekiqCronConfigWorker JID-ef9c45f915e96b41187a1157: start","job_status":"start","scheduling_latency_s":0.001521} {"severity":"INFO","time":"2020-02-06T18:06:04.422Z","queue":"cronjob:geo_sidekiq_cron_config","class":"Geo::SidekiqCronConfigWorker","retry":false,"queue_namespace":"cronjob","jid":"ef9c45f915e96b41187a1157","created_at":"2020-02-06T18:06:04.392Z","correlation_id":"e3339397185fe81397c673e5858e330d","enqueued_at":"2020-02-06T18:06:04.393Z","pid":26165,"message":"Geo::SidekiqCronConfigWorker JID-ef9c45f915e96b41187a1157: done: 0.027777 sec","job_status":"done","scheduling_latency_s":0.001521,"duration":0.027777,"cpu_s":0.013122,"completed_at":"2020-02-06T18:06:04.422Z","db_duration":0.6738480005878955,"db_duration_s":0.0006738480005878955} {"severity":"INFO","time":"2020-02-06T18:06:04.427Z","queue":"cronjob:update_all_mirrors","class":"UpdateAllMirrorsWorker","retry":false,"queue_namespace":"cronjob","jid":"97291ce7c09b27601f95f13e","created_at":"2020-02-06T18:06:04.358Z","correlation_id":"1598c20d9d05619e22283b3f6fdd28c3","enqueued_at":"2020-02-06T18:06:04.358Z","pid":26165,"message":"UpdateAllMirrorsWorker JID-97291ce7c09b27601f95f13e: done: 0.066283 sec","job_status":"done","scheduling_latency_s":0.00288,"duration":0.066283,"cpu_s":0.015178,"completed_at":"2020-02-06T18:06:04.427Z","db_duration":6.117351003922522,"db_duration_s":0.006117351003922522}
==> /var/log/gitlab/gitlab-rails/sidekiq.log <== {"severity":"INFO","time":"2020-02-06T18:06:04.394Z","queue":"cronjob:geo_sidekiq_cron_config","class":"Geo::SidekiqCronConfigWorker","retry":false,"queue_namespace":"cronjob","jid":"ef9c45f915e96b41187a1157","created_at":"2020-02-06T18:06:04.392Z","correlation_id":"e3339397185fe81397c673e5858e330d","enqueued_at":"2020-02-06T18:06:04.393Z","pid":26165,"message":"Geo::SidekiqCronConfigWorker JID-ef9c45f915e96b41187a1157: start","job_status":"start","scheduling_latency_s":0.001521} {"severity":"INFO","time":"2020-02-06T18:06:04.422Z","queue":"cronjob:geo_sidekiq_cron_config","class":"Geo::SidekiqCronConfigWorker","retry":false,"queue_namespace":"cronjob","jid":"ef9c45f915e96b41187a1157","created_at":"2020-02-06T18:06:04.392Z","correlation_id":"e3339397185fe81397c673e5858e330d","enqueued_at":"2020-02-06T18:06:04.393Z","pid":26165,"message":"Geo::SidekiqCronConfigWorker JID-ef9c45f915e96b41187a1157: done: 0.027777 sec","job_status":"done","scheduling_latency_s":0.001521,"duration":0.027777,"cpu_s":0.013122,"completed_at":"2020-02-06T18:06:04.422Z","db_duration":0.6738480005878955,"db_duration_s":0.0006738480005878955} {"severity":"INFO","time":"2020-02-06T18:06:04.427Z","queue":"cronjob:update_all_mirrors","class":"UpdateAllMirrorsWorker","retry":false,"queue_namespace":"cronjob","jid":"97291ce7c09b27601f95f13e","created_at":"2020-02-06T18:06:04.358Z","correlation_id":"1598c20d9d05619e22283b3f6fdd28c3","enqueued_at":"2020-02-06T18:06:04.358Z","pid":26165,"message":"UpdateAllMirrorsWorker JID-97291ce7c09b27601f95f13e: done: 0.066283 sec","job_status":"done","scheduling_latency_s":0.00288,"duration":0.066283,"cpu_s":0.015178,"completed_at":"2020-02-06T18:06:04.427Z","db_duration":6.117351003922522,"db_duration_s":0.006117351003922522}
==> /var/log/gitlab/gitlab-rails/sidekiq_exporter.log <== [2020-02-06T18:06:04.395+0000] 127.0.0.1 - - [06/Feb/2020:18:06:04 UTC] "GET /metrics HTTP/1.1" 200 25 "-" "Prometheus/2.12.0"
==> /var/log/gitlab/gitlab-rails/production.log <== Started GET "/-/metrics" for 127.0.0.1 at 2020-02-06 18:06:13 +0000 Processing by MetricsController#index as HTML Completed 200 OK in 1ms (Views: 0.3ms | ActiveRecord: 0.0ms | Elasticsearch: 0.0ms)
==> /var/log/gitlab/gitlab-rails/production_json.log <== {"method":"GET","path":"/-/metrics","format":"html","controller":"MetricsController","action":"index","status":200,"duration":1.1,"view":0.28,"db":0.0,"time":"2020-02-06T18:06:13.305Z","params":[],"remote_ip":null,"user_id":null,"username":null,"ua":null,"queue_duration":null,"correlation_id":"6eaea9d4-6560-4dd3-b198-e3b6fdf59721","cpu_s":0.049552270000001286}
==> /var/log/gitlab/gitlab-rails/sidekiq_exporter.log <== [2020-02-06T18:06:19.258+0000] 127.0.0.1 - - [06/Feb/2020:18:06:19 UTC] "GET /metrics HTTP/1.1" 200 25 "-" "Prometheus/2.12.0"
==> /var/log/gitlab/gitlab-rails/production.log <== Started GET "/-/metrics" for 127.0.0.1 at 2020-02-06 18:06:28 +0000 Processing by MetricsController#index as HTML Completed 200 OK in 2ms (Views: 0.5ms | ActiveRecord: 0.0ms | Elasticsearch: 0.0ms)
==> /var/log/gitlab/gitlab-rails/production_json.log <== {"method":"GET","path":"/-/metrics","format":"html","controller":"MetricsController","action":"index","status":200,"duration":1.91,"view":0.5,"db":0.0,"time":"2020-02-06T18:06:28.261Z","params":[],"remote_ip":null,"user_id":null,"username":null,"ua":null,"queue_duration":null,"correlation_id":"476dbce8-1b08-496b-9098-935811fa6473","cpu_s":0.011825689000000139}
==> /var/log/gitlab/gitlab-rails/sidekiq_exporter.log <== [2020-02-06T18:06:34.257+0000] 127.0.0.1 - - [06/Feb/2020:18:06:34 UTC] "GET /metrics HTTP/1.1" 200 25 "-" "Prometheus/2.12.0"
==> /var/log/gitlab/gitlab-rails/production.log <== Started GET "/-/metrics" for 127.0.0.1 at 2020-02-06 18:06:43 +0000 Processing by MetricsController#index as HTML Completed 200 OK in 1ms (Views: 0.3ms | ActiveRecord: 0.0ms | Elasticsearch: 0.0ms)
==> /var/log/gitlab/gitlab-rails/production_json.log <== {"method":"GET","path":"/-/metrics","format":"html","controller":"MetricsController","action":"index","status":200,"duration":0.94,"view":0.28,"db":0.0,"time":"2020-02-06T18:06:43.302Z","params":[],"remote_ip":null,"user_id":null,"username":null,"ua":null,"queue_duration":null,"correlation_id":"ba0631e4-91af-446a-93e0-7513be9485c9","cpu_s":0.04549218500000052} ==> /var/log/gitlab/gitlab-rails/sidekiq_exporter.log <== [2020-02-06T18:06:49.259+0000] 127.0.0.1 - - [06/Feb/2020:18:06:49 UTC] "GET /metrics HTTP/1.1" 200 25 "-" "Prometheus/2.12.0"
==> /var/log/gitlab/gitlab-rails/production.log <== Started GET "/-/metrics" for 127.0.0.1 at 2020-02-06 18:06:58 +0000 Processing by MetricsController#index as HTML Completed 200 OK in 2ms (Views: 0.5ms | ActiveRecord: 0.0ms | Elasticsearch: 0.0ms)
==> /var/log/gitlab/gitlab-rails/production_json.log <== {"method":"GET","path":"/-/metrics","format":"html","controller":"MetricsController","action":"index","status":200,"duration":1.86,"view":0.54,"db":0.0,"time":"2020-02-06T18:06:58.261Z","params":[],"remote_ip":null,"user_id":null,"username":null,"ua":null,"queue_duration":null,"correlation_id":"1e49aa8b-ea84-4e7e-8dca-01069ab25b24","cpu_s":0.012212009000000634}
==> /var/log/gitlab/gitlab-rails/sidekiq_exporter.log <== [2020-02-06T18:07:04.258+0000] 127.0.0.1 - - [06/Feb/2020:18:07:04 UTC] "GET /metrics HTTP/1.1" 200 25 "-" "Prometheus/2.12.0"
==> /var/log/gitlab/gitlab-rails/production.log <== Started GET "/-/metrics" for 127.0.0.1 at 2020-02-06 18:07:13 +0000 Processing by MetricsController#index as HTML Completed 200 OK in 1ms (Views: 0.2ms | ActiveRecord: 0.0ms | Elasticsearch: 0.0ms)
==> /var/log/gitlab/gitlab-rails/production_json.log <== {"method":"GET","path":"/-/metrics","format":"html","controller":"MetricsController","action":"index","status":200,"duration":0.8,"view":0.21,"db":0.0,"time":"2020-02-06T18:07:13.297Z","params":[],"remote_ip":null,"user_id":null,"username":null,"ua":null,"queue_duration":null,"correlation_id":"74f31414-b5ff-44c4-982b-91b3593f24d6","cpu_s":0.04035648500000022}
==> /var/log/gitlab/gitlab-rails/sidekiq_exporter.log <== [2020-02-06T18:07:19.259+0000] 127.0.0.1 - - [06/Feb/2020:18:07:19 UTC] "GET /metrics HTTP/1.1" 200 25 "-" "Prometheus/2.12.0"
==> /var/log/gitlab/sidekiq/current <== {"severity":"INFO","time":"2020-02-06T18:07:25.270Z","queue":"cronjob:update_all_mirrors","class":"UpdateAllMirrorsWorker","retry":false,"queue_namespace":"cronjob","jid":"5ac3e190f792a726560870b9","created_at":"2020-02-06T18:07:25.267Z","correlation_id":"0baaf3d3bf47558fca9f4d3364174680","enqueued_at":"2020-02-06T18:07:25.268Z","pid":26165,"message":"UpdateAllMirrorsWorker JID-5ac3e190f792a726560870b9: start","job_status":"start","scheduling_latency_s":0.001857} {"severity":"INFO","time":"2020-02-06T18:07:25.300Z","queue":"cronjob:geo_sidekiq_cron_config","class":"Geo::SidekiqCronConfigWorker","retry":false,"queue_namespace":"cronjob","jid":"915bc6b50637c239d2cda6ec","created_at":"2020-02-06T18:07:25.299Z","correlation_id":"3bb231dc8d09c6b5d9ab6956d92df5da","enqueued_at":"2020-02-06T18:07:25.299Z","pid":26165,"message":"Geo::SidekiqCronConfigWorker JID-915bc6b50637c239d2cda6ec: start","job_status":"start","scheduling_latency_s":0.000709} {"severity":"INFO","time":"2020-02-06T18:07:25.324Z","queue":"cronjob:geo_sidekiq_cron_config","class":"Geo::SidekiqCronConfigWorker","retry":false,"queue_namespace":"cronjob","jid":"915bc6b50637c239d2cda6ec","created_at":"2020-02-06T18:07:25.299Z","correlation_id":"3bb231dc8d09c6b5d9ab6956d92df5da","enqueued_at":"2020-02-06T18:07:25.299Z","pid":26165,"message":"Geo::SidekiqCronConfigWorker JID-915bc6b50637c239d2cda6ec: done: 0.024769 sec","job_status":"done","scheduling_latency_s":0.000709,"duration":0.024769,"cpu_s":0.011784,"completed_at":"2020-02-06T18:07:25.324Z","db_duration":0,"db_duration_s":0} {"severity":"INFO","time":"2020-02-06T18:07:25.331Z","queue":"cronjob:update_all_mirrors","class":"UpdateAllMirrorsWorker","retry":false,"queue_namespace":"cronjob","jid":"5ac3e190f792a726560870b9","created_at":"2020-02-06T18:07:25.267Z","correlation_id":"0baaf3d3bf47558fca9f4d3364174680","enqueued_at":"2020-02-06T18:07:25.268Z","pid":26165,"message":"UpdateAllMirrorsWorker JID-5ac3e190f792a726560870b9: done: 0.060778 sec","job_status":"done","scheduling_latency_s":0.001857,"duration":0.060778,"cpu_s":0.013953,"completed_at":"2020-02-06T18:07:25.331Z","db_duration":4.545010990113951,"db_duration_s":0.004545010990113951} ==> /var/log/gitlab/gitlab-rails/sidekiq.log <== {"severity":"INFO","time":"2020-02-06T18:07:25.270Z","queue":"cronjob:update_all_mirrors","class":"UpdateAllMirrorsWorker","retry":false,"queue_namespace":"cronjob","jid":"5ac3e190f792a726560870b9","created_at":"2020-02-06T18:07:25.267Z","correlation_id":"0baaf3d3bf47558fca9f4d3364174680","enqueued_at":"2020-02-06T18:07:25.268Z","pid":26165,"message":"UpdateAllMirrorsWorker JID-5ac3e190f792a726560870b9: start","job_status":"start","scheduling_latency_s":0.001857} {"severity":"INFO","time":"2020-02-06T18:07:25.300Z","queue":"cronjob:geo_sidekiq_cron_config","class":"Geo::SidekiqCronConfigWorker","retry":false,"queue_namespace":"cronjob","jid":"915bc6b50637c239d2cda6ec","created_at":"2020-02-06T18:07:25.299Z","correlation_id":"3bb231dc8d09c6b5d9ab6956d92df5da","enqueued_at":"2020-02-06T18:07:25.299Z","pid":26165,"message":"Geo::SidekiqCronConfigWorker JID-915bc6b50637c239d2cda6ec: start","job_status":"start","scheduling_latency_s":0.000709} {"severity":"INFO","time":"2020-02-06T18:07:25.324Z","queue":"cronjob:geo_sidekiq_cron_config","class":"Geo::SidekiqCronConfigWorker","retry":false,"queue_namespace":"cronjob","jid":"915bc6b50637c239d2cda6ec","created_at":"2020-02-06T18:07:25.299Z","correlation_id":"3bb231dc8d09c6b5d9ab6956d92df5da","enqueued_at":"2020-02-06T18:07:25.299Z","pid":26165,"message":"Geo::SidekiqCronConfigWorker JID-915bc6b50637c239d2cda6ec: done: 0.024769 sec","job_status":"done","scheduling_latency_s":0.000709,"duration":0.024769,"cpu_s":0.011784,"completed_at":"2020-02-06T18:07:25.324Z","db_duration":0,"db_duration_s":0} {"severity":"INFO","time":"2020-02-06T18:07:25.331Z","queue":"cronjob:update_all_mirrors","class":"UpdateAllMirrorsWorker","retry":false,"queue_namespace":"cronjob","jid":"5ac3e190f792a726560870b9","created_at":"2020-02-06T18:07:25.267Z","correlation_id":"0baaf3d3bf47558fca9f4d3364174680","enqueued_at":"2020-02-06T18:07:25.268Z","pid":26165,"message":"UpdateAllMirrorsWorker JID-5ac3e190f792a726560870b9: done: 0.060778 sec","job_status":"done","scheduling_latency_s":0.001857,"duration":0.060778,"cpu_s":0.013953,"completed_at":"2020-02-06T18:07:25.331Z","db_duration":4.545010990113951,"db_duration_s":0.004545010990113951}
==> /var/log/gitlab/gitlab-rails/production.log <== Started GET "/-/metrics" for 127.0.0.1 at 2020-02-06 18:07:28 +0000 Processing by MetricsController#index as HTML Completed 200 OK in 2ms (Views: 0.7ms | ActiveRecord: 0.0ms | Elasticsearch: 0.0ms)
==> /var/log/gitlab/gitlab-rails/production_json.log <== {"method":"GET","path":"/-/metrics","format":"html","controller":"MetricsController","action":"index","status":200,"duration":2.52,"view":0.68,"db":0.0,"time":"2020-02-06T18:07:28.262Z","params":[],"remote_ip":null,"user_id":null,"username":null,"ua":null,"queue_duration":null,"correlation_id":"638132a5-8162-4134-ae6d-0253d682c0d6","cpu_s":0.01334462099999989}
==> /var/log/gitlab/gitlab-rails/sidekiq_exporter.log <== [2020-02-06T18:07:34.261+0000] 127.0.0.1 - - [06/Feb/2020:18:07:34 UTC] "GET /metrics HTTP/1.1" 200 25 "-" "Prometheus/2.12.0"
==> /var/log/gitlab/gitlab-rails/production.log <== Started GET "/-/metrics" for 127.0.0.1 at 2020-02-06 18:07:43 +0000 Processing by MetricsController#index as HTML Completed 200 OK in 1ms (Views: 0.2ms | ActiveRecord: 0.0ms | Elasticsearch: 0.0ms)
==> /var/log/gitlab/gitlab-rails/production_json.log <== {"method":"GET","path":"/-/metrics","format":"html","controller":"MetricsController","action":"index","status":200,"duration":0.88,"view":0.23,"db":0.0,"time":"2020-02-06T18:07:43.296Z","params":[],"remote_ip":null,"user_id":null,"username":null,"ua":null,"queue_duration":null,"correlation_id":"2ed1e195-2cfb-423c-92e9-f1d1847013bf","cpu_s":0.04093136200000025}
==> /var/log/gitlab/gitlab-rails/sidekiq_exporter.log <== [2020-02-06T18:07:49.259+0000] 127.0.0.1 - - [06/Feb/2020:18:07:49 UTC] "GET /metrics HTTP/1.1" 200 25 "-" "Prometheus/2.12.0"
==> /var/log/gitlab/gitlab-rails/production.log <== Started GET "/-/metrics" for 127.0.0.1 at 2020-02-06 18:07:58 +0000 Processing by MetricsController#index as HTML Completed 200 OK in 1ms (Views: 0.2ms | ActiveRecord: 0.0ms | Elasticsearch: 0.0ms)
==> /var/log/gitlab/gitlab-rails/production_json.log <== {"method":"GET","path":"/-/metrics","format":"html","controller":"MetricsController","action":"index","status":200,"duration":0.83,"view":0.22,"db":0.0,"time":"2020-02-06T18:07:58.300Z","params":[],"remote_ip":null,"user_id":null,"username":null,"ua":null,"queue_duration":null,"correlation_id":"e5ca06be-4410-4d0b-900b-fac5f36c7b89","cpu_s":0.043236401999999785}
==> /var/log/gitlab/redis/current <== 2020-02-06_18:08:00.01353 12284:M 06 Feb 18:08:00.013 * 10 changes in 300 seconds. Saving... 2020-02-06_18:08:00.01428 12284:M 06 Feb 18:08:00.014 * Background saving started by pid 14211 2020-02-06_18:08:00.03344 14211:C 06 Feb 18:08:00.033 * DB saved on disk 2020-02-06_18:08:00.03453 14211:C 06 Feb 18:08:00.034 * RDB: 1 MB of memory used by copy-on-write 2020-02-06_18:08:00.11572 12284:M 06 Feb 18:08:00.115 * Background saving terminated with success
==> /var/log/gitlab/sidekiq/current <== {"severity":"INFO","time":"2020-02-06T18:08:03.854Z","queue":"cronjob:update_all_mirrors","class":"UpdateAllMirrorsWorker","retry":false,"queue_namespace":"cronjob","jid":"129754daf9f59bb81895d2ef","created_at":"2020-02-06T18:08:03.853Z","correlation_id":"22bcca423bcb896770e6d646d1a62f12","enqueued_at":"2020-02-06T18:08:03.854Z","pid":26165,"message":"UpdateAllMirrorsWorker JID-129754daf9f59bb81895d2ef: start","job_status":"start","scheduling_latency_s":0.000668} {"severity":"INFO","time":"2020-02-06T18:08:03.884Z","queue":"cronjob:geo_sidekiq_cron_config","class":"Geo::SidekiqCronConfigWorker","retry":false,"queue_namespace":"cronjob","jid":"02cc3c07fd79a0b602aff068","created_at":"2020-02-06T18:08:03.882Z","correlation_id":"31ad458e093af2293ee6104995881576","enqueued_at":"2020-02-06T18:08:03.883Z","pid":26165,"message":"Geo::SidekiqCronConfigWorker JID-02cc3c07fd79a0b602aff068: start","job_status":"start","scheduling_latency_s":0.001015} {"severity":"INFO","time":"2020-02-06T18:08:03.907Z","queue":"cronjob:geo_sidekiq_cron_config","class":"Geo::SidekiqCronConfigWorker","retry":false,"queue_namespace":"cronjob","jid":"02cc3c07fd79a0b602aff068","created_at":"2020-02-06T18:08:03.882Z","correlation_id":"31ad458e093af2293ee6104995881576","enqueued_at":"2020-02-06T18:08:03.883Z","pid":26165,"message":"Geo::SidekiqCronConfigWorker JID-02cc3c07fd79a0b602aff068: done: 0.022713 sec","job_status":"done","scheduling_latency_s":0.001015,"duration":0.022713,"cpu_s":0.010348,"completed_at":"2020-02-06T18:08:03.907Z","db_duration":0,"db_duration_s":0} {"severity":"INFO","time":"2020-02-06T18:08:03.912Z","queue":"cronjob:update_all_mirrors","class":"UpdateAllMirrorsWorker","retry":false,"queue_namespace":"cronjob","jid":"129754daf9f59bb81895d2ef","created_at":"2020-02-06T18:08:03.853Z","correlation_id":"22bcca423bcb896770e6d646d1a62f12","enqueued_at":"2020-02-06T18:08:03.854Z","pid":26165,"message":"UpdateAllMirrorsWorker JID-129754daf9f59bb81895d2ef: done: 0.058163 sec","job_status":"done","scheduling_latency_s":0.000668,"duration":0.058163,"cpu_s":0.013538,"completed_at":"2020-02-06T18:08:03.912Z","db_duration":4.3291309993946925,"db_duration_s":0.0043291309993946925}
==> /var/log/gitlab/gitlab-rails/sidekiq.log <== {"severity":"INFO","time":"2020-02-06T18:08:03.854Z","queue":"cronjob:update_all_mirrors","class":"UpdateAllMirrorsWorker","retry":false,"queue_namespace":"cronjob","jid":"129754daf9f59bb81895d2ef","created_at":"2020-02-06T18:08:03.853Z","correlation_id":"22bcca423bcb896770e6d646d1a62f12","enqueued_at":"2020-02-06T18:08:03.854Z","pid":26165,"message":"UpdateAllMirrorsWorker JID-129754daf9f59bb81895d2ef: start","job_status":"start","scheduling_latency_s":0.000668} {"severity":"INFO","time":"2020-02-06T18:08:03.884Z","queue":"cronjob:geo_sidekiq_cron_config","class":"Geo::SidekiqCronConfigWorker","retry":false,"queue_namespace":"cronjob","jid":"02cc3c07fd79a0b602aff068","created_at":"2020-02-06T18:08:03.882Z","correlation_id":"31ad458e093af2293ee6104995881576","enqueued_at":"2020-02-06T18:08:03.883Z","pid":26165,"message":"Geo::SidekiqCronConfigWorker JID-02cc3c07fd79a0b602aff068: start","job_status":"start","scheduling_latency_s":0.001015} {"severity":"INFO","time":"2020-02-06T18:08:03.907Z","queue":"cronjob:geo_sidekiq_cron_config","class":"Geo::SidekiqCronConfigWorker","retry":false,"queue_namespace":"cronjob","jid":"02cc3c07fd79a0b602aff068","created_at":"2020-02-06T18:08:03.882Z","correlation_id":"31ad458e093af2293ee6104995881576","enqueued_at":"2020-02-06T18:08:03.883Z","pid":26165,"message":"Geo::SidekiqCronConfigWorker JID-02cc3c07fd79a0b602aff068: done: 0.022713 sec","job_status":"done","scheduling_latency_s":0.001015,"duration":0.022713,"cpu_s":0.010348,"completed_at":"2020-02-06T18:08:03.907Z","db_duration":0,"db_duration_s":0} {"severity":"INFO","time":"2020-02-06T18:08:03.912Z","queue":"cronjob:update_all_mirrors","class":"UpdateAllMirrorsWorker","retry":false,"queue_namespace":"cronjob","jid":"129754daf9f59bb81895d2ef","created_at":"2020-02-06T18:08:03.853Z","correlation_id":"22bcca423bcb896770e6d646d1a62f12","enqueued_at":"2020-02-06T18:08:03.854Z","pid":26165,"message":"UpdateAllMirrorsWorker JID-129754daf9f59bb81895d2ef: done: 0.058163 sec","job_status":"done","scheduling_latency_s":0.000668,"duration":0.058163,"cpu_s":0.013538,"completed_at":"2020-02-06T18:08:03.912Z","db_duration":4.3291309993946925,"db_duration_s":0.0043291309993946925}
==> /var/log/gitlab/gitlab-rails/sidekiq_exporter.log <== [2020-02-06T18:08:04.258+0000] 127.0.0.1 - - [06/Feb/2020:18:08:04 UTC] "GET /metrics HTTP/1.1" 200 25 "-" "Prometheus/2.12.0" ==> /var/log/gitlab/gitlab-rails/production.log <== Started GET "/-/metrics" for 127.0.0.1 at 2020-02-06 18:08:13 +0000 Processing by MetricsController#index as HTML Completed 200 OK in 2ms (Views: 0.5ms | ActiveRecord: 0.0ms | Elasticsearch: 0.0ms)
==> /var/log/gitlab/gitlab-rails/production_json.log <== {"method":"GET","path":"/-/metrics","format":"html","controller":"MetricsController","action":"index","status":200,"duration":1.93,"view":0.54,"db":0.0,"time":"2020-02-06T18:08:13.321Z","params":[],"remote_ip":null,"user_id":null,"username":null,"ua":null,"queue_duration":null,"correlation_id":"d62a9594-a27a-4fbf-be1e-870c8e931cb8","cpu_s":0.06002879900000124}
==> /var/log/gitlab/gitlab-rails/sidekiq_exporter.log <== [2020-02-06T18:08:19.259+0000] 127.0.0.1 - - [06/Feb/2020:18:08:19 UTC] "GET /metrics HTTP/1.1" 200 25 "-" "Prometheus/2.12.0"
==> /var/log/gitlab/gitlab-rails/production.log <== Started GET "/-/metrics" for 127.0.0.1 at 2020-02-06 18:08:28 +0000 Processing by MetricsController#index as HTML Completed 200 OK in 2ms (Views: 0.5ms | ActiveRecord: 0.0ms | Elasticsearch: 0.0ms)
==> /var/log/gitlab/gitlab-rails/production_json.log <== {"method":"GET","path":"/-/metrics","format":"html","controller":"MetricsController","action":"index","status":200,"duration":1.9,"view":0.53,"db":0.0,"time":"2020-02-06T18:08:28.261Z","params":[],"remote_ip":null,"user_id":null,"username":null,"ua":null,"queue_duration":null,"correlation_id":"8a5fce33-c2a5-46ba-a5e8-374b5a1c944f","cpu_s":0.01218270200000049}
==> /var/log/gitlab/gitlab-rails/sidekiq_exporter.log <== [2020-02-06T18:08:34.260+0000] 127.0.0.1 - - [06/Feb/2020:18:08:34 UTC] "GET /metrics HTTP/1.1" 200 25 "-" "Prometheus/2.12.0"
==> /var/log/gitlab/gitlab-rails/production.log <== Started GET "/-/metrics" for 127.0.0.1 at 2020-02-06 18:08:43 +0000 Processing by MetricsController#index as HTML Completed 200 OK in 1ms (Views: 0.2ms | ActiveRecord: 0.0ms | Elasticsearch: 0.0ms)
==> /var/log/gitlab/gitlab-rails/production_json.log <== {"method":"GET","path":"/-/metrics","format":"html","controller":"MetricsController","action":"index","status":200,"duration":0.72,"view":0.19,"db":0.0,"time":"2020-02-06T18:08:43.252Z","params":[],"remote_ip":null,"user_id":null,"username":null,"ua":null,"queue_duration":null,"correlation_id":"96e0b8eb-a582-43af-9184-9b7d6faa9be0","cpu_s":0.004598913000000593}
==> /var/log/gitlab/gitlab-rails/sidekiq_exporter.log <== [2020-02-06T18:08:49.258+0000] 127.0.0.1 - - [06/Feb/2020:18:08:49 UTC] "GET /metrics HTTP/1.1" 200 25 "-" "Prometheus/2.12.0"
==> /var/log/gitlab/gitlab-rails/production.log <== Started GET "/-/metrics" for 127.0.0.1 at 2020-02-06 18:08:58 +0000 Processing by MetricsController#index as HTML Completed 200 OK in 1ms (Views: 0.2ms | ActiveRecord: 0.0ms | Elasticsearch: 0.0ms)
==> /var/log/gitlab/gitlab-rails/production_json.log <== {"method":"GET","path":"/-/metrics","format":"html","controller":"MetricsController","action":"index","status":200,"duration":0.72,"view":0.2,"db":0.0,"time":"2020-02-06T18:08:58.294Z","params":[],"remote_ip":null,"user_id":null,"username":null,"ua":null,"queue_duration":null,"correlation_id":"daf320ba-8e72-4a78-8792-c07eea8e2438","cpu_s":0.03892464399999973}
==> /var/log/gitlab/gitlab-rails/sidekiq_exporter.log <== [2020-02-06T18:09:04.259+0000] 127.0.0.1 - - [06/Feb/2020:18:09:04 UTC] "GET /metrics HTTP/1.1" 200 25 "-" "Prometheus/2.12.0"
==> /var/log/gitlab/gitlab-rails/production.log <== Started GET "/-/metrics" for 127.0.0.1 at 2020-02-06 18:09:13 +0000 Processing by MetricsController#index as HTML Completed 200 OK in 1ms (Views: 0.2ms | ActiveRecord: 0.0ms | Elasticsearch: 0.0ms)
==> /var/log/gitlab/gitlab-rails/production_json.log <== {"method":"GET","path":"/-/metrics","format":"html","controller":"MetricsController","action":"index","status":200,"duration":0.68,"view":0.21,"db":0.0,"time":"2020-02-06T18:09:13.329Z","params":[],"remote_ip":null,"user_id":null,"username":null,"ua":null,"queue_duration":null,"correlation_id":"ced83d82-e88e-4927-9bb6-256fc46cfee9","cpu_s":0.0752096550000001}
==> /var/log/gitlab/sidekiq/current <== {"severity":"INFO","time":"2020-02-06T18:09:16.222Z","queue":"cronjob:update_all_mirrors","class":"UpdateAllMirrorsWorker","retry":false,"queue_namespace":"cronjob","jid":"8328426068a64b3263cccca3","created_at":"2020-02-06T18:09:16.221Z","correlation_id":"1d03d657c172a49d135b704578697c19","enqueued_at":"2020-02-06T18:09:16.221Z","pid":26165,"message":"UpdateAllMirrorsWorker JID-8328426068a64b3263cccca3: start","job_status":"start","scheduling_latency_s":0.000763} {"severity":"INFO","time":"2020-02-06T18:09:16.279Z","queue":"cronjob:geo_sidekiq_cron_config","class":"Geo::SidekiqCronConfigWorker","retry":false,"queue_namespace":"cronjob","jid":"6735dd9512946e830824cbea","created_at":"2020-02-06T18:09:16.277Z","correlation_id":"c3d42b5aafdca3b47cefe25699f60818","enqueued_at":"2020-02-06T18:09:16.278Z","pid":26165,"message":"Geo::SidekiqCronConfigWorker JID-6735dd9512946e830824cbea: start","job_status":"start","scheduling_latency_s":0.001379} {"severity":"INFO","time":"2020-02-06T18:09:16.342Z","queue":"cronjob:update_all_mirrors","class":"UpdateAllMirrorsWorker","retry":false,"queue_namespace":"cronjob","jid":"8328426068a64b3263cccca3","created_at":"2020-02-06T18:09:16.221Z","correlation_id":"1d03d657c172a49d135b704578697c19","enqueued_at":"2020-02-06T18:09:16.221Z","pid":26165,"message":"UpdateAllMirrorsWorker JID-8328426068a64b3263cccca3: done: 0.119985 sec","job_status":"done","scheduling_latency_s":0.000763,"duration":0.119985,"cpu_s":0.024044,"completed_at":"2020-02-06T18:09:16.342Z","db_duration":24.28374999726657,"db_duration_s":0.024283749997266568} {"severity":"INFO","time":"2020-02-06T18:09:16.345Z","queue":"cronjob:geo_sidekiq_cron_config","class":"Geo::SidekiqCronConfigWorker","retry":false,"queue_namespace":"cronjob","jid":"6735dd9512946e830824cbea","created_at":"2020-02-06T18:09:16.277Z","correlation_id":"c3d42b5aafdca3b47cefe25699f60818","enqueued_at":"2020-02-06T18:09:16.278Z","pid":26165,"message":"Geo::SidekiqCronConfigWorker JID-6735dd9512946e830824cbea: done: 0.065654 sec","job_status":"done","scheduling_latency_s":0.001379,"duration":0.065654,"cpu_s":0.023256,"completed_at":"2020-02-06T18:09:16.345Z","db_duration":0,"db_duration_s":0}
==> /var/log/gitlab/gitlab-rails/sidekiq.log <== {"severity":"INFO","time":"2020-02-06T18:09:16.222Z","queue":"cronjob:update_all_mirrors","class":"UpdateAllMirrorsWorker","retry":false,"queue_namespace":"cronjob","jid":"8328426068a64b3263cccca3","created_at":"2020-02-06T18:09:16.221Z","correlation_id":"1d03d657c172a49d135b704578697c19","enqueued_at":"2020-02-06T18:09:16.221Z","pid":26165,"message":"UpdateAllMirrorsWorker JID-8328426068a64b3263cccca3: start","job_status":"start","scheduling_latency_s":0.000763} {"severity":"INFO","time":"2020-02-06T18:09:16.279Z","queue":"cronjob:geo_sidekiq_cron_config","class":"Geo::SidekiqCronConfigWorker","retry":false,"queue_namespace":"cronjob","jid":"6735dd9512946e830824cbea","created_at":"2020-02-06T18:09:16.277Z","correlation_id":"c3d42b5aafdca3b47cefe25699f60818","enqueued_at":"2020-02-06T18:09:16.278Z","pid":26165,"message":"Geo::SidekiqCronConfigWorker JID-6735dd9512946e830824cbea: start","job_status":"start","scheduling_latency_s":0.001379} {"severity":"INFO","time":"2020-02-06T18:09:16.342Z","queue":"cronjob:update_all_mirrors","class":"UpdateAllMirrorsWorker","retry":false,"queue_namespace":"cronjob","jid":"8328426068a64b3263cccca3","created_at":"2020-02-06T18:09:16.221Z","correlation_id":"1d03d657c172a49d135b704578697c19","enqueued_at":"2020-02-06T18:09:16.221Z","pid":26165,"message":"UpdateAllMirrorsWorker JID-8328426068a64b3263cccca3: done: 0.119985 sec","job_status":"done","scheduling_latency_s":0.000763,"duration":0.119985,"cpu_s":0.024044,"completed_at":"2020-02-06T18:09:16.342Z","db_duration":24.28374999726657,"db_duration_s":0.024283749997266568} {"severity":"INFO","time":"2020-02-06T18:09:16.345Z","queue":"cronjob:geo_sidekiq_cron_config","class":"Geo::SidekiqCronConfigWorker","retry":false,"queue_namespace":"cronjob","jid":"6735dd9512946e830824cbea","created_at":"2020-02-06T18:09:16.277Z","correlation_id":"c3d42b5aafdca3b47cefe25699f60818","enqueued_at":"2020-02-06T18:09:16.278Z","pid":26165,"message":"Geo::SidekiqCronConfigWorker JID-6735dd9512946e830824cbea: done: 0.065654 sec","job_status":"done","scheduling_latency_s":0.001379,"duration":0.065654,"cpu_s":0.023256,"completed_at":"2020-02-06T18:09:16.345Z","db_duration":0,"db_duration_s":0}
==> /var/log/gitlab/gitlab-rails/sidekiq_exporter.log <== [2020-02-06T18:09:19.259+0000] 127.0.0.1 - - [06/Feb/2020:18:09:19 UTC] "GET /metrics HTTP/1.1" 200 25 "-" "Prometheus/2.12.0" ==> /var/log/gitlab/gitlab-rails/production.log <== Started GET "/-/metrics" for 127.0.0.1 at 2020-02-06 18:09:28 +0000 Processing by MetricsController#index as HTML Completed 200 OK in 1ms (Views: 0.3ms | ActiveRecord: 0.0ms | Elasticsearch: 0.0ms)
==> /var/log/gitlab/gitlab-rails/production_json.log <== {"method":"GET","path":"/-/metrics","format":"html","controller":"MetricsController","action":"index","status":200,"duration":1.05,"view":0.28,"db":0.0,"time":"2020-02-06T18:09:28.256Z","params":[],"remote_ip":null,"user_id":null,"username":null,"ua":null,"queue_duration":null,"correlation_id":"2d7add2e-c3d8-4323-840f-c428a295eeb3","cpu_s":0.007606052999999946}
==> /var/log/gitlab/gitlab-rails/sidekiq_exporter.log <== [2020-02-06T18:09:34.259+0000] 127.0.0.1 - - [06/Feb/2020:18:09:34 UTC] "GET /metrics HTTP/1.1" 200 25 "-" "Prometheus/2.12.0"
==> /var/log/gitlab/gitlab-rails/production.log <== Started GET "/-/metrics" for 127.0.0.1 at 2020-02-06 18:09:43 +0000 Processing by MetricsController#index as HTML Completed 200 OK in 2ms (Views: 0.6ms | ActiveRecord: 0.0ms | Elasticsearch: 0.0ms)
==> /var/log/gitlab/gitlab-rails/production_json.log <== {"method":"GET","path":"/-/metrics","format":"html","controller":"MetricsController","action":"index","status":200,"duration":1.93,"view":0.56,"db":0.0,"time":"2020-02-06T18:09:43.312Z","params":[],"remote_ip":null,"user_id":null,"username":null,"ua":null,"queue_duration":null,"correlation_id":"053c0263-076b-4eeb-bc5e-c510366734de","cpu_s":0.054586982000000006}
==> /var/log/gitlab/gitlab-rails/sidekiq_exporter.log <== [2020-02-06T18:09:49.258+0000] 127.0.0.1 - - [06/Feb/2020:18:09:49 UTC] "GET /metrics HTTP/1.1" 200 25 "-" "Prometheus/2.12.0"
==> /var/log/gitlab/gitlab-rails/production.log <== Started GET "/-/metrics" for 127.0.0.1 at 2020-02-06 18:09:58 +0000 Processing by MetricsController#index as HTML Completed 200 OK in 1ms (Views: 0.3ms | ActiveRecord: 0.0ms | Elasticsearch: 0.0ms)
==> /var/log/gitlab/gitlab-rails/production_json.log <== {"method":"GET","path":"/-/metrics","format":"html","controller":"MetricsController","action":"index","status":200,"duration":1.08,"view":0.29,"db":0.0,"time":"2020-02-06T18:09:58.292Z","params":[],"remote_ip":null,"user_id":null,"username":null,"ua":null,"queue_duration":null,"correlation_id":"1a1bf213-f717-42ab-ac9e-bb7cdeb64b00","cpu_s":0.03865613100000154}
==> /var/log/gitlab/gitlab-rails/sidekiq_exporter.log <== [2020-02-06T18:10:04.259+0000] 127.0.0.1 - - [06/Feb/2020:18:10:04 UTC] "GET /metrics HTTP/1.1" 200 25 "-" "Prometheus/2.12.0"
==> /var/log/gitlab/sidekiq/current <== {"severity":"INFO","time":"2020-02-06T18:10:06.151Z","queue":"cronjob:update_all_mirrors","class":"UpdateAllMirrorsWorker","retry":false,"queue_namespace":"cronjob","jid":"3fab7de39b4967e7da9e4838","created_at":"2020-02-06T18:10:06.150Z","correlation_id":"32af0cd68bf0910c04d48755cf8d7536","enqueued_at":"2020-02-06T18:10:06.150Z","pid":26165,"message":"UpdateAllMirrorsWorker JID-3fab7de39b4967e7da9e4838: start","job_status":"start","scheduling_latency_s":0.000693} {"severity":"INFO","time":"2020-02-06T18:10:06.175Z","queue":"cronjob:pages_domain_ssl_renewal_cron","class":"PagesDomainSslRenewalCronWorker","retry":false,"queue_namespace":"cronjob","jid":"f37928e89532431e6716d858","created_at":"2020-02-06T18:10:06.174Z","correlation_id":"fd183c0bef63eb2d3ae5bcf5a6b6cdf4","enqueued_at":"2020-02-06T18:10:06.174Z","pid":26165,"message":"PagesDomainSslRenewalCronWorker JID-f37928e89532431e6716d858: start","job_status":"start","scheduling_latency_s":0.00104} {"severity":"INFO","time":"2020-02-06T18:10:06.184Z","queue":"cronjob:geo_sidekiq_cron_config","class":"Geo::SidekiqCronConfigWorker","retry":false,"queue_namespace":"cronjob","jid":"b484d725599ce361b41e8ec0","created_at":"2020-02-06T18:10:06.183Z","correlation_id":"9b64020a32b63d36e310832f5b214a49","enqueued_at":"2020-02-06T18:10:06.183Z","pid":26165,"message":"Geo::SidekiqCronConfigWorker JID-b484d725599ce361b41e8ec0: start","job_status":"start","scheduling_latency_s":0.001104} {"severity":"INFO","time":"2020-02-06T18:10:06.272Z","queue":"cronjob:pages_domain_ssl_renewal_cron","class":"PagesDomainSslRenewalCronWorker","retry":false,"queue_namespace":"cronjob","jid":"f37928e89532431e6716d858","created_at":"2020-02-06T18:10:06.174Z","correlation_id":"fd183c0bef63eb2d3ae5bcf5a6b6cdf4","enqueued_at":"2020-02-06T18:10:06.174Z","pid":26165,"message":"PagesDomainSslRenewalCronWorker JID-f37928e89532431e6716d858: done: 0.096468 sec","job_status":"done","scheduling_latency_s":0.00104,"duration":0.096468,"cpu_s":0.013407,"completed_at":"2020-02-06T18:10:06.272Z" ,"db_duration":7.721850022790022,"db_duration_s":0.007721850022790022} {"severity":"INFO","time":"2020-02-06T18:10:06.272Z","queue":"cronjob:update_all_mirrors","class":"UpdateAllMirrorsWorker","retry":false,"queue_namespace":"cronjob","jid":"3fab7de39b4967e7da9e4838","created_at":"2020-02-06T18:10:06.150Z","correlation_id":"32af0cd68bf0910c04d48755cf8d7536","enqueued_at":"2020-02-06T18:10:06.150Z","pid":26165,"message":"UpdateAllMirrorsWorker JID-3fab7de39b4967e7da9e4838: done: 0.121581 sec","job_status":"done","scheduling_latency_s":0.000693,"duration":0.121581,"cpu_s":0.014112,"completed_at":"2020-02-06T18:10:06.272Z","db_duration":11.661321012070403,"db_duration_s":0.011661321012070403} {"severity":"INFO","time":"2020-02-06T18:10:06.284Z","queue":"cronjob:geo_sidekiq_cron_config","class":"Geo::SidekiqCronConfigWorker","retry":false,"queue_namespace":"cronjob","jid":"b484d725599ce361b41e8ec0","created_at":"2020-02-06T18:10:06.183Z","correlation_id":"9b64020a32b63d36e310832f5b214a49","enqueued_at":"2020-02-06T18:10:06.183Z","pid":26165,"message":"Geo::SidekiqCronConfigWorker JID-b484d725599ce361b41e8ec0: done: 0.099571 sec","job_status":"done","scheduling_latency_s":0.001104,"duration":0.099571,"cpu_s":0.067442,"completed_at":"2020-02-06T18:10:06.284Z","db_duration":19.538531996659003,"db_duration_s":0.019538531996659003}
==> /var/log/gitlab/gitlab-rails/sidekiq.log <== {"severity":"INFO","time":"2020-02-06T18:10:06.151Z","queue":"cronjob:update_all_mirrors","class":"UpdateAllMirrorsWorker","retry":false,"queue_namespace":"cronjob","jid":"3fab7de39b4967e7da9e4838","created_at":"2020-02-06T18:10:06.150Z","correlation_id":"32af0cd68bf0910c04d48755cf8d7536","enqueued_at":"2020-02-06T18:10:06.150Z","pid":26165,"message":"UpdateAllMirrorsWorker JID-3fab7de39b4967e7da9e4838: start","job_status":"start","scheduling_latency_s":0.000693} {"severity":"INFO","time":"2020-02-06T18:10:06.175Z","queue":"cronjob:pages_domain_ssl_renewal_cron","class":"PagesDomainSslRenewalCronWorker","retry":false,"queue_namespace":"cronjob","jid":"f37928e89532431e6716d858","created_at":"2020-02-06T18:10:06.174Z","correlation_id":"fd183c0bef63eb2d3ae5bcf5a6b6cdf4","enqueued_at":"2020-02-06T18:10:06.174Z","pid":26165,"message":"PagesDomainSslRenewalCronWorker JID-f37928e89532431e6716d858: start","job_status":"start","scheduling_latency_s":0.00104} {"severity":"INFO","time":"2020-02-06T18:10:06.184Z","queue":"cronjob:geo_sidekiq_cron_config","class":"Geo::SidekiqCronConfigWorker","retry":false,"queue_namespace":"cronjob","jid":"b484d725599ce361b41e8ec0","created_at":"2020-02-06T18:10:06.183Z","correlation_id":"9b64020a32b63d36e310832f5b214a49","enqueued_at":"2020-02-06T18:10:06.183Z","pid":26165,"message":"Geo::SidekiqCronConfigWorker JID-b484d725599ce361b41e8ec0: start","job_status":"start","scheduling_latency_s":0.001104} {"severity":"INFO","time":"2020-02-06T18:10:06.272Z","queue":"cronjob:pages_domain_ssl_renewal_cron","class":"PagesDomainSslRenewalCronWorker","retry":false,"queue_namespace":"cronjob","jid":"f37928e89532431e6716d858","created_at":"2020-02-06T18:10:06.174Z","correlation_id":"fd183c0bef63eb2d3ae5bcf5a6b6cdf4","enqueued_at":"2020-02-06T18:10:06.174Z","pid":26165,"message":"PagesDomainSslRenewalCronWorker JID-f37928e89532431e6716d858: done: 0.096468 sec","job_status":"done","scheduling_latency_s":0.00104,"duration":0.096468,"cpu_s":0.013407,"completed_at":"2020-02-06T18:10:06.272Z","db_duration":7.721850022790022,"db_duration_s":0.007721850022790022} {"severity":"INFO","time":"2020-02-06T18:10:06.272Z","queue":"cronjob:update_all_mirrors","class":"UpdateAllMirrorsWorker","retry":false,"queue_namespace":"cronjob","jid":"3fab7de39b4967e7da9e4838","created_at":"2020-02-06T18:10:06.150Z","correlation_id":"32af0cd68bf0910c04d48755cf8d7536","enqueued_at":"2020-02-06T18:10:06.150Z","pid":26165,"message":"UpdateAllMirrorsWorker JID-3fab7de39b4967e7da9e4838: done: 0.121581 sec","job_status":"done","scheduling_latency_s":0.000693,"duration":0.121581,"cpu_s":0.014112,"completed_at":"2020-02-06T18:10:06.272Z","db_duration":11.661321012070403,"db_duration_s":0.011661321012070403} {"severity":"INFO","time":"2020-02-06T18:10:06.284Z","queue":"cronjob:geo_sidekiq_cron_config","class":"Geo::SidekiqCronConfigWorker","retry":false,"queue_namespace":"cronjob","jid":"b484d725599ce361b41e8ec0","created_at":"2020-02-06T18:10:06.183Z","correlation_id":"9b64020a32b63d36e310832f5b214a49","enqueued_at":"2020-02-06T18:10:06.183Z","pid":26165,"message":"Geo::SidekiqCronConfigWorker JID-b484d725599ce361b41e8ec0: done: 0.099571 sec","job_status":"done","scheduling_latency_s":0.001104,"duration":0.099571,"cpu_s":0.067442,"completed_at":"2020-02-06T18:10:06.284Z","db_duration":19.538531996659003,"db_duration_s":0.019538531996659003}
==> /var/log/gitlab/gitlab-rails/production.log <== Started GET "/-/metrics" for 127.0.0.1 at 2020-02-06 18:10:13 +0000 Processing by MetricsController#index as HTML Completed 200 OK in 1ms (Views: 0.5ms | ActiveRecord: 0.0ms | Elasticsearch: 0.0ms)
==> /var/log/gitlab/gitlab-rails/production_json.log <== {"method":"GET","path":"/-/metrics","format":"html","controller":"MetricsController","action":"index","status":200,"duration":1.74,"view":0.5,"db":0.0,"time":"2020-02-06T18:10:13.261Z","params":[],"remote_ip":null,"user_id":null,"username":null,"ua":null,"queue_duration":null,"correlation_id":"2014a89b-414a-402c-b916-dc6d3c30c705","cpu_s":0.011722830000000073}
==> /var/log/gitlab/gitlab-rails/sidekiq_exporter.log <== [2020-02-06T18:10:19.259+0000] 127.0.0.1 - - [06/Feb/2020:18:10:19 UTC] "GET /metrics HTTP/1.1" 200 25 "-" "Prometheus/2.12.0" ==> /var/log/gitlab/gitlab-rails/production.log <== Started GET "/-/metrics" for 127.0.0.1 at 2020-02-06 18:10:28 +0000 Processing by MetricsController#index as HTML Completed 200 OK in 1ms (Views: 0.3ms | ActiveRecord: 0.0ms | Elasticsearch: 0.0ms)
==> /var/log/gitlab/gitlab-rails/production_json.log <== {"method":"GET","path":"/-/metrics","format":"html","controller":"MetricsController","action":"index","status":200,"duration":0.94,"view":0.25,"db":0.0,"time":"2020-02-06T18:10:28.294Z","params":[],"remote_ip":null,"user_id":null,"username":null,"ua":null,"queue_duration":null,"correlation_id":"86919904-e803-4b5d-876b-d8b7a35dd454","cpu_s":0.03937249800000053}
==> /var/log/gitlab/gitlab-rails/sidekiq_exporter.log <== [2020-02-06T18:10:34.259+0000] 127.0.0.1 - - [06/Feb/2020:18:10:34 UTC] "GET /metrics HTTP/1.1" 200 25 "-" "Prometheus/2.12.0"
==> /var/log/gitlab/gitlab-rails/production.log <== Started GET "/-/metrics" for 127.0.0.1 at 2020-02-06 18:10:43 +0000 Processing by MetricsController#index as HTML Completed 200 OK in 1ms (Views: 0.4ms | ActiveRecord: 0.0ms | Elasticsearch: 0.0ms)
==> /var/log/gitlab/gitlab-rails/production_json.log <== {"method":"GET","path":"/-/metrics","format":"html","controller":"MetricsController","action":"index","status":200,"duration":1.28,"view":0.38,"db":0.0,"time":"2020-02-06T18:10:43.272Z","params":[],"remote_ip":null,"user_id":null,"username":null,"ua":null,"queue_duration":null,"correlation_id":"c39670fb-257e-407a-9701-91a4a16494a3","cpu_s":0.020759404000001425}
==> /var/log/gitlab/gitlab-rails/sidekiq_exporter.log <== [2020-02-06T18:10:49.259+0000] 127.0.0.1 - - [06/Feb/2020:18:10:49 UTC] "GET /metrics HTTP/1.1" 200 25 "-" "Prometheus/2.12.0"
==> /var/log/gitlab/gitlab-rails/production.log <== Started GET "/-/metrics" for 127.0.0.1 at 2020-02-06 18:10:58 +0000 Processing by MetricsController#index as HTML Completed 200 OK in 2ms (Views: 0.5ms | ActiveRecord: 0.0ms | Elasticsearch: 0.0ms)
==> /var/log/gitlab/gitlab-rails/production_json.log <== {"method":"GET","path":"/-/metrics","format":"html","controller":"MetricsController","action":"index","status":200,"duration":1.81,"view":0.51,"db":0.0,"time":"2020-02-06T18:10:58.260Z","params":[],"remote_ip":null,"user_id":null,"username":null,"ua":null,"queue_duration":null,"correlation_id":"549f9b7a-7e94-4348-a880-0c6285319887","cpu_s":0.011340966000000563}
==> /var/log/gitlab/gitlab-rails/sidekiq_exporter.log <== [2020-02-06T18:11:04.259+0000] 127.0.0.1 - - [06/Feb/2020:18:11:04 UTC] "GET /metrics HTTP/1.1" 200 25 "-" "Prometheus/2.12.0"
==> /var/log/gitlab/gitlab-rails/production.log <== Started GET "/-/metrics" for 127.0.0.1 at 2020-02-06 18:11:13 +0000 Processing by MetricsController#index as HTML Completed 200 OK in 1ms (Views: 0.4ms | ActiveRecord: 0.0ms | Elasticsearch: 0.0ms)
==> /var/log/gitlab/gitlab-rails/production_json.log <== {"method":"GET","path":"/-/metrics","format":"html","controller":"MetricsController","action":"index","status":200,"duration":1.31,"view":0.36,"db":0.0,"time":"2020-02-06T18:11:13.300Z","params":[],"remote_ip":null,"user_id":null,"username":null,"ua":null,"queue_duration":null,"correlation_id":"f0a7118b-7f15-4085-85fa-6f28c0c7c9b5","cpu_s":0.04484570999999882} ==> /var/log/gitlab/gitlab-rails/sidekiq_exporter.log <== [2020-02-06T18:11:19.259+0000] 127.0.0.1 - - [06/Feb/2020:18:11:19 UTC] "GET /metrics HTTP/1.1" 200 25 "-" "Prometheus/2.12.0"
==> /var/log/gitlab/sidekiq/current <== {"severity":"INFO","time":"2020-02-06T18:11:20.442Z","queue":"cronjob:update_all_mirrors","class":"UpdateAllMirrorsWorker","retry":false,"queue_namespace":"cronjob","jid":"5e7055eb12727eb9598eabf8","created_at":"2020-02-06T18:11:20.441Z","correlation_id":"d616316530407bcc0016a10ff555b8bb","enqueued_at":"2020-02-06T18:11:20.441Z","pid":26165,"message":"UpdateAllMirrorsWorker JID-5e7055eb12727eb9598eabf8: start","job_status":"start","scheduling_latency_s":0.000682} {"severity":"INFO","time":"2020-02-06T18:11:20.472Z","queue":"cronjob:geo_sidekiq_cron_config","class":"Geo::SidekiqCronConfigWorker","retry":false,"queue_namespace":"cronjob","jid":"0afe7df6df8732b7c34e044a","created_at":"2020-02-06T18:11:20.471Z","correlation_id":"898160d9a51b45d6cd812996b3404787","enqueued_at":"2020-02-06T18:11:20.471Z","pid":26165,"message":"Geo::SidekiqCronConfigWorker JID-0afe7df6df8732b7c34e044a: start","job_status":"start","scheduling_latency_s":0.001132} {"severity":"INFO","time":"2020-02-06T18:11:20.493Z","queue":"cronjob:geo_sidekiq_cron_config","class":"Geo::SidekiqCronConfigWorker","retry":false,"queue_namespace":"cronjob","jid":"0afe7df6df8732b7c34e044a","created_at":"2020-02-06T18:11:20.471Z","correlation_id":"898160d9a51b45d6cd812996b3404787","enqueued_at":"2020-02-06T18:11:20.471Z","pid":26165,"message":"Geo::SidekiqCronConfigWorker JID-0afe7df6df8732b7c34e044a: done: 0.020603 sec","job_status":"done","scheduling_latency_s":0.001132,"duration":0.020603,"cpu_s":0.011265,"completed_at":"2020-02-06T18:11:20.493Z","db_duration":0.4923450032947585,"db_duration_s":0.0004923450032947585} {"severity":"INFO","time":"2020-02-06T18:11:20.507Z","queue":"cronjob:update_all_mirrors","class":"UpdateAllMirrorsWorker","retry":false,"queue_namespace":"cronjob","jid":"5e7055eb12727eb9598eabf8","created_at":"2020-02-06T18:11:20.441Z","correlation_id":"d616316530407bcc0016a10ff555b8bb","enqueued_at":"2020-02-06T18:11:20.441Z","pid":26165,"message":"UpdateAllMirrorsWorker JID-5e7055eb12727eb9598eabf8: done: 0.064648 sec","job_status":"done","scheduling_latency_s":0.000682,"duration":0.064648,"cpu_s":0.014956,"completed_at":"2020-02-06T18:11:20.507Z","db_duration":8.46363500750158,"db_duration_s":0.00846363500750158}
==> /var/log/gitlab/gitlab-rails/sidekiq.log <== {"severity":"INFO","time":"2020-02-06T18:11:20.442Z","queue":"cronjob:update_all_mirrors","class":"UpdateAllMirrorsWorker","retry":false,"queue_namespace":"cronjob","jid":"5e7055eb12727eb9598eabf8","created_at":"2020-02-06T18:11:20.441Z","correlation_id":"d616316530407bcc0016a10ff555b8bb","enqueued_at":"2020-02-06T18:11:20.441Z","pid":26165,"message":"UpdateAllMirrorsWorker JID-5e7055eb12727eb9598eabf8: start","job_status":"start","scheduling_latency_s":0.000682} {"severity":"INFO","time":"2020-02-06T18:11:20.472Z","queue":"cronjob:geo_sidekiq_cron_config","class":"Geo::SidekiqCronConfigWorker","retry":false,"queue_namespace":"cronjob","jid":"0afe7df6df8732b7c34e044a","created_at":"2020-02-06T18:11:20.471Z","correlation_id":"898160d9a51b45d6cd812996b3404787","enqueued_at":"2020-02-06T18:11:20.471Z","pid":26165,"message":"Geo::SidekiqCronConfigWorker JID-0afe7df6df8732b7c34e044a: start","job_status":"start","scheduling_latency_s":0.001132} {"severity":"INFO","time":"2020-02-06T18:11:20.493Z","queue":"cronjob:geo_sidekiq_cron_config","class":"Geo::SidekiqCronConfigWorker","retry":false,"queue_namespace":"cronjob","jid":"0afe7df6df8732b7c34e044a","created_at":"2020-02-06T18:11:20.471Z","correlation_id":"898160d9a51b45d6cd812996b3404787","enqueued_at":"2020-02-06T18:11:20.471Z","pid":26165,"message":"Geo::SidekiqCronConfigWorker JID-0afe7df6df8732b7c34e044a: done: 0.020603 sec","job_status":"done","scheduling_latency_s":0.001132,"duration":0.020603,"cpu_s":0.011265,"completed_at":"2020-02-06T18:11:20.493Z","db_duration":0.4923450032947585,"db_duration_s":0.0004923450032947585} {"severity":"INFO","time":"2020-02-06T18:11:20.507Z","queue":"cronjob:update_all_mirrors","class":"UpdateAllMirrorsWorker","retry":false,"queue_namespace":"cronjob","jid":"5e7055eb12727eb9598eabf8","created_at":"2020-02-06T18:11:20.441Z","correlation_id":"d616316530407bcc0016a10ff555b8bb","enqueued_at":"2020-02-06T18:11:20.441Z","pid":26165,"message":"UpdateAllMirrorsWorker JID-5e7055eb12727eb9598eabf8: done: 0.064648 sec","job_status":"done","scheduling_latency_s":0.000682,"duration":0.064648,"cpu_s":0.014956,"completed_at":"2020-02-06T18:11:20.507Z","db_duration":8.46363500750158,"db_duration_s":0.00846363500750158}
==> /var/log/gitlab/gitlab-rails/production.log <== Started GET "/-/metrics" for 127.0.0.1 at 2020-02-06 18:11:28 +0000 Processing by MetricsController#index as HTML Completed 200 OK in 1ms (Views: 0.4ms | ActiveRecord: 0.0ms | Elasticsearch: 0.0ms)
==> /var/log/gitlab/gitlab-rails/production_json.log <== {"method":"GET","path":"/-/metrics","format":"html","controller":"MetricsController","action":"index","status":200,"duration":1.34,"view":0.36,"db":0.0,"time":"2020-02-06T18:11:28.261Z","params":[],"remote_ip":null,"user_id":null,"username":null,"ua":null,"queue_duration":null,"correlation_id":"bd160106-afc2-4769-9a91-5e9fa4cba529","cpu_s":0.012372037000000446} ==> /var/log/gitlab/gitlab-rails/sidekiq_exporter.log <== [2020-02-06T18:11:34.258+0000] 127.0.0.1 - - [06/Feb/2020:18:11:34 UTC] "GET /metrics HTTP/1.1" 200 25 "-" "Prometheus/2.12.0"
==> /var/log/gitlab/gitlab-rails/production.log <== Started GET "/-/metrics" for 127.0.0.1 at 2020-02-06 18:11:43 +0000 Processing by MetricsController#index as HTML Completed 200 OK in 1ms (Views: 0.2ms | ActiveRecord: 0.0ms | Elasticsearch: 0.0ms)
==> /var/log/gitlab/gitlab-rails/production_json.log <== {"method":"GET","path":"/-/metrics","format":"html","controller":"MetricsController","action":"index","status":200,"duration":0.67,"view":0.2,"db":0.0,"time":"2020-02-06T18:11:43.346Z","params":[],"remote_ip":null,"user_id":null,"username":null,"ua":null,"queue_duration":null,"correlation_id":"1a85dde8-1421-4f0b-aa71-c2c0171aedfc","cpu_s":0.08272464899999932}
==> /var/log/gitlab/gitlab-rails/sidekiq_exporter.log <== [2020-02-06T18:11:49.257+0000] 127.0.0.1 - - [06/Feb/2020:18:11:49 UTC] "GET /metrics HTTP/1.1" 200 25 "-" "Prometheus/2.12.0"
==> /var/log/gitlab/gitlab-rails/production.log <== Started GET "/-/metrics" for 127.0.0.1 at 2020-02-06 18:11:58 +0000 Processing by MetricsController#index as HTML Completed 200 OK in 2ms (Views: 0.7ms | ActiveRecord: 0.0ms | Elasticsearch: 0.0ms)
==> /var/log/gitlab/gitlab-rails/production_json.log <== {"method":"GET","path":"/-/metrics","format":"html","controller":"MetricsController","action":"index","status":200,"duration":2.43,"view":0.65,"db":0.0,"time":"2020-02-06T18:11:58.264Z","params":[],"remote_ip":null,"user_id":null,"username":null,"ua":null,"queue_duration":null,"correlation_id":"7b11f9e1-9c9a-41fa-a978-39ce7f17428b","cpu_s":0.01492148700000051}
==> /var/log/gitlab/gitlab-rails/sidekiq_exporter.log <== [2020-02-06T18:12:04.258+0000] 127.0.0.1 - - [06/Feb/2020:18:12:04 UTC] "GET /metrics HTTP/1.1" 200 25 "-" "Prometheus/2.12.0"
==> /var/log/gitlab/gitlab-rails/production.log <== Started GET "/-/metrics" for 127.0.0.1 at 2020-02-06 18:12:13 +0000 Processing by MetricsController#index as HTML Completed 200 OK in 2ms (Views: 0.5ms | ActiveRecord: 0.0ms | Elasticsearch: 0.0ms)
==> /var/log/gitlab/gitlab-rails/production_json.log <== {"method":"GET","path":"/-/metrics","format":"html","controller":"MetricsController","action":"index","status":200,"duration":1.88,"view":0.48,"db":0.0,"time":"2020-02-06T18:12:13.314Z","params":[],"remote_ip":null,"user_id":null,"username":null,"ua":null,"queue_duration":null,"correlation_id":"e545487d-4289-44b8-8b43-25bcef03049e","cpu_s":0.05733492300000087}
==> /var/log/gitlab/gitlab-rails/sidekiq_exporter.log <== [2020-02-06T18:12:19.259+0000] 127.0.0.1 - - [06/Feb/2020:18:12:19 UTC] "GET /metrics HTTP/1.1" 200 25 "-" "Prometheus/2.12.0"
==> /var/log/gitlab/sidekiq/current <== {"severity":"INFO","time":"2020-02-06T18:12:25.862Z","queue":"cronjob:update_all_mirrors","class":"UpdateAllMirrorsWorker","retry":false,"queue_namespace":"cronjob","jid":"f56839664a51b74072145744","created_at":"2020-02-06T18:12:25.861Z","correlation_id":"d94a016368c9dd43839b4ce450ba491a","enqueued_at":"2020-02-06T18:12:25.861Z","pid":26165,"message":"UpdateAllMirrorsWorker JID-f56839664a51b74072145744: start","job_status":"start","scheduling_latency_s":0.000651} {"severity":"INFO","time":"2020-02-06T18:12:25.892Z","queue":"cronjob:geo_sidekiq_cron_config","class":"Geo::SidekiqCronConfigWorker","retry":false,"queue_namespace":"cronjob","jid":"18d124732842da6a3bf1edf0","created_at":"2020-02-06T18:12:25.891Z","correlation_id":"4e1dda72275dcc940e1f8233415aed83","enqueued_at":"2020-02-06T18:12:25.891Z","pid":26165,"message":"Geo::SidekiqCronConfigWorker JID-18d124732842da6a3bf1edf0: start","job_status":"start","scheduling_latency_s":0.000939} {"severity":"INFO","time":"2020-02-06T18:12:25.923Z","queue":"cronjob:update_all_mirrors","class":"UpdateAllMirrorsWorker","retry":false,"queue_namespace":"cronjob","jid":"f56839664a51b74072145744","created_at":"2020-02-06T18:12:25.861Z","correlation_id":"d94a016368c9dd43839b4ce450ba491a","enqueued_at":"2020-02-06T18:12:25.861Z","pid":26165,"message":"UpdateAllMirrorsWorker JID-f56839664a51b74072145744: done: 0.061097 sec","job_status":"done","scheduling_latency_s":0.000651,"duration":0.061097,"cpu_s":0.014321,"completed_at":"2020-02-06T18:12:25.923Z","db_duration":4.634611992514692,"db_duration_s":0.004634611992514692} {"severity":"INFO","time":"2020-02-06T18:12:25.923Z","queue":"cronjob:geo_sidekiq_cron_config","class":"Geo::SidekiqCronConfigWorker","retry":false,"queue_namespace":"cronjob","jid":"18d124732842da6a3bf1edf0","created_at":"2020-02-06T18:12:25.891Z","correlation_id":"4e1dda72275dcc940e1f8233415aed83","enqueued_at":"2020-02-06T18:12:25.891Z","pid":26165,"message":"Geo::SidekiqCronConfigWorker JID-18d124732842da6a3bf1edf0: done: 0.031458 sec","job_status":"done","scheduling_latency_s":0.000939,"duration":0.031458,"cpu_s":0.013015,"completed_at":"2020-02-06T18:12:25.923Z","db_duration":0,"db_duration_s":0}
==> /var/log/gitlab/gitlab-rails/sidekiq.log <== {"severity":"INFO","time":"2020-02-06T18:12:25.862Z","queue":"cronjob:update_all_mirrors","class":"UpdateAllMirrorsWorker","retry":false,"queue_namespace":"cronjob","jid":"f56839664a51b74072145744","created_at":"2020-02-06T18:12:25.861Z","correlation_id":"d94a016368c9dd43839b4ce450ba491a","enqueued_at":"2020-02-06T18:12:25.861Z","pid":26165,"message":"UpdateAllMirrorsWorker JID-f56839664a51b74072145744: start","job_status":"start","scheduling_latency_s":0.000651} {"severity":"INFO","time":"2020-02-06T18:12:25.892Z","queue":"cronjob:geo_sidekiq_cron_config","class":"Geo::SidekiqCronConfigWorker","retry":false,"queue_namespace":"cronjob","jid":"18d124732842da6a3bf1edf0","created_at":"2020-02-06T18:12:25.891Z","correlation_id":"4e1dda72275dcc940e1f8233415aed83","enqueued_at":"2020-02-06T18:12:25.891Z","pid":26165,"message":"Geo::SidekiqCronConfigWorker JID-18d124732842da6a3bf1edf0: start","job_status":"start","scheduling_latency_s":0.000939} {"severity":"INFO","time":"2020-02-06T18:12:25.923Z","queue":"cronjob:update_all_mirrors","class":"UpdateAllMirrorsWorker","retry":false,"queue_namespace":"cronjob","jid":"f56839664a51b74072145744","created_at":"2020-02-06T18:12:25.861Z","correlation_id":"d94a016368c9dd43839b4ce450ba491a","enqueued_at":"2020-02-06T18:12:25.861Z","pid":26165,"message":"UpdateAllMirrorsWorker JID-f56839664a51b74072145744: done: 0.061097 sec","job_status":"done","scheduling_latency_s":0.000651,"duration":0.061097,"cpu_s":0.014321,"completed_at":"2020-02-06T18:12:25.923Z","db_duration":4.634611992514692,"db_duration_s":0.004634611992514692} {"severity":"INFO","time":"2020-02-06T18:12:25.923Z","queue":"cronjob:geo_sidekiq_cron_config","class":"Geo::SidekiqCronConfigWorker","retry":false,"queue_namespace":"cronjob","jid":"18d124732842da6a3bf1edf0","created_at":"2020-02-06T18:12:25.891Z","correlation_id":"4e1dda72275dcc940e1f8233415aed83","enqueued_at":"2020-02-06T18:12:25.891Z","pid":26165,"message":"Geo::SidekiqCronConfigWorker JID-18d124732842da6a3bf1edf0: done: 0.031458 sec","job_status":"done","scheduling_latency_s":0.000939,"duration":0.031458,"cpu_s":0.013015,"completed_at":"2020-02-06T18:12:25.923Z","db_duration":0,"db_duration_s":0}
==> /var/log/gitlab/gitlab-rails/production.log <== Started GET "/-/metrics" for 127.0.0.1 at 2020-02-06 18:12:28 +0000 Processing by MetricsController#index as HTML Completed 200 OK in 2ms (Views: 0.5ms | ActiveRecord: 0.0ms | Elasticsearch: 0.0ms)
==> /var/log/gitlab/gitlab-rails/production_json.log <== {"method":"GET","path":"/-/metrics","format":"html","controller":"MetricsController","action":"index","status":200,"duration":1.76,"view":0.47,"db":0.0,"time":"2020-02-06T18:12:28.331Z","params":[],"remote_ip":null,"user_id":null,"username":null,"ua":null,"queue_duration":null,"correlation_id":"f22c3751-e4c6-4eaa-821f-b55bb170cf4d","cpu_s":0.07208011200000009}
==> /var/log/gitlab/gitlab-rails/sidekiq_exporter.log <== [2020-02-06T18:12:34.259+0000] 127.0.0.1 - - [06/Feb/2020:18:12:34 UTC] "GET /metrics HTTP/1.1" 200 25 "-" "Prometheus/2.12.0"
==> /var/log/gitlab/gitlab-rails/production.log <== Started GET "/-/metrics" for 127.0.0.1 at 2020-02-06 18:12:43 +0000 Processing by MetricsController#index as HTML Completed 200 OK in 2ms (Views: 0.6ms | ActiveRecord: 0.0ms | Elasticsearch: 0.0ms)
==> /var/log/gitlab/gitlab-rails/production_json.log <== {"method":"GET","path":"/-/metrics","format":"html","controller":"MetricsController","action":"index","status":200,"duration":2.34,"view":0.56,"db":0.0,"time":"2020-02-06T18:12:43.262Z","params":[],"remote_ip":null,"user_id":null,"username":null,"ua":null,"queue_duration":null,"correlation_id":"a9f9af10-0c5e-4bb4-a1df-fe6c5ab00c9f","cpu_s":0.012289731999999276} ==> /var/log/gitlab/gitlab-rails/sidekiq_exporter.log <== [2020-02-06T18:12:49.258+0000] 127.0.0.1 - - [06/Feb/2020:18:12:49 UTC] "GET /metrics HTTP/1.1" 200 25 "-" "Prometheus/2.12.0"
==> /var/log/gitlab/gitlab-rails/production.log <== Started GET "/-/metrics" for 127.0.0.1 at 2020-02-06 18:12:58 +0000 Processing by MetricsController#index as HTML Completed 200 OK in 2ms (Views: 0.7ms | ActiveRecord: 0.0ms | Elasticsearch: 0.0ms)
==> /var/log/gitlab/gitlab-rails/production_json.log <== {"method":"GET","path":"/-/metrics","format":"html","controller":"MetricsController","action":"index","status":200,"duration":2.58,"view":0.72,"db":0.0,"time":"2020-02-06T18:12:58.313Z","params":[],"remote_ip":null,"user_id":null,"username":null,"ua":null,"queue_duration":null,"correlation_id":"39ee6ff3-5ac1-4a66-9807-555aeb6260cf","cpu_s":0.05589346300000031}
==> /var/log/gitlab/redis/current <== 2020-02-06_18:13:01.02925 12284:M 06 Feb 18:13:01.029 * 10 changes in 300 seconds. Saving... 2020-02-06_18:13:01.02985 12284:M 06 Feb 18:13:01.029 * Background saving started by pid 14795 2020-02-06_18:13:01.05441 14795:C 06 Feb 18:13:01.054 * DB saved on disk 2020-02-06_18:13:01.05681 14795:C 06 Feb 18:13:01.056 * RDB: 1 MB of memory used by copy-on-write 2020-02-06_18:13:01.13088 12284:M 06 Feb 18:13:01.130 * Background saving terminated with success
==> /var/log/gitlab/gitlab-rails/sidekiq_exporter.log <== [2020-02-06T18:13:04.262+0000] 127.0.0.1 - - [06/Feb/2020:18:13:04 UTC] "GET /metrics HTTP/1.1" 200 25 "-" "Prometheus/2.12.0"
==> /var/log/gitlab/sidekiq/current <== {"severity":"INFO","time":"2020-02-06T18:13:11.045Z","queue":"cronjob:update_all_mirrors","class":"UpdateAllMirrorsWorker","retry":false,"queue_namespace":"cronjob","jid":"4c540e9e1713b86f96f6d4d8","created_at":"2020-02-06T18:13:11.044Z","correlation_id":"52187398456f4d1ae1f63221771f7ef0","enqueued_at":"2020-02-06T18:13:11.044Z","pid":26165,"message":"UpdateAllMirrorsWorker JID-4c540e9e1713b86f96f6d4d8: start","job_status":"start","scheduling_latency_s":0.000679} {"severity":"INFO","time":"2020-02-06T18:13:11.076Z","queue":"cronjob:geo_sidekiq_cron_config","class":"Geo::SidekiqCronConfigWorker","retry":false,"queue_namespace":"cronjob","jid":"e916b53d2fbc945ca361114f","created_at":"2020-02-06T18:13:11.075Z","correlation_id":"6120df3e6f02a64e2d5ef76224a1e1fb","enqueued_at":"2020-02-06T18:13:11.075Z","pid":26165,"message":"Geo::SidekiqCronConfigWorker JID-e916b53d2fbc945ca361114f: start","job_status":"start","scheduling_latency_s":0.000831} {"severity":"INFO","time":"2020-02-06T18:13:11.151Z","queue":"cronjob:geo_sidekiq_cron_config","class":"Geo::SidekiqCronConfigWorker","retry":false,"queue_namespace":"cronjob","jid":"e916b53d2fbc945ca361114f","created_at":"2020-02-06T18:13:11.075Z","correlation_id":"6120df3e6f02a64e2d5ef76224a1e1fb","enqueued_at":"2020-02-06T18:13:11.075Z","pid":26165,"message":"Geo::SidekiqCronConfigWorker JID-e916b53d2fbc945ca361114f: done: 0.074836 sec","job_status":"done","scheduling_latency_s":0.000831,"duration":0.074836,"cpu_s":0.052681,"completed_at":"2020-02-06T18:13:11.151Z","db_duration":0,"db_duration_s":0} {"severity":"INFO","time":"2020-02-06T18:13:11.151Z","queue":"cronjob:update_all_mirrors","class":"UpdateAllMirrorsWorker","retry":false,"queue_namespace":"cronjob","jid":"4c540e9e1713b86f96f6d4d8","created_at":"2020-02-06T18:13:11.044Z","correlation_id":"52187398456f4d1ae1f63221771f7ef0","enqueued_at":"2020-02-06T18:13:11.044Z","pid":26165,"message":"UpdateAllMirrorsWorker JID-4c540e9e1713b86f96f6d4d8: done: 0.105823 sec","job_status":"done","scheduling_latency_s":0.000679,"duration":0.105823,"cpu_s":0.014838,"completed_at":"2020-02-06T18:13:11.151Z","db_duration":52.863580000121146,"db_duration_s":0.052863580000121146}
==> /var/log/gitlab/gitlab-rails/sidekiq.log <== {"severity":"INFO","time":"2020-02-06T18:13:11.045Z","queue":"cronjob:update_all_mirrors","class":"UpdateAllMirrorsWorker","retry":false,"queue_namespace":"cronjob","jid":"4c540e9e1713b86f96f6d4d8","created_at":"2020-02-06T18:13:11.044Z","correlation_id":"52187398456f4d1ae1f63221771f7ef0","enqueued_at":"2020-02-06T18:13:11.044Z","pid":26165,"message":"UpdateAllMirrorsWorker JID-4c540e9e1713b86f96f6d4d8: start","job_status":"start","scheduling_latency_s":0.000679} {"severity":"INFO","time":"2020-02-06T18:13:11.076Z","queue":"cronjob:geo_sidekiq_cron_config","class":"Geo::SidekiqCronConfigWorker","retry":false,"queue_namespace":"cronjob","jid":"e916b53d2fbc945ca361114f","created_at":"2020-02-06T18:13:11.075Z","correlation_id":"6120df3e6f02a64e2d5ef76224a1e1fb","enqueued_at":"2020-02-06T18:13:11.075Z","pid":26165,"message":"Geo::SidekiqCronConfigWorker JID-e916b53d2fbc945ca361114f: start","job_status":"start","scheduling_latency_s":0.000831} {"severity":"INFO","time":"2020-02-06T18:13:11.151Z","queue":"cronjob:geo_sidekiq_cron_config","class":"Geo::SidekiqCronConfigWorker","retry":false,"queue_namespace":"cronjob","jid":"e916b53d2fbc945ca361114f","created_at":"2020-02-06T18:13:11.075Z","correlation_id":"6120df3e6f02a64e2d5ef76224a1e1fb","enqueued_at":"2020-02-06T18:13:11.075Z","pid":26165,"message":"Geo::SidekiqCronConfigWorker JID-e916b53d2fbc945ca361114f: done: 0.074836 sec","job_status":"done","scheduling_latency_s":0.000831,"duration":0.074836,"cpu_s":0.052681,"completed_at":"2020-02-06T18:13:11.151Z","db_duration":0,"db_duration_s":0} {"severity":"INFO","time":"2020-02-06T18:13:11.151Z","queue":"cronjob:update_all_mirrors","class":"UpdateAllMirrorsWorker","retry":false,"queue_namespace":"cronjob","jid":"4c540e9e1713b86f96f6d4d8","created_at":"2020-02-06T18:13:11.044Z","correlation_id":"52187398456f4d1ae1f63221771f7ef0","enqueued_at":"2020-02-06T18:13:11.044Z","pid":26165,"message":"UpdateAllMirrorsWorker JID-4c540e9e1713b86f96f6d4d8: done: 0.105823 sec","job_status":"done","scheduling_latency_s":0.000679,"duration":0.105823,"cpu_s":0.014838,"completed_at":"2020-02-06T18:13:11.151Z","db_duration":52.863580000121146,"db_duration_s":0.052863580000121146}
==> /var/log/gitlab/gitlab-rails/production.log <== Started GET "/-/metrics" for 127.0.0.1 at 2020-02-06 18:13:13 +0000 Processing by MetricsController#index as HTML Completed 200 OK in 1ms (Views: 0.3ms | ActiveRecord: 0.0ms | Elasticsearch: 0.0ms)
==> /var/log/gitlab/gitlab-rails/production_json.log <== {"method":"GET","path":"/-/metrics","format":"html","controller":"MetricsController","action":"index","status":200,"duration":1.55,"view":0.32,"db":0.0,"time":"2020-02-06T18:13:13.255Z","params":[],"remote_ip":null,"user_id":null,"username":null,"ua":null,"queue_duration":null,"correlation_id":"63c865d2-1e2a-4504-b432-905f17dafb50","cpu_s":0.006944471999998925}
==> /var/log/gitlab/gitlab-rails/sidekiq_exporter.log <== [2020-02-06T18:13:19.259+0000] 127.0.0.1 - - [06/Feb/2020:18:13:19 UTC] "GET /metrics HTTP/1.1" 200 25 "-" "Prometheus/2.12.0"
==> /var/log/gitlab/gitlab-rails/production.log <== Started GET "/-/metrics" for 127.0.0.1 at 2020-02-06 18:13:28 +0000 Processing by MetricsController#index as HTML Completed 200 OK in 2ms (Views: 0.6ms | ActiveRecord: 0.0ms | Elasticsearch: 0.0ms)
==> /var/log/gitlab/gitlab-rails/production_json.log <== {"method":"GET","path":"/-/metrics","format":"html","controller":"MetricsController","action":"index","status":200,"duration":2.36,"view":0.64,"db":0.0,"time":"2020-02-06T18:13:28.264Z","params":[],"remote_ip":null,"user_id":null,"username":null,"ua":null,"queue_duration":null,"correlation_id":"f77f0a44-ae49-4722-8de3-1a7d97301cb9","cpu_s":0.014677361000000388}
==> /var/log/gitlab/gitlab-rails/sidekiq_exporter.log <== [2020-02-06T18:13:34.258+0000] 127.0.0.1 - - [06/Feb/2020:18:13:34 UTC] "GET /metrics HTTP/1.1" 200 25 "-" "Prometheus/2.12.0"
==> /var/log/gitlab/gitlab-rails/production.log <== Started GET "/-/metrics" for 127.0.0.1 at 2020-02-06 18:13:43 +0000 Processing by MetricsController#index as HTML Completed 200 OK in 1ms (Views: 0.3ms | ActiveRecord: 0.0ms | Elasticsearch: 0.0ms)
==> /var/log/gitlab/gitlab-rails/production_json.log <== {"method":"GET","path":"/-/metrics","format":"html","controller":"MetricsController","action":"index","status":200,"duration":0.86,"view":0.25,"db":0.0,"time":"2020-02-06T18:13:43.316Z","params":[],"remote_ip":null,"user_id":null,"username":null,"ua":null,"queue_duration":null,"correlation_id":"c0ec4188-7d09-4b8b-8bc6-24fc0e20c3da","cpu_s":0.058388411999999335}
==> /var/log/gitlab/gitlab-rails/sidekiq_exporter.log <== [2020-02-06T18:13:49.259+0000] 127.0.0.1 - - [06/Feb/2020:18:13:49 UTC] "GET /metrics HTTP/1.1" 200 25 "-" "Prometheus/2.12.0"
==> /var/log/gitlab/gitlab-rails/production.log <== Started GET "/-/metrics" for 127.0.0.1 at 2020-02-06 18:13:58 +0000 Processing by MetricsController#index as HTML Completed 200 OK in 1ms (Views: 0.2ms | ActiveRecord: 0.0ms | Elasticsearch: 0.0ms) ==> /var/log/gitlab/gitlab-rails/production_json.log <== {"method":"GET","path":"/-/metrics","format":"html","controller":"MetricsController","action":"index","status":200,"duration":0.72,"view":0.19,"db":0.0,"time":"2020-02-06T18:13:58.264Z","params":[],"remote_ip":null,"user_id":null,"username":null,"ua":null,"queue_duration":null,"correlation_id":"8426f6a5-4a4e-43b4-b3fc-049ed3956fa9","cpu_s":0.014707953000000273}
==> /var/log/gitlab/gitlab-rails/sidekiq_exporter.log <== [2020-02-06T18:14:04.259+0000] 127.0.0.1 - - [06/Feb/2020:18:14:04 UTC] "GET /metrics HTTP/1.1" 200 25 "-" "Prometheus/2.12.0"
==> /var/log/gitlab/sidekiq/current <== {"severity":"INFO","time":"2020-02-06T18:14:10.112Z","queue":"cronjob:update_all_mirrors","class":"UpdateAllMirrorsWorker","retry":false,"queue_namespace":"cronjob","jid":"256bdaa373ebd06032ce9b76","created_at":"2020-02-06T18:14:10.111Z","correlation_id":"996c1c75aed88f249543cafb3d5c7e46","enqueued_at":"2020-02-06T18:14:10.112Z","pid":26165,"message":"UpdateAllMirrorsWorker JID-256bdaa373ebd06032ce9b76: start","job_status":"start","scheduling_latency_s":0.000687} {"severity":"INFO","time":"2020-02-06T18:14:10.146Z","queue":"cronjob:geo_sidekiq_cron_config","class":"Geo::SidekiqCronConfigWorker","retry":false,"queue_namespace":"cronjob","jid":"97640275aa0055a36c630a53","created_at":"2020-02-06T18:14:10.144Z","correlation_id":"1c461fb2e343d561edc65dae9f6014eb","enqueued_at":"2020-02-06T18:14:10.144Z","pid":26165,"message":"Geo::SidekiqCronConfigWorker JID-97640275aa0055a36c630a53: start","job_status":"start","scheduling_latency_s":0.002089} {"severity":"INFO","time":"2020-02-06T18:14:10.184Z","queue":"cronjob:geo_sidekiq_cron_config","class":"Geo::SidekiqCronConfigWorker","retry":false,"queue_namespace":"cronjob","jid":"97640275aa0055a36c630a53","created_at":"2020-02-06T18:14:10.144Z","correlation_id":"1c461fb2e343d561edc65dae9f6014eb","enqueued_at":"2020-02-06T18:14:10.144Z","pid":26165,"message":"Geo::SidekiqCronConfigWorker JID-97640275aa0055a36c630a53: done: 0.037293 sec","job_status":"done","scheduling_latency_s":0.002089,"duration":0.037293,"cpu_s":0.0134,"completed_at":"2020-02-06T18:14:10.184Z","db_duration":0,"db_duration_s":0} {"severity":"INFO","time":"2020-02-06T18:14:10.185Z","queue":"cronjob:update_all_mirrors","class":"UpdateAllMirrorsWorker","retry":false,"queue_namespace":"cronjob","jid":"256bdaa373ebd06032ce9b76","created_at":"2020-02-06T18:14:10.111Z","correlation_id":"996c1c75aed88f249543cafb3d5c7e46","enqueued_at":"2020-02-06T18:14:10.112Z","pid":26165,"message":"UpdateAllMirrorsWorker JID-256bdaa373ebd06032ce9b76: done: 0.072956 sec","job_status":"done","scheduling_latency_s":0.000687,"duration":0.072956,"cpu_s":0.021063,"completed_at":"2020-02-06T18:14:10.185Z","db_duration":13.41296399186831,"db_duration_s":0.01341296399186831}
==> /var/log/gitlab/gitlab-rails/sidekiq.log <== {"severity":"INFO","time":"2020-02-06T18:14:10.112Z","queue":"cronjob:update_all_mirrors","class":"UpdateAllMirrorsWorker","retry":false,"queue_namespace":"cronjob","jid":"256bdaa373ebd06032ce9b76","created_at":"2020-02-06T18:14:10.111Z","correlation_id":"996c1c75aed88f249543cafb3d5c7e46","enqueued_at":"2020-02-06T18:14:10.112Z","pid":26165,"message":"UpdateAllMirrorsWorker JID-256bdaa373ebd06032ce9b76: start","job_status":"start","scheduling_latency_s":0.000687} {"severity":"INFO","time":"2020-02-06T18:14:10.146Z","queue":"cronjob:geo_sidekiq_cron_config","class":"Geo::SidekiqCronConfigWorker","retry":false,"queue_namespace":"cronjob","jid":"97640275aa0055a36c630a53","created_at":"2020-02-06T18:14:10.144Z","correlation_id":"1c461fb2e343d561edc65dae9f6014eb","enqueued_at":"2020-02-06T18:14:10.144Z","pid":26165,"message":"Geo::SidekiqCronConfigWorker JID-97640275aa0055a36c630a53: start","job_status":"start","scheduling_latency_s":0.002089} {"severity":"INFO","time":"2020-02-06T18:14:10.184Z","queue":"cronjob:geo_sidekiq_cron_config","class":"Geo::SidekiqCronConfigWorker","retry":false,"queue_namespace":"cronjob","jid":"97640275aa0055a36c630a53","created_at":"2020-02-06T18:14:10.144Z","correlation_id":"1c461fb2e343d561edc65dae9f6014eb","enqueued_at":"2020-02-06T18:14:10.144Z","pid":26165,"message":"Geo::SidekiqCronConfigWorker JID-97640275aa0055a36c630a53: done: 0.037293 sec","job_status":"done","scheduling_latency_s":0.002089,"duration":0.037293,"cpu_s":0.0134,"completed_at":"2020-02-06T18:14:10.184Z","db_duration":0,"db_duration_s":0} {"severity":"INFO","time":"2020-02-06T18:14:10.185Z","queue":"cronjob:update_all_mirrors","class":"UpdateAllMirrorsWorker","retry":false,"queue_namespace":"cronjob","jid":"256bdaa373ebd06032ce9b76","created_at":"2020-02-06T18:14:10.111Z","correlation_id":"996c1c75aed88f249543cafb3d5c7e46","enqueued_at":"2020-02-06T18:14:10.112Z","pid":26165,"message":"UpdateAllMirrorsWorker JID-256bdaa373ebd06032ce9b76: done: 0.072956 sec","job_status":"done","scheduling_latency_s":0.000687,"duration":0.072956,"cpu_s":0.021063,"completed_at":"2020-02-06T18:14:10.185Z","db_duration":13.41296399186831,"db_duration_s":0.01341296399186831}
==> /var/log/gitlab/gitlab-rails/production.log <== Started GET "/-/metrics" for 127.0.0.1 at 2020-02-06 18:14:13 +0000 Processing by MetricsController#index as HTML Completed 200 OK in 1ms (Views: 0.4ms | ActiveRecord: 0.0ms | Elasticsearch: 0.0ms)
==> /var/log/gitlab/gitlab-rails/production_json.log <== {"method":"GET","path":"/-/metrics","format":"html","controller":"MetricsController","action":"index","status":200,"duration":1.21,"view":0.39,"db":0.0,"time":"2020-02-06T18:14:13.317Z","params":[],"remote_ip":null,"user_id":null,"username":null,"ua":null,"queue_duration":null,"correlation_id":"0b00ded6-18b7-4406-82e8-f790a314860e","cpu_s":0.05531247300000075}
Output of checks
Results of GitLab environment info
Expand for output related to GitLab environment info
System information System: Ubuntu 18.04 Proxy: no Current User: git Using RVM: no Ruby Version: 2.6.5p114 Gem Version: 2.7.10 Bundler Version:1.17.3 Rake Version: 12.3.3 Redis Version: 5.0.7 Git Version: 2.24.1 Sidekiq Version:5.2.7 Go Version: unknownGitLab information Version: 12.7.5-ee Revision: 19edff260da Directory: /opt/gitlab/embedded/service/gitlab-rails DB Adapter: PostgreSQL DB Version: 10.9 URL: https://redacted HTTP Clone URL: https://redacted/some-group/some-project.git SSH Clone URL: git@redacted:some-group/some-project.git Elasticsearch: no Geo: no Using LDAP: no Using Omniauth: yes Omniauth Providers:
GitLab Shell Version: 11.0.0 Repository storage paths:
- default: /media/share/git-data/repositories GitLab Shell path: /opt/gitlab/embedded/service/gitlab-shell Git: /opt/gitlab/embedded/bin/git
Results of GitLab application Check
Expand for output related to the GitLab application check
Checking GitLab subtasks ...Checking GitLab Shell ...
GitLab Shell: ... GitLab Shell version >= 11.0.0 ? ... OK (11.0.0) Running /opt/gitlab/embedded/service/gitlab-shell/bin/check Internal API available: OK Redis available via internal API: OK gitlab-shell self-check successful
Checking GitLab Shell ... Finished
Checking Gitaly ...
Gitaly: ... default ... OK
Checking Gitaly ... Finished
Checking Sidekiq ...
Sidekiq: ... Running? ... yes Number of Sidekiq processes ... 1
Checking Sidekiq ... Finished
Checking Incoming Email ...
Incoming Email: ... Reply by email is disabled in config/gitlab.yml
Checking Incoming Email ... Finished
Checking LDAP ...
LDAP: ... LDAP is disabled in config/gitlab.yml
Checking LDAP ... Finished
Checking GitLab App ...
Git configured correctly? ... yes Database config exists? ... yes All migrations up? ... yes Database contains orphaned GroupMembers? ... no GitLab config exists? ... yes GitLab config up to date? ... yes Log directory writable? ... yes Tmp directory writable? ... yes Uploads directory exists? ... yes Uploads directory has correct permissions? ... yes Uploads directory tmp has correct permissions? ... yes Init script exists? ... skipped (omnibus-gitlab has no init script) Init script up-to-date? ... skipped (omnibus-gitlab has no init script) Projects have namespace: ... 1/2 ... yes 2/4 ... yes 1/5 ... yes 2/6 ... yes 2/13 ... yes 2/17 ... yes 1/18 ... yes 1/19 ... yes 2/20 ... yes 2/21 ... yes 4/24 ... yes 1/25 ... yes Redis version >= 2.8.0? ... yes Ruby version >= 2.5.3 ? ... yes (2.6.5) Git version >= 2.22.0 ? ... yes (2.24.1) Git user has default SSH configuration? ... yes Active users: ... 9 Is authorized keys file accessible? ... yes Elasticsearch version 5.6 - 6.x? ... skipped (elasticsearch is disabled)
Checking GitLab App ... Finished
Checking GitLab subtasks ... Finished
I could provide container file or let somebody from the team access server and LXD containers (with Gitlab and Apache)