Push Event WebHooks fail when tested and do not work
Summary
Project webhooks do not work.
Steps to reproduce
This webhook is for Puppet Enterprise Code Manager
I set up a hook as described here:
https://puppet.com/docs/pe/2019.1/code_mgr_webhook.html#triggering-code-manager-with-a-webhook
What is the current bug behavior?
Clicking "Test" give me an OOPS/500 screen and a 503-Service Unavailable in the logs.
A test commit to a branch produces the same 503-Service Unavailable in the logs.
What is the expected correct behavior?
The test should succees The commit should trigger Puppet to update its code.
Relevant logs
Relevant logs from Test
==> /var/log/gitlab/gitlab-rails/production.log <== Completed 500 Internal Server Error in 157ms (ActiveRecord: 7.4ms | Elasticsearch: 0.0ms)Net::HTTPFatalError (503 "Service Unavailable"):
lib/gitlab/http.rb:24:in
perform_request' app/services/web_hook_service.rb:77:inmake_request' app/services/web_hook_service.rb:30:inexecute' app/models/hooks/web_hook.rb:26:inexecute' app/services/test_hooks/base_service.rb:24:inblock in execute' app/services/test_hooks/base_service.rb:21:incatch' app/services/test_hooks/base_service.rb:21:inexecute' lib/gitlab/metrics/instrumentation.rb:161:inblock in execute' lib/gitlab/metrics/method_call.rb:36:inmeasure' lib/gitlab/metrics/instrumentation.rb:161:inexecute' app/controllers/projects/hooks_controller.rb:43:intest' ee/lib/gitlab/ip_address_state.rb:10:inwith' ee/app/controllers/ee/application_controller.rb:46:inset_current_ip_address' lib/gitlab/session.rb:11:inwith_session' app/controllers/application_controller.rb:461:inset_session_storage' lib/gitlab/i18n.rb:55:inwith_locale' lib/gitlab/i18n.rb:61:inwith_user_locale' app/controllers/application_controller.rb:455:inset_locale' lib/gitlab/middleware/rails_queue_duration.rb:27:incall' lib/gitlab/metrics/rack_middleware.rb:17:inblock in call' lib/gitlab/metrics/transaction.rb:62:inrun' lib/gitlab/metrics/rack_middleware.rb:17:incall' lib/gitlab/request_profiler/middleware.rb:17:incall' ee/lib/gitlab/jira/middleware.rb:19:incall' lib/gitlab/middleware/go.rb:20:incall' lib/gitlab/etag_caching/middleware.rb:13:incall' lib/gitlab/middleware/correlation_id.rb:16:inblock in call' lib/gitlab/middleware/correlation_id.rb:15:incall' lib/gitlab/middleware/multipart.rb:117:incall' lib/gitlab/middleware/read_only/controller.rb:48:incall' lib/gitlab/middleware/read_only.rb:18:incall' lib/gitlab/middleware/basic_health_check.rb:25:incall' lib/gitlab/request_context.rb:32:incall' lib/gitlab/metrics/requests_rack_middleware.rb:49:incall' lib/gitlab/middleware/release_env.rb:12:in `call'==> /var/log/gitlab/gitlab-rails/production_json.log <== {"method":"POST","path":"/puppet/control-repo/hooks/1/test","format":"html","controller":"Projects::HooksController","action":"test","status":500,"error":"Net::HTTPFatalError: 503 "Service Unavailable"","duration":157.19,"view":0.0,"db":7.43,"time":"2019-10-23T11:49:43.473Z","params":[{"key":"_method","value":"post"},{"key":"authenticity_token","value":"[FILTERED]"},{"key":"trigger","value":"push_events"},{"key":"namespace_id","value":"puppet"},{"key":"project_id","value":"control-repo"},{"key":"id","value":"1"}],"remote_ip":"198.118.208.75","user_id":2,"username":"ME","ua":"Mozilla/5.0 (X11; Linux x86_64; rv:60.0) Gecko/20100101 Firefox/60.0","queue_duration":16.27,"gitaly_calls":4,"gitaly_duration":8.13,"correlation_id":"grASU0KyxV2","cpu_s":0.10970030299999678}
==> /var/log/gitlab/gitaly/current <== {"correlation_id":"grASU0KyxV2","grpc.code":"OK","grpc.meta.auth_version":"v2","grpc.meta.client_name":"gitlab-web","grpc.method":"FindCommits","grpc.request.deadline":"2019-10-23T11:50:13Z","grpc.request.fullMethod":"/gitaly.CommitService/FindCommits","grpc.request.glProjectPath":"puppet/control-repo","grpc.request.glRepository":"project-14","grpc.request.repoPath":"puppet/control-repo.git","grpc.request.repoStorage":"default","grpc.request.topLevelGroup":"puppet","grpc.service":"gitaly.CommitService","grpc.start_time":"2019-10-23T11:49:43Z","grpc.time_ms":23.37,"level":"info","msg":"finished streaming call with code OK","peer.address":"@","pid":6283,"span.kind":"server","system":"grpc","time":"2019-10-23T11:49:43Z"} {"correlation_id":"grASU0KyxV2","grpc.code":"OK","grpc.meta.auth_version":"v2","grpc.meta.client_name":"gitlab-web","grpc.method":"CommitDelta","grpc.request.deadline":"2019-10-23T11:49:53Z","grpc.request.fullMethod":"/gitaly.DiffService/CommitDelta","grpc.request.glProjectPath":"puppet/control-repo","grpc.request.glRepository":"project-14","grpc.request.repoPath":"puppet/control-repo.git","grpc.request.repoStorage":"default","grpc.request.topLevelGroup":"puppet","grpc.service":"gitaly.DiffService","grpc.start_time":"2019-10-23T11:49:43Z","grpc.time_ms":24.43,"level":"info","msg":"finished streaming call with code OK","peer.address":"@","pid":6283,"span.kind":"server","system":"grpc","time":"2019-10-23T11:49:43Z"} {"correlation_id":"grASU0KyxV2","grpc.code":"OK","grpc.meta.auth_version":"v2","grpc.meta.client_name":"gitlab-web","grpc.method":"CommitDelta","grpc.request.deadline":"2019-10-23T11:49:53Z","grpc.request.fullMethod":"/gitaly.DiffService/CommitDelta","grpc.request.glProjectPath":"puppet/control-repo","grpc.request.glRepository":"project-14","grpc.request.repoPath":"puppet/control-repo.git","grpc.request.repoStorage":"default","grpc.request.topLevelGroup":"puppet","grpc.service":"gitaly.DiffService","grpc.start_time":"2019-10-23T11:49:43Z","grpc.time_ms":10.264,"level":"info","msg":"finished streaming call with code OK","peer.address":"@","pid":6283,"span.kind":"server","system":"grpc","time":"2019-10-23T11:49:43Z"} {"correlation_id":"grASU0KyxV2","grpc.code":"OK","grpc.meta.auth_version":"v2","grpc.meta.client_name":"gitlab-web","grpc.method":"CommitDelta","grpc.request.deadline":"2019-10-23T11:49:53Z","grpc.request.fullMethod":"/gitaly.DiffService/CommitDelta","grpc.request.glProjectPath":"puppet/control-repo","grpc.request.glRepository":"project-14","grpc.request.repoPath":"puppet/control-repo.git","grpc.request.repoStorage":"default","grpc.request.topLevelGroup":"puppet","grpc.service":"gitaly.DiffService","grpc.start_time":"2019-10-23T11:49:43Z","grpc.time_ms":4.901,"level":"info","msg":"finished streaming call with code OK","peer.address":"@","pid":6283,"span.kind":"server","system":"grpc","time":"2019-10-23T11:49:43Z"}
==> /var/log/gitlab/gitlab-workhorse/current <== {"correlation_id":"grASU0KyxV2","duration_ms":202,"host":"gitlab-server.foo.org","level":"info","method":"POST","msg":"access","proto":"HTTP/1.1","referrer":"https://gitlab-server.foo.org/puppet/control-repo/-/settings/integrations","remote_addr":"198.118.208.75:0","remote_ip":"198.118.208.75","status":500,"system":"http","time":"2019-10-23T07:49:43-04:00","uri":"/puppet/control-repo/hooks/1/test?trigger=push_events","user_agent":"Mozilla/5.0 (X11; Linux x86_64; rv:60.0) Gecko/20100101 Firefox/60.0","written_bytes":2926} {"correlation_id":"iZgTcCRLri3","error":"badgateway: failed after 0s: context canceled","level":"error","method":"GET","msg":"error","time":"2019-10-23T07:49:43-04:00","uri":"/favicon.ico"} {"correlation_id":"iZgTcCRLri3","duration_ms":20,"host":"gitlab-server.foo.org","level":"info","method":"GET","msg":"access","proto":"HTTP/1.1","referrer":"","remote_addr":"198.118.208.75:0","remote_ip":"198.118.208.75","status":502,"system":"http","time":"2019-10-23T07:49:43-04:00","uri":"/favicon.ico","user_agent":"Mozilla/5.0 (X11; Linux x86_64; rv:60.0) Gecko/20100101 Firefox/60.0","written_bytes":2940}
==> /var/log/gitlab/nginx/gitlab_access.log <== 198.118.208.75 - - [23/Oct/2019:07:49:43 -0400] "POST /puppet/control-repo/hooks/1/test?trigger=push_events HTTP/2.0" 500 2926 "https://gitlab-server.foo.org/puppet/control-repo/-/settings/integrations" "Mozilla/5.0 (X11; Linux x86_64; rv:60.0) Gecko/20100101 Firefox/60.0" 198.118.208.75 - - [23/Oct/2019:07:49:43 -0400] "GET /favicon.ico HTTP/2.0" 499 0 "" "Mozilla/5.0 (X11; Linux x86_64; rv:60.0) Gecko/20100101 Firefox/60.0"
Relevant logs from commit
==> /var/log/gitlab/unicorn/current <== 2019-10-22_15:57:16.15261 adopted new unicorn master 6005 2019-10-22_15:57:37.22346 Received TERM from runit, sending to process group (-PID) 2019-10-22_15:57:37.22593 Terminated 2019-10-22_15:57:37.22630 forwarding TERM to unicorn master 6005 2019-10-22_15:57:37.25069 Waiting for unicorn to die... 2019-10-22_15:57:38.23116 forwarding CONT to unicorn master 6005 2019-10-22_15:57:38.23119 /opt/gitlab/embedded/bin/gitlab-unicorn-wrapper: line 132: kill: (6005) - No such process 2019-10-22_15:57:38.23120 wrapper for unicorn master 6005 exiting 2019-10-22_15:58:02.89093 starting new unicorn master 2019-10-22_15:58:59.31152 adopted new unicorn master 6541==> /var/log/gitlab/sidekiq/state <==
==> /var/log/gitlab/sidekiq/current <== {"severity":"INFO","time":"2019-10-23T12:34:30.376Z","queue":"cronjob:geo_sidekiq_cron_config","class":"Geo::SidekiqCronConfigWorker","retry":false,"queue_namespace":"cronjob","jid":"7d440a3279329c2741113271","created_at":"2019-10-23T12:34:29.920821Z","correlation_id":"b2348c06-8c13-475c-9a07-6bf4ebae10ef","enqueued_at":"2019-10-23T12:34:29.921188Z","pid":6504,"message":"Geo::SidekiqCronConfigWorker JID-7d440a3279329c2741113271: done: 0.453645 sec","job_status":"done","scheduling_latency_s":0.002323,"duration":0.453645,"cpu_s":0.035731,"completed_at":"2019-10-23T12:34:30.376885Z"} {"severity":"INFO","time":"2019-10-23T12:34:30.386Z","queue":"cronjob:update_all_mirrors","class":"UpdateAllMirrorsWorker","retry":false,"queue_namespace":"cronjob","jid":"5c8b9ef606c878682a2fc343","created_at":"2019-10-23T12:34:30.278824Z","correlation_id":"6e8c0f50-e252-4a81-be45-f7d32319830b","enqueued_at":"2019-10-23T12:34:30.282912Z","pid":6504,"message":"UpdateAllMirrorsWorker JID-5c8b9ef606c878682a2fc343: done: 0.100542 sec","job_status":"done","scheduling_latency_s":0.003621,"duration":0.100542,"cpu_s":0.030579,"completed_at":"2019-10-23T12:34:30.386814Z"} {"severity":"INFO","time":"2019-10-23T12:35:30.280Z","queue":"cronjob:geo_sidekiq_cron_config","class":"Geo::SidekiqCronConfigWorker","retry":false,"queue_namespace":"cronjob","jid":"e91d071fd5f901fbf211b891","created_at":"2019-10-23T12:35:30.277948Z","correlation_id":"e9bb048a-12e5-4ae9-9e4e-60b1bb797eb2","enqueued_at":"2019-10-23T12:35:30.278513Z","pid":6504,"message":"Geo::SidekiqCronConfigWorker JID-e91d071fd5f901fbf211b891: start","job_status":"start","scheduling_latency_s":0.002155} {"severity":"INFO","time":"2019-10-23T12:35:30.649Z","queue":"cronjob:update_all_mirrors","class":"UpdateAllMirrorsWorker","retry":false,"queue_namespace":"cronjob","jid":"452a2a2f44be95068bfdbd5d","created_at":"2019-10-23T12:35:30.645386Z","correlation_id":"0d10f6d9-95e7-4fbf-a2e2-6a46bb0daaaa","enqueued_at":"2019-10-23T12:35:30.645879Z","pid":6504,"message":"UpdateAllMirrorsWorker JID-452a2a2f44be95068bfdbd5d: start","job_status":"start","scheduling_latency_s":0.003747} {"severity":"INFO","time":"2019-10-23T12:35:30.752Z","queue":"cronjob:update_all_mirrors","class":"UpdateAllMirrorsWorker","retry":false,"queue_namespace":"cronjob","jid":"452a2a2f44be95068bfdbd5d","created_at":"2019-10-23T12:35:30.645386Z","correlation_id":"0d10f6d9-95e7-4fbf-a2e2-6a46bb0daaaa","enqueued_at":"2019-10-23T12:35:30.645879Z","pid":6504,"message":"UpdateAllMirrorsWorker JID-452a2a2f44be95068bfdbd5d: done: 0.102961 sec","job_status":"done","scheduling_latency_s":0.003747,"duration":0.102961,"cpu_s":0.0305,"completed_at":"2019-10-23T12:35:30.752350Z"} {"severity":"INFO","time":"2019-10-23T12:35:30.753Z","queue":"cronjob:geo_sidekiq_cron_config","class":"Geo::SidekiqCronConfigWorker","retry":false,"queue_namespace":"cronjob","jid":"e91d071fd5f901fbf211b891","created_at":"2019-10-23T12:35:30.277948Z","correlation_id":"e9bb048a-12e5-4ae9-9e4e-60b1bb797eb2","enqueued_at":"2019-10-23T12:35:30.278513Z","pid":6504,"message":"Geo::SidekiqCronConfigWorker JID-e91d071fd5f901fbf211b891: done: 0.473359 sec","job_status":"done","scheduling_latency_s":0.002155,"duration":0.473359,"cpu_s":0.035746,"completed_at":"2019-10-23T12:35:30.753736Z"} {"severity":"INFO","time":"2019-10-23T12:36:02.743Z","queue":"cronjob:geo_sidekiq_cron_config","class":"Geo::SidekiqCronConfigWorker","retry":false,"queue_namespace":"cronjob","jid":"5db3962b4c289cdac9fb8952","created_at":"2019-10-23T12:36:02.740637Z","correlation_id":"cd0ab9dd-ab34-453e-877b-80e9c8e1868f","enqueued_at":"2019-10-23T12:36:02.741102Z","pid":6504,"message":"Geo::SidekiqCronConfigWorker JID-5db3962b4c289cdac9fb8952: start","job_status":"start","scheduling_latency_s":0.002513} {"severity":"INFO","time":"2019-10-23T12:36:03.104Z","queue":"cronjob:update_all_mirrors","class":"UpdateAllMirrorsWorker","retry":false,"queue_namespace":"cronjob","jid":"a5eb0a0824090fb6fbb66c85","created_at":"2019-10-23T12:36:03.099559Z","correlation_id":"5a9a0547-b88b-40ed-801d-879934b03e3e","enqueued_at":"2019-10-23T12:36:03.100282Z","pid":6504,"message":"UpdateAllMirrorsWorker JID-a5eb0a0824090fb6fbb66c85: start","job_status":"start","scheduling_latency_s":0.00409} {"severity":"INFO","time":"2019-10-23T12:36:03.202Z","queue":"cronjob:geo_sidekiq_cron_config","class":"Geo::SidekiqCronConfigWorker","retry":false,"queue_namespace":"cronjob","jid":"5db3962b4c289cdac9fb8952","created_at":"2019-10-23T12:36:02.740637Z","correlation_id":"cd0ab9dd-ab34-453e-877b-80e9c8e1868f","enqueued_at":"2019-10-23T12:36:02.741102Z","pid":6504,"message":"Geo::SidekiqCronConfigWorker JID-5db3962b4c289cdac9fb8952: done: 0.459346 sec","job_status":"done","scheduling_latency_s":0.002513,"duration":0.459346,"cpu_s":0.034562,"completed_at":"2019-10-23T12:36:03.202695Z"} {"severity":"INFO","time":"2019-10-23T12:36:03.210Z","queue":"cronjob:update_all_mirrors","class":"UpdateAllMirrorsWorker","retry":false,"queue_namespace":"cronjob","jid":"a5eb0a0824090fb6fbb66c85","created_at":"2019-10-23T12:36:03.099559Z","correlation_id":"5a9a0547-b88b-40ed-801d-879934b03e3e","enqueued_at":"2019-10-23T12:36:03.100282Z","pid":6504,"message":"UpdateAllMirrorsWorker JID-a5eb0a0824090fb6fbb66c85: done: 0.106178 sec","job_status":"done","scheduling_latency_s":0.00409,"duration":0.106178,"cpu_s":0.028911,"completed_at":"2019-10-23T12:36:03.210321Z"}
==> /var/log/gitlab/gitlab-workhorse/current <== {"correlation_id":"wVGWa6NlmKa","duration_ms":857,"host":"gitlab-server.DOMAIN","level":"info","method":"GET","msg":"access","proto":"HTTP/1.1","referrer":"https://gitlab-server.DOMAIN/puppet/control-repo/tree/dev","remote_addr":"1.2.3.4:0","remote_ip":"1.2.3.4","status":200,"system":"http","time":"2019-10-23T08:36:00-04:00","uri":"/puppet/control-repo/blob/dev/README.md","user_agent":"Mozilla/5.0 (X11; Linux x86_64; rv:60.0) Gecko/20100101 Firefox/60.0","written_bytes":48786} {"correlation_id":"r8NN4p7hFe9","encoding":"gzip","file":"/opt/gitlab/embedded/service/gitlab-rails/public/assets/webpack/pages.projects.blob.show.22f05ee0.chunk.js","level":"info","method":"GET","msg":"Send static file","time":"2019-10-23T08:36:00-04:00","uri":"/assets/webpack/pages.projects.blob.show.22f05ee0.chunk.js"} {"correlation_id":"r8NN4p7hFe9","duration_ms":4,"host":"gitlab-server.DOMAIN","level":"info","method":"GET","msg":"access","proto":"HTTP/1.1","referrer":"https://gitlab-server.DOMAIN/puppet/control-repo/blob/dev/README.md","remote_addr":"1.2.3.4:0","remote_ip":"1.2.3.4","status":200,"system":"http","time":"2019-10-23T08:36:00-04:00","uri":"/assets/webpack/pages.projects.blob.show.22f05ee0.chunk.js","user_agent":"Mozilla/5.0 (X11; Linux x86_64; rv:60.0) Gecko/20100101 Firefox/60.0","written_bytes":12780} {"correlation_id":"NDr4y3P4fU7","duration_ms":138,"host":"gitlab-server.DOMAIN","level":"info","method":"GET","msg":"access","proto":"HTTP/1.1","referrer":"https://gitlab-server.DOMAIN/puppet/control-repo/blob/dev/README.md","remote_addr":"1.2.3.4:0","remote_ip":"1.2.3.4","status":200,"system":"http","time":"2019-10-23T08:36:01-04:00","uri":"/puppet/control-repo/commits/4f4baaf55ed7b684f609d4723ac725adfda1e124/signatures?limit=1","user_agent":"Mozilla/5.0 (X11; Linux x86_64; rv:60.0) Gecko/20100101 Firefox/60.0","written_bytes":17} {"correlation_id":"HAKllAlCU63","duration_ms":247,"host":"gitlab-server.DOMAIN","level":"info","method":"GET","msg":"access","proto":"HTTP/1.1","referrer":"https://gitlab-server.DOMAIN/puppet/control-repo/blob/dev/README.md","remote_addr":"1.2.3.4:0","remote_ip":"1.2.3.4","status":304,"system":"http","time":"2019-10-23T08:36:01-04:00","uri":"/puppet/control-repo/blob/dev/README.md?format=json\u0026viewer=rich","user_agent":"Mozilla/5.0 (X11; Linux x86_64; rv:60.0) Gecko/20100101 Firefox/60.0","written_bytes":0} {"correlation_id":"CRcH0X4f9V9","duration_ms":464,"host":"gitlab-server.DOMAIN","level":"info","method":"GET","msg":"access","proto":"HTTP/1.1","referrer":"https://gitlab-server.DOMAIN/puppet/control-repo/blob/dev/README.md","remote_addr":"1.2.3.4:0","remote_ip":"1.2.3.4","status":200,"system":"http","time":"2019-10-23T08:36:04-04:00","uri":"/puppet/control-repo/edit/dev/README.md","user_agent":"Mozilla/5.0 (X11; Linux x86_64; rv:60.0) Gecko/20100101 Firefox/60.0","written_bytes":68994} {"correlation_id":"E91DRMslmp2","encoding":"gzip","file":"/opt/gitlab/embedded/service/gitlab-rails/public/assets/webpack/pages.projects.blob.edit.b8426cf9.chunk.js","level":"info","method":"GET","msg":"Send static file","time":"2019-10-23T08:36:04-04:00","uri":"/assets/webpack/pages.projects.blob.edit.b8426cf9.chunk.js"} {"correlation_id":"E91DRMslmp2","duration_ms":1,"host":"gitlab-server.DOMAIN","level":"info","method":"GET","msg":"access","proto":"HTTP/1.1","referrer":"https://gitlab-server.DOMAIN/puppet/control-repo/edit/dev/README.md","remote_addr":"1.2.3.4:0","remote_ip":"1.2.3.4","status":200,"system":"http","time":"2019-10-23T08:36:04-04:00","uri":"/assets/webpack/pages.projects.blob.edit.b8426cf9.chunk.js","user_agent":"Mozilla/5.0 (X11; Linux x86_64; rv:60.0) Gecko/20100101 Firefox/60.0","written_bytes":7829} {"correlation_id":"L70WBH7G931","encoding":"gzip","file":"/opt/gitlab/embedded/service/gitlab-rails/public/assets/ace/mode-markdown-c33305e8a6835b17c07f224eace192a8931be0f70608f2c596ce66e711f6c4d4.js","level":"info","method":"GET","msg":"Send static file","time":"2019-10-23T08:36:04-04:00","uri":"/assets/ace/mode-markdown-c33305e8a6835b17c07f224eace192a8931be0f70608f2c596ce66e711f6c4d4.js"} {"correlation_id":"L70WBH7G931","duration_ms":1,"host":"gitlab-server.DOMAIN","level":"info","method":"GET","msg":"access","proto":"HTTP/1.1","referrer":"https://gitlab-server.DOMAIN/puppet/control-repo/edit/dev/README.md","remote_addr":"1.2.3.4:0","remote_ip":"1.2.3.4","status":200,"system":"http","time":"2019-10-23T08:36:04-04:00","uri":"/assets/ace/mode-markdown-c33305e8a6835b17c07f224eace192a8931be0f70608f2c596ce66e711f6c4d4.js","user_agent":"Mozilla/5.0 (X11; Linux x86_64; rv:60.0) Gecko/20100101 Firefox/60.0","written_bytes":18711}
==> /var/log/gitlab/gitlab-workhorse/state <==
==> /var/log/gitlab/nginx/current <==
==> /var/log/gitlab/nginx/error.log <==
==> /var/log/gitlab/nginx/gitlab_access.log <== 1.2.3.4 - - [23/Oct/2019:08:35:56 -0400] "GET /puppet/control-repo/blob/dev/README.md?format=json&viewer=rich HTTP/2.0" 200 936 "https://gitlab-server.DOMAIN/puppet/control-repo/tree/dev" "Mozilla/5.0 (X11; Linux x86_64; rv:60.0) Gecko/20100101 Firefox/60.0" 1.2.3.4 - - [23/Oct/2019:08:35:57 -0400] "GET /puppet/control-repo/refs/dev/logs_tree/?format=js HTTP/2.0" 200 1433 "https://gitlab-server.DOMAIN/puppet/control-repo/tree/dev" "Mozilla/5.0 (X11; Linux x86_64; rv:60.0) Gecko/20100101 Firefox/60.0" 1.2.3.4 - - [23/Oct/2019:08:35:57 -0400] "GET /puppet/control-repo/commits/4f4baaf55ed7b684f609d4723ac725adfda1e124/signatures HTTP/2.0" 200 1128 "https://gitlab-server.DOMAIN/puppet/control-repo/tree/dev" "Mozilla/5.0 (X11; Linux x86_64; rv:60.0) Gecko/20100101 Firefox/60.0" 1.2.3.4 - - [23/Oct/2019:08:36:00 -0400] "GET /puppet/control-repo/blob/dev/README.md HTTP/2.0" 200 12988 "https://gitlab-server.DOMAIN/puppet/control-repo/tree/dev" "Mozilla/5.0 (X11; Linux x86_64; rv:60.0) Gecko/20100101 Firefox/60.0" 1.2.3.4 - - [23/Oct/2019:08:36:00 -0400] "GET /assets/webpack/pages.projects.blob.show.22f05ee0.chunk.js HTTP/2.0" 200 12780 "https://gitlab-server.DOMAIN/puppet/control-repo/blob/dev/README.md" "Mozilla/5.0 (X11; Linux x86_64; rv:60.0) Gecko/20100101 Firefox/60.0" 1.2.3.4 - - [23/Oct/2019:08:36:01 -0400] "GET /puppet/control-repo/commits/4f4baaf55ed7b684f609d4723ac725adfda1e124/signatures?limit=1 HTTP/2.0" 200 17 "https://gitlab-server.DOMAIN/puppet/control-repo/blob/dev/README.md" "Mozilla/5.0 (X11; Linux x86_64; rv:60.0) Gecko/20100101 Firefox/60.0" 1.2.3.4 - - [23/Oct/2019:08:36:01 -0400] "GET /puppet/control-repo/blob/dev/README.md?format=json&viewer=rich HTTP/2.0" 304 0 "https://gitlab-server.DOMAIN/puppet/control-repo/blob/dev/README.md" "Mozilla/5.0 (X11; Linux x86_64; rv:60.0) Gecko/20100101 Firefox/60.0" 1.2.3.4 - - [23/Oct/2019:08:36:04 -0400] "GET /puppet/control-repo/edit/dev/README.md HTTP/2.0" 200 15565 "https://gitlab-server.DOMAIN/puppet/control-repo/blob/dev/README.md" "Mozilla/5.0 (X11; Linux x86_64; rv:60.0) Gecko/20100101 Firefox/60.0" 1.2.3.4 - - [23/Oct/2019:08:36:04 -0400] "GET /assets/webpack/pages.projects.blob.edit.b8426cf9.chunk.js HTTP/2.0" 200 7829 "https://gitlab-server.DOMAIN/puppet/control-repo/edit/dev/README.md" "Mozilla/5.0 (X11; Linux x86_64; rv:60.0) Gecko/20100101 Firefox/60.0" 1.2.3.4 - - [23/Oct/2019:08:36:04 -0400] "GET /assets/ace/mode-markdown-c33305e8a6835b17c07f224eace192a8931be0f70608f2c596ce66e711f6c4d4.js HTTP/2.0" 200 0 "https://gitlab-server.DOMAIN/puppet/control-repo/edit/dev/README.md" "Mozilla/5.0 (X11; Linux x86_64; rv:60.0) Gecko/20100101 Firefox/60.0"
==> /var/log/gitlab/nginx/gitlab_error.log <==
==> /var/log/gitlab/nginx/access.log <==
==> /var/log/gitlab/node-exporter/state <==
==> /var/log/gitlab/node-exporter/current <== 2019-10-22_15:57:59.91065 time="2019-10-22T11:57:59-04:00" level=info msg=" - sockstat" source="node_exporter.go:104" 2019-10-22_15:57:59.91066 time="2019-10-22T11:57:59-04:00" level=info msg=" - stat" source="node_exporter.go:104" 2019-10-22_15:57:59.91066 time="2019-10-22T11:57:59-04:00" level=info msg=" - textfile" source="node_exporter.go:104" 2019-10-22_15:57:59.91066 time="2019-10-22T11:57:59-04:00" level=info msg=" - time" source="node_exporter.go:104" 2019-10-22_15:57:59.91067 time="2019-10-22T11:57:59-04:00" level=info msg=" - timex" source="node_exporter.go:104" 2019-10-22_15:57:59.91068 time="2019-10-22T11:57:59-04:00" level=info msg=" - uname" source="node_exporter.go:104" 2019-10-22_15:57:59.91068 time="2019-10-22T11:57:59-04:00" level=info msg=" - vmstat" source="node_exporter.go:104" 2019-10-22_15:57:59.91068 time="2019-10-22T11:57:59-04:00" level=info msg=" - xfs" source="node_exporter.go:104" 2019-10-22_15:57:59.91069 time="2019-10-22T11:57:59-04:00" level=info msg=" - zfs" source="node_exporter.go:104" 2019-10-22_15:57:59.91069 time="2019-10-22T11:57:59-04:00" level=info msg="Listening on localhost:9100" source="node_exporter.go:170"
==> /var/log/gitlab/gitlab-monitor/state <==
==> /var/log/gitlab/gitlab-monitor/current <== 2019-09-24_19:45:06.32528 Puma starting in single mode... 2019-09-24_19:45:06.32529 * Version 3.12.0 (ruby 2.5.3-p105), codename: Llamas in Pajamas 2019-09-24_19:45:06.32529 * Min threads: 0, max threads: 16 2019-09-24_19:45:06.32529 * Environment: development 2019-09-24_19:45:06.32530 * Listening on tcp://localhost:9168 2019-09-24_19:45:06.32530 Use Ctrl-C to stop 2019-09-25_11:36:42.56658 - Gracefully stopping, waiting for requests to finish 2019-09-25_11:36:42.68591 === puma shutdown: 2019-09-25 07:36:42 -0400 === 2019-09-25_11:36:42.68593 - Goodbye! 2019-09-25_11:36:42.68616 == Sinatra has ended his set (crowd applauds)
==> /var/log/gitlab/redis-exporter/state <==
==> /var/log/gitlab/redis-exporter/current <== 2019-10-22_12:02:25.84763 time="2019-10-22T08:02:25-04:00" level=info msg="Redis Metrics Exporter 1.1.0 build date: sha1: Go: go1.12.7" 2019-10-22_12:02:25.84835 time="2019-10-22T08:02:25-04:00" level=info msg="Providing metrics at localhost:9121/metrics" 2019-10-22_12:52:45.66518 time="2019-10-22T08:52:45-04:00" level=info msg="Redis Metrics Exporter 1.1.0 build date: sha1: Go: go1.12.7" 2019-10-22_12:52:45.68155 time="2019-10-22T08:52:45-04:00" level=info msg="Providing metrics at localhost:9121/metrics" 2019-10-22_15:48:24.40268 time="2019-10-22T11:48:24-04:00" level=info msg="Redis Metrics Exporter 1.1.0 build date: sha1: Go: go1.12.7" 2019-10-22_15:48:24.41262 time="2019-10-22T11:48:24-04:00" level=info msg="Providing metrics at localhost:9121/metrics" 2019-10-22_15:56:17.69443 time="2019-10-22T11:56:17-04:00" level=info msg="Redis Metrics Exporter 1.1.0 build date: sha1: Go: go1.12.7" 2019-10-22_15:56:17.69553 time="2019-10-22T11:56:17-04:00" level=info msg="Providing metrics at localhost:9121/metrics" 2019-10-22_15:58:02.03021 time="2019-10-22T11:58:02-04:00" level=info msg="Redis Metrics Exporter 1.1.0 build date: sha1: Go: go1.12.7" 2019-10-22_15:58:02.03023 time="2019-10-22T11:58:02-04:00" level=info msg="Providing metrics at localhost:9121/metrics"
==> /var/log/gitlab/prometheus/state <==
==> /var/log/gitlab/prometheus/current <== 2019-10-23_12:27:05.41750 level=warn ts=2019-10-23T12:27:05.417Z caller=manager.go:567 component="rule manager" group="GitLab Saturation Ratios" msg="Error on ingesting results from rule evaluation with different value but same timestamp" numDropped=1 2019-10-23_12:28:05.41764 level=warn ts=2019-10-23T12:28:05.417Z caller=manager.go:567 component="rule manager" group="GitLab Saturation Ratios" msg="Error on ingesting results from rule evaluation with different value but same timestamp" numDropped=1 2019-10-23_12:29:05.41777 level=warn ts=2019-10-23T12:29:05.417Z caller=manager.go:567 component="rule manager" group="GitLab Saturation Ratios" msg="Error on ingesting results from rule evaluation with different value but same timestamp" numDropped=1 2019-10-23_12:30:05.41785 level=warn ts=2019-10-23T12:30:05.417Z caller=manager.go:567 component="rule manager" group="GitLab Saturation Ratios" msg="Error on ingesting results from rule evaluation with different value but same timestamp" numDropped=1 2019-10-23_12:31:05.41605 level=warn ts=2019-10-23T12:31:05.415Z caller=manager.go:567 component="rule manager" group="GitLab Saturation Ratios" msg="Error on ingesting results from rule evaluation with different value but same timestamp" numDropped=1 2019-10-23_12:32:05.41701 level=warn ts=2019-10-23T12:32:05.416Z caller=manager.go:567 component="rule manager" group="GitLab Saturation Ratios" msg="Error on ingesting results from rule evaluation with different value but same timestamp" numDropped=1 2019-10-23_12:33:05.41695 level=warn ts=2019-10-23T12:33:05.416Z caller=manager.go:567 component="rule manager" group="GitLab Saturation Ratios" msg="Error on ingesting results from rule evaluation with different value but same timestamp" numDropped=1 2019-10-23_12:34:05.41688 level=warn ts=2019-10-23T12:34:05.416Z caller=manager.go:567 component="rule manager" group="GitLab Saturation Ratios" msg="Error on ingesting results from rule evaluation with different value but same timestamp" numDropped=1 2019-10-23_12:35:05.41578 level=warn ts=2019-10-23T12:35:05.415Z caller=manager.go:567 component="rule manager" group="GitLab Saturation Ratios" msg="Error on ingesting results from rule evaluation with different value but same timestamp" numDropped=1 2019-10-23_12:36:05.41679 level=warn ts=2019-10-23T12:36:05.416Z caller=manager.go:567 component="rule manager" group="GitLab Saturation Ratios" msg="Error on ingesting results from rule evaluation with different value but same timestamp" numDropped=1
==> /var/log/gitlab/alertmanager/state <==
==> /var/log/gitlab/alertmanager/current <== 2019-10-23_12:36:29.87182 level=warn ts=2019-10-23T12:36:29.871Z caller=cluster.go:154 component=cluster err="couldn't deduce an advertise address: no private IP found, explicit advertise addr not provided" 2019-10-23_12:36:29.87388 level=error ts=2019-10-23T12:36:29.873Z caller=main.go:222 msg="unable to initialize gossip mesh" err="create memberlist: Failed to get final advertise address: No private IP address found, and explicit IP not provided" 2019-10-23_12:36:30.94655 level=info ts=2019-10-23T12:36:30.946Z caller=main.go:197 msg="Starting Alertmanager" version="(version=0.18.0, branch=master, revision=)" 2019-10-23_12:36:30.94658 level=info ts=2019-10-23T12:36:30.946Z caller=main.go:198 build_context="(go=go1.12.7, user=GitLab-Omnibus, date=)" 2019-10-23_12:36:30.94753 level=warn ts=2019-10-23T12:36:30.947Z caller=cluster.go:154 component=cluster err="couldn't deduce an advertise address: no private IP found, explicit advertise addr not provided" 2019-10-23_12:36:30.94934 level=error ts=2019-10-23T12:36:30.949Z caller=main.go:222 msg="unable to initialize gossip mesh" err="create memberlist: Failed to get final advertise address: No private IP address found, and explicit IP not provided" 2019-10-23_12:36:32.02827 level=info ts=2019-10-23T12:36:32.026Z caller=main.go:197 msg="Starting Alertmanager" version="(version=0.18.0, branch=master, revision=)" 2019-10-23_12:36:32.02832 level=info ts=2019-10-23T12:36:32.026Z caller=main.go:198 build_context="(go=go1.12.7, user=GitLab-Omnibus, date=)" 2019-10-23_12:36:32.02832 level=warn ts=2019-10-23T12:36:32.027Z caller=cluster.go:154 component=cluster err="couldn't deduce an advertise address: no private IP found, explicit advertise addr not provided" 2019-10-23_12:36:32.02976 level=error ts=2019-10-23T12:36:32.029Z caller=main.go:222 msg="unable to initialize gossip mesh" err="create memberlist: Failed to get final advertise address: No private IP address found, and explicit IP not provided"
==> /var/log/gitlab/postgres-exporter/state <==
==> /var/log/gitlab/postgres-exporter/current <== 2019-10-22_15:48:46.30428 time="2019-10-22T11:48:46-04:00" level=info msg="Established new database connection to "/var/opt/gitlab/postgresql:5432"." source="postgres_exporter.go:777" 2019-10-22_15:48:46.32398 time="2019-10-22T11:48:46-04:00" level=info msg="Semantic Version Changed on "/var/opt/gitlab/postgresql:5432": 0.0.0 -> 10.9.0" source="postgres_exporter.go:1229" 2019-10-22_15:56:16.84555 time="2019-10-22T11:56:16-04:00" level=info msg="Established new database connection to "/var/opt/gitlab/postgresql:5432"." source="postgres_exporter.go:777" 2019-10-22_15:56:16.85472 time="2019-10-22T11:56:16-04:00" level=info msg="Semantic Version Changed on "/var/opt/gitlab/postgresql:5432": 0.0.0 -> 10.9.0" source="postgres_exporter.go:1229" 2019-10-22_15:56:16.90209 time="2019-10-22T11:56:16-04:00" level=info msg="Starting Server: localhost:9187" source="postgres_exporter.go:1437" 2019-10-22_15:58:00.33308 time="2019-10-22T11:58:00-04:00" level=info msg="Established new database connection to "/var/opt/gitlab/postgresql:5432"." source="postgres_exporter.go:777" 2019-10-22_15:58:00.33310 time="2019-10-22T11:58:00-04:00" level=error msg="Error opening connection to database (user=gitlab-psql%20host=/var/opt/gitlab/postgresql%20database=postgres%20sslmode=allow): dial unix /var/opt/gitlab/postgresql/.s.PGSQL.5432: connect: no such file or directory" source="postgres_exporter.go:1336" 2019-10-22_15:58:00.33311 time="2019-10-22T11:58:00-04:00" level=info msg="Starting Server: localhost:9187" source="postgres_exporter.go:1437" 2019-10-22_15:58:16.30436 time="2019-10-22T11:58:16-04:00" level=info msg="Established new database connection to "/var/opt/gitlab/postgresql:5432"." source="postgres_exporter.go:777" 2019-10-22_15:58:16.33355 time="2019-10-22T11:58:16-04:00" level=info msg="Semantic Version Changed on "/var/opt/gitlab/postgresql:5432": 0.0.0 -> 10.9.0" source="postgres_exporter.go:1229"
==> /var/log/gitlab/gitlab-exporter/state <==
==> /var/log/gitlab/gitlab-exporter/current <== 2019-10-22_15:57:27.09853 === puma shutdown: 2019-10-22 11:57:27 -0400 === 2019-10-22_15:57:27.09856 - Goodbye! 2019-10-22_15:57:27.09856 == Sinatra has ended his set (crowd applauds) 2019-10-22_15:57:58.54752 == Sinatra (v2.0.7) has taken the stage on 9168 for development with backup from Puma 2019-10-22_15:57:58.54753 Puma starting in single mode... 2019-10-22_15:57:58.54755 * Version 3.12.0 (ruby 2.6.3-p62), codename: Llamas in Pajamas 2019-10-22_15:57:58.54755 * Min threads: 0, max threads: 16 2019-10-22_15:57:58.54756 * Environment: development 2019-10-22_15:57:58.54756 * Listening on tcp://localhost:9168 2019-10-22_15:57:58.54756 Use Ctrl-C to stop
==> /var/log/gitlab/grafana/state <==
==> /var/log/gitlab/grafana/current <== 2019-10-22_15:57:58.69753 t=2019-10-22T11:57:58-0400 lvl=info msg="Initializing SearchService" logger=server 2019-10-22_15:57:58.69753 t=2019-10-22T11:57:58-0400 lvl=info msg="Initializing ServerLockService" logger=server 2019-10-22_15:57:58.69753 t=2019-10-22T11:57:58-0400 lvl=info msg="Initializing TracingService" logger=server 2019-10-22_15:57:58.69754 t=2019-10-22T11:57:58-0400 lvl=info msg="Initializing UsageStatsService" logger=server 2019-10-22_15:57:58.69754 t=2019-10-22T11:57:58-0400 lvl=info msg="Initializing UserAuthTokenService" logger=server 2019-10-22_15:57:58.69755 t=2019-10-22T11:57:58-0400 lvl=info msg="Initializing CleanUpService" logger=server 2019-10-22_15:57:58.69755 t=2019-10-22T11:57:58-0400 lvl=info msg="Initializing NotificationService" logger=server 2019-10-22_15:57:58.69849 t=2019-10-22T11:57:58-0400 lvl=info msg="Initializing provisioningServiceImpl" logger=server 2019-10-22_15:57:58.77360 t=2019-10-22T11:57:58-0400 lvl=info msg="Initializing Stream Manager" 2019-10-22_15:57:58.77364 t=2019-10-22T11:57:58-0400 lvl=info msg="HTTP Server Listen" logger=http.server address=localhost:3000 protocol=http subUrl=/-/grafana socket=
==> /var/log/gitlab/alertmanager/current <== 2019-10-23_12:36:33.11154 level=info ts=2019-10-23T12:36:33.109Z caller=main.go:197 msg="Starting Alertmanager" version="(version=0.18.0, branch=master, revision=)" 2019-10-23_12:36:33.11157 level=info ts=2019-10-23T12:36:33.109Z caller=main.go:198 build_context="(go=go1.12.7, user=GitLab-Omnibus, date=)" 2019-10-23_12:36:33.11157 level=warn ts=2019-10-23T12:36:33.111Z caller=cluster.go:154 component=cluster err="couldn't deduce an advertise address: no private IP found, explicit advertise addr not provided" 2019-10-23_12:36:33.11309 level=error ts=2019-10-23T12:36:33.112Z caller=main.go:222 msg="unable to initialize gossip mesh" err="create memberlist: Failed to get final advertise address: No private IP address found, and explicit IP not provided" 2019-10-23_12:36:34.19084 level=info ts=2019-10-23T12:36:34.190Z caller=main.go:197 msg="Starting Alertmanager" version="(version=0.18.0, branch=master, revision=)" 2019-10-23_12:36:34.19091 level=info ts=2019-10-23T12:36:34.190Z caller=main.go:198 build_context="(go=go1.12.7, user=GitLab-Omnibus, date=)" 2019-10-23_12:36:34.19238 level=warn ts=2019-10-23T12:36:34.192Z caller=cluster.go:154 component=cluster err="couldn't deduce an advertise address: no private IP found, explicit advertise addr not provided" 2019-10-23_12:36:34.19434 level=error ts=2019-10-23T12:36:34.194Z caller=main.go:222 msg="unable to initialize gossip mesh" err="create memberlist: Failed to get final advertise address: No private IP address found, and explicit IP not provided" 2019-10-23_12:36:35.26880 level=info ts=2019-10-23T12:36:35.268Z caller=main.go:197 msg="Starting Alertmanager" version="(version=0.18.0, branch=master, revision=)" 2019-10-23_12:36:35.26884 level=info ts=2019-10-23T12:36:35.268Z caller=main.go:198 build_context="(go=go1.12.7, user=GitLab-Omnibus, date=)" 2019-10-23_12:36:35.26992 level=warn ts=2019-10-23T12:36:35.269Z caller=cluster.go:154 component=cluster err="couldn't deduce an advertise address: no private IP found, explicit advertise addr not provided" 2019-10-23_12:36:35.27168 level=error ts=2019-10-23T12:36:35.271Z caller=main.go:222 msg="unable to initialize gossip mesh" err="create memberlist: Failed to get final advertise address: No private IP address found, and explicit IP not provided" 2019-10-23_12:36:36.34688 level=info ts=2019-10-23T12:36:36.346Z caller=main.go:197 msg="Starting Alertmanager" version="(version=0.18.0, branch=master, revision=)" 2019-10-23_12:36:36.34693 level=info ts=2019-10-23T12:36:36.346Z caller=main.go:198 build_context="(go=go1.12.7, user=GitLab-Omnibus, date=)" 2019-10-23_12:36:36.34806 level=warn ts=2019-10-23T12:36:36.347Z caller=cluster.go:154 component=cluster err="couldn't deduce an advertise address: no private IP found, explicit advertise addr not provided" 2019-10-23_12:36:36.34989 level=error ts=2019-10-23T12:36:36.349Z caller=main.go:222 msg="unable to initialize gossip mesh" err="create memberlist: Failed to get final advertise address: No private IP address found, and explicit IP not provided"
==> /var/log/gitlab/gitlab-rails/production.log <== Started PUT "/puppet/control-repo/update/dev/README.md" for 1.2.3.4 at 2019-10-23 08:36:37 -0400 Processing by Gitlab::RequestForgeryProtection::Controller#index as HTML Parameters: {"utf8"=>"✓", "authenticity_token"=>"[FILTERED]", "file_path"=>"README.md", "encoding"=>"text", "commit_message"=>"Update README.md", "branch_name"=>"dev", "original_branch"=>"dev", "last_commit_sha"=>"4f4baaf55ed7b684f609d4723ac725adfda1e124", "content"=>"[FILTERED]", "from_merge_request_iid"=>""} Completed 200 OK in 2ms (ActiveRecord: 0.0ms | Elasticsearch: 0.0ms) Processing by Projects::BlobController#update as HTML Parameters: {"utf8"=>"✓", "authenticity_token"=>"[FILTERED]", "file_path"=>"README.md", "encoding"=>"text", "commit_message"=>"Update README.md", "branch_name"=>"dev", "original_branch"=>"dev", "last_commit_sha"=>"4f4baaf55ed7b684f609d4723ac725adfda1e124", "content"=>"[FILTERED]", "from_merge_request_iid"=>"", "namespace_id"=>"puppet", "project_id"=>"control-repo", "id"=>"dev/README.md"}
==> /var/log/gitlab/gitlab-rails/production_json.log <== {"method":"PUT","path":"/puppet/control-repo/update/dev/README.md","format":"html","controller":"Gitlab::RequestForgeryProtection::Controller","action":"index","status":200,"duration":1.74,"view":0.0,"db":0.0,"time":"2019-10-23T12:36:37.103Z","params":[{"key":"utf8","value":"✓"},{"key":"_method","value":"put"},{"key":"authenticity_token","value":"[FILTERED]"},{"key":"file_path","value":"README.md"},{"key":"encoding","value":"text"},{"key":"commit_message","value":"Update README.md"},{"key":"branch_name","value":"dev"},{"key":"original_branch","value":"dev"},{"key":"last_commit_sha","value":"4f4baaf55ed7b684f609d4723ac725adfda1e124"},{"key":"content","value":"[FILTERED]"},{"key":"from_merge_request_iid","value":""}],"remote_ip":null,"user_id":null,"username":null,"ua":null,"queue_duration":null,"correlation_id":null,"cpu_s":0.006638223999999582}
==> /var/log/gitlab/gitaly/current <== {"correlation_id":"cGtn0Xtms23","grpc.code":"OK","grpc.meta.auth_version":"v2","grpc.meta.client_name":"gitlab-web","grpc.method":"FindAllTagNames","grpc.request.deadline":"2019-10-23T12:36:47Z","grpc.request.fullMethod":"/gitaly.RefService/FindAllTagNames","grpc.request.glProjectPath":"puppet/control-repo","grpc.request.glRepository":"project-14","grpc.request.repoPath":"puppet/control-repo.git","grpc.request.repoStorage":"default","grpc.request.topLevelGroup":"puppet","grpc.service":"gitaly.RefService","grpc.start_time":"2019-10-23T12:36:37Z","grpc.time_ms":18.811,"level":"info","msg":"finished streaming call with code OK","peer.address":"@","pid":6283,"span.kind":"server","system":"grpc","time":"2019-10-23T12:36:37Z"} {"correlation_id":"cGtn0Xtms23","grpc.code":"OK","grpc.meta.auth_version":"v2","grpc.meta.client_name":"gitlab-web","grpc.method":"FindCommits","grpc.request.deadline":"2019-10-23T12:37:07Z","grpc.request.fullMethod":"/gitaly.CommitService/FindCommits","grpc.request.glProjectPath":"puppet/control-repo","grpc.request.glRepository":"project-14","grpc.request.repoPath":"puppet/control-repo.git","grpc.request.repoStorage":"default","grpc.request.topLevelGroup":"puppet","grpc.service":"gitaly.CommitService","grpc.start_time":"2019-10-23T12:36:37Z","grpc.time_ms":6.612,"level":"info","msg":"finished streaming call with code OK","peer.address":"@","pid":6283,"span.kind":"server","system":"grpc","time":"2019-10-23T12:36:37Z"} {"correlation_id":"cGtn0Xtms23","grpc.code":"OK","grpc.meta.auth_version":"v2","grpc.meta.client_name":"gitlab-web","grpc.method":"FindCommits","grpc.request.deadline":"2019-10-23T12:37:07Z","grpc.request.fullMethod":"/gitaly.CommitService/FindCommits","grpc.request.glProjectPath":"puppet/control-repo","grpc.request.glRepository":"project-14","grpc.request.repoPath":"puppet/control-repo.git","grpc.request.repoStorage":"default","grpc.request.topLevelGroup":"puppet","grpc.service":"gitaly.CommitService","grpc.start_time":"2019-10-23T12:36:37Z","grpc.time_ms":5.121,"level":"info","msg":"finished streaming call with code OK","peer.address":"@","pid":6283,"span.kind":"server","system":"grpc","time":"2019-10-23T12:36:37Z"} {"grpc.code":"OK","grpc.meta.auth_version":"v2","grpc.meta.client_name":"gitaly-ruby","grpc.method":"RepositoryExists","grpc.request.fullMethod":"/gitaly.RepositoryService/RepositoryExists","grpc.request.glProjectPath":"puppet/control-repo","grpc.request.glRepository":"project-14","grpc.request.repoPath":"puppet/control-repo.git","grpc.request.repoStorage":"default","grpc.request.topLevelGroup":"puppet","grpc.service":"gitaly.RepositoryService","grpc.start_time":"2019-10-23T12:36:37Z","grpc.time_ms":0.117,"level":"info","msg":"finished unary call with code OK","peer.address":"@","pid":6283,"span.kind":"server","system":"grpc","time":"2019-10-23T12:36:37Z"} {"grpc.code":"OK","grpc.meta.auth_version":"v2","grpc.meta.client_name":"gitaly-ruby","grpc.method":"HasLocalBranches","grpc.request.fullMethod":"/gitaly.RepositoryService/HasLocalBranches","grpc.request.glProjectPath":"puppet/control-repo","grpc.request.glRepository":"project-14","grpc.request.repoPath":"puppet/control-repo.git","grpc.request.repoStorage":"default","grpc.request.topLevelGroup":"puppet","grpc.service":"gitaly.RepositoryService","grpc.start_time":"2019-10-23T12:36:37Z","grpc.time_ms":4.149,"level":"info","msg":"finished unary call with code OK","peer.address":"@","pid":6283,"span.kind":"server","system":"grpc","time":"2019-10-23T12:36:37Z"} {"grpc.code":"OK","grpc.meta.auth_version":"v2","grpc.meta.client_name":"gitaly-ruby","grpc.method":"RefExists","grpc.request.fullMethod":"/gitaly.RefService/RefExists","grpc.request.glProjectPath":"puppet/control-repo","grpc.request.glRepository":"project-14","grpc.request.repoPath":"puppet/control-repo.git","grpc.request.repoStorage":"default","grpc.request.topLevelGroup":"puppet","grpc.service":"gitaly.RefService","grpc.start_time":"2019-10-23T12:36:37Z","grpc.time_ms":3.524,"level":"info","msg":"finished unary call with code OK","peer.address":"@","pid":6283,"span.kind":"server","system":"grpc","time":"2019-10-23T12:36:37Z"} {"grpc.code":"OK","grpc.meta.auth_version":"v2","grpc.meta.client_name":"gitaly-ruby","grpc.method":"RepositoryExists","grpc.request.fullMethod":"/gitaly.RepositoryService/RepositoryExists","grpc.request.glProjectPath":"puppet/control-repo","grpc.request.glRepository":"project-14","grpc.request.repoPath":"puppet/control-repo.git","grpc.request.repoStorage":"default","grpc.request.topLevelGroup":"puppet","grpc.service":"gitaly.RepositoryService","grpc.start_time":"2019-10-23T12:36:37Z","grpc.time_ms":0.817,"level":"info","msg":"finished unary call with code OK","peer.address":"@","pid":6283,"span.kind":"server","system":"grpc","time":"2019-10-23T12:36:37Z"} {"grpc.code":"OK","grpc.meta.auth_version":"v2","grpc.meta.client_name":"gitaly-ruby","grpc.method":"HasLocalBranches","grpc.request.fullMethod":"/gitaly.RepositoryService/HasLocalBranches","grpc.request.glProjectPath":"puppet/control-repo","grpc.request.glRepository":"project-14","grpc.request.repoPath":"puppet/control-repo.git","grpc.request.repoStorage":"default","grpc.request.topLevelGroup":"puppet","grpc.service":"gitaly.RepositoryService","grpc.start_time":"2019-10-23T12:36:37Z","grpc.time_ms":4.165,"level":"info","msg":"finished unary call with code OK","peer.address":"@","pid":6283,"span.kind":"server","system":"grpc","time":"2019-10-23T12:36:37Z"} {"level":"info","msg":"I, [2019-10-22T11:58:01.210851 #6320] INFO -- sentry: ** [Raven] Raven 2.9.0 configured not to capture errors: DSN not set","supervisor.args":["bundle","exec","bin/ruby-cd","/var/opt/gitlab/gitaly","/opt/gitlab/embedded/service/gitaly-ruby/bin/gitaly-ruby","6283","/tmp/gitaly-ruby256855346/socket.1"],"supervisor.name":"gitaly-ruby.1","time":"2019-10-23T12:36:37Z"} {"level":"info","msg":"GRPC-RUBY: WARN: Using gitaly-proto 1.67.0","supervisor.args":["bundle","exec","bin/ruby-cd","/var/opt/gitlab/gitaly","/opt/gitlab/embedded/service/gitaly-ruby/bin/gitaly-ruby","6283","/tmp/gitaly-ruby256855346/socket.1"],"supervisor.name":"gitaly-ruby.1","time":"2019-10-23T12:36:37Z"}
==> /var/log/gitlab/alertmanager/current <== 2019-10-23_12:36:37.42671 level=info ts=2019-10-23T12:36:37.426Z caller=main.go:197 msg="Starting Alertmanager" version="(version=0.18.0, branch=master, revision=)" 2019-10-23_12:36:37.42675 level=info ts=2019-10-23T12:36:37.426Z caller=main.go:198 build_context="(go=go1.12.7, user=GitLab-Omnibus, date=)" 2019-10-23_12:36:37.42797 level=warn ts=2019-10-23T12:36:37.427Z caller=cluster.go:154 component=cluster err="couldn't deduce an advertise address: no private IP found, explicit advertise addr not provided" 2019-10-23_12:36:37.42980 level=error ts=2019-10-23T12:36:37.429Z caller=main.go:222 msg="unable to initialize gossip mesh" err="create memberlist: Failed to get final advertise address: No private IP address found, and explicit IP not provided"
==> /var/log/gitlab/gitlab-shell/gitlab-shell.log <== {"method":"POST","url":"http://127.0.0.1:8080/api/v4/internal/allowed","duration":0.156120496,"gitaly_embedded":true,"pid":26337,"level":"info","msg":"finished HTTP request","time":"2019-10-23T08:36:38-04:00"} {"method":"POST","url":"http://127.0.0.1:8080/api/v4/internal/pre_receive","duration":0.035232216,"gitaly_embedded":true,"pid":26337,"level":"info","msg":"finished HTTP request","time":"2019-10-23T08:36:38-04:00"}
==> /var/log/gitlab/gitlab-rails/production.log <== Started POST "/api/v4/internal/allowed" for 127.0.0.1 at 2019-10-23 08:36:37 -0400 Processing by Gitlab::RequestForgeryProtection::Controller#index as / Parameters: {"changes"=>"4f4baaf55ed7b684f609d4723ac725adfda1e124 1398412c20110f345f9e1e26c26fcc594b89d209 refs/heads/dev\n", "gl_repository"=>"project-14", "project"=>"/var/opt/gitlab/git-data/repositories/puppet/control-repo.git", "protocol"=>"web", "env"=>"{"GIT_ALTERNATE_OBJECT_DIRECTORIES":[],"GIT_ALTERNATE_OBJECT_DIRECTORIES_RELATIVE":[],"GIT_OBJECT_DIRECTORY":null,"GIT_OBJECT_DIRECTORY_RELATIVE":null}", "user_id"=>"2", "secret_token"=>"[FILTERED]"} Can't verify CSRF token authenticity. This CSRF token verification failure is handled internally by
GitLab::RequestForgeryProtectionUnlike the logs may suggest, this does not result in an actual 422 response to the user For API requests, the only effect is thatcurrent_userwill benilfor the duration of the request Completed 422 Unprocessable Entity in 1ms (ActiveRecord: 0.0ms | Elasticsearch: 0.0ms) Started POST "/api/v4/internal/pre_receive" for 127.0.0.1 at 2019-10-23 08:36:38 -0400 Processing by Gitlab::RequestForgeryProtection::Controller#index as / Parameters: {"gl_repository"=>"project-14", "secret_token"=>"[FILTERED]"} Can't verify CSRF token authenticity. This CSRF token verification failure is handled internally byGitLab::RequestForgeryProtectionUnlike the logs may suggest, this does not result in an actual 422 response to the user For API requests, the only effect is thatcurrent_userwill benilfor the duration of the request Completed 422 Unprocessable Entity in 1ms (ActiveRecord: 0.0ms | Elasticsearch: 0.0ms)==> /var/log/gitlab/gitlab-rails/production_json.log <== {"method":"POST","path":"/api/v4/internal/allowed","format":"/","controller":"Gitlab::RequestForgeryProtection::Controller","action":"index","status":422,"error":"ActionController::InvalidAuthenticityToken: ActionController::InvalidAuthenticityToken","duration":1.26,"view":0.0,"db":0.0,"time":"2019-10-23T12:36:37.861Z","params":[{"key":"changes","value":"4f4baaf55ed7b684f609d4723ac725adfda1e124 1398412c20110f345f9e1e26c26fcc594b89d209 refs/heads/dev\n"},{"key":"gl_repository","value":"project-14"},{"key":"project","value":"/var/opt/gitlab/git-data/repositories/puppet/control-repo.git"},{"key":"protocol","value":"web"},{"key":"env","value":"{"GIT_ALTERNATE_OBJECT_DIRECTORIES":[],"GIT_ALTERNATE_OBJECT_DIRECTORIES_RELATIVE":[],"GIT_OBJECT_DIRECTORY":null,"GIT_OBJECT_DIRECTORY_RELATIVE":null}"},{"key":"user_id","value":"2"},{"key":"secret_token","value":"[FILTERED]"}],"remote_ip":null,"user_id":null,"username":null,"ua":null,"queue_duration":null,"correlation_id":null,"cpu_s":0.005180803000001788} {"method":"POST","path":"/api/v4/internal/pre_receive","format":"/","controller":"Gitlab::RequestForgeryProtection::Controller","action":"index","status":422,"error":"ActionController::InvalidAuthenticityToken: ActionController::InvalidAuthenticityToken","duration":0.98,"view":0.0,"db":0.0,"time":"2019-10-23T12:36:38.018Z","params":[{"key":"gl_repository","value":"project-14"},{"key":"secret_token","value":"[FILTERED]"}],"remote_ip":null,"user_id":null,"username":null,"ua":null,"queue_duration":null,"correlation_id":null,"cpu_s":0.004531714000002296}
==> /var/log/gitlab/gitlab-rails/api_json.log <== {"time":"2019-10-23T12:36:38.005Z","severity":"INFO","duration":106.61,"db":30.1,"view":76.50999999999999,"status":200,"method":"POST","path":"/api/v4/internal/allowed","params":[{"key":"action","value":"git-receive-pack"},{"key":"changes","value":"4f4baaf55ed7b684f609d4723ac725adfda1e124 1398412c20110f345f9e1e26c26fcc594b89d209 refs/heads/dev\n"},{"key":"gl_repository","value":"project-14"},{"key":"project","value":"/var/opt/gitlab/git-data/repositories/puppet/control-repo.git"},{"key":"protocol","value":"web"},{"key":"env","value":"{"GIT_ALTERNATE_OBJECT_DIRECTORIES":[],"GIT_ALTERNATE_OBJECT_DIRECTORIES_RELATIVE":[],"GIT_OBJECT_DIRECTORY":null,"GIT_OBJECT_DIRECTORY_RELATIVE":null}"},{"key":"user_id","value":"2"},{"key":"secret_token","value":"[FILTERED]"}],"host":"127.0.0.1","remote_ip":"127.0.0.1","ua":"Ruby","route":"/api/:version/internal/allowed","gitaly_calls":2,"gitaly_duration":10.96,"correlation_id":"5cde89c3-3ce0-44f6-8c77-f49aa9cea300"} {"time":"2019-10-23T12:36:38.043Z","severity":"INFO","duration":2.98,"db":0,"view":2.98,"status":200,"method":"POST","path":"/api/v4/internal/pre_receive","params":[{"key":"gl_repository","value":"project-14"},{"key":"secret_token","value":"[FILTERED]"}],"host":"127.0.0.1","remote_ip":"127.0.0.1","ua":"Ruby","route":"/api/:version/internal/pre_receive","correlation_id":"a5e43f19-7ddc-4bdd-b5c6-566be1db63c8"}
==> /var/log/gitlab/gitlab-rails/sidekiq_exporter.log <== [2019-10-23T08:36:37.684-0400] 127.0.0.1 - - [23/Oct/2019:08:36:37 EDT] "GET /metrics HTTP/1.1" 200 15764 "-" "Prometheus/2.12.0"
==> /var/log/gitlab/gitaly/current <== {"correlation_id":"5cde89c3-3ce0-44f6-8c77-f49aa9cea300","grpc.code":"OK","grpc.meta.auth_version":"v2","grpc.meta.client_name":"gitlab-web","grpc.method":"RepositoryExists","grpc.request.deadline":"2019-10-23T12:36:47Z","grpc.request.fullMethod":"/gitaly.RepositoryService/RepositoryExists","grpc.request.glProjectPath":"puppet/control-repo","grpc.request.glRepository":"project-14","grpc.request.repoPath":"puppet/control-repo.git","grpc.request.repoStorage":"default","grpc.request.topLevelGroup":"puppet","grpc.service":"gitaly.RepositoryService","grpc.start_time":"2019-10-23T12:36:37Z","grpc.time_ms":0.162,"level":"info","msg":"finished unary call with code OK","peer.address":"@","pid":6283,"span.kind":"server","system":"grpc","time":"2019-10-23T12:36:37Z"} {"correlation_id":"5cde89c3-3ce0-44f6-8c77-f49aa9cea300","grpc.code":"OK","grpc.meta.auth_version":"v2","grpc.meta.client_name":"gitlab-web","grpc.method":"Cleanup","grpc.request.deadline":"2019-10-23T12:36:48Z","grpc.request.fullMethod":"/gitaly.RepositoryService/Cleanup","grpc.request.glProjectPath":"puppet/control-repo","grpc.request.glRepository":"project-14","grpc.request.repoPath":"puppet/control-repo.git","grpc.request.repoStorage":"default","grpc.request.topLevelGroup":"puppet","grpc.service":"gitaly.RepositoryService","grpc.start_time":"2019-10-23T12:36:37Z","grpc.time_ms":4.194,"level":"info","msg":"finished unary call with code OK","peer.address":"@","pid":6283,"span.kind":"server","system":"grpc","time":"2019-10-23T12:36:37Z"}
==> /var/log/gitlab/alertmanager/current <== 2019-10-23_12:36:38.50593 level=info ts=2019-10-23T12:36:38.505Z caller=main.go:197 msg="Starting Alertmanager" version="(version=0.18.0, branch=master, revision=)" 2019-10-23_12:36:38.50597 level=info ts=2019-10-23T12:36:38.505Z caller=main.go:198 build_context="(go=go1.12.7, user=GitLab-Omnibus, date=)" 2019-10-23_12:36:38.50781 level=warn ts=2019-10-23T12:36:38.507Z caller=cluster.go:154 component=cluster err="couldn't deduce an advertise address: no private IP found, explicit advertise addr not provided" 2019-10-23_12:36:38.50999 level=error ts=2019-10-23T12:36:38.509Z caller=main.go:222 msg="unable to initialize gossip mesh" err="create memberlist: Failed to get final advertise address: No private IP address found, and explicit IP not provided"
==> /var/log/gitlab/gitlab-shell/gitlab-shell.log <== {"method":"POST","url":"http://127.0.0.1:8080/api/v4/internal/post_receive","duration":0.08625931,"gitaly_embedded":true,"pid":26361,"level":"info","msg":"finished HTTP request","time":"2019-10-23T08:36:38-04:00"}
==> /var/log/gitlab/gitlab-rails/sidekiq.log <== {"severity":"INFO","time":"2019-10-23T12:36:38.891Z","class":"PostReceive","retry":3,"queue":"post_receive","jid":"fc188e4a4cb80ad000c0369c","created_at":"2019-10-23T12:36:38.886161Z","correlation_id":"37667733-2f9c-43a4-bc03-6cb909b6decd","enqueued_at":"2019-10-23T12:36:38.888801Z","pid":6504,"message":"PostReceive JID-fc188e4a4cb80ad000c0369c: start","job_status":"start","scheduling_latency_s":0.002395} {"severity":"INFO","time":"2019-10-23T12:36:39.471Z","class":"ProjectCacheWorker","retry":3,"queue":"project_cache","jid":"268415669707bd8071d0519c","created_at":"2019-10-23T12:36:39.468903Z","correlation_id":"37667733-2f9c-43a4-bc03-6cb909b6decd","enqueued_at":"2019-10-23T12:36:39.469232Z","pid":6504,"message":"ProjectCacheWorker JID-268415669707bd8071d0519c: start","job_status":"start","scheduling_latency_s":0.002207} {"severity":"INFO","time":"2019-10-23T12:36:39.510Z","class":"UpdateMergeRequestsWorker","retry":3,"queue":"update_merge_requests","jid":"85814e8ca06f03ba191d6ee0","created_at":"2019-10-23T12:36:39.506186Z","correlation_id":"37667733-2f9c-43a4-bc03-6cb909b6decd","enqueued_at":"2019-10-23T12:36:39.506645Z","pid":6504,"message":"UpdateMergeRequestsWorker JID-85814e8ca06f03ba191d6ee0: start","job_status":"start","scheduling_latency_s":0.003413}
==> /var/log/gitlab/gitlab-rails/production.log <== Started POST "/api/v4/internal/post_receive" for 127.0.0.1 at 2019-10-23 08:36:38 -0400 Processing by Gitlab::RequestForgeryProtection::Controller#index as / Parameters: {"gl_repository"=>"project-14", "identifier"=>"user-2", "changes"=>"4f4baaf55ed7b684f609d4723ac725adfda1e124 1398412c20110f345f9e1e26c26fcc594b89d209 refs/heads/dev\n", "secret_token"=>"[FILTERED]"} Can't verify CSRF token authenticity. This CSRF token verification failure is handled internally by
GitLab::RequestForgeryProtectionUnlike the logs may suggest, this does not result in an actual 422 response to the user For API requests, the only effect is thatcurrent_userwill benilfor the duration of the request Completed 422 Unprocessable Entity in 1ms (ActiveRecord: 0.0ms | Elasticsearch: 0.0ms) Redirected to https://gitlab-server.DOMAIN/puppet/control-repo/blob/dev/README.md Completed 302 Found in 1809ms (ActiveRecord: 9.3ms | Elasticsearch: 0.0ms) Started GET "/puppet/control-repo/blob/dev/README.md" for 1.2.3.4 at 2019-10-23 08:36:38 -0400 Processing by Gitlab::RequestForgeryProtection::Controller#index as HTML Completed 200 OK in 1ms (ActiveRecord: 0.0ms | Elasticsearch: 0.0ms) Processing by Projects::BlobController#show as HTML Parameters: {"namespace_id"=>"puppet", "project_id"=>"control-repo", "id"=>"dev/README.md"} Completed 200 OK in 454ms (Views: 346.2ms | ActiveRecord: 19.3ms | Elasticsearch: 0.0ms) Updating statistics for project 14==> /var/log/gitlab/gitlab-rails/production_json.log <== {"method":"POST","path":"/api/v4/internal/post_receive","format":"/","controller":"Gitlab::RequestForgeryProtection::Controller","action":"index","status":422,"error":"ActionController::InvalidAuthenticityToken: ActionController::InvalidAuthenticityToken","duration":1.17,"view":0.0,"db":0.0,"time":"2019-10-23T12:36:38.841Z","params":[{"key":"gl_repository","value":"project-14"},{"key":"identifier","value":"user-2"},{"key":"changes","value":"4f4baaf55ed7b684f609d4723ac725adfda1e124 1398412c20110f345f9e1e26c26fcc594b89d209 refs/heads/dev\n"},{"key":"secret_token","value":"[FILTERED]"}],"remote_ip":null,"user_id":null,"username":null,"ua":null,"queue_duration":null,"correlation_id":null,"cpu_s":0.0056456169999989925} {"method":"PUT","path":"/puppet/control-repo/update/dev/README.md","format":"html","controller":"Projects::BlobController","action":"update","status":302,"duration":1808.89,"view":0.0,"db":9.28,"location":"https://gitlab-server.DOMAIN/puppet/control-repo/blob/dev/README.md","time":"2019-10-23T12:36:38.943Z","params":[{"key":"utf8","value":"✓"},{"key":"_method","value":"put"},{"key":"authenticity_token","value":"[FILTERED]"},{"key":"file_path","value":"README.md"},{"key":"encoding","value":"text"},{"key":"commit_message","value":"Update README.md"},{"key":"branch_name","value":"dev"},{"key":"original_branch","value":"dev"},{"key":"last_commit_sha","value":"4f4baaf55ed7b684f609d4723ac725adfda1e124"},{"key":"content","value":"[FILTERED]"},{"key":"from_merge_request_iid","value":""},{"key":"namespace_id","value":"puppet"},{"key":"project_id","value":"control-repo"},{"key":"id","value":"dev/README.md"}],"remote_ip":"1.2.3.4","user_id":2,"username":"dewhit11","ua":"Mozilla/5.0 (X11; Linux x86_64; rv:60.0) Gecko/20100101 Firefox/60.0","queue_duration":17.81,"gitaly_calls":4,"gitaly_duration":1677.96,"rugged_calls":3,"rugged_duration_ms":2.78,"correlation_id":"cGtn0Xtms23","cpu_s":0.13559951099999523} {"method":"GET","path":"/puppet/control-repo/blob/dev/README.md","format":"html","controller":"Gitlab::RequestForgeryProtection::Controller","action":"index","status":200,"duration":0.7,"view":0.0,"db":0.0,"time":"2019-10-23T12:36:38.993Z","params":[],"remote_ip":null,"user_id":null,"username":null,"ua":null,"queue_duration":null,"correlation_id":null,"cpu_s":0.026831207000000745} {"method":"GET","path":"/puppet/control-repo/blob/dev/README.md","format":"html","controller":"Projects::BlobController","action":"show","status":200,"duration":454.63,"view":346.23,"db":19.31,"time":"2019-10-23T12:36:39.477Z","params":[{"key":"namespace_id","value":"puppet"},{"key":"project_id","value":"control-repo"},{"key":"id","value":"dev/README.md"}],"remote_ip":"1.2.3.4","user_id":2,"username":"dewhit11","ua":"Mozilla/5.0 (X11; Linux x86_64; rv:60.0) Gecko/20100101 Firefox/60.0","queue_duration":40.8,"gitaly_calls":5,"gitaly_duration":16.13,"rugged_calls":7,"rugged_duration_ms":3.9,"correlation_id":"GGKHQowCG7a","cpu_s":0.46634293200000343}
==> /var/log/gitlab/gitlab-rails/api_json.log <== {"time":"2019-10-23T12:36:38.913Z","severity":"INFO","duration":44.21,"db":6.76,"view":37.45,"status":200,"method":"POST","path":"/api/v4/internal/post_receive","params":[{"key":"gl_repository","value":"project-14"},{"key":"identifier","value":"user-2"},{"key":"changes","value":"4f4baaf55ed7b684f609d4723ac725adfda1e124 1398412c20110f345f9e1e26c26fcc594b89d209 refs/heads/dev\n"},{"key":"secret_token","value":"[FILTERED]"}],"host":"127.0.0.1","remote_ip":"127.0.0.1","ua":"Ruby","route":"/api/:version/internal/post_receive","correlation_id":"37667733-2f9c-43a4-bc03-6cb909b6decd"}
==> /var/log/gitlab/gitaly/current <== {"correlation_id":"cGtn0Xtms23","grpc.code":"OK","grpc.meta.auth_version":"v2","grpc.meta.client_name":"gitlab-web","grpc.method":"UserCommitFiles","grpc.request.deadline":"2019-10-23T12:37:32Z","grpc.request.fullMethod":"/gitaly.OperationService/UserCommitFiles","grpc.service":"gitaly.OperationService","grpc.start_time":"2019-10-23T12:36:37Z","grpc.time_ms":1672.356,"level":"info","msg":"finished streaming call with code OK","peer.address":"@","pid":6283,"span.kind":"server","system":"grpc","time":"2019-10-23T12:36:38Z"} {"correlation_id":"GGKHQowCG7a","grpc.code":"OK","grpc.meta.auth_version":"v2","grpc.meta.client_name":"gitlab-web","grpc.method":"FindAllTagNames","grpc.request.deadline":"2019-10-23T12:36:49Z","grpc.request.fullMethod":"/gitaly.RefService/FindAllTagNames","grpc.request.glProjectPath":"puppet/control-repo","grpc.request.glRepository":"project-14","grpc.request.repoPath":"puppet/control-repo.git","grpc.request.repoStorage":"default","grpc.request.topLevelGroup":"puppet","grpc.service":"gitaly.RefService","grpc.start_time":"2019-10-23T12:36:39Z","grpc.time_ms":4.092,"level":"info","msg":"finished streaming call with code OK","peer.address":"@","pid":6283,"span.kind":"server","system":"grpc","time":"2019-10-23T12:36:39Z"} {"correlation_id":"GGKHQowCG7a","grpc.code":"OK","grpc.meta.auth_version":"v2","grpc.meta.client_name":"gitlab-web","grpc.method":"LastCommitForPath","grpc.request.deadline":"2019-10-23T12:36:49Z","grpc.request.fullMethod":"/gitaly.CommitService/LastCommitForPath","grpc.request.glProjectPath":"puppet/control-repo","grpc.request.glRepository":"project-14","grpc.request.repoPath":"puppet/control-repo.git","grpc.request.repoStorage":"default","grpc.request.topLevelGroup":"puppet","grpc.service":"gitaly.CommitService","grpc.start_time":"2019-10-23T12:36:39Z","grpc.time_ms":8.447,"level":"info","msg":"finished unary call with code OK","peer.address":"@","pid":6283,"span.kind":"server","system":"grpc","time":"2019-10-23T12:36:39Z"} {"correlation_id":"GGKHQowCG7a","grpc.code":"OK","grpc.meta.auth_version":"v2","grpc.meta.client_name":"gitlab-web","grpc.method":"GetCommitSignatures","grpc.request.deadline":"2019-10-23T12:36:49Z","grpc.request.fullMethod":"/gitaly.CommitService/GetCommitSignatures","grpc.request.glProjectPath":"puppet/control-repo","grpc.request.glRepository":"project-14","grpc.request.repoPath":"puppet/control-repo.git","grpc.request.repoStorage":"default","grpc.request.topLevelGroup":"puppet","grpc.service":"gitaly.CommitService","grpc.start_time":"2019-10-23T12:36:39Z","grpc.time_ms":0.955,"level":"info","msg":"finished streaming call with code OK","peer.address":"@","pid":6283,"span.kind":"server","system":"grpc","time":"2019-10-23T12:36:39Z"} {"correlation_id":"GGKHQowCG7a","grpc.code":"OK","grpc.meta.auth_version":"v2","grpc.meta.client_name":"gitlab-web","grpc.method":"RepositoryExists","grpc.request.deadline":"2019-10-23T12:36:49Z","grpc.request.fullMethod":"/gitaly.RepositoryService/RepositoryExists","grpc.request.glProjectPath":"puppet/control-repo.wiki","grpc.request.glRepository":"wiki-14","grpc.request.repoPath":"puppet/control-repo.wiki.git","grpc.request.repoStorage":"default","grpc.request.topLevelGroup":"puppet","grpc.service":"gitaly.RepositoryService","grpc.start_time":"2019-10-23T12:36:39Z","grpc.time_ms":0.108,"level":"info","msg":"finished unary call with code OK","peer.address":"@","pid":6283,"span.kind":"server","system":"grpc","time":"2019-10-23T12:36:39Z"} {"correlation_id":"GGKHQowCG7a","grpc.code":"OK","grpc.meta.auth_version":"v2","grpc.meta.client_name":"gitlab-web","grpc.method":"WikiFindPage","grpc.request.deadline":"2019-10-23T12:36:49Z","grpc.request.fullMethod":"/gitaly.WikiService/WikiFindPage","grpc.request.glProjectPath":"puppet/control-repo.wiki","grpc.request.glRepository":"wiki-14","grpc.request.repoPath":"puppet/control-repo.wiki.git","grpc.request.repoStorage":"default","grpc.request.topLevelGroup":"puppet","grpc.service":"gitaly.WikiService","grpc.start_time":"2019-10-23T12:36:39Z","grpc.time_ms":2.521,"level":"info","msg":"finished streaming call with code OK","peer.address":"@","pid":6283,"span.kind":"server","system":"grpc","time":"2019-10-23T12:36:39Z"} {"correlation_id":"37667733-2f9c-43a4-bc03-6cb909b6decd","grpc.code":"OK","grpc.meta.auth_version":"v2","grpc.meta.client_name":"gitlab-sidekiq","grpc.method":"FindAllBranchNames","grpc.request.deadline":"2019-10-23T12:36:49Z","grpc.request.fullMethod":"/gitaly.RefService/FindAllBranchNames","grpc.request.glProjectPath":"puppet/control-repo","grpc.request.glRepository":"project-14","grpc.request.repoPath":"puppet/control-repo.git","grpc.request.repoStorage":"default","grpc.request.topLevelGroup":"puppet","grpc.service":"gitaly.RefService","grpc.start_time":"2019-10-23T12:36:39Z","grpc.time_ms":4.768,"level":"info","msg":"finished streaming call with code OK","peer.address":"@","pid":6283,"span.kind":"server","system":"grpc","time":"2019-10-23T12:36:39Z"}
==> /var/log/gitlab/sidekiq/current <== {"severity":"INFO","time":"2019-10-23T12:36:38.891Z","class":"PostReceive","retry":3,"queue":"post_receive","jid":"fc188e4a4cb80ad000c0369c","created_at":"2019-10-23T12:36:38.886161Z","correlation_id":"37667733-2f9c-43a4-bc03-6cb909b6decd","enqueued_at":"2019-10-23T12:36:38.888801Z","pid":6504,"message":"PostReceive JID-fc188e4a4cb80ad000c0369c: start","job_status":"start","scheduling_latency_s":0.002395} {"severity":"INFO","time":"2019-10-23T12:36:39.471Z","class":"ProjectCacheWorker","retry":3,"queue":"project_cache","jid":"268415669707bd8071d0519c","created_at":"2019-10-23T12:36:39.468903Z","correlation_id":"37667733-2f9c-43a4-bc03-6cb909b6decd","enqueued_at":"2019-10-23T12:36:39.469232Z","pid":6504,"message":"ProjectCacheWorker JID-268415669707bd8071d0519c: start","job_status":"start","scheduling_latency_s":0.002207} {"severity":"INFO","time":"2019-10-23T12:36:39.510Z","class":"UpdateMergeRequestsWorker","retry":3,"queue":"update_merge_requests","jid":"85814e8ca06f03ba191d6ee0","created_at":"2019-10-23T12:36:39.506186Z","correlation_id":"37667733-2f9c-43a4-bc03-6cb909b6decd","enqueued_at":"2019-10-23T12:36:39.506645Z","pid":6504,"message":"UpdateMergeRequestsWorker JID-85814e8ca06f03ba191d6ee0: start","job_status":"start","scheduling_latency_s":0.003413}
==> /var/log/gitlab/gitlab-workhorse/current <== {"correlation_id":"cGtn0Xtms23","duration_ms":1855,"host":"gitlab-server.DOMAIN","level":"info","method":"POST","msg":"access","proto":"HTTP/1.1","referrer":"https://gitlab-server.DOMAIN/puppet/control-repo/edit/dev/README.md","remote_addr":"1.2.3.4:0","remote_ip":"1.2.3.4","status":302,"system":"http","time":"2019-10-23T08:36:38-04:00","uri":"/puppet/control-repo/update/dev/README.md","user_agent":"Mozilla/5.0 (X11; Linux x86_64; rv:60.0) Gecko/20100101 Firefox/60.0","written_bytes":150} {"correlation_id":"GGKHQowCG7a","duration_ms":521,"host":"gitlab-server.DOMAIN","level":"info","method":"GET","msg":"access","proto":"HTTP/1.1","referrer":"https://gitlab-server.DOMAIN/puppet/control-repo/edit/dev/README.md","remote_addr":"1.2.3.4:0","remote_ip":"1.2.3.4","status":200,"system":"http","time":"2019-10-23T08:36:39-04:00","uri":"/puppet/control-repo/blob/dev/README.md","user_agent":"Mozilla/5.0 (X11; Linux x86_64; rv:60.0) Gecko/20100101 Firefox/60.0","written_bytes":49104} {"correlation_id":"ZPzX8fVaLP4","encoding":"gzip","file":"/opt/gitlab/embedded/service/gitlab-rails/public/assets/webpack/pages.projects.blob.show.22f05ee0.chunk.js","level":"info","method":"GET","msg":"Send static file","time":"2019-10-23T08:36:39-04:00","uri":"/assets/webpack/pages.projects.blob.show.22f05ee0.chunk.js"} {"correlation_id":"ZPzX8fVaLP4","duration_ms":1,"host":"gitlab-server.DOMAIN","level":"info","method":"GET","msg":"access","proto":"HTTP/1.1","referrer":"https://gitlab-server.DOMAIN/puppet/control-repo/blob/dev/README.md","remote_addr":"1.2.3.4:0","remote_ip":"1.2.3.4","status":200,"system":"http","time":"2019-10-23T08:36:39-04:00","uri":"/assets/webpack/pages.projects.blob.show.22f05ee0.chunk.js","user_agent":"Mozilla/5.0 (X11; Linux x86_64; rv:60.0) Gecko/20100101 Firefox/60.0","written_bytes":12780}
==> /var/log/gitlab/nginx/gitlab_access.log <== 1.2.3.4 - - [23/Oct/2019:08:36:38 -0400] "POST /puppet/control-repo/update/dev/README.md HTTP/2.0" 302 150 "https://gitlab-server.DOMAIN/puppet/control-repo/edit/dev/README.md" "Mozilla/5.0 (X11; Linux x86_64; rv:60.0) Gecko/20100101 Firefox/60.0" 1.2.3.4 - - [23/Oct/2019:08:36:39 -0400] "GET /puppet/control-repo/blob/dev/README.md HTTP/2.0" 200 13071 "https://gitlab-server.DOMAIN/puppet/control-repo/edit/dev/README.md" "Mozilla/5.0 (X11; Linux x86_64; rv:60.0) Gecko/20100101 Firefox/60.0" 1.2.3.4 - - [23/Oct/2019:08:36:39 -0400] "GET /assets/webpack/pages.projects.blob.show.22f05ee0.chunk.js HTTP/2.0" 200 0 "https://gitlab-server.DOMAIN/puppet/control-repo/blob/dev/README.md" "Mozilla/5.0 (X11; Linux x86_64; rv:60.0) Gecko/20100101 Firefox/60.0"
==> /var/log/gitlab/alertmanager/current <== 2019-10-23_12:36:39.59390 level=info ts=2019-10-23T12:36:39.593Z caller=main.go:197 msg="Starting Alertmanager" version="(version=0.18.0, branch=master, revision=)" 2019-10-23_12:36:39.59394 level=info ts=2019-10-23T12:36:39.593Z caller=main.go:198 build_context="(go=go1.12.7, user=GitLab-Omnibus, date=)" 2019-10-23_12:36:39.59756 level=warn ts=2019-10-23T12:36:39.597Z caller=cluster.go:154 component=cluster err="couldn't deduce an advertise address: no private IP found, explicit advertise addr not provided" 2019-10-23_12:36:39.60033 level=error ts=2019-10-23T12:36:39.600Z caller=main.go:222 msg="unable to initialize gossip mesh" err="create memberlist: Failed to get final advertise address: No private IP address found, and explicit IP not provided"
==> /var/log/gitlab/gitlab-rails/sidekiq.log <== {"severity":"INFO","time":"2019-10-23T12:36:39.683Z","class":"Namespaces::ScheduleAggregationWorker","retry":3,"queue":"update_namespace_statistics:namespaces_schedule_aggregation","queue_namespace":"update_namespace_statistics","jid":"8406c4d69cededa58714bbc8","created_at":"2019-10-23T12:36:39.679121Z","correlation_id":"37667733-2f9c-43a4-bc03-6cb909b6decd","enqueued_at":"2019-10-23T12:36:39.679501Z","pid":6504,"message":"Namespaces::ScheduleAggregationWorker JID-8406c4d69cededa58714bbc8: start","job_status":"start","scheduling_latency_s":0.003821} {"severity":"INFO","time":"2019-10-23T12:36:39.864Z","class":"ProjectCacheWorker","retry":3,"queue":"project_cache","jid":"268415669707bd8071d0519c","created_at":"2019-10-23T12:36:39.468903Z","correlation_id":"37667733-2f9c-43a4-bc03-6cb909b6decd","enqueued_at":"2019-10-23T12:36:39.469232Z","pid":6504,"message":"ProjectCacheWorker JID-268415669707bd8071d0519c: done: 0.393604 sec","job_status":"done","scheduling_latency_s":0.002207,"gitaly_calls":1,"gitaly_duration":35.11,"duration":0.393604,"cpu_s":0.073839,"completed_at":"2019-10-23T12:36:39.864795Z"} {"severity":"INFO","time":"2019-10-23T12:36:40.035Z","class":"Namespaces::RootStatisticsWorker","retry":3,"queue":"update_namespace_statistics:namespaces_root_statistics","queue_namespace":"update_namespace_statistics","jid":"1627fc0ed25148ea5b8ce441","created_at":"2019-10-23T12:36:40.032638Z","correlation_id":"37667733-2f9c-43a4-bc03-6cb909b6decd","enqueued_at":"2019-10-23T12:36:40.033002Z","pid":6504,"message":"Namespaces::RootStatisticsWorker JID-1627fc0ed25148ea5b8ce441: start","job_status":"start","scheduling_latency_s":0.002384} {"severity":"INFO","time":"2019-10-23T12:36:40.084Z","class":"Namespaces::ScheduleAggregationWorker","retry":3,"queue":"update_namespace_statistics:namespaces_schedule_aggregation","queue_namespace":"update_namespace_statistics","jid":"8406c4d69cededa58714bbc8","created_at":"2019-10-23T12:36:39.679121Z","correlation_id":"37667733-2f9c-43a4-bc03-6cb909b6decd","enqueued_at":"2019-10-23T12:36:39.679501Z","pid":6504,"message":"Namespaces::ScheduleAggregationWorker JID-8406c4d69cededa58714bbc8: done: 0.401609 sec","job_status":"done","scheduling_latency_s":0.003821,"duration":0.401609,"cpu_s":0.026736,"completed_at":"2019-10-23T12:36:40.084700Z"} {"severity":"INFO","time":"2019-10-23T12:36:40.136Z","class":"UpdateMergeRequestsWorker","retry":3,"queue":"update_merge_requests","jid":"85814e8ca06f03ba191d6ee0","created_at":"2019-10-23T12:36:39.506186Z","correlation_id":"37667733-2f9c-43a4-bc03-6cb909b6decd","enqueued_at":"2019-10-23T12:36:39.506645Z","pid":6504,"message":"UpdateMergeRequestsWorker JID-85814e8ca06f03ba191d6ee0: done: 0.626376 sec","job_status":"done","scheduling_latency_s":0.003413,"gitaly_calls":1,"gitaly_duration":79.26,"duration":0.626376,"cpu_s":0.151446,"completed_at":"2019-10-23T12:36:40.136225Z"} {"severity":"INFO","time":"2019-10-23T12:36:40.219Z","class":"Namespaces::RootStatisticsWorker","retry":3,"queue":"update_namespace_statistics:namespaces_root_statistics","queue_namespace":"update_namespace_statistics","jid":"1627fc0ed25148ea5b8ce441","created_at":"2019-10-23T12:36:40.032638Z","correlation_id":"37667733-2f9c-43a4-bc03-6cb909b6decd","enqueued_at":"2019-10-23T12:36:40.033002Z","pid":6504,"message":"Namespaces::RootStatisticsWorker JID-1627fc0ed25148ea5b8ce441: done: 0.184219 sec","job_status":"done","scheduling_latency_s":0.002384,"duration":0.184219,"cpu_s":0.038421,"completed_at":"2019-10-23T12:36:40.219421Z"}
==> /var/log/gitlab/gitlab-rails/production.log <== Started GET "/puppet/control-repo/commits/1398412c20110f345f9e1e26c26fcc594b89d209/signatures?limit=1" for 1.2.3.4 at 2019-10-23 08:36:39 -0400 Started GET "/puppet/control-repo/blob/dev/README.md?format=json&viewer=rich" for 1.2.3.4 at 2019-10-23 08:36:39 -0400 Processing by Gitlab::RequestForgeryProtection::Controller#index as JSON Parameters: {"viewer"=>"rich"} Completed 200 OK in 0ms (ActiveRecord: 0.0ms | Elasticsearch: 0.0ms) Processing by Projects::BlobController#show as JSON Parameters: {"viewer"=>"rich", "namespace_id"=>"puppet", "project_id"=>"control-repo", "id"=>"dev/README.md"} Processing by Gitlab::RequestForgeryProtection::Controller#index as JSON Parameters: {"limit"=>"1"} Completed 200 OK in 0ms (ActiveRecord: 0.0ms | Elasticsearch: 0.0ms) Processing by Projects::CommitsController#signatures as JSON Parameters: {"limit"=>"1", "namespace_id"=>"puppet", "project_id"=>"control-repo", "id"=>"1398412c20110f345f9e1e26c26fcc594b89d209"} Completed 200 OK in 87ms (Views: 0.5ms | ActiveRecord: 3.4ms | Elasticsearch: 0.0ms) Completed 200 OK in 107ms (Views: 0.3ms | ActiveRecord: 11.3ms | Elasticsearch: 0.0ms)
==> /var/log/gitlab/gitlab-rails/production_json.log <== {"method":"GET","path":"/puppet/control-repo/blob/dev/README.md","format":"json","controller":"Gitlab::RequestForgeryProtection::Controller","action":"index","status":200,"duration":0.52,"view":0.0,"db":0.0,"time":"2019-10-23T12:36:39.767Z","params":[{"key":"viewer","value":"rich"}],"remote_ip":null,"user_id":null,"username":null,"ua":null,"queue_duration":null,"correlation_id":null,"cpu_s":0.0038052370000087876} {"method":"GET","path":"/puppet/control-repo/commits/1398412c20110f345f9e1e26c26fcc594b89d209/signatures","format":"json","controller":"Gitlab::RequestForgeryProtection::Controller","action":"index","status":200,"duration":0.65,"view":0.0,"db":0.0,"time":"2019-10-23T12:36:39.794Z","params":[{"key":"limit","value":"1"}],"remote_ip":null,"user_id":null,"username":null,"ua":null,"queue_duration":null,"correlation_id":null,"cpu_s":0.023843773999999485} {"method":"GET","path":"/puppet/control-repo/blob/dev/README.md","format":"json","controller":"Projects::BlobController","action":"show","status":200,"duration":87.25,"view":0.48,"db":3.42,"time":"2019-10-23T12:36:39.877Z","params":[{"key":"viewer","value":"rich"},{"key":"namespace_id","value":"puppet"},{"key":"project_id","value":"control-repo"},{"key":"id","value":"dev/README.md"}],"remote_ip":"1.2.3.4","user_id":2,"username":"dewhit11","ua":"Mozilla/5.0 (X11; Linux x86_64; rv:60.0) Gecko/20100101 Firefox/60.0","queue_duration":11.67,"gitaly_calls":3,"gitaly_duration":5.9,"rugged_calls":3,"rugged_duration_ms":2.0,"correlation_id":"HfKSroNf9na","cpu_s":0.08759858500000917} {"method":"GET","path":"/puppet/control-repo/commits/1398412c20110f345f9e1e26c26fcc594b89d209/signatures","format":"json","controller":"Projects::CommitsController","action":"signatures","status":200,"duration":106.8,"view":0.27,"db":11.35,"time":"2019-10-23T12:36:39.929Z","params":[{"key":"limit","value":"1"},{"key":"namespace_id","value":"puppet"},{"key":"project_id","value":"control-repo"},{"key":"id","value":"1398412c20110f345f9e1e26c26fcc594b89d209"}],"remote_ip":"1.2.3.4","user_id":2,"username":"dewhit11","ua":"Mozilla/5.0 (X11; Linux x86_64; rv:60.0) Gecko/20100101 Firefox/60.0","queue_duration":39.06,"gitaly_calls":3,"gitaly_duration":3.48,"rugged_calls":1,"rugged_duration_ms":2.32,"correlation_id":"gMhh6JKlse3","cpu_s":0.12418464900000004}
==> /var/log/gitlab/gitaly/current <== {"correlation_id":"37667733-2f9c-43a4-bc03-6cb909b6decd","grpc.code":"OK","grpc.meta.auth_version":"v2","grpc.meta.client_name":"gitlab-sidekiq","grpc.method":"CommitsBetween","grpc.request.deadline":"2019-10-23T12:37:09Z","grpc.request.fullMethod":"/gitaly.CommitService/CommitsBetween","grpc.request.glProjectPath":"puppet/control-repo","grpc.request.glRepository":"project-14","grpc.request.repoPath":"puppet/control-repo.git","grpc.request.repoStorage":"default","grpc.request.topLevelGroup":"puppet","grpc.service":"gitaly.CommitService","grpc.start_time":"2019-10-23T12:36:39Z","grpc.time_ms":7.968,"level":"info","msg":"finished streaming call with code OK","peer.address":"@","pid":6283,"span.kind":"server","system":"grpc","time":"2019-10-23T12:36:39Z"} {"correlation_id":"37667733-2f9c-43a4-bc03-6cb909b6decd","grpc.code":"OK","grpc.meta.auth_version":"v2","grpc.meta.client_name":"gitlab-sidekiq","grpc.method":"RepositorySize","grpc.request.deadline":"2019-10-23T12:37:09Z","grpc.request.fullMethod":"/gitaly.RepositoryService/RepositorySize","grpc.request.glProjectPath":"puppet/control-repo","grpc.request.glRepository":"project-14","grpc.request.repoPath":"puppet/control-repo.git","grpc.request.repoStorage":"default","grpc.request.topLevelGroup":"puppet","grpc.service":"gitaly.RepositoryService","grpc.start_time":"2019-10-23T12:36:39Z","grpc.time_ms":7.247,"level":"info","msg":"finished unary call with code OK","peer.address":"@","pid":6283,"span.kind":"server","system":"grpc","time":"2019-10-23T12:36:39Z"} {"correlation_id":"HfKSroNf9na","grpc.code":"OK","grpc.meta.auth_version":"v2","grpc.meta.client_name":"gitlab-web","grpc.method":"HasLocalBranches","grpc.request.deadline":"2019-10-23T12:36:49Z","grpc.request.fullMethod":"/gitaly.RepositoryService/HasLocalBranches","grpc.request.glProjectPath":"puppet/control-repo","grpc.request.glRepository":"project-14","grpc.request.repoPath":"puppet/control-repo.git","grpc.request.repoStorage":"default","grpc.request.topLevelGroup":"puppet","grpc.service":"gitaly.RepositoryService","grpc.start_time":"2019-10-23T12:36:39Z","grpc.time_ms":2.885,"level":"info","msg":"finished unary call with code OK","peer.address":"@","pid":6283,"span.kind":"server","system":"grpc","time":"2019-10-23T12:36:39Z"} {"correlation_id":"HfKSroNf9na","grpc.code":"OK","grpc.meta.auth_version":"v2","grpc.meta.client_name":"gitlab-web","grpc.method":"FindAllTagNames","grpc.request.deadline":"2019-10-23T12:36:49Z","grpc.request.fullMethod":"/gitaly.RefService/FindAllTagNames","grpc.request.glProjectPath":"puppet/control-repo","grpc.request.glRepository":"project-14","grpc.request.repoPath":"puppet/control-repo.git","grpc.request.repoStorage":"default","grpc.request.topLevelGroup":"puppet","grpc.service":"gitaly.RefService","grpc.start_time":"2019-10-23T12:36:39Z","grpc.time_ms":2.639,"level":"info","msg":"finished streaming call with code OK","peer.address":"@","pid":6283,"span.kind":"server","system":"grpc","time":"2019-10-23T12:36:39Z"} {"correlation_id":"gMhh6JKlse3","grpc.code":"OK","grpc.meta.auth_version":"v2","grpc.meta.client_name":"gitlab-web","grpc.method":"FindAllTagNames","grpc.request.deadline":"2019-10-23T12:36:49Z","grpc.request.fullMethod":"/gitaly.RefService/FindAllTagNames","grpc.request.glProjectPath":"puppet/control-repo","grpc.request.glRepository":"project-14","grpc.request.repoPath":"puppet/control-repo.git","grpc.request.repoStorage":"default","grpc.request.topLevelGroup":"puppet","grpc.service":"gitaly.RefService","grpc.start_time":"2019-10-23T12:36:39Z","grpc.time_ms":5.229,"level":"info","msg":"finished streaming call with code OK","peer.address":"@","pid":6283,"span.kind":"server","system":"grpc","time":"2019-10-23T12:36:39Z"} {"correlation_id":"HfKSroNf9na","grpc.code":"OK","grpc.meta.auth_version":"v2","grpc.meta.client_name":"gitlab-web","grpc.method":"FindCommits","grpc.request.deadline":"2019-10-23T12:37:09Z","grpc.request.fullMethod":"/gitaly.CommitService/FindCommits","grpc.request.glProjectPath":"puppet/control-repo","grpc.request.glRepository":"project-14","grpc.request.repoPath":"puppet/control-repo.git","grpc.request.repoStorage":"default","grpc.request.topLevelGroup":"puppet","grpc.service":"gitaly.CommitService","grpc.start_time":"2019-10-23T12:36:39Z","grpc.time_ms":13.501,"level":"info","msg":"finished streaming call with code OK","peer.address":"@","pid":6283,"span.kind":"server","system":"grpc","time":"2019-10-23T12:36:39Z"} {"correlation_id":"37667733-2f9c-43a4-bc03-6cb909b6decd","grpc.code":"OK","grpc.meta.auth_version":"v2","grpc.meta.client_name":"gitlab-sidekiq","grpc.method":"CommitsBetween","grpc.request.deadline":"2019-10-23T12:37:09Z","grpc.request.fullMethod":"/gitaly.CommitService/CommitsBetween","grpc.request.glProjectPath":"puppet/control-repo","grpc.request.glRepository":"project-14","grpc.request.repoPath":"puppet/control-repo.git","grpc.request.repoStorage":"default","grpc.request.topLevelGroup":"puppet","grpc.service":"gitaly.CommitService","grpc.start_time":"2019-10-23T12:36:39Z","grpc.time_ms":11.46,"level":"info","msg":"finished streaming call with code OK","peer.address":"@","pid":6283,"span.kind":"server","system":"grpc","time":"2019-10-23T12:36:39Z"} {"correlation_id":"gMhh6JKlse3","grpc.code":"OK","grpc.meta.auth_version":"v2","grpc.meta.client_name":"gitlab-web","grpc.method":"FindCommits","grpc.request.deadline":"2019-10-23T12:37:09Z","grpc.request.fullMethod":"/gitaly.CommitService/FindCommits","grpc.request.glProjectPath":"puppet/control-repo","grpc.request.glRepository":"project-14","grpc.request.repoPath":"puppet/control-repo.git","grpc.request.repoStorage":"default","grpc.request.topLevelGroup":"puppet","grpc.service":"gitaly.CommitService","grpc.start_time":"2019-10-23T12:36:39Z","grpc.time_ms":10.658,"level":"info","msg":"finished streaming call with code OK","peer.address":"@","pid":6283,"span.kind":"server","system":"grpc","time":"2019-10-23T12:36:39Z"} {"correlation_id":"gMhh6JKlse3","grpc.code":"OK","grpc.meta.auth_version":"v2","grpc.meta.client_name":"gitlab-web","grpc.method":"GetCommitSignatures","grpc.request.deadline":"2019-10-23T12:36:50Z","grpc.request.fullMethod":"/gitaly.CommitService/GetCommitSignatures","grpc.request.glProjectPath":"puppet/control-repo","grpc.request.glRepository":"project-14","grpc.request.repoPath":"puppet/control-repo.git","grpc.request.repoStorage":"default","grpc.request.topLevelGroup":"puppet","grpc.service":"gitaly.CommitService","grpc.start_time":"2019-10-23T12:36:39Z","grpc.time_ms":0.837,"level":"info","msg":"finished streaming call with code OK","peer.address":"@","pid":6283,"span.kind":"server","system":"grpc","time":"2019-10-23T12:36:39Z"} {"correlation_id":"37667733-2f9c-43a4-bc03-6cb909b6decd","grpc.code":"OK","grpc.meta.auth_version":"v2","grpc.meta.client_name":"gitlab-sidekiq","grpc.method":"ServerInfo","grpc.request.deadline":"2019-10-23T12:36:50Z","grpc.request.fullMethod":"/gitaly.ServerService/ServerInfo","grpc.service":"gitaly.ServerService","grpc.start_time":"2019-10-23T12:36:40Z","grpc.time_ms":4.075,"level":"info","msg":"finished unary call with code OK","peer.address":"@","pid":6283,"span.kind":"server","system":"grpc","time":"2019-10-23T12:36:40Z"} {"correlation_id":"37667733-2f9c-43a4-bc03-6cb909b6decd","grpc.code":"OK","grpc.meta.auth_version":"v2","grpc.meta.client_name":"gitlab-sidekiq","grpc.method":"FindAllTagNames","grpc.request.deadline":"2019-10-23T12:36:50Z","grpc.request.fullMethod":"/gitaly.RefService/FindAllTagNames","grpc.request.glProjectPath":"puppet/control-repo","grpc.request.glRepository":"project-14","grpc.request.repoPath":"puppet/control-repo.git","grpc.request.repoStorage":"default","grpc.request.topLevelGroup":"puppet","grpc.service":"gitaly.RefService","grpc.start_time":"2019-10-23T12:36:40Z","grpc.time_ms":3.027,"level":"info","msg":"finished streaming call with code OK","peer.address":"@","pid":6283,"span.kind":"server","system":"grpc","time":"2019-10-23T12:36:40Z"}
==> /var/log/gitlab/sidekiq/current <== {"severity":"INFO","time":"2019-10-23T12:36:39.683Z","class":"Namespaces::ScheduleAggregationWorker","retry":3,"queue":"update_namespace_statistics:namespaces_schedule_aggregation","queue_namespace":"update_namespace_statistics","jid":"8406c4d69cededa58714bbc8","created_at":"2019-10-23T12:36:39.679121Z","correlation_id":"37667733-2f9c-43a4-bc03-6cb909b6decd","enqueued_at":"2019-10-23T12:36:39.679501Z","pid":6504,"message":"Namespaces::ScheduleAggregationWorker JID-8406c4d69cededa58714bbc8: start","job_status":"start","scheduling_latency_s":0.003821} {"severity":"INFO","time":"2019-10-23T12:36:39.864Z","class":"ProjectCacheWorker","retry":3,"queue":"project_cache","jid":"268415669707bd8071d0519c","created_at":"2019-10-23T12:36:39.468903Z","correlation_id":"37667733-2f9c-43a4-bc03-6cb909b6decd","enqueued_at":"2019-10-23T12:36:39.469232Z","pid":6504,"message":"ProjectCacheWorker JID-268415669707bd8071d0519c: done: 0.393604 sec","job_status":"done","scheduling_latency_s":0.002207,"gitaly_calls":1,"gitaly_duration":35.11,"duration":0.393604,"cpu_s":0.073839,"completed_at":"2019-10-23T12:36:39.864795Z"} {"severity":"INFO","time":"2019-10-23T12:36:40.035Z","class":"Namespaces::RootStatisticsWorker","retry":3,"queue":"update_namespace_statistics:namespaces_root_statistics","queue_namespace":"update_namespace_statistics","jid":"1627fc0ed25148ea5b8ce441","created_at":"2019-10-23T12:36:40.032638Z","correlation_id":"37667733-2f9c-43a4-bc03-6cb909b6decd","enqueued_at":"2019-10-23T12:36:40.033002Z","pid":6504,"message":"Namespaces::RootStatisticsWorker JID-1627fc0ed25148ea5b8ce441: start","job_status":"start","scheduling_latency_s":0.002384} {"severity":"INFO","time":"2019-10-23T12:36:40.084Z","class":"Namespaces::ScheduleAggregationWorker","retry":3,"queue":"update_namespace_statistics:namespaces_schedule_aggregation","queue_namespace":"update_namespace_statistics","jid":"8406c4d69cededa58714bbc8","created_at":"2019-10-23T12:36:39.679121Z","correlation_id":"37667733-2f9c-43a4-bc03-6cb909b6decd","enqueued_at":"2019-10-23T12:36:39.679501Z","pid":6504,"message":"Namespaces::ScheduleAggregationWorker JID-8406c4d69cededa58714bbc8: done: 0.401609 sec","job_status":"done","scheduling_latency_s":0.003821,"duration":0.401609,"cpu_s":0.026736,"completed_at":"2019-10-23T12:36:40.084700Z"} {"severity":"INFO","time":"2019-10-23T12:36:40.136Z","class":"UpdateMergeRequestsWorker","retry":3,"queue":"update_merge_requests","jid":"85814e8ca06f03ba191d6ee0","created_at":"2019-10-23T12:36:39.506186Z","correlation_id":"37667733-2f9c-43a4-bc03-6cb909b6decd","enqueued_at":"2019-10-23T12:36:39.506645Z","pid":6504,"message":"UpdateMergeRequestsWorker JID-85814e8ca06f03ba191d6ee0: done: 0.626376 sec","job_status":"done","scheduling_latency_s":0.003413,"gitaly_calls":1,"gitaly_duration":79.26,"duration":0.626376,"cpu_s":0.151446,"completed_at":"2019-10-23T12:36:40.136225Z"} {"severity":"INFO","time":"2019-10-23T12:36:40.219Z","class":"Namespaces::RootStatisticsWorker","retry":3,"queue":"update_namespace_statistics:namespaces_root_statistics","queue_namespace":"update_namespace_statistics","jid":"1627fc0ed25148ea5b8ce441","created_at":"2019-10-23T12:36:40.032638Z","correlation_id":"37667733-2f9c-43a4-bc03-6cb909b6decd","enqueued_at":"2019-10-23T12:36:40.033002Z","pid":6504,"message":"Namespaces::RootStatisticsWorker JID-1627fc0ed25148ea5b8ce441: done: 0.184219 sec","job_status":"done","scheduling_latency_s":0.002384,"duration":0.184219,"cpu_s":0.038421,"completed_at":"2019-10-23T12:36:40.219421Z"}
==> /var/log/gitlab/gitlab-workhorse/current <== {"correlation_id":"HfKSroNf9na","duration_ms":116,"host":"gitlab-server.DOMAIN","level":"info","method":"GET","msg":"access","proto":"HTTP/1.1","referrer":"https://gitlab-server.DOMAIN/puppet/control-repo/blob/dev/README.md","remote_addr":"1.2.3.4:0","remote_ip":"1.2.3.4","status":200,"system":"http","time":"2019-10-23T08:36:39-04:00","uri":"/puppet/control-repo/blob/dev/README.md?format=json\u0026viewer=rich","user_agent":"Mozilla/5.0 (X11; Linux x86_64; rv:60.0) Gecko/20100101 Firefox/60.0","written_bytes":1032} {"correlation_id":"gMhh6JKlse3","duration_ms":169,"host":"gitlab-server.DOMAIN","level":"info","method":"GET","msg":"access","proto":"HTTP/1.1","referrer":"https://gitlab-server.DOMAIN/puppet/control-repo/blob/dev/README.md","remote_addr":"1.2.3.4:0","remote_ip":"1.2.3.4","status":200,"system":"http","time":"2019-10-23T08:36:39-04:00","uri":"/puppet/control-repo/commits/1398412c20110f345f9e1e26c26fcc594b89d209/signatures?limit=1","user_agent":"Mozilla/5.0 (X11; Linux x86_64; rv:60.0) Gecko/20100101 Firefox/60.0","written_bytes":17}
==> /var/log/gitlab/nginx/gitlab_access.log <== 1.2.3.4 - - [23/Oct/2019:08:36:39 -0400] "GET /puppet/control-repo/blob/dev/README.md?format=json&viewer=rich HTTP/2.0" 200 1032 "https://gitlab-server.DOMAIN/puppet/control-repo/blob/dev/README.md" "Mozilla/5.0 (X11; Linux x86_64; rv:60.0) Gecko/20100101 Firefox/60.0" 1.2.3.4 - - [23/Oct/2019:08:36:39 -0400] "GET /puppet/control-repo/commits/1398412c20110f345f9e1e26c26fcc594b89d209/signatures?limit=1 HTTP/2.0" 200 17 "https://gitlab-server.DOMAIN/puppet/control-repo/blob/dev/README.md" "Mozilla/5.0 (X11; Linux x86_64; rv:60.0) Gecko/20100101 Firefox/60.0"
==> /var/log/gitlab/gitlab-rails/sidekiq.log <== {"severity":"WARN","time":"2019-10-23T12:36:41.161Z","class":"Git::BranchHooksService","correlation_id":"37667733-2f9c-43a4-bc03-6cb909b6decd","project_id":14,"project_path":"puppet/control-repo","message":"Error creating pipeline","errors":"Missing .gitlab-ci.yml file","pipeline_params":{"before":"4f4baaf55ed7b684f609d4723ac725adfda1e124","after":"1398412c20110f345f9e1e26c26fcc594b89d209","ref":"refs/heads/dev","push_options":{},"checkout_sha":"1398412c20110f345f9e1e26c26fcc594b89d209"}} {"severity":"INFO","time":"2019-10-23T12:36:41.288Z","class":"WebHookWorker","retry":4,"queue":"web_hook","dead":false,"jid":"d894b634cf4acfd59600d54a","created_at":"2019-10-23T12:36:41.278700Z","correlation_id":"37667733-2f9c-43a4-bc03-6cb909b6decd","enqueued_at":"2019-10-23T12:36:41.282349Z","pid":6504,"message":"WebHookWorker JID-d894b634cf4acfd59600d54a: start","job_status":"start","scheduling_latency_s":0.005753} {"severity":"INFO","time":"2019-10-23T12:36:41.377Z","class":"GitGarbageCollectWorker","retry":false,"queue":"git_garbage_collect","jid":"11bf43e2d471ae29c65693d9","created_at":"2019-10-23T12:36:41.370536Z","correlation_id":"37667733-2f9c-43a4-bc03-6cb909b6decd","enqueued_at":"2019-10-23T12:36:41.370821Z","pid":6504,"message":"GitGarbageCollectWorker JID-11bf43e2d471ae29c65693d9: start","job_status":"start","scheduling_latency_s":0.006245} {"severity":"WARN","time":"2019-10-23T12:36:41.389Z","class":"WebHookWorker","retry":4,"queue":"web_hook","dead":false,"jid":"d894b634cf4acfd59600d54a","created_at":"2019-10-23T12:36:41.278700Z","correlation_id":"37667733-2f9c-43a4-bc03-6cb909b6decd","enqueued_at":"2019-10-23T12:36:41.282349Z","pid":6504,"message":"WebHookWorker JID-d894b634cf4acfd59600d54a: fail: 0.10154 sec","job_status":"fail","scheduling_latency_s":0.005753,"duration":0.10154,"cpu_s":0.045363,"completed_at":"2019-10-23T12:36:41.389406Z","error_message":"Sidekiq::JobRetry::Skip","error_class":"Sidekiq::JobRetry::Skip"} {"severity":"WARN","time":"2019-10-23T12:36:41.394Z","error_class":"Net::HTTPFatalError","error_message":"503 "Service Unavailable"","context":"Job raised exception","jobstr":"{"class":"WebHookWorker","args":[1,{"object_kind":"push","event_name":"push","before":"4f4baaf55ed7b684f609d4723ac725adfda1e124","after":"1398412c20110f345f9e1e26c26fcc594b89d209","ref":"refs/heads/dev","checkout_sha":"1398412c20110f345f9e1e26c26fcc594b89d209","message":null,"user_id":2,"user_name":"Dan White","user_username":"dewhit11","user_email":"","user_avatar":"https://secure.gravatar.com/avatar/60c4b43dc7a426293eb7417d934dd5ae?s=80\u0026d=identicon","project_id":14,"project":{"id":14,"name":"control-repo","description":"Puppet Control Repository","web_url":"https://gitlab-server.DOMAIN/puppet/control-repo","avatar_url":null,"git_ssh_url":"git@gitlab-server.DOMAIN:puppet/control-repo.git","git_http_url":"https://gitlab-server.DOMAIN/puppet/control-repo.git","namespace":"puppet","visibility_level":10,"path_with_namespace":"puppet/control-repo","default_branch":"production","ci_config_path":null,"homepage":"https://gitlab-server.DOMAIN/puppet/control-repo","url":"git@gitlab-server.DOMAIN:puppet/control-repo.git","ssh_url":"git@gitlab-server.DOMAIN:puppet/control-repo.git","http_url":"https://gitlab-server.DOMAIN/puppet/control-repo.git"},"commits":[{"id":"1398412c20110f345f9e1e26c26fcc594b89d209","message":"Update README.md","timestamp":"2019-10-23T12:36:37Z","url":"https://gitlab-server.DOMAIN/puppet/control-repo/commit/1398412c20110f345f9e1e26c26fcc594b89d209","author":{"name":"Dan White","email":"daniel.e.white@nasa.gov"},"added":[],"modified":["README.md"],"removed":[]}],"total_commits_count":1,"push_options":{},"repository":{"name":"control-repo","url":"git@gitlab-server.DOMAIN:puppet/control-repo.git","description":"Puppet Control Repository","homepage":"https://gitlab-server.DOMAIN/puppet/control-repo","git_http_url":"https://gitlab-server.DOMAIN/puppet/control-repo.git","git_ssh_url":"git@gitlab-server.DOMAIN:puppet/control-repo.git","visibility_level":10}},"push_hooks"],"retry":4,"queue":"web_hook","dead":false,"jid":"d894b634cf4acfd59600d54a","created_at":1571834201.2787008,"correlation_id":"37667733-2f9c-43a4-bc03-6cb909b6decd","enqueued_at":1571834201.2823493}","class":"WebHookWorker","args":[1,{"object_kind":"push","event_name":"push","before":"4f4baaf55ed7b684f609d4723ac725adfda1e124","after":"1398412c20110f345f9e1e26c26fcc594b89d209","ref":"refs/heads/dev","checkout_sha":"1398412c20110f345f9e1e26c26fcc594b89d209","message":null,"user_id":2,"user_name":"Dan White","user_username":"dewhit11","user_email":"","user_avatar":"https://secure.gravatar.com/avatar/60c4b43dc7a426293eb7417d934dd5ae?s=80\u0026d=identicon","project_id":14,"project":{"id":14,"name":"control-repo","description":"Puppet Control Repository","web_url":"https://gitlab-server.DOMAIN/puppet/control-repo","avatar_url":null,"git_ssh_url":"git@gitlab-server.DOMAIN:puppet/control-repo.git","git_http_url":"https://gitlab-server.DOMAIN/puppet/control-repo.git","namespace":"puppet","visibility_level":10,"path_with_namespace":"puppet/control-repo","default_branch":"production","ci_config_path":null,"homepage":"https://gitlab-server.DOMAIN/puppet/control-repo","url":"git@gitlab-server.DOMAIN:puppet/control-repo.git","ssh_url":"git@gitlab-server.DOMAIN:puppet/control-repo.git","http_url":"https://gitlab-server.DOMAIN/puppet/control-repo.git"},"commits":[{"id":"1398412c20110f345f9e1e26c26fcc594b89d209","message":"Update README.md","timestamp":"2019-10-23T12:36:37Z","url":"https://gitlab-server.DOMAIN/puppet/control-repo/commit/1398412c20110f345f9e1e26c26fcc594b89d209","author":{"name":"Dan White","email":"daniel.e.white@nasa.gov"},"added":[],"modified":["README.md"],"removed":[]}],"total_commits_count":1,"push_options":{},"repository":{"name":"control-repo","url":"git@gitlab-server.DOMAIN:puppet/control-repo.git","description":"Puppet Control Repository","homepage":"https://gitlab-server.DOMAIN/puppet/control-repo","git_http_url":"https://gitlab-server.DOMAIN/puppet/control-repo.git","git_ssh_url":"git@gitlab-server.DOMAIN:puppet/control-repo.git","visibility_level":10}},"push_hooks"],"retry":4,"queue":"web_hook","dead":false,"jid":"d894b634cf4acfd59600d54a","created_at":1571834201.2787008,"correlation_id":"37667733-2f9c-43a4-bc03-6cb909b6decd","enqueued_at":1571834201.2823493,"error_backtrace":["lib/gitlab/http.rb:24:in
perform_request'","app/services/web_hook_service.rb:77:inmake_request'","app/services/web_hook_service.rb:30:inexecute'","app/workers/web_hook_worker.rb:13:inperform'","lib/gitlab/sidekiq_daemon/monitor.rb:43:in `within_job'"]} {"severity":"INFO","time":"2019-10-23T12:36:41.486Z","class":"PostReceive","retry":3,"queue":"post_receive","jid":"fc188e4a4cb80ad000c0369c","created_at":"2019-10-23T12:36:38.886161Z","correlation_id":"37667733-2f9c-43a4-bc03-6cb909b6decd","enqueued_at":"2019-10-23T12:36:38.888801Z","pid":6504,"message":"PostReceive JID-fc188e4a4cb80ad000c0369c: done: 2.595672 sec","job_status":"done","scheduling_latency_s":0.002395,"gitaly_calls":6,"gitaly_duration":77.76,"rugged_calls":3,"rugged_duration_ms":5.12,"duration":2.595672,"cpu_s":1.755024,"completed_at":"2019-10-23T12:36:41.486590Z"}==> /var/log/gitlab/gitaly/current <== {"correlation_id":"37667733-2f9c-43a4-bc03-6cb909b6decd","grpc.code":"OK","grpc.meta.auth_version":"v2","grpc.meta.client_name":"gitlab-sidekiq","grpc.method":"CommitDelta","grpc.request.deadline":"2019-10-23T12:36:51Z","grpc.request.fullMethod":"/gitaly.DiffService/CommitDelta","grpc.request.glProjectPath":"puppet/control-repo","grpc.request.glRepository":"project-14","grpc.request.repoPath":"puppet/control-repo.git","grpc.request.repoStorage":"default","grpc.request.topLevelGroup":"puppet","grpc.service":"gitaly.DiffService","grpc.start_time":"2019-10-23T12:36:41Z","grpc.time_ms":7.898,"level":"info","msg":"finished streaming call with code OK","peer.address":"@","pid":6283,"span.kind":"server","system":"grpc","time":"2019-10-23T12:36:41Z"} {"correlation_id":"37667733-2f9c-43a4-bc03-6cb909b6decd","grpc.code":"OK","grpc.meta.auth_version":"v2","grpc.meta.client_name":"gitlab-sidekiq","grpc.method":"FilterShasWithSignatures","grpc.request.deadline":"2019-10-23T12:36:51Z","grpc.request.fullMethod":"/gitaly.CommitService/FilterShasWithSignatures","grpc.request.glProjectPath":"puppet/control-repo","grpc.request.glRepository":"project-14","grpc.request.repoPath":"puppet/control-repo.git","grpc.request.repoStorage":"default","grpc.request.topLevelGroup":"puppet","grpc.service":"gitaly.CommitService","grpc.start_time":"2019-10-23T12:36:41Z","grpc.time_ms":7.979,"level":"info","msg":"finished streaming call with code OK","peer.address":"@","pid":6283,"span.kind":"server","system":"grpc","time":"2019-10-23T12:36:41Z"} {"correlation_id":"37667733-2f9c-43a4-bc03-6cb909b6decd","grpc.code":"OK","grpc.meta.auth_version":"v2","grpc.meta.client_name":"gitlab-sidekiq","grpc.method":"RepackIncremental","grpc.request.deadline":"2019-10-23T18:36:41Z","grpc.request.fullMethod":"/gitaly.RepositoryService/RepackIncremental","grpc.request.glProjectPath":"puppet/control-repo","grpc.request.glRepository":"project-14","grpc.request.repoPath":"puppet/control-repo.git","grpc.request.repoStorage":"default","grpc.request.topLevelGroup":"puppet","grpc.service":"gitaly.RepositoryService","grpc.start_time":"2019-10-23T12:36:41Z","grpc.time_ms":35.051,"level":"info","msg":"finished unary call with code OK","peer.address":"@","pid":6283,"span.kind":"server","system":"grpc","time":"2019-10-23T12:36:41Z"} {"correlation_id":"37667733-2f9c-43a4-bc03-6cb909b6decd","grpc.meta.auth_version":"v2","grpc.meta.client_name":"gitlab-sidekiq","grpc.method":"RepackIncremental","grpc.request.deadline":"2019-10-23T18:36:41Z","grpc.request.fullMethod":"/gitaly.RepositoryService/RepackIncremental","grpc.request.glProjectPath":"puppet/control-repo","grpc.request.glRepository":"project-14","grpc.request.repoPath":"puppet/control-repo.git","grpc.request.repoStorage":"default","grpc.request.topLevelGroup":"puppet","grpc.service":"gitaly.RepositoryService","grpc.start_time":"2019-10-23T12:36:41Z","level":"error","msg":"Marked 1 islands, done.\n","peer.address":"@","pid":6283,"span.kind":"server","system":"grpc","time":"2019-10-23T12:36:41Z"}
==> /var/log/gitlab/sidekiq/current <== {"severity":"WARN","time":"2019-10-23T12:36:41.161Z","class":"Git::BranchHooksService","correlation_id":"37667733-2f9c-43a4-bc03-6cb909b6decd","project_id":14,"project_path":"puppet/control-repo","message":"Error creating pipeline","errors":"Missing .gitlab-ci.yml file","pipeline_params":{"before":"4f4baaf55ed7b684f609d4723ac725adfda1e124","after":"1398412c20110f345f9e1e26c26fcc594b89d209","ref":"refs/heads/dev","push_options":{},"checkout_sha":"1398412c20110f345f9e1e26c26fcc594b89d209"}} {"severity":"INFO","time":"2019-10-23T12:36:41.288Z","class":"WebHookWorker","retry":4,"queue":"web_hook","dead":false,"jid":"d894b634cf4acfd59600d54a","created_at":"2019-10-23T12:36:41.278700Z","correlation_id":"37667733-2f9c-43a4-bc03-6cb909b6decd","enqueued_at":"2019-10-23T12:36:41.282349Z","pid":6504,"message":"WebHookWorker JID-d894b634cf4acfd59600d54a: start","job_status":"start","scheduling_latency_s":0.005753} {"severity":"INFO","time":"2019-10-23T12:36:41.377Z","class":"GitGarbageCollectWorker","retry":false,"queue":"git_garbage_collect","jid":"11bf43e2d471ae29c65693d9","created_at":"2019-10-23T12:36:41.370536Z","correlation_id":"37667733-2f9c-43a4-bc03-6cb909b6decd","enqueued_at":"2019-10-23T12:36:41.370821Z","pid":6504,"message":"GitGarbageCollectWorker JID-11bf43e2d471ae29c65693d9: start","job_status":"start","scheduling_latency_s":0.006245} {"severity":"WARN","time":"2019-10-23T12:36:41.389Z","class":"WebHookWorker","retry":4,"queue":"web_hook","dead":false,"jid":"d894b634cf4acfd59600d54a","created_at":"2019-10-23T12:36:41.278700Z","correlation_id":"37667733-2f9c-43a4-bc03-6cb909b6decd","enqueued_at":"2019-10-23T12:36:41.282349Z","pid":6504,"message":"WebHookWorker JID-d894b634cf4acfd59600d54a: fail: 0.10154 sec","job_status":"fail","scheduling_latency_s":0.005753,"duration":0.10154,"cpu_s":0.045363,"completed_at":"2019-10-23T12:36:41.389406Z","error_message":"Sidekiq::JobRetry::Skip","error_class":"Sidekiq::JobRetry::Skip"} {"severity":"WARN","time":"2019-10-23T12:36:41.394Z","error_class":"Net::HTTPFatalError","error_message":"503 "Service Unavailable"","context":"Job raised exception","jobstr":"{"class":"WebHookWorker","args":[1,{"object_kind":"push","event_name":"push","before":"4f4baaf55ed7b684f609d4723ac725adfda1e124","after":"1398412c20110f345f9e1e26c26fcc594b89d209","ref":"refs/heads/dev","checkout_sha":"1398412c20110f345f9e1e26c26fcc594b89d209","message":null,"user_id":2,"user_name":"Dan White","user_username":"dewhit11","user_email":"","user_avatar":"https://secure.gravatar.com/avatar/60c4b43dc7a426293eb7417d934dd5ae?s=80\u0026d=identicon","project_id":14,"project":{"id":14,"name":"control-repo","description":"Puppet Control Repository","web_url":"https://gitlab-server.DOMAIN/puppet/control-repo","avatar_url":null,"git_ssh_url":"git@gitlab-server.DOMAIN:puppet/control-repo.git","git_http_url":"https://gitlab-server.DOMAIN/puppet/control-repo.git","namespace":"puppet","visibility_level":10,"path_with_namespace":"puppet/control-repo","default_branch":"production","ci_config_path":null,"homepage":"https://gitlab-server.DOMAIN/puppet/control-repo","url":"git@gitlab-server.DOMAIN:puppet/control-repo.git","ssh_url":"git@gitlab-server.DOMAIN:puppet/control-repo.git","http_url":"https://gitlab-server.DOMAIN/puppet/control-repo.git"},"commits":[{"id":"1398412c20110f345f9e1e26c26fcc594b89d209","message":"Update README.md","timestamp":"2019-10-23T12:36:37Z","url":"https://gitlab-server.DOMAIN/puppet/control-repo/commit/1398412c20110f345f9e1e26c26fcc594b89d209","author":{"name":"Dan White","email":"daniel.e.white@nasa.gov"},"added":[],"modified":["README.md"],"removed":[]}],"total_commits_count":1,"push_options":{},"repository":{"name":"control-repo","url":"git@gitlab-server.DOMAIN:puppet/control-repo.git","description":"Puppet Control Repository","homepage":"https://gitlab-server.DOMAIN/puppet/control-repo","git_http_url":"https://gitlab-server.DOMAIN/puppet/control-repo.git","git_ssh_url":"git@gitlab-server.DOMAIN:puppet/control-repo.git","visibility_level":10}},"push_hooks"],"retry":4,"queue":"web_hook","dead":false,"jid":"d894b634cf4acfd59600d54a","created_at":1571834201.2787008,"correlation_id":"37667733-2f9c-43a4-bc03-6cb909b6decd","enqueued_at":1571834201.2823493}","class":"WebHookWorker","args":[1,{"object_kind":"push","event_name":"push","before":"4f4baaf55ed7b684f609d4723ac725adfda1e124","after":"1398412c20110f345f9e1e26c26fcc594b89d209","ref":"refs/heads/dev","checkout_sha":"1398412c20110f345f9e1e26c26fcc594b89d209","message":null,"user_id":2,"user_name":"Dan White","user_username":"dewhit11","user_email":"","user_avatar":"https://secure.gravatar.com/avatar/60c4b43dc7a426293eb7417d934dd5ae?s=80\u0026d=identicon","project_id":14,"project":{"id":14,"name":"control-repo","description":"Puppet Control Repository","web_url":"https://gitlab-server.DOMAIN/puppet/control-repo","avatar_url":null,"git_ssh_url":"git@gitlab-server.DOMAIN:puppet/control-repo.git","git_http_url":"https://gitlab-server.DOMAIN/puppet/control-repo.git","namespace":"puppet","visibility_level":10,"path_with_namespace":"puppet/control-repo","default_branch":"production","ci_config_path":null,"homepage":"https://gitlab-server.DOMAIN/puppet/control-repo","url":"git@gitlab-server.DOMAIN:puppet/control-repo.git","ssh_url":"git@gitlab-server.DOMAIN:puppet/control-repo.git","http_url":"https://gitlab-server.DOMAIN/puppet/control-repo.git"},"commits":[{"id":"1398412c20110f345f9e1e26c26fcc594b89d209","message":"Update README.md","timestamp":"2019-10-23T12:36:37Z","url":"https://gitlab-server.DOMAIN/puppet/control-repo/commit/1398412c20110f345f9e1e26c26fcc594b89d209","author":{"name":"Dan White","email":"daniel.e.white@nasa.gov"},"added":[],"modified":["README.md"],"removed":[]}],"total_commits_count":1,"push_options":{},"repository":{"name":"control-repo","url":"git@gitlab-server.DOMAIN:puppet/control-repo.git","description":"Puppet Control Repository","homepage":"https://gitlab-server.DOMAIN/puppet/control-repo","git_http_url":"https://gitlab-server.DOMAIN/puppet/control-repo.git","git_ssh_url":"git@gitlab-server.DOMAIN:puppet/control-repo.git","visibility_level":10}},"push_hooks"],"retry":4,"queue":"web_hook","dead":false,"jid":"d894b634cf4acfd59600d54a","created_at":1571834201.2787008,"correlation_id":"37667733-2f9c-43a4-bc03-6cb909b6decd","enqueued_at":1571834201.2823493,"error_backtrace":["lib/gitlab/http.rb:24:in
perform_request'","app/services/web_hook_service.rb:77:inmake_request'","app/services/web_hook_service.rb:30:inexecute'","app/workers/web_hook_worker.rb:13:inperform'","lib/gitlab/sidekiq_daemon/monitor.rb:43:in `within_job'"]} {"severity":"INFO","time":"2019-10-23T12:36:41.486Z","class":"PostReceive","retry":3,"queue":"post_receive","jid":"fc188e4a4cb80ad000c0369c","created_at":"2019-10-23T12:36:38.886161Z","correlation_id":"37667733-2f9c-43a4-bc03-6cb909b6decd","enqueued_at":"2019-10-23T12:36:38.888801Z","pid":6504,"message":"PostReceive JID-fc188e4a4cb80ad000c0369c: done: 2.595672 sec","job_status":"done","scheduling_latency_s":0.002395,"gitaly_calls":6,"gitaly_duration":77.76,"rugged_calls":3,"rugged_duration_ms":5.12,"duration":2.595672,"cpu_s":1.755024,"completed_at":"2019-10-23T12:36:41.486590Z"}==> /var/log/gitlab/alertmanager/current <== 2019-10-23_12:36:40.69057 level=info ts=2019-10-23T12:36:40.687Z caller=main.go:197 msg="Starting Alertmanager" version="(version=0.18.0, branch=master, revision=)" 2019-10-23_12:36:40.69060 level=info ts=2019-10-23T12:36:40.687Z caller=main.go:198 build_context="(go=go1.12.7, user=GitLab-Omnibus, date=)" 2019-10-23_12:36:40.69061 level=warn ts=2019-10-23T12:36:40.688Z caller=cluster.go:154 component=cluster err="couldn't deduce an advertise address: no private IP found, explicit advertise addr not provided" 2019-10-23_12:36:40.69061 level=error ts=2019-10-23T12:36:40.690Z caller=main.go:222 msg="unable to initialize gossip mesh" err="create memberlist: Failed to get final advertise address: No private IP address found, and explicit IP not provided"
==> /var/log/gitlab/gitlab-rails/sidekiq.log <== {"severity":"INFO","time":"2019-10-23T12:36:41.665Z","class":"GitGarbageCollectWorker","retry":false,"queue":"git_garbage_collect","jid":"11bf43e2d471ae29c65693d9","created_at":"2019-10-23T12:36:41.370536Z","correlation_id":"37667733-2f9c-43a4-bc03-6cb909b6decd","enqueued_at":"2019-10-23T12:36:41.370821Z","pid":6504,"message":"GitGarbageCollectWorker JID-11bf43e2d471ae29c65693d9: done: 0.288186 sec","job_status":"done","scheduling_latency_s":0.006245,"gitaly_calls":1,"gitaly_duration":40.45,"duration":0.288186,"cpu_s":0.189742,"completed_at":"2019-10-23T12:36:41.665009Z"}
==> /var/log/gitlab/sidekiq/current <== {"severity":"INFO","time":"2019-10-23T12:36:41.665Z","class":"GitGarbageCollectWorker","retry":false,"queue":"git_garbage_collect","jid":"11bf43e2d471ae29c65693d9","created_at":"2019-10-23T12:36:41.370536Z","correlation_id":"37667733-2f9c-43a4-bc03-6cb909b6decd","enqueued_at":"2019-10-23T12:36:41.370821Z","pid":6504,"message":"GitGarbageCollectWorker JID-11bf43e2d471ae29c65693d9: done: 0.288186 sec","job_status":"done","scheduling_latency_s":0.006245,"gitaly_calls":1,"gitaly_duration":40.45,"duration":0.288186,"cpu_s":0.189742,"completed_at":"2019-10-23T12:36:41.665009Z"}
==> /var/log/gitlab/alertmanager/current <== 2019-10-23_12:36:41.78685 level=info ts=2019-10-23T12:36:41.786Z caller=main.go:197 msg="Starting Alertmanager" version="(version=0.18.0, branch=master, revision=)" 2019-10-23_12:36:41.78689 level=info ts=2019-10-23T12:36:41.786Z caller=main.go:198 build_context="(go=go1.12.7, user=GitLab-Omnibus, date=)" 2019-10-23_12:36:41.78814 level=warn ts=2019-10-23T12:36:41.788Z caller=cluster.go:154 component=cluster err="couldn't deduce an advertise address: no private IP found, explicit advertise addr not provided" 2019-10-23_12:36:41.79653 level=error ts=2019-10-23T12:36:41.796Z caller=main.go:222 msg="unable to initialize gossip mesh" err="create memberlist: Failed to get final advertise address: No private IP address found, and explicit IP not provided"
==> /var/log/gitlab/gitlab-rails/production.log <== Started GET "/-/metrics" for 127.0.0.1 at 2019-10-23 08:36:42 -0400 Processing by Gitlab::RequestForgeryProtection::Controller#index as HTML Completed 200 OK in 0ms (ActiveRecord: 0.0ms | Elasticsearch: 0.0ms) Processing by MetricsController#index as HTML Completed 200 OK in 35ms (Views: 0.8ms | ActiveRecord: 0.0ms | Elasticsearch: 0.0ms)
==> /var/log/gitlab/gitlab-rails/production_json.log <== {"method":"GET","path":"/-/metrics","format":"html","controller":"Gitlab::RequestForgeryProtection::Controller","action":"index","status":200,"duration":0.6,"view":0.0,"db":0.0,"time":"2019-10-23T12:36:42.713Z","params":[],"remote_ip":null,"user_id":null,"username":null,"ua":null,"queue_duration":null,"correlation_id":null,"cpu_s":0.004105084999991959} {"method":"GET","path":"/-/metrics","format":"html","controller":"MetricsController","action":"index","status":200,"duration":35.55,"view":0.78,"db":0.0,"time":"2019-10-23T12:36:42.755Z","params":[],"remote_ip":null,"user_id":null,"username":null,"ua":null,"queue_duration":null,"correlation_id":"f09608b1-76ac-4437-9db4-26c73fa9349c","cpu_s":0.04576720399998635}
==> /var/log/gitlab/alertmanager/current <== 2019-10-23_12:36:42.87456 level=info ts=2019-10-23T12:36:42.873Z caller=main.go:197 msg="Starting Alertmanager" version="(version=0.18.0, branch=master, revision=)" 2019-10-23_12:36:42.87459 level=info ts=2019-10-23T12:36:42.873Z caller=main.go:198 build_context="(go=go1.12.7, user=GitLab-Omnibus, date=)" 2019-10-23_12:36:42.87496 level=warn ts=2019-10-23T12:36:42.874Z caller=cluster.go:154 component=cluster err="couldn't deduce an advertise address: no private IP found, explicit advertise addr not provided" 2019-10-23_12:36:42.87682 level=error ts=2019-10-23T12:36:42.876Z caller=main.go:222 msg="unable to initialize gossip mesh" err="create memberlist: Failed to get final advertise address: No private IP address found, and explicit IP not provided" 2019-10-23_12:36:43.95781 level=info ts=2019-10-23T12:36:43.957Z caller=main.go:197 msg="Starting Alertmanager" version="(version=0.18.0, branch=master, revision=)" 2019-10-23_12:36:43.95785 level=info ts=2019-10-23T12:36:43.957Z caller=main.go:198 build_context="(go=go1.12.7, user=GitLab-Omnibus, date=)" 2019-10-23_12:36:43.95898 level=warn ts=2019-10-23T12:36:43.958Z caller=cluster.go:154 component=cluster err="couldn't deduce an advertise address: no private IP found, explicit advertise addr not provided" 2019-10-23_12:36:43.96097 level=error ts=2019-10-23T12:36:43.960Z caller=main.go:222 msg="unable to initialize gossip mesh" err="create memberlist: Failed to get final advertise address: No private IP address found, and explicit IP not provided" 2019-10-23_12:36:45.03528 level=info ts=2019-10-23T12:36:45.035Z caller=main.go:197 msg="Starting Alertmanager" version="(version=0.18.0, branch=master, revision=)" 2019-10-23_12:36:45.03532 level=info ts=2019-10-23T12:36:45.035Z caller=main.go:198 build_context="(go=go1.12.7, user=GitLab-Omnibus, date=)" 2019-10-23_12:36:45.03645 level=warn ts=2019-10-23T12:36:45.036Z caller=cluster.go:154 component=cluster err="couldn't deduce an advertise address: no private IP found, explicit advertise addr not provided" 2019-10-23_12:36:45.03833 level=error ts=2019-10-23T12:36:45.038Z caller=main.go:222 msg="unable to initialize gossip mesh" err="create memberlist: Failed to get final advertise address: No private IP address found, and explicit IP not provided" 2019-10-23_12:36:46.11128 level=info ts=2019-10-23T12:36:46.111Z caller=main.go:197 msg="Starting Alertmanager" version="(version=0.18.0, branch=master, revision=)" 2019-10-23_12:36:46.11132 level=info ts=2019-10-23T12:36:46.111Z caller=main.go:198 build_context="(go=go1.12.7, user=GitLab-Omnibus, date=)" 2019-10-23_12:36:46.11243 level=warn ts=2019-10-23T12:36:46.112Z caller=cluster.go:154 component=cluster err="couldn't deduce an advertise address: no private IP found, explicit advertise addr not provided" 2019-10-23_12:36:46.11423 level=error ts=2019-10-23T12:36:46.114Z caller=main.go:222 msg="unable to initialize gossip mesh" err="create memberlist: Failed to get final advertise address: No private IP address found, and explicit IP not provided" 2019-10-23_12:36:47.19665 level=info ts=2019-10-23T12:36:47.196Z caller=main.go:197 msg="Starting Alertmanager" version="(version=0.18.0, branch=master, revision=)" 2019-10-23_12:36:47.19669 level=info ts=2019-10-23T12:36:47.196Z caller=main.go:198 build_context="(go=go1.12.7, user=GitLab-Omnibus, date=)" 2019-10-23_12:36:47.19785 level=warn ts=2019-10-23T12:36:47.197Z caller=cluster.go:154 component=cluster err="couldn't deduce an advertise address: no private IP found, explicit advertise addr not provided" 2019-10-23_12:36:47.19976 level=error ts=2019-10-23T12:36:47.199Z caller=main.go:222 msg="unable to initialize gossip mesh" err="create memberlist: Failed to get final advertise address: No private IP address found, and explicit IP not provided" 2019-10-23_12:36:48.27920 level=info ts=2019-10-23T12:36:48.279Z caller=main.go:197 msg="Starting Alertmanager" version="(version=0.18.0, branch=master, revision=)" 2019-10-23_12:36:48.27925 level=info ts=2019-10-23T12:36:48.279Z caller=main.go:198 build_context="(go=go1.12.7, user=GitLab-Omnibus, date=)" 2019-10-23_12:36:48.28035 level=warn ts=2019-10-23T12:36:48.280Z caller=cluster.go:154 component=cluster err="couldn't deduce an advertise address: no private IP found, explicit advertise addr not provided" 2019-10-23_12:36:48.28212 level=error ts=2019-10-23T12:36:48.282Z caller=main.go:222 msg="unable to initialize gossip mesh" err="create memberlist: Failed to get final advertise address: No private IP address found, and explicit IP not provided" 2019-10-23_12:36:49.35866 level=info ts=2019-10-23T12:36:49.358Z caller=main.go:197 msg="Starting Alertmanager" version="(version=0.18.0, branch=master, revision=)" 2019-10-23_12:36:49.35870 level=info ts=2019-10-23T12:36:49.358Z caller=main.go:198 build_context="(go=go1.12.7, user=GitLab-Omnibus, date=)" 2019-10-23_12:36:49.35980 level=warn ts=2019-10-23T12:36:49.359Z caller=cluster.go:154 component=cluster err="couldn't deduce an advertise address: no private IP found, explicit advertise addr not provided" 2019-10-23_12:36:49.36163 level=error ts=2019-10-23T12:36:49.361Z caller=main.go:222 msg="unable to initialize gossip mesh" err="create memberlist: Failed to get final advertise address: No private IP address found, and explicit IP not provided" 2019-10-23_12:36:50.44180 level=info ts=2019-10-23T12:36:50.441Z caller=main.go:197 msg="Starting Alertmanager" version="(version=0.18.0, branch=master, revision=)" 2019-10-23_12:36:50.44184 level=info ts=2019-10-23T12:36:50.441Z caller=main.go:198 build_context="(go=go1.12.7, user=GitLab-Omnibus, date=)" 2019-10-23_12:36:50.44300 level=warn ts=2019-10-23T12:36:50.442Z caller=cluster.go:154 component=cluster err="couldn't deduce an advertise address: no private IP found, explicit advertise addr not provided" 2019-10-23_12:36:50.44507 level=error ts=2019-10-23T12:36:50.444Z caller=main.go:222 msg="unable to initialize gossip mesh" err="create memberlist: Failed to get final advertise address: No private IP address found, and explicit IP not provided" 2019-10-23_12:36:51.52231 level=info ts=2019-10-23T12:36:51.522Z caller=main.go:197 msg="Starting Alertmanager" version="(version=0.18.0, branch=master, revision=)" 2019-10-23_12:36:51.52235 level=info ts=2019-10-23T12:36:51.522Z caller=main.go:198 build_context="(go=go1.12.7, user=GitLab-Omnibus, date=)" 2019-10-23_12:36:51.52349 level=warn ts=2019-10-23T12:36:51.523Z caller=cluster.go:154 component=cluster err="couldn't deduce an advertise address: no private IP found, explicit advertise addr not provided" 2019-10-23_12:36:51.52533 level=error ts=2019-10-23T12:36:51.525Z caller=main.go:222 msg="unable to initialize gossip mesh" err="create memberlist: Failed to get final advertise address: No private IP address found, and explicit IP not provided" 2019-10-23_12:36:52.60255 level=info ts=2019-10-23T12:36:52.602Z caller=main.go:197 msg="Starting Alertmanager" version="(version=0.18.0, branch=master, revision=)" 2019-10-23_12:36:52.60257 level=info ts=2019-10-23T12:36:52.602Z caller=main.go:198 build_context="(go=go1.12.7, user=GitLab-Omnibus, date=)" 2019-10-23_12:36:52.60340 level=warn ts=2019-10-23T12:36:52.603Z caller=cluster.go:154 component=cluster err="couldn't deduce an advertise address: no private IP found, explicit advertise addr not provided" 2019-10-23_12:36:52.60527 level=error ts=2019-10-23T12:36:52.605Z caller=main.go:222 msg="unable to initialize gossip mesh" err="create memberlist: Failed to get final advertise address: No private IP address found, and explicit IP not provided"
==> /var/log/gitlab/gitlab-rails/sidekiq_exporter.log <== [2019-10-23T08:36:52.690-0400] 127.0.0.1 - - [23/Oct/2019:08:36:52 EDT] "GET /metrics HTTP/1.1" 200 19005 "-" "Prometheus/2.12.0"
==> /var/log/gitlab/alertmanager/current <== 2019-10-23_12:36:53.68244 level=info ts=2019-10-23T12:36:53.682Z caller=main.go:197 msg="Starting Alertmanager" version="(version=0.18.0, branch=master, revision=)" 2019-10-23_12:36:53.68250 level=info ts=2019-10-23T12:36:53.682Z caller=main.go:198 build_context="(go=go1.12.7, user=GitLab-Omnibus, date=)" 2019-10-23_12:36:53.68370 level=warn ts=2019-10-23T12:36:53.683Z caller=cluster.go:154 component=cluster err="couldn't deduce an advertise address: no private IP found, explicit advertise addr not provided" 2019-10-23_12:36:53.68561 level=error ts=2019-10-23T12:36:53.685Z caller=main.go:222 msg="unable to initialize gossip mesh" err="create memberlist: Failed to get final advertise address: No private IP address found, and explicit IP not provided" 2019-10-23_12:36:54.76107 level=info ts=2019-10-23T12:36:54.760Z caller=main.go:197 msg="Starting Alertmanager" version="(version=0.18.0, branch=master, revision=)" 2019-10-23_12:36:54.76111 level=info ts=2019-10-23T12:36:54.761Z caller=main.go:198 build_context="(go=go1.12.7, user=GitLab-Omnibus, date=)" 2019-10-23_12:36:54.76224 level=warn ts=2019-10-23T12:36:54.762Z caller=cluster.go:154 component=cluster err="couldn't deduce an advertise address: no private IP found, explicit advertise addr not provided" 2019-10-23_12:36:54.76410 level=error ts=2019-10-23T12:36:54.763Z caller=main.go:222 msg="unable to initialize gossip mesh" err="create memberlist: Failed to get final advertise address: No private IP address found, and explicit IP not provided" 2019-10-23_12:36:55.85073 level=info ts=2019-10-23T12:36:55.850Z caller=main.go:197 msg="Starting Alertmanager" version="(version=0.18.0, branch=master, revision=)" 2019-10-23_12:36:55.85077 level=info ts=2019-10-23T12:36:55.850Z caller=main.go:198 build_context="(go=go1.12.7, user=GitLab-Omnibus, date=)" 2019-10-23_12:36:55.85190 level=warn ts=2019-10-23T12:36:55.851Z caller=cluster.go:154 component=cluster err="couldn't deduce an advertise address: no private IP found, explicit advertise addr not provided" 2019-10-23_12:36:55.85373 level=error ts=2019-10-23T12:36:55.853Z caller=main.go:222 msg="unable to initialize gossip mesh" err="create memberlist: Failed to get final advertise address: No private IP address found, and explicit IP not provided" 2019-10-23_12:36:56.93374 level=info ts=2019-10-23T12:36:56.933Z caller=main.go:197 msg="Starting Alertmanager" version="(version=0.18.0, branch=master, revision=)" 2019-10-23_12:36:56.93378 level=info ts=2019-10-23T12:36:56.933Z caller=main.go:198 build_context="(go=go1.12.7, user=GitLab-Omnibus, date=)" 2019-10-23_12:36:56.93492 level=warn ts=2019-10-23T12:36:56.934Z caller=cluster.go:154 component=cluster err="couldn't deduce an advertise address: no private IP found, explicit advertise addr not provided" 2019-10-23_12:36:56.93708 level=error ts=2019-10-23T12:36:56.936Z caller=main.go:222 msg="unable to initialize gossip mesh" err="create memberlist: Failed to get final advertise address: No private IP address found, and explicit IP not provided"
==> /var/log/gitlab/gitlab-rails/production.log <== Started GET "/-/metrics" for 127.0.0.1 at 2019-10-23 08:36:57 -0400 Processing by Gitlab::RequestForgeryProtection::Controller#index as HTML Completed 200 OK in 0ms (ActiveRecord: 0.0ms | Elasticsearch: 0.0ms) Processing by MetricsController#index as HTML Completed 200 OK in 35ms (Views: 0.7ms | ActiveRecord: 0.0ms | Elasticsearch: 0.0ms)
==> /var/log/gitlab/gitlab-rails/production_json.log <== {"method":"GET","path":"/-/metrics","format":"html","controller":"Gitlab::RequestForgeryProtection::Controller","action":"index","status":200,"duration":0.65,"view":0.0,"db":0.0,"time":"2019-10-23T12:36:57.709Z","params":[],"remote_ip":null,"user_id":null,"username":null,"ua":null,"queue_duration":null,"correlation_id":null,"cpu_s":0.00425833799999964} {"method":"GET","path":"/-/metrics","format":"html","controller":"MetricsController","action":"index","status":200,"duration":35.68,"view":0.72,"db":0.0,"time":"2019-10-23T12:36:57.751Z","params":[],"remote_ip":null,"user_id":null,"username":null,"ua":null,"queue_duration":null,"correlation_id":"200e397c-a3f4-430b-b33a-1fed0dba79a1","cpu_s":0.0460236320000007}
==> /var/log/gitlab/alertmanager/current <== 2019-10-23_12:36:58.01255 level=info ts=2019-10-23T12:36:58.012Z caller=main.go:197 msg="Starting Alertmanager" version="(version=0.18.0, branch=master, revision=)" 2019-10-23_12:36:58.01259 level=info ts=2019-10-23T12:36:58.012Z caller=main.go:198 build_context="(go=go1.12.7, user=GitLab-Omnibus, date=)" 2019-10-23_12:36:58.01371 level=warn ts=2019-10-23T12:36:58.013Z caller=cluster.go:154 component=cluster err="couldn't deduce an advertise address: no private IP found, explicit advertise addr not provided" 2019-10-23_12:36:58.01550 level=error ts=2019-10-23T12:36:58.015Z caller=main.go:222 msg="unable to initialize gossip mesh" err="create memberlist: Failed to get final advertise address: No private IP address found, and explicit IP not provided" 2019-10-23_12:36:59.09401 level=info ts=2019-10-23T12:36:59.093Z caller=main.go:197 msg="Starting Alertmanager" version="(version=0.18.0, branch=master, revision=)" 2019-10-23_12:36:59.09405 level=info ts=2019-10-23T12:36:59.093Z caller=main.go:198 build_context="(go=go1.12.7, user=GitLab-Omnibus, date=)" 2019-10-23_12:36:59.09519 level=warn ts=2019-10-23T12:36:59.095Z caller=cluster.go:154 component=cluster err="couldn't deduce an advertise address: no private IP found, explicit advertise addr not provided" 2019-10-23_12:36:59.09701 level=error ts=2019-10-23T12:36:59.096Z caller=main.go:222 msg="unable to initialize gossip mesh" err="create memberlist: Failed to get final advertise address: No private IP address found, and explicit IP not provided" 2019-10-23_12:37:00.18042 level=info ts=2019-10-23T12:37:00.180Z caller=main.go:197 msg="Starting Alertmanager" version="(version=0.18.0, branch=master, revision=)" 2019-10-23_12:37:00.18046 level=info ts=2019-10-23T12:37:00.180Z caller=main.go:198 build_context="(go=go1.12.7, user=GitLab-Omnibus, date=)" 2019-10-23_12:37:00.18160 level=warn ts=2019-10-23T12:37:00.181Z caller=cluster.go:154 component=cluster err="couldn't deduce an advertise address: no private IP found, explicit advertise addr not provided" 2019-10-23_12:37:00.18338 level=error ts=2019-10-23T12:37:00.183Z caller=main.go:222 msg="unable to initialize gossip mesh" err="create memberlist: Failed to get final advertise address: No private IP address found, and explicit IP not provided" 2019-10-23_12:37:01.26745 level=info ts=2019-10-23T12:37:01.267Z caller=main.go:197 msg="Starting Alertmanager" version="(version=0.18.0, branch=master, revision=)" 2019-10-23_12:37:01.26751 level=info ts=2019-10-23T12:37:01.267Z caller=main.go:198 build_context="(go=go1.12.7, user=GitLab-Omnibus, date=)" 2019-10-23_12:37:01.26863 level=warn ts=2019-10-23T12:37:01.268Z caller=cluster.go:154 component=cluster err="couldn't deduce an advertise address: no private IP found, explicit advertise addr not provided" 2019-10-23_12:37:01.27073 level=error ts=2019-10-23T12:37:01.270Z caller=main.go:222 msg="unable to initialize gossip mesh" err="create memberlist: Failed to get final advertise address: No private IP address found, and explicit IP not provided" 2019-10-23_12:37:02.34838 level=info ts=2019-10-23T12:37:02.348Z caller=main.go:197 msg="Starting Alertmanager" version="(version=0.18.0, branch=master, revision=)" 2019-10-23_12:37:02.34842 level=info ts=2019-10-23T12:37:02.348Z caller=main.go:198 build_context="(go=go1.12.7, user=GitLab-Omnibus, date=)" 2019-10-23_12:37:02.34963 level=warn ts=2019-10-23T12:37:02.349Z caller=cluster.go:154 component=cluster err="couldn't deduce an advertise address: no private IP found, explicit advertise addr not provided" 2019-10-23_12:37:02.35141 level=error ts=2019-10-23T12:37:02.351Z caller=main.go:222 msg="unable to initialize gossip mesh" err="create memberlist: Failed to get final advertise address: No private IP address found, and explicit IP not provided" 2019-10-23_12:37:03.43469 level=info ts=2019-10-23T12:37:03.434Z caller=main.go:197 msg="Starting Alertmanager" version="(version=0.18.0, branch=master, revision=)" 2019-10-23_12:37:03.43473 level=info ts=2019-10-23T12:37:03.434Z caller=main.go:198 build_context="(go=go1.12.7, user=GitLab-Omnibus, date=)" 2019-10-23_12:37:03.43587 level=warn ts=2019-10-23T12:37:03.435Z caller=cluster.go:154 component=cluster err="couldn't deduce an advertise address: no private IP found, explicit advertise addr not provided" 2019-10-23_12:37:03.43802 level=error ts=2019-10-23T12:37:03.437Z caller=main.go:222 msg="unable to initialize gossip mesh" err="create memberlist: Failed to get final advertise address: No private IP address found, and explicit IP not provided" 2019-10-23_12:37:04.51190 level=info ts=2019-10-23T12:37:04.511Z caller=main.go:197 msg="Starting Alertmanager" version="(version=0.18.0, branch=master, revision=)" 2019-10-23_12:37:04.51195 level=info ts=2019-10-23T12:37:04.511Z caller=main.go:198 build_context="(go=go1.12.7, user=GitLab-Omnibus, date=)" 2019-10-23_12:37:04.51309 level=warn ts=2019-10-23T12:37:04.513Z caller=cluster.go:154 component=cluster err="couldn't deduce an advertise address: no private IP found, explicit advertise addr not provided" 2019-10-23_12:37:04.51500 level=error ts=2019-10-23T12:37:04.514Z caller=main.go:222 msg="unable to initialize gossip mesh" err="create memberlist: Failed to get final advertise address: No private IP address found, and explicit IP not provided"
==> /var/log/gitlab/prometheus/current <== 2019-10-23_12:37:05.42565 level=warn ts=2019-10-23T12:37:05.422Z caller=manager.go:567 component="rule manager" group="GitLab Saturation Ratios" msg="Error on ingesting results from rule evaluation with different value but same timestamp" numDropped=1
==> /var/log/gitlab/alertmanager/current <== 2019-10-23_12:37:05.58813 level=info ts=2019-10-23T12:37:05.587Z caller=main.go:197 msg="Starting Alertmanager" version="(version=0.18.0, branch=master, revision=)" 2019-10-23_12:37:05.58817 level=info ts=2019-10-23T12:37:05.588Z caller=main.go:198 build_context="(go=go1.12.7, user=GitLab-Omnibus, date=)" 2019-10-23_12:37:05.58928 level=warn ts=2019-10-23T12:37:05.589Z caller=cluster.go:154 component=cluster err="couldn't deduce an advertise address: no private IP found, explicit advertise addr not provided" 2019-10-23_12:37:05.59108 level=error ts=2019-10-23T12:37:05.590Z caller=main.go:222 msg="unable to initialize gossip mesh" err="create memberlist: Failed to get final advertise address: No private IP address found, and explicit IP not provided" 2019-10-23_12:37:06.66682 level=info ts=2019-10-23T12:37:06.666Z caller=main.go:197 msg="Starting Alertmanager" version="(version=0.18.0, branch=master, revision=)" 2019-10-23_12:37:06.66686 level=info ts=2019-10-23T12:37:06.666Z caller=main.go:198 build_context="(go=go1.12.7, user=GitLab-Omnibus, date=)" 2019-10-23_12:37:06.66796 level=warn ts=2019-10-23T12:37:06.667Z caller=cluster.go:154 component=cluster err="couldn't deduce an advertise address: no private IP found, explicit advertise addr not provided" 2019-10-23_12:37:06.66989 level=error ts=2019-10-23T12:37:06.669Z caller=main.go:222 msg="unable to initialize gossip mesh" err="create memberlist: Failed to get final advertise address: No private IP address found, and explicit IP not provided"
==> /var/log/gitlab/gitlab-rails/sidekiq_exporter.log <== [2019-10-23T08:37:07.693-0400] 127.0.0.1 - - [23/Oct/2019:08:37:07 EDT] "GET /metrics HTTP/1.1" 200 19005 "-" "Prometheus/2.12.0"
==> /var/log/gitlab/redis/current <== 2019-10-23_12:37:08.09163 6491:M 23 Oct 08:37:08.091 * 10 changes in 300 seconds. Saving... 2019-10-23_12:37:08.09528 6491:M 23 Oct 08:37:08.091 * Background saving started by pid 26686 2019-10-23_12:37:08.10832 26686:C 23 Oct 08:37:08.108 * DB saved on disk 2019-10-23_12:37:08.10905 26686:C 23 Oct 08:37:08.108 * RDB: 0 MB of memory used by copy-on-write 2019-10-23_12:37:08.19268 6491:M 23 Oct 08:37:08.192 * Background saving terminated with success
==> /var/log/gitlab/alertmanager/current <== 2019-10-23_12:37:07.74651 level=info ts=2019-10-23T12:37:07.746Z caller=main.go:197 msg="Starting Alertmanager" version="(version=0.18.0, branch=master, revision=)" 2019-10-23_12:37:07.74655 level=info ts=2019-10-23T12:37:07.746Z caller=main.go:198 build_context="(go=go1.12.7, user=GitLab-Omnibus, date=)" 2019-10-23_12:37:07.74782 level=warn ts=2019-10-23T12:37:07.747Z caller=cluster.go:154 component=cluster err="couldn't deduce an advertise address: no private IP found, explicit advertise addr not provided" 2019-10-23_12:37:07.74987 level=error ts=2019-10-23T12:37:07.749Z caller=main.go:222 msg="unable to initialize gossip mesh" err="create memberlist: Failed to get final advertise address: No private IP address found, and explicit IP not provided" 2019-10-23_12:37:08.82727 level=info ts=2019-10-23T12:37:08.827Z caller=main.go:197 msg="Starting Alertmanager" version="(version=0.18.0, branch=master, revision=)" 2019-10-23_12:37:08.82731 level=info ts=2019-10-23T12:37:08.827Z caller=main.go:198 build_context="(go=go1.12.7, user=GitLab-Omnibus, date=)" 2019-10-23_12:37:08.82840 level=warn ts=2019-10-23T12:37:08.828Z caller=cluster.go:154 component=cluster err="couldn't deduce an advertise address: no private IP found, explicit advertise addr not provided" 2019-10-23_12:37:08.83024 level=error ts=2019-10-23T12:37:08.830Z caller=main.go:222 msg="unable to initialize gossip mesh" err="create memberlist: Failed to get final advertise address: No private IP address found, and explicit IP not provided" 2019-10-23_12:37:09.90505 level=info ts=2019-10-23T12:37:09.904Z caller=main.go:197 msg="Starting Alertmanager" version="(version=0.18.0, branch=master, revision=)" 2019-10-23_12:37:09.90509 level=info ts=2019-10-23T12:37:09.905Z caller=main.go:198 build_context="(go=go1.12.7, user=GitLab-Omnibus, date=)" 2019-10-23_12:37:09.90626 level=warn ts=2019-10-23T12:37:09.906Z caller=cluster.go:154 component=cluster err="couldn't deduce an advertise address: no private IP found, explicit advertise addr not provided" 2019-10-23_12:37:09.90817 level=error ts=2019-10-23T12:37:09.908Z caller=main.go:222 msg="unable to initialize gossip mesh" err="create memberlist: Failed to get final advertise address: No private IP address found, and explicit IP not provided"
==> /var/log/gitlab/gitlab-rails/sidekiq.log <== {"severity":"INFO","time":"2019-10-23T12:37:11.050Z","queue":"cronjob:geo_sidekiq_cron_config","class":"Geo::SidekiqCronConfigWorker","retry":false,"queue_namespace":"cronjob","jid":"27a6a3c7339d874f568a031e","created_at":"2019-10-23T12:37:11.047185Z","correlation_id":"6daf588a-e4bf-484f-a7cb-138478298048","enqueued_at":"2019-10-23T12:37:11.047687Z","pid":6504,"message":"Geo::SidekiqCronConfigWorker JID-27a6a3c7339d874f568a031e: start","job_status":"start","scheduling_latency_s":0.002532} {"severity":"INFO","time":"2019-10-23T12:37:11.568Z","queue":"cronjob:update_all_mirrors","class":"UpdateAllMirrorsWorker","retry":false,"queue_namespace":"cronjob","jid":"e80ecd9aa7bc34edd9c1442d","created_at":"2019-10-23T12:37:11.563169Z","correlation_id":"21e471b7-1950-493d-9971-23f16d640897","enqueued_at":"2019-10-23T12:37:11.564666Z","pid":6504,"message":"UpdateAllMirrorsWorker JID-e80ecd9aa7bc34edd9c1442d: start","job_status":"start","scheduling_latency_s":0.003664}
==> /var/log/gitlab/sidekiq/current <== {"severity":"INFO","time":"2019-10-23T12:37:11.050Z","queue":"cronjob:geo_sidekiq_cron_config","class":"Geo::SidekiqCronConfigWorker","retry":false,"queue_namespace":"cronjob","jid":"27a6a3c7339d874f568a031e","created_at":"2019-10-23T12:37:11.047185Z","correlation_id":"6daf588a-e4bf-484f-a7cb-138478298048","enqueued_at":"2019-10-23T12:37:11.047687Z","pid":6504,"message":"Geo::SidekiqCronConfigWorker JID-27a6a3c7339d874f568a031e: start","job_status":"start","scheduling_latency_s":0.002532} {"severity":"INFO","time":"2019-10-23T12:37:11.568Z","queue":"cronjob:update_all_mirrors","class":"UpdateAllMirrorsWorker","retry":false,"queue_namespace":"cronjob","jid":"e80ecd9aa7bc34edd9c1442d","created_at":"2019-10-23T12:37:11.563169Z","correlation_id":"21e471b7-1950-493d-9971-23f16d640897","enqueued_at":"2019-10-23T12:37:11.564666Z","pid":6504,"message":"UpdateAllMirrorsWorker JID-e80ecd9aa7bc34edd9c1442d: start","job_status":"start","scheduling_latency_s":0.003664}
==> /var/log/gitlab/alertmanager/current <== 2019-10-23_12:37:10.98474 level=info ts=2019-10-23T12:37:10.984Z caller=main.go:197 msg="Starting Alertmanager" version="(version=0.18.0, branch=master, revision=)" 2019-10-23_12:37:10.98478 level=info ts=2019-10-23T12:37:10.984Z caller=main.go:198 build_context="(go=go1.12.7, user=GitLab-Omnibus, date=)" 2019-10-23_12:37:10.98589 level=warn ts=2019-10-23T12:37:10.985Z caller=cluster.go:154 component=cluster err="couldn't deduce an advertise address: no private IP found, explicit advertise addr not provided" 2019-10-23_12:37:10.98773 level=error ts=2019-10-23T12:37:10.987Z caller=main.go:222 msg="unable to initialize gossip mesh" err="create memberlist: Failed to get final advertise address: No private IP address found, and explicit IP not provided"
==> /var/log/gitlab/gitlab-rails/sidekiq.log <== {"severity":"INFO","time":"2019-10-23T12:37:11.658Z","queue":"cronjob:geo_sidekiq_cron_config","class":"Geo::SidekiqCronConfigWorker","retry":false,"queue_namespace":"cronjob","jid":"27a6a3c7339d874f568a031e","created_at":"2019-10-23T12:37:11.047185Z","correlation_id":"6daf588a-e4bf-484f-a7cb-138478298048","enqueued_at":"2019-10-23T12:37:11.047687Z","pid":6504,"message":"Geo::SidekiqCronConfigWorker JID-27a6a3c7339d874f568a031e: done: 0.608039 sec","job_status":"done","scheduling_latency_s":0.002532,"duration":0.608039,"cpu_s":0.030814,"completed_at":"2019-10-23T12:37:11.658033Z"} {"severity":"INFO","time":"2019-10-23T12:37:11.666Z","queue":"cronjob:update_all_mirrors","class":"UpdateAllMirrorsWorker","retry":false,"queue_namespace":"cronjob","jid":"e80ecd9aa7bc34edd9c1442d","created_at":"2019-10-23T12:37:11.563169Z","correlation_id":"21e471b7-1950-493d-9971-23f16d640897","enqueued_at":"2019-10-23T12:37:11.564666Z","pid":6504,"message":"UpdateAllMirrorsWorker JID-e80ecd9aa7bc34edd9c1442d: done: 0.097988 sec","job_status":"done","scheduling_latency_s":0.003664,"duration":0.097988,"cpu_s":0.02955,"completed_at":"2019-10-23T12:37:11.666095Z"}
==> /var/log/gitlab/sidekiq/current <== {"severity":"INFO","time":"2019-10-23T12:37:11.658Z","queue":"cronjob:geo_sidekiq_cron_config","class":"Geo::SidekiqCronConfigWorker","retry":false,"queue_namespace":"cronjob","jid":"27a6a3c7339d874f568a031e","created_at":"2019-10-23T12:37:11.047185Z","correlation_id":"6daf588a-e4bf-484f-a7cb-138478298048","enqueued_at":"2019-10-23T12:37:11.047687Z","pid":6504,"message":"Geo::SidekiqCronConfigWorker JID-27a6a3c7339d874f568a031e: done: 0.608039 sec","job_status":"done","scheduling_latency_s":0.002532,"duration":0.608039,"cpu_s":0.030814,"completed_at":"2019-10-23T12:37:11.658033Z"} {"severity":"INFO","time":"2019-10-23T12:37:11.666Z","queue":"cronjob:update_all_mirrors","class":"UpdateAllMirrorsWorker","retry":false,"queue_namespace":"cronjob","jid":"e80ecd9aa7bc34edd9c1442d","created_at":"2019-10-23T12:37:11.563169Z","correlation_id":"21e471b7-1950-493d-9971-23f16d640897","enqueued_at":"2019-10-23T12:37:11.564666Z","pid":6504,"message":"UpdateAllMirrorsWorker JID-e80ecd9aa7bc34edd9c1442d: done: 0.097988 sec","job_status":"done","scheduling_latency_s":0.003664,"duration":0.097988,"cpu_s":0.02955,"completed_at":"2019-10-23T12:37:11.666095Z"}
==> /var/log/gitlab/alertmanager/current <== 2019-10-23_12:37:12.06955 level=info ts=2019-10-23T12:37:12.069Z caller=main.go:197 msg="Starting Alertmanager" version="(version=0.18.0, branch=master, revision=)" 2019-10-23_12:37:12.06958 level=info ts=2019-10-23T12:37:12.069Z caller=main.go:198 build_context="(go=go1.12.7, user=GitLab-Omnibus, date=)" 2019-10-23_12:37:12.07039 level=warn ts=2019-10-23T12:37:12.070Z caller=cluster.go:154 component=cluster err="couldn't deduce an advertise address: no private IP found, explicit advertise addr not provided" 2019-10-23_12:37:12.07228 level=error ts=2019-10-23T12:37:12.072Z caller=main.go:222 msg="unable to initialize gossip mesh" err="create memberlist: Failed to get final advertise address: No private IP address found, and explicit IP not provided"
==> /var/log/gitlab/gitlab-rails/production.log <== Started GET "/-/metrics" for 127.0.0.1 at 2019-10-23 08:37:12 -0400 Processing by Gitlab::RequestForgeryProtection::Controller#index as HTML Completed 200 OK in 0ms (ActiveRecord: 0.0ms | Elasticsearch: 0.0ms) Processing by MetricsController#index as HTML Completed 200 OK in 34ms (Views: 0.6ms | ActiveRecord: 0.0ms | Elasticsearch: 0.0ms)
==> /var/log/gitlab/gitlab-rails/production_json.log <== {"method":"GET","path":"/-/metrics","format":"html","controller":"Gitlab::RequestForgeryProtection::Controller","action":"index","status":200,"duration":0.61,"view":0.0,"db":0.0,"time":"2019-10-23T12:37:12.709Z","params":[],"remote_ip":null,"user_id":null,"username":null,"ua":null,"queue_duration":null,"correlation_id":null,"cpu_s":0.0040008139999940795} {"method":"GET","path":"/-/metrics","format":"html","controller":"MetricsController","action":"index","status":200,"duration":34.63,"view":0.61,"db":0.0,"time":"2019-10-23T12:37:12.750Z","params":[],"remote_ip":null,"user_id":null,"username":null,"ua":null,"queue_duration":null,"correlation_id":"7bc073d2-fb20-4b4c-964f-cf36f77e0fae","cpu_s":0.04445163500000149}
==> /var/log/gitlab/alertmanager/current <== 2019-10-23_12:37:13.15597 level=info ts=2019-10-23T12:37:13.155Z caller=main.go:197 msg="Starting Alertmanager" version="(version=0.18.0, branch=master, revision=)" 2019-10-23_12:37:13.15601 level=info ts=2019-10-23T12:37:13.155Z caller=main.go:198 build_context="(go=go1.12.7, user=GitLab-Omnibus, date=)" 2019-10-23_12:37:13.15716 level=warn ts=2019-10-23T12:37:13.157Z caller=cluster.go:154 component=cluster err="couldn't deduce an advertise address: no private IP found, explicit advertise addr not provided" 2019-10-23_12:37:13.15908 level=error ts=2019-10-23T12:37:13.158Z caller=main.go:222 msg="unable to initialize gossip mesh" err="create memberlist: Failed to get final advertise address: No private IP address found, and explicit IP not provided" 2019-10-23_12:37:14.23499 level=info ts=2019-10-23T12:37:14.232Z caller=main.go:197 msg="Starting Alertmanager" version="(version=0.18.0, branch=master, revision=)" 2019-10-23_12:37:14.23501 level=info ts=2019-10-23T12:37:14.232Z caller=main.go:198 build_context="(go=go1.12.7, user=GitLab-Omnibus, date=)" 2019-10-23_12:37:14.23502 level=warn ts=2019-10-23T12:37:14.234Z caller=cluster.go:154 component=cluster err="couldn't deduce an advertise address: no private IP found, explicit advertise addr not provided" 2019-10-23_12:37:14.23610 level=error ts=2019-10-23T12:37:14.235Z caller=main.go:222 msg="unable to initialize gossip mesh" err="create memberlist: Failed to get final advertise address: No private IP address found, and explicit IP not provided" 2019-10-23_12:37:15.31064 level=info ts=2019-10-23T12:37:15.310Z caller=main.go:197 msg="Starting Alertmanager" version="(version=0.18.0, branch=master, revision=)" 2019-10-23_12:37:15.31070 level=info ts=2019-10-23T12:37:15.310Z caller=main.go:198 build_context="(go=go1.12.7, user=GitLab-Omnibus, date=)" 2019-10-23_12:37:15.31185 level=warn ts=2019-10-23T12:37:15.311Z caller=cluster.go:154 component=cluster err="couldn't deduce an advertise address: no private IP found, explicit advertise addr not provided" 2019-10-23_12:37:15.32044 level=error ts=2019-10-23T12:37:15.320Z caller=main.go:222 msg="unable to initialize gossip mesh" err="create memberlist: Failed to get final advertise address: No private IP address found, and explicit IP not provided" 2019-10-23_12:37:16.39525 level=info ts=2019-10-23T12:37:16.395Z caller=main.go:197 msg="Starting Alertmanager" version="(version=0.18.0, branch=master, revision=)" 2019-10-23_12:37:16.39529 level=info ts=2019-10-23T12:37:16.395Z caller=main.go:198 build_context="(go=go1.12.7, user=GitLab-Omnibus, date=)" 2019-10-23_12:37:16.39639 level=warn ts=2019-10-23T12:37:16.396Z caller=cluster.go:154 component=cluster err="couldn't deduce an advertise address: no private IP found, explicit advertise addr not provided" 2019-10-23_12:37:16.39822 level=error ts=2019-10-23T12:37:16.398Z caller=main.go:222 msg="unable to initialize gossip mesh" err="create memberlist: Failed to get final advertise address: No private IP address found, and explicit IP not provided"
==> /var/log/gitlab/gitlab-rails/sidekiq.log <== {"severity":"INFO","time":"2019-10-23T12:37:16.978Z","class":"WebHookWorker","retry":4,"queue":"web_hook","dead":false,"jid":"d894b634cf4acfd59600d54a","created_at":"2019-10-23T12:36:41.278700Z","correlation_id":"37667733-2f9c-43a4-bc03-6cb909b6decd","enqueued_at":"2019-10-23T12:37:16.975144Z","error_message":"503 "Service Unavailable"","error_class":"Net::HTTPFatalError","failed_at":1571834201.382918,"retry_count":0,"pid":6504,"message":"WebHookWorker JID-d894b634cf4acfd59600d54a: start","job_status":"start","scheduling_latency_s":0.00289} {"severity":"WARN","time":"2019-10-23T12:37:17.022Z","class":"WebHookWorker","retry":4,"queue":"web_hook","dead":false,"jid":"d894b634cf4acfd59600d54a","created_at":"2019-10-23T12:36:41.278700Z","correlation_id":"37667733-2f9c-43a4-bc03-6cb909b6decd","enqueued_at":"2019-10-23T12:37:16.975144Z","error_message":"Sidekiq::JobRetry::Skip","error_class":"Sidekiq::JobRetry::Skip","failed_at":"2019-10-23T12:36:41.382917Z","retry_count":0,"pid":6504,"message":"WebHookWorker JID-d894b634cf4acfd59600d54a: fail: 0.045019 sec","job_status":"fail","scheduling_latency_s":0.00289,"duration":0.045019,"cpu_s":0.035393,"completed_at":"2019-10-23T12:37:17.022759Z"} {"severity":"WARN","time":"2019-10-23T12:37:17.026Z","error_class":"Net::HTTPFatalError","error_message":"503 "Service Unavailable"","context":"Job raised exception","jobstr":"{"class":"WebHookWorker","args":[1,{"object_kind":"push","event_name":"push","before":"4f4baaf55ed7b684f609d4723ac725adfda1e124","after":"1398412c20110f345f9e1e26c26fcc594b89d209","ref":"refs/heads/dev","checkout_sha":"1398412c20110f345f9e1e26c26fcc594b89d209","message":null,"user_id":2,"user_name":"Dan White","user_username":"dewhit11","user_email":"","user_avatar":"https://secure.gravatar.com/avatar/60c4b43dc7a426293eb7417d934dd5ae?s=80\u0026d=identicon","project_id":14,"project":{"id":14,"name":"control-repo","description":"Puppet Control Repository","web_url":"https://gitlab-server.DOMAIN/puppet/control-repo","avatar_url":null,"git_ssh_url":"git@gitlab-server.DOMAIN:puppet/control-repo.git","git_http_url":"https://gitlab-server.DOMAIN/puppet/control-repo.git","namespace":"puppet","visibility_level":10,"path_with_namespace":"puppet/control-repo","default_branch":"production","ci_config_path":null,"homepage":"https://gitlab-server.DOMAIN/puppet/control-repo","url":"git@gitlab-server.DOMAIN:puppet/control-repo.git","ssh_url":"git@gitlab-server.DOMAIN:puppet/control-repo.git","http_url":"https://gitlab-server.DOMAIN/puppet/control-repo.git"},"commits":[{"id":"1398412c20110f345f9e1e26c26fcc594b89d209","message":"Update README.md","timestamp":"2019-10-23T12:36:37Z","url":"https://gitlab-server.DOMAIN/puppet/control-repo/commit/1398412c20110f345f9e1e26c26fcc594b89d209","author":{"name":"Dan White","email":"daniel.e.white@nasa.gov"},"added":[],"modified":["README.md"],"removed":[]}],"total_commits_count":1,"push_options":{},"repository":{"name":"control-repo","url":"git@gitlab-server.DOMAIN:puppet/control-repo.git","description":"Puppet Control Repository","homepage":"https://gitlab-server.DOMAIN/puppet/control-repo","git_http_url":"https://gitlab-server.DOMAIN/puppet/control-repo.git","git_ssh_url":"git@gitlab-server.DOMAIN:puppet/control-repo.git","visibility_level":10}},"push_hooks"],"retry":4,"queue":"web_hook","dead":false,"jid":"d894b634cf4acfd59600d54a","created_at":1571834201.2787008,"correlation_id":"37667733-2f9c-43a4-bc03-6cb909b6decd","enqueued_at":1571834236.9751441,"error_message":"503 \"Service Unavailable\"","error_class":"Net::HTTPFatalError","failed_at":1571834201.382918,"retry_count":0}","class":"WebHookWorker","args":[1,{"object_kind":"push","event_name":"push","before":"4f4baaf55ed7b684f609d4723ac725adfda1e124","after":"1398412c20110f345f9e1e26c26fcc594b89d209","ref":"refs/heads/dev","checkout_sha":"1398412c20110f345f9e1e26c26fcc594b89d209","message":null,"user_id":2,"user_name":"Dan White","user_username":"dewhit11","user_email":"","user_avatar":"https://secure.gravatar.com/avatar/60c4b43dc7a426293eb7417d934dd5ae?s=80\u0026d=identicon","project_id":14,"project":{"id":14,"name":"control-repo","description":"Puppet Control Repository","web_url":"https://gitlab-server.DOMAIN/puppet/control-repo","avatar_url":null,"git_ssh_url":"git@gitlab-server.DOMAIN:puppet/control-repo.git","git_http_url":"https://gitlab-server.DOMAIN/puppet/control-repo.git","namespace":"puppet","visibility_level":10,"path_with_namespace":"puppet/control-repo","default_branch":"production","ci_config_path":null,"homepage":"https://gitlab-server.DOMAIN/puppet/control-repo","url":"git@gitlab-server.DOMAIN:puppet/control-repo.git","ssh_url":"git@gitlab-server.DOMAIN:puppet/control-repo.git","http_url":"https://gitlab-server.DOMAIN/puppet/control-repo.git"},"commits":[{"id":"1398412c20110f345f9e1e26c26fcc594b89d209","message":"Update README.md","timestamp":"2019-10-23T12:36:37Z","url":"https://gitlab-server.DOMAIN/puppet/control-repo/commit/1398412c20110f345f9e1e26c26fcc594b89d209","author":{"name":"Dan White","email":"daniel.e.white@nasa.gov"},"added":[],"modified":["README.md"],"removed":[]}],"total_commits_count":1,"push_options":{},"repository":{"name":"control-repo","url":"git@gitlab-server.DOMAIN:puppet/control-repo.git","description":"Puppet Control Repository","homepage":"https://gitlab-server.DOMAIN/puppet/control-repo","git_http_url":"https://gitlab-server.DOMAIN/puppet/control-repo.git","git_ssh_url":"git@gitlab-server.DOMAIN:puppet/control-repo.git","visibility_level":10}},"push_hooks"],"retry":4,"queue":"web_hook","dead":false,"jid":"d894b634cf4acfd59600d54a","created_at":1571834201.2787008,"correlation_id":"37667733-2f9c-43a4-bc03-6cb909b6decd","enqueued_at":1571834236.9751441,"failed_at":1571834201.382918,"retry_count":0,"error_backtrace":["lib/gitlab/http.rb:24:in
perform_request'","app/services/web_hook_service.rb:77:inmake_request'","app/services/web_hook_service.rb:30:inexecute'","app/workers/web_hook_worker.rb:13:inperform'","lib/gitlab/sidekiq_daemon/monitor.rb:43:in `within_job'"]}==> /var/log/gitlab/sidekiq/current <== {"severity":"INFO","time":"2019-10-23T12:37:16.978Z","class":"WebHookWorker","retry":4,"queue":"web_hook","dead":false,"jid":"d894b634cf4acfd59600d54a","created_at":"2019-10-23T12:36:41.278700Z","correlation_id":"37667733-2f9c-43a4-bc03-6cb909b6decd","enqueued_at":"2019-10-23T12:37:16.975144Z","error_message":"503 "Service Unavailable"","error_class":"Net::HTTPFatalError","failed_at":1571834201.382918,"retry_count":0,"pid":6504,"message":"WebHookWorker JID-d894b634cf4acfd59600d54a: start","job_status":"start","scheduling_latency_s":0.00289} {"severity":"WARN","time":"2019-10-23T12:37:17.022Z","class":"WebHookWorker","retry":4,"queue":"web_hook","dead":false,"jid":"d894b634cf4acfd59600d54a","created_at":"2019-10-23T12:36:41.278700Z","correlation_id":"37667733-2f9c-43a4-bc03-6cb909b6decd","enqueued_at":"2019-10-23T12:37:16.975144Z","error_message":"Sidekiq::JobRetry::Skip","error_class":"Sidekiq::JobRetry::Skip","failed_at":"2019-10-23T12:36:41.382917Z","retry_count":0,"pid":6504,"message":"WebHookWorker JID-d894b634cf4acfd59600d54a: fail: 0.045019 sec","job_status":"fail","scheduling_latency_s":0.00289,"duration":0.045019,"cpu_s":0.035393,"completed_at":"2019-10-23T12:37:17.022759Z"} {"severity":"WARN","time":"2019-10-23T12:37:17.026Z","error_class":"Net::HTTPFatalError","error_message":"503 "Service Unavailable"","context":"Job raised exception","jobstr":"{"class":"WebHookWorker","args":[1,{"object_kind":"push","event_name":"push","before":"4f4baaf55ed7b684f609d4723ac725adfda1e124","after":"1398412c20110f345f9e1e26c26fcc594b89d209","ref":"refs/heads/dev","checkout_sha":"1398412c20110f345f9e1e26c26fcc594b89d209","message":null,"user_id":2,"user_name":"Dan White","user_username":"dewhit11","user_email":"","user_avatar":"https://secure.gravatar.com/avatar/60c4b43dc7a426293eb7417d934dd5ae?s=80\u0026d=identicon","project_id":14,"project":{"id":14,"name":"control-repo","description":"Puppet Control Repository","web_url":"https://gitlab-server.DOMAIN/puppet/control-repo","avatar_url":null,"git_ssh_url":"git@gitlab-server.DOMAIN:puppet/control-repo.git","git_http_url":"https://gitlab-server.DOMAIN/puppet/control-repo.git","namespace":"puppet","visibility_level":10,"path_with_namespace":"puppet/control-repo","default_branch":"production","ci_config_path":null,"homepage":"https://gitlab-server.DOMAIN/puppet/control-repo","url":"git@gitlab-server.DOMAIN:puppet/control-repo.git","ssh_url":"git@gitlab-server.DOMAIN:puppet/control-repo.git","http_url":"https://gitlab-server.DOMAIN/puppet/control-repo.git"},"commits":[{"id":"1398412c20110f345f9e1e26c26fcc594b89d209","message":"Update README.md","timestamp":"2019-10-23T12:36:37Z","url":"https://gitlab-server.DOMAIN/puppet/control-repo/commit/1398412c20110f345f9e1e26c26fcc594b89d209","author":{"name":"Dan White","email":"daniel.e.white@nasa.gov"},"added":[],"modified":["README.md"],"removed":[]}],"total_commits_count":1,"push_options":{},"repository":{"name":"control-repo","url":"git@gitlab-server.DOMAIN:puppet/control-repo.git","description":"Puppet Control Repository","homepage":"https://gitlab-server.DOMAIN/puppet/control-repo","git_http_url":"https://gitlab-server.DOMAIN/puppet/control-repo.git","git_ssh_url":"git@gitlab-server.DOMAIN:puppet/control-repo.git","visibility_level":10}},"push_hooks"],"retry":4,"queue":"web_hook","dead":false,"jid":"d894b634cf4acfd59600d54a","created_at":1571834201.2787008,"correlation_id":"37667733-2f9c-43a4-bc03-6cb909b6decd","enqueued_at":1571834236.9751441,"error_message":"503 \"Service Unavailable\"","error_class":"Net::HTTPFatalError","failed_at":1571834201.382918,"retry_count":0}","class":"WebHookWorker","args":[1,{"object_kind":"push","event_name":"push","before":"4f4baaf55ed7b684f609d4723ac725adfda1e124","after":"1398412c20110f345f9e1e26c26fcc594b89d209","ref":"refs/heads/dev","checkout_sha":"1398412c20110f345f9e1e26c26fcc594b89d209","message":null,"user_id":2,"user_name":"Dan White","user_username":"dewhit11","user_email":"","user_avatar":"https://secure.gravatar.com/avatar/60c4b43dc7a426293eb7417d934dd5ae?s=80\u0026d=identicon","project_id":14,"project":{"id":14,"name":"control-repo","description":"Puppet Control Repository","web_url":"https://gitlab-server.DOMAIN/puppet/control-repo","avatar_url":null,"git_ssh_url":"git@gitlab-server.DOMAIN:puppet/control-repo.git","git_http_url":"https://gitlab-server.DOMAIN/puppet/control-repo.git","namespace":"puppet","visibility_level":10,"path_with_namespace":"puppet/control-repo","default_branch":"production","ci_config_path":null,"homepage":"https://gitlab-server.DOMAIN/puppet/control-repo","url":"git@gitlab-server.DOMAIN:puppet/control-repo.git","ssh_url":"git@gitlab-server.DOMAIN:puppet/control-repo.git","http_url":"https://gitlab-server.DOMAIN/puppet/control-repo.git"},"commits":[{"id":"1398412c20110f345f9e1e26c26fcc594b89d209","message":"Update README.md","timestamp":"2019-10-23T12:36:37Z","url":"https://gitlab-server.DOMAIN/puppet/control-repo/commit/1398412c20110f345f9e1e26c26fcc594b89d209","author":{"name":"Dan White","email":"daniel.e.white@nasa.gov"},"added":[],"modified":["README.md"],"removed":[]}],"total_commits_count":1,"push_options":{},"repository":{"name":"control-repo","url":"git@gitlab-server.DOMAIN:puppet/control-repo.git","description":"Puppet Control Repository","homepage":"https://gitlab-server.DOMAIN/puppet/control-repo","git_http_url":"https://gitlab-server.DOMAIN/puppet/control-repo.git","git_ssh_url":"git@gitlab-server.DOMAIN:puppet/control-repo.git","visibility_level":10}},"push_hooks"],"retry":4,"queue":"web_hook","dead":false,"jid":"d894b634cf4acfd59600d54a","created_at":1571834201.2787008,"correlation_id":"37667733-2f9c-43a4-bc03-6cb909b6decd","enqueued_at":1571834236.9751441,"failed_at":1571834201.382918,"retry_count":0,"error_backtrace":["lib/gitlab/http.rb:24:in
perform_request'","app/services/web_hook_service.rb:77:inmake_request'","app/services/web_hook_service.rb:30:inexecute'","app/workers/web_hook_worker.rb:13:inperform'","lib/gitlab/sidekiq_daemon/monitor.rb:43:in `within_job'"]}==> /var/log/gitlab/alertmanager/current <== 2019-10-23_12:37:17.47837 level=info ts=2019-10-23T12:37:17.478Z caller=main.go:197 msg="Starting Alertmanager" version="(version=0.18.0, branch=master, revision=)" 2019-10-23_12:37:17.47843 level=info ts=2019-10-23T12:37:17.478Z caller=main.go:198 build_context="(go=go1.12.7, user=GitLab-Omnibus, date=)" 2019-10-23_12:37:17.47957 level=warn ts=2019-10-23T12:37:17.479Z caller=cluster.go:154 component=cluster err="couldn't deduce an advertise address: no private IP found, explicit advertise addr not provided" 2019-10-23_12:37:17.48169 level=error ts=2019-10-23T12:37:17.481Z caller=main.go:222 msg="unable to initialize gossip mesh" err="create memberlist: Failed to get final advertise address: No private IP address found, and explicit IP not provided" 2019-10-23_12:37:18.55869 level=info ts=2019-10-23T12:37:18.558Z caller=main.go:197 msg="Starting Alertmanager" version="(version=0.18.0, branch=master, revision=)" 2019-10-23_12:37:18.55873 level=info ts=2019-10-23T12:37:18.558Z caller=main.go:198 build_context="(go=go1.12.7, user=GitLab-Omnibus, date=)" 2019-10-23_12:37:18.55993 level=warn ts=2019-10-23T12:37:18.559Z caller=cluster.go:154 component=cluster err="couldn't deduce an advertise address: no private IP found, explicit advertise addr not provided" 2019-10-23_12:37:18.56183 level=error ts=2019-10-23T12:37:18.561Z caller=main.go:222 msg="unable to initialize gossip mesh" err="create memberlist: Failed to get final advertise address: No private IP address found, and explicit IP not provided" 2019-10-23_12:37:19.64577 level=info ts=2019-10-23T12:37:19.645Z caller=main.go:197 msg="Starting Alertmanager" version="(version=0.18.0, branch=master, revision=)" 2019-10-23_12:37:19.64581 level=info ts=2019-10-23T12:37:19.645Z caller=main.go:198 build_context="(go=go1.12.7, user=GitLab-Omnibus, date=)" 2019-10-23_12:37:19.64689 level=warn ts=2019-10-23T12:37:19.646Z caller=cluster.go:154 component=cluster err="couldn't deduce an advertise address: no private IP found, explicit advertise addr not provided" 2019-10-23_12:37:19.64866 level=error ts=2019-10-23T12:37:19.648Z caller=main.go:222 msg="unable to initialize gossip mesh" err="create memberlist: Failed to get final advertise address: No private IP address found, and explicit IP not provided" 2019-10-23_12:37:20.72660 level=info ts=2019-10-23T12:37:20.726Z caller=main.go:197 msg="Starting Alertmanager" version="(version=0.18.0, branch=master, revision=)" 2019-10-23_12:37:20.72664 level=info ts=2019-10-23T12:37:20.726Z caller=main.go:198 build_context="(go=go1.12.7, user=GitLab-Omnibus, date=)" 2019-10-23_12:37:20.72779 level=warn ts=2019-10-23T12:37:20.727Z caller=cluster.go:154 component=cluster err="couldn't deduce an advertise address: no private IP found, explicit advertise addr not provided" 2019-10-23_12:37:20.72965 level=error ts=2019-10-23T12:37:20.729Z caller=main.go:222 msg="unable to initialize gossip mesh" err="create memberlist: Failed to get final advertise address: No private IP address found, and explicit IP not provided" 2019-10-23_12:37:21.80388 level=info ts=2019-10-23T12:37:21.803Z caller=main.go:197 msg="Starting Alertmanager" version="(version=0.18.0, branch=master, revision=)" 2019-10-23_12:37:21.80392 level=info ts=2019-10-23T12:37:21.803Z caller=main.go:198 build_context="(go=go1.12.7, user=GitLab-Omnibus, date=)" 2019-10-23_12:37:21.80509 level=warn ts=2019-10-23T12:37:21.805Z caller=cluster.go:154 component=cluster err="couldn't deduce an advertise address: no private IP found, explicit advertise addr not provided" 2019-10-23_12:37:21.80690 level=error ts=2019-10-23T12:37:21.806Z caller=main.go:222 msg="unable to initialize gossip mesh" err="create memberlist: Failed to get final advertise address: No private IP address found, and explicit IP not provided"
Details of package version
Provide the package version installation details
With some details redacted (hostname, IP addresses) gitlabsos.lb-gsf14-gitlab-1.lab.nascom.nasa.gov_20191023080659.tar.gzEnvironment details
- Operating System: CentOS 7.7
- Installation Target:
- VM: VMWare
- Installation Type:
- Upgrade from version 11.x
- Is there any other software running on the machine: No
- Is this a single or multiple node installation? Single
- Resources
- CPU: 4
- Memory total: 4G
Configuration details
Provide the relevant sections of `/etc/gitlab/gitlab.rb`
external_url 'https://gitlab-server.DOMAIN'
gitlab_rails['manage_backup_path'] = true
gitlab_rails['backup_path'] = "/opt/backup"
gitlab_rails['backup_archive_permissions'] = 0644
gitlab_rails['backup_pg_schema'] = 'public'
gitlab_rails['backup_keep_time'] = 604800
gitlab_rails['env'] = {
'http_proxy' => 'http://proxy-server:port',
'https_proxy' => 'http://proxy-server:port',
'BUNDLE_GEMFILE' => "/opt/gitlab/embedded/service/gitlab-rails/Gemfile",
'PATH' => "/opt/gitlab/bin:/opt/gitlab/embedded/bin:/bin:/usr/bin"
}
gitlab_workhorse['env'] = {
'http_proxy' => 'http://proxy-server:port',
'https_proxy' => 'http://proxy-server:port',
'PATH' => '/opt/gitlab/bin:/opt/gitlab/embedded/bin:/bin:/usr/bin',
'SSL_CERT_DIR' => '/opt/gitlab/embedded/ssl/certs/'
}
nginx['redirect_http_to_https'] = true
# Self-signed cert
nginx['ssl_certificate'] = "/etc/gitlab/ssl/#{node['fqdn']}.crt"
nginx['ssl_certificate_key'] = "/etc/gitlab/ssl/#{node['fqdn']}.key"
gitaly['env'] = {
'http_proxy' => 'http://proxy-server:port',
'https_proxy' => 'http://proxy-server:port',
'no_proxy' => 'unix',
'PATH' => "/opt/gitlab/bin:/opt/gitlab/embedded/bin:/bin:/usr/bin",
'HOME' => '/var/opt/gitlab'
}
Proxy environment values
ftp_proxy=proxy-server:port FTP_PROXY=proxy-server:port http_proxy=proxy-server:port HTTP_PROXY=proxy-server:port https_proxy=proxy-server:port HTTPS_PROXY=proxy-server:port no_proxy=1.2.,.DOMAIN NO_PROXY=1.2.,.DOMAIN socks_proxy=proxy-server:port SOCKS_PROXY=proxy-server:port