Error on prometheus after update
Summary
After latest update, gitlab's prometheus logs in loop.
Steps to reproduce
I just use the official gitlab ce omnibus image from the docker hub (currently gitlab-ce=12.7.6-ce.0). I updated it to latest today and it failed.
What is the current bug behavior?
It logs in loop:
2020-02-20_11:25:59.09496 level=error ts=2020-02-20T11:25:59.094Z caller=main.go:733 err="opening storage failed: get segment range: segments are not sequential"What is the expected correct behavior?
It usually works and updates my installation.
Relevant logs
Relevant logs
==> /var/log/gitlab/prometheus/current <== [520/91325] 2020-02-20_11:25:59.09273 level=info ts=2020-02-20T11:25:59.092Z caller=main.go:293 msg="no time or size retention was set so using the default time retention" duration=15d 2020-02-20_11:25:59.09275 level=info ts=2020-02-20T11:25:59.092Z caller=main.go:329 msg="Starting Prometheus" version="(version=2.12.0, branch=master, revision=)" 2020-02-20_11:25:59.09276 level=info ts=2020-02-20T11:25:59.092Z caller=main.go:330 build_context="(go=go1.12.13, user=GitLab-Omnibus, date=)" 2020-02-20_11:25:59.09278 level=info ts=2020-02-20T11:25:59.092Z caller=main.go:331 host_details="(Linux 4.15.0-88-generic #88-Ubuntu SMP Tue Feb 11 20:11:34 UTC 2020 x86_64 gitlab.example.com (none))" 2020-02-20_11:25:59.09280 level=info ts=2020-02-20T11:25:59.092Z caller=main.go:332 fd_limits="(soft=50000, hard=50000)" 2020-02-20_11:25:59.09282 level=info ts=2020-02-20T11:25:59.092Z caller=main.go:333 vm_limits="(soft=unlimited, hard=unlimited)" 2020-02-20_11:25:59.09360 level=info ts=2020-02-20T11:25:59.093Z caller=main.go:654 msg="Starting TSDB ..." 2020-02-20_11:25:59.09361 level=info ts=2020-02-20T11:25:59.093Z caller=web.go:448 component=web msg="Start listening for connections" address=localhost:9090 2020-02-20_11:25:59.09380 level=info ts=2020-02-20T11:25:59.093Z caller=repair.go:59 component=tsdb msg="found healthy block" mint=1558958400000 maxt=1559023200000 ulid=01DBYTNXEG33JE58J3VA8A22SD 2020-02-20_11:25:59.09384 level=info ts=2020-02-20T11:25:59.093Z caller=repair.go:59 component=tsdb msg="found healthy block" mint=1559023200000 maxt=1559088000000 ulid=01DC0RFEJWXKWXQPBK4E7AD6AY 2020-02-20_11:25:59.09387 level=info ts=2020-02-20T11:25:59.093Z caller=repair.go:59 component=tsdb msg="found healthy block" mint=1559088000000 maxt=1559152800000 ulid=01DC2P8ZTYJ5V9BNZ58BSCVTYJ 2020-02-20_11:25:59.09391 level=info ts=2020-02-20T11:25:59.093Z caller=repair.go:59 component=tsdb msg="found healthy block" mint=1559152800000 maxt=1559217600000 ulid=01DC4M2H2NN2TGW66RMKNHYZVB 2020-02-20_11:25:59.09394 level=info ts=2020-02-20T11:25:59.093Z caller=repair.go:59 component=tsdb msg="found healthy block" mint=1559217600000 maxt=1559282400000 ulid=01DC6HW29M54Y2CE3P11WSPM3Q 2020-02-20_11:25:59.09397 level=info ts=2020-02-20T11:25:59.093Z caller=repair.go:59 component=tsdb msg="found healthy block" mint=1559282400000 maxt=1559347200000 ulid=01DC8FNKHQTFSBGX0BTTTH30N5 2020-02-20_11:25:59.09401 level=info ts=2020-02-20T11:25:59.094Z caller=repair.go:59 component=tsdb msg="found healthy block" mint=1559347200000 maxt=1559412000000 ulid=01DCADF4TMVGV8M7QYPCJYR1PS 2020-02-20_11:25:59.09404 level=info ts=2020-02-20T11:25:59.094Z caller=repair.go:59 component=tsdb msg="found healthy block" mint=1559412000000 maxt=1559476800000 ulid=01DCCB8P22KFGA96JKJGCTSE6V 2020-02-20_11:25:59.09407 level=info ts=2020-02-20T11:25:59.094Z caller=repair.go:59 component=tsdb msg="found healthy block" mint=1559476800000 maxt=1559541600000 ulid=01DCE927AVZ4G7FNWBSBJZW0AA 2020-02-20_11:25:59.09410 level=info ts=2020-02-20T11:25:59.094Z caller=repair.go:59 component=tsdb msg="found healthy block" mint=1559541600000 maxt=1559606400000 ulid=01DCG6VSDDB5DPJMCYB46WDFT7 2020-02-20_11:25:59.09413 level=info ts=2020-02-20T11:25:59.094Z caller=repair.go:59 component=tsdb msg="found healthy block" mint=1559606400000 maxt=1559671200000 ulid=01DCJ4N9T7KB9BKFMYFHRKF0ES 2020-02-20_11:25:59.09416 level=info ts=2020-02-20T11:25:59.094Z caller=repair.go:59 component=tsdb msg="found healthy block" mint=1559671200000 maxt=1559736000000 ulid=01DCM2EVWMK4DEJAQCVWNSDWF2 2020-02-20_11:25:59.09420 level=info ts=2020-02-20T11:25:59.094Z caller=repair.go:59 component=tsdb msg="found healthy block" mint=1559736000000 maxt=1559800800000 ulid=01DCP08CE0N9EQ365S50ZDBFYB 2020-02-20_11:25:59.09423 level=info ts=2020-02-20T11:25:59.094Z caller=repair.go:59 component=tsdb msg="found healthy block" mint=1559800800000 maxt=1559865600000 ulid=01DCQY1XJZ0EKBRCY8RR3X2210 2020-02-20_11:25:59.09426 level=info ts=2020-02-20T11:25:59.094Z caller=repair.go:59 component=tsdb msg="found healthy block" mint=1559865600000 maxt=1559930400000 ulid=01DCSVVET8318XPZDY851NSPCM 2020-02-20_11:25:59.09429 level=info ts=2020-02-20T11:25:59.094Z caller=repair.go:59 component=tsdb msg="found healthy block" mint=1559930400000 maxt=1559995200000 ulid=01DCVSN038F34P2NPABAP56JJK 2020-02-20_11:25:59.09432 level=info ts=2020-02-20T11:25:59.094Z caller=repair.go:59 component=tsdb msg="found healthy block" mint=1559995200000 maxt=1560060000000 ulid=01DCXQEJ534Z1E3WV68X4ST35X 2020-02-20_11:25:59.09435 level=info ts=2020-02-20T11:25:59.094Z caller=repair.go:59 component=tsdb msg="found healthy block" mint=1560060000000 maxt=1560124800000 ulid=01DCZN82JFTC9AX8TETK5A2JVN 2020-02-20_11:25:59.09438 level=info ts=2020-02-20T11:25:59.094Z caller=repair.go:59 component=tsdb msg="found healthy block" mint=1560124800000 maxt=1560189600000 ulid=01DD1K1KSQFKZYWKRX72AQD7XX 2020-02-20_11:25:59.09441 level=info ts=2020-02-20T11:25:59.094Z caller=repair.go:59 component=tsdb msg="found healthy block" mint=1560189600000 maxt=1560254400000 ulid=01DD3GV51YF1H2T2NM116ZA8K6 2020-02-20_11:25:59.09444 level=info ts=2020-02-20T11:25:59.094Z caller=repair.go:59 component=tsdb msg="found healthy block" mint=1560254400000 maxt=1560276000000 ulid=01DD45EANP59XM2W6BZCY8ENA2 2020-02-20_11:25:59.09447 level=info ts=2020-02-20T11:25:59.094Z caller=repair.go:59 component=tsdb msg="found healthy block" mint=1560297600000 maxt=1560304800000 ulid=01DD4T1G8545C6Y8SV2B4GVFMT 2020-02-20_11:25:59.09449 level=info ts=2020-02-20T11:25:59.094Z caller=repair.go:59 component=tsdb msg="found healthy block" mint=1560276000000 maxt=1560297600000 ulid=01DD4T1GDTM9FPS7HKQV30HCVM 2020-02-20_11:25:59.09451 level=info ts=2020-02-20T11:25:59.094Z caller=repair.go:59 component=tsdb msg="found healthy block" mint=1560304800000 maxt=1560312000000 ulid=01DD50X7G2ECDSK68W7PBJKGAK 2020-02-20_11:25:59.09454 level=info ts=2020-02-20T11:25:59.094Z caller=repair.go:59 component=tsdb msg="found healthy block" mint=1560312000000 maxt=1560319200000 ulid=01DD57RYR341QNTPJATB2GRPZ7 2020-02-20_11:25:59.09473 level=info ts=2020-02-20T11:25:59.094Z caller=main.go:523 msg="Stopping scrape discovery manager..." 2020-02-20_11:25:59.09474 level=info ts=2020-02-20T11:25:59.094Z caller=main.go:537 msg="Stopping notify discovery manager..." 2020-02-20_11:25:59.09474 level=info ts=2020-02-20T11:25:59.094Z caller=main.go:559 msg="Stopping scrape manager..." 2020-02-20_11:25:59.09475 level=info ts=2020-02-20T11:25:59.094Z caller=main.go:533 msg="Notify discovery manager stopped" 2020-02-20_11:25:59.09476 level=info ts=2020-02-20T11:25:59.094Z caller=main.go:519 msg="Scrape discovery manager stopped" 2020-02-20_11:25:59.09477 level=info ts=2020-02-20T11:25:59.094Z caller=manager.go:815 component="rule manager" msg="Stopping rule manager..." 2020-02-20_11:25:59.09478 level=info ts=2020-02-20T11:25:59.094Z caller=manager.go:821 component="rule manager" msg="Rule manager stopped" 2020-02-20_11:25:59.09478 level=info ts=2020-02-20T11:25:59.094Z caller=notifier.go:602 component=notifier msg="Stopping notification manager..." 2020-02-20_11:25:59.09479 level=info ts=2020-02-20T11:25:59.094Z caller=main.go:724 msg="Notifier manager stopped" 2020-02-20_11:25:59.09481 level=info ts=2020-02-20T11:25:59.094Z caller=main.go:553 msg="Scrape manager stopped" 2020-02-20_11:25:59.09496 level=error ts=2020-02-20T11:25:59.094Z caller=main.go:733 err="opening storage failed: get segment range: segments are not sequential"
Details of package version
Provide the package version installation details
Use Traefik as the inverse proxy
external_url 'https://{{ gitlab_domain }}' nginx['enable'] = false gitlab_workhorse['listen_network'] = "tcp" gitlab_workhorse['listen_addr'] = "0.0.0.0:8181"
Enable docker image registry
gitlab_rails['registry_enabled'] = true registry_external_url 'https://{{registry_domain}}' registry_nginx['enable'] = false registry['enable'] = true registry['registry_http_addr'] = "0.0.0.0:5000" registry['storage_delete_enabled'] = true registry['storage'] = { 's3' => { 'accesskey' => '{{registry_aws.key}}', 'bucket' => '{{registry_aws.bucket}}', 'region' => '{{registry_aws.region}}', 'secretkey' => '{{registry_aws.secret}}', } }
Use SMTP relay container
gitlab_rails['smtp_address'] = "smtp" gitlab_rails['smtp_authentication'] = "none" gitlab_rails['smtp_domain'] = "{{ backup_smtp_domain }}" gitlab_rails['smtp_enable_starttls_auto'] = false gitlab_rails['smtp_enable'] = true gitlab_rails['smtp_port'] = 25 gitlab_rails['smtp_ssl'] = false gitlab_rails['smtp_tls'] = false
Backups are stored remotely, no need to keep many locally
gitlab_rails['backup_keep_time'] = 3600 # 1h
Edited by Jairo Llopis