Skip to content

Prometheus eats disk space in /var/opt/gitlab/prometheus/data

I am running a server on Ubuntu 18.04 with GitLab CE 11.8.

My GitLab config:

grep prometheus /etc/gitlab/gitlab.rb
# gitlab_workhorse['prometheus_listen_addr'] = "localhost:9229"
##! Prometheus metrics for Pages docs: https://gitlab.com/gitlab-org/gitlab-pages/#enable-prometheus-metrics
##! Docs: https://docs.gitlab.com/ce/administration/monitoring/prometheus/
# prometheus['enable'] = true
# prometheus['monitor_kubernetes'] = true
# prometheus['username'] = 'gitlab-prometheus'
# prometheus['uid'] = nil
# prometheus['gid'] = nil
# prometheus['shell'] = '/bin/sh'
# prometheus['home'] = '/var/opt/gitlab/prometheus'
# prometheus['log_directory'] = '/var/log/gitlab/prometheus'
# prometheus['rules_files'] = ['/var/opt/gitlab/prometheus/rules/*.rules']
# prometheus['scrape_interval'] = 15
# prometheus['scrape_timeout'] = 15
# prometheus['chunk_encoding_version'] = 2
# See: https://prometheus.io/docs/operating/configuration/#<scrape_config>
# prometheus['scrape_configs'] = [
# See: https://prometheus.io/docs/operating/storage/#memory-usage
# prometheus['target_heap_size'] = (
# prometheus['flags'] = {
#   'storage.local.path' => "#{node['gitlab']['prometheus']['home']}/data",
#   'storage.local.target-heap-size' => node['gitlab']['prometheus']['target-heap-size'],
#   'config.file' => "#{node['gitlab']['prometheus']['home']}/prometheus.yml"
# prometheus['listen_address'] = 'localhost:9090'
##! Docs: https://docs.gitlab.com/ce/administration/monitoring/prometheus/alertmanager.html
##! Docs: https://docs.gitlab.com/ce/administration/monitoring/prometheus/node_exporter.html
##! Docs: https://docs.gitlab.com/ce/administration/monitoring/prometheus/redis_exporter.html
##! Docs: https://docs.gitlab.com/ce/administration/monitoring/prometheus/postgres_exporter.html
##! Docs: https://docs.gitlab.com/ee/administration/monitoring/prometheus/pgbouncer_exporter.html
##! Docs: https://docs.gitlab.com/ce/administration/monitoring/prometheus/gitlab_monitor_exporter.html
# To completely disable prometheus, and all of it's exporters, set to false
# prometheus_monitoring['enable'] = true
# gitaly['prometheus_listen_addr'] = "localhost:9236"
# gitaly['prometheus_grpc_latency_buckets'] = "[0.001, 0.005, 0.025, 0.1, 0.5, 1.0, 10.0, 30.0, 60.0, 300.0, 1500.0]"

For some reason, my /var/opt/gitlab/prometheus/data directory has grown to 27 GB, eating up all disk space, thereby rendering the server (and GitLab) unusable.

What causes this? The data is continuously being written to.

Edited by Werner Robitza