Prometheus eats disk space in /var/opt/gitlab/prometheus/data
I am running a server on Ubuntu 18.04 with GitLab CE 11.8.
My GitLab config:
grep prometheus /etc/gitlab/gitlab.rb
# gitlab_workhorse['prometheus_listen_addr'] = "localhost:9229"
##! Prometheus metrics for Pages docs: https://gitlab.com/gitlab-org/gitlab-pages/#enable-prometheus-metrics
##! Docs: https://docs.gitlab.com/ce/administration/monitoring/prometheus/
# prometheus['enable'] = true
# prometheus['monitor_kubernetes'] = true
# prometheus['username'] = 'gitlab-prometheus'
# prometheus['uid'] = nil
# prometheus['gid'] = nil
# prometheus['shell'] = '/bin/sh'
# prometheus['home'] = '/var/opt/gitlab/prometheus'
# prometheus['log_directory'] = '/var/log/gitlab/prometheus'
# prometheus['rules_files'] = ['/var/opt/gitlab/prometheus/rules/*.rules']
# prometheus['scrape_interval'] = 15
# prometheus['scrape_timeout'] = 15
# prometheus['chunk_encoding_version'] = 2
# See: https://prometheus.io/docs/operating/configuration/#<scrape_config>
# prometheus['scrape_configs'] = [
# See: https://prometheus.io/docs/operating/storage/#memory-usage
# prometheus['target_heap_size'] = (
# prometheus['flags'] = {
# 'storage.local.path' => "#{node['gitlab']['prometheus']['home']}/data",
# 'storage.local.target-heap-size' => node['gitlab']['prometheus']['target-heap-size'],
# 'config.file' => "#{node['gitlab']['prometheus']['home']}/prometheus.yml"
# prometheus['listen_address'] = 'localhost:9090'
##! Docs: https://docs.gitlab.com/ce/administration/monitoring/prometheus/alertmanager.html
##! Docs: https://docs.gitlab.com/ce/administration/monitoring/prometheus/node_exporter.html
##! Docs: https://docs.gitlab.com/ce/administration/monitoring/prometheus/redis_exporter.html
##! Docs: https://docs.gitlab.com/ce/administration/monitoring/prometheus/postgres_exporter.html
##! Docs: https://docs.gitlab.com/ee/administration/monitoring/prometheus/pgbouncer_exporter.html
##! Docs: https://docs.gitlab.com/ce/administration/monitoring/prometheus/gitlab_monitor_exporter.html
# To completely disable prometheus, and all of it's exporters, set to false
# prometheus_monitoring['enable'] = true
# gitaly['prometheus_listen_addr'] = "localhost:9236"
# gitaly['prometheus_grpc_latency_buckets'] = "[0.001, 0.005, 0.025, 0.1, 0.5, 1.0, 10.0, 30.0, 60.0, 300.0, 1500.0]"
For some reason, my /var/opt/gitlab/prometheus/data
directory has grown to 27 GB, eating up all disk space, thereby rendering the server (and GitLab) unusable.
What causes this? The data is continuously being written to.