Confirm Tamland projections for Kibana search capacity is correct
There was an incident on staging where Kibana search became unavailable because it ran out of disk space: production#7333 (closed)
It appears that we do have the production-version of this metric here: https://gitlab.com/gitlab-com/runbooks/-/blob/master/libsonnet/saturation-monitoring/elastic_disk_space.libsonnet but we are not sure that it is measuring the right thing: https://thanos-query.ops.gitlab.net/graph?g0.expr=sum%20by%20(env%2C%20type)%20(gitlab_component_saturation%3Aratio_quantile95_1h%7Bcomponent%3D%22elastic_disk_space%22%7D)&g0.tab=0&g0.stacked=0&g0.range_input=12h&g0.max_source_resolution=0s&g0.deduplicate=1&g0.partial_response=0&g0.store_matches=%5B%5D
We see free space dropping here: https://dashboards.gitlab.net/d/alerts-sat_elastic_disk_space/alerts-elastic_disk_space-saturation-detail?orgId=1&var-PROMETHEUS_DS=Global&var-environment=gstg&var-type=logging&var-stage=main and our denominator might be wrong.