Redis bigkeys analysis: data into prometheus
From #320 (closed):
Extract the core output data collected in #360 (closed) and get it into prometheus.
For reasons explained in #320 (comment 339887270) we want a custom Prometheus exporter (as cheap and quick as possible) that reports summary data stored in redis by the periodic extraction job, rather than trying to piggy back on node exporter via text files. Adding an additional datasource to https://gitlab.com/gitlab-org/gitlab-exporter/ is likely the best way to achieve that efficiently.
The data from big-keys is:
- Keyed biggest of each type, by count/bytes/members etc. Exporting the key name is problematic, not because of cardinality per-se, but because it will change every time the biggest key changes and some of the key names are quite long and potentially weird.
- Number of keys of each type (list, hash, strings, steams, sets, zsets): this is just counts of keys, not size-in-storage (only string gives us that in bytes) e.g. the items in lists could be any arbitrary size.
- Counts of elements of each type (e.g. total members across all 'sets'). Needs two labels: type (list,hash,string, streams, sets, zsets) and units (items, fields, bytes, entries, members). Metric name must be have a 'count' suffix to avoid mis-interpretation.
Create some simple dashboards once we have some basic data collected.
Edited by Craig Miskell