Optimise active record subscriber logs to reduce log volume
We emit many fields per database but for some instances, those fields are empty. e.g. embeddings db is empty (all zeroes) for most rails and sidekiq calls.
Ideas:
- add field to log line only if value is > 0
- drop roles-only fields since they are essentially duplicates of role+db_config_name fields (!166417 (comment 2119326534))
Impact (from #500768 (comment 2179275845)):
This came up as part of a Capacity Planning investigation https://gitlab.com/gitlab-com/gl-infra/capacity-planning-trackers/gitlab-com/-/issues/1669#note_2176044074.
Impact
An MR added two fields to the default rails message size, increasing it from 4.495 KiB to 4.765 KiB, an increase of 6%.
That 0.270 KiB increase was multiplied over approximately 4.7 billion log entries a day.
That works out to 1269000000 KiB per day or 1.2 Tb extra per day and has spiked disk space in our logging cluster (which stores 7 days worth of data) from 72.95% of 320.63 Tb to 82.52% of the same space. (This number is larger than the percent increase because of how Elastic stores data -- in most cases, it's keeping two copies of all the data for HA reasons).
If we decrease even a small number of these extra fields, we'll save a significant amount in storage over all and thus have a better, easier to manage logging cluster and spend less money to do so.