Skip to content

Change shard size to report avg per index

what/why

This MR changes the shard size report in mr metrics man to always report average shard size for each index. This is in line with the alerts from the monitoring cluster.

before

Note: I had to add > before the backticks so the code would display properly, not sure if there's a better way to have backticks within backticks

DRY_RUN=1 ruby mr_metric_man.rb
> Running queries for this time frame:
  - gte: 2024-06-26T12:00:00Z
  - lte: 2024-06-27T12:10:30Z
*24 hour updates*
- :turtle: *3,279* <https://log.gprd.gitlab.net/app/discover#/view/00101480-4bf4-11ec-a012-eb2e5674cacf?_g=(filters%3A!()%2CrefreshInterval%3A(pause%3A!t%2Cvalue%3A0)%2Ctime%3A(from%3Anow-24h%2Fh%2Cto%3Anow))|searches with 5+s durations> Amount of traffic within SLO: `99.56%` :large_green_circle:
- :boom: *110* <https://log.gprd.gitlab.net/app/discover#/view/0573a930-b1d1-11ec-afaf-2bca15dfbf33?_g=(filters%3A!()%2CrefreshInterval%3A(pause%3A!t%2Cvalue%3A0)%2Ctime%3A(from%3Anow-24h%2Fh%2Cto%3Anow))|searches blew up> with these <https://log.gprd.gitlab.net/app/discover#/view/32dd16a0-4989-11ec-a012-eb2e5674cacf?_g=(filters%3A!()%2CrefreshInterval%3A(pause%3A!t%2Cvalue%3A0)%2Ctime%3A(from%3Anow-24h%2Fh%2Cto%3Anow))|error messages>
- :elasticsearch: <https://00a4ef3362214c44a044feaa539b4686.us-central1.gcp.cloud.es.io:9243/app/monitoring#/overview?_g=(cluster_uuid:nkvJVBhsSwWfoqyHIA_raQ,refreshInterval:(pause:!f,value:10000),time:(from:now-15m,to:now))|prod elastic cluster> is :large_green_circle:
- :kibana: <https://cloud.elastic.co/deployments/f2f6ef644c28466cb6653383369d7160|prod elastic deployment> is :large_green_circle:
- :jigsaw: shard size warnings :warning:

>```
too small	3.63gb		gitlab-production-notes-20240125-1800-reindex-1000076-0
too small	0.08gb		gitlab-production-epics-20240201-1830-reindex-1000078-0
too small	1.08gb		gitlab-production-wikis-20240206-1030-reindex-1000080-0
too small	0.95gb		gitlab-production-issues-20240130-1207-reindex-1000077-0
too small	0.77gb		gitlab-production-projects-20240531-1650-reindex-1000113-0
too small	0.0gb		gitlab-production-work_items-20240606-1525
too small	4.53gb		gitlab-production-merge_requests-20240205-1030-reindex-1000079-0
too small	5.8gb		gitlab-production-commits-20221104-1910-reindex-3-0
too small	0.74gb		gitlab-production-users-20231218-1500-reindex-1000074-0
>```
- :mag_right: no pending migrations

after

DRY_RUN=1 ruby mr_metric_man.rb
> Running queries for this time frame:
  - gte: 2024-06-26T15:00:00Z
  - lte: 2024-06-27T15:16:15Z
*24 hour updates*
- :turtle: *3,298* <https://log.gprd.gitlab.net/app/discover#/view/00101480-4bf4-11ec-a012-eb2e5674cacf?_g=(filters%3A!()%2CrefreshInterval%3A(pause%3A!t%2Cvalue%3A0)%2Ctime%3A(from%3Anow-24h%2Fh%2Cto%3Anow))|searches with 5+s durations> Amount of traffic within SLO: `99.54%` :large_green_circle:
- :boom: *121* <https://log.gprd.gitlab.net/app/discover#/view/0573a930-b1d1-11ec-afaf-2bca15dfbf33?_g=(filters%3A!()%2CrefreshInterval%3A(pause%3A!t%2Cvalue%3A0)%2Ctime%3A(from%3Anow-24h%2Fh%2Cto%3Anow))|searches blew up> with these <https://log.gprd.gitlab.net/app/discover#/view/32dd16a0-4989-11ec-a012-eb2e5674cacf?_g=(filters%3A!()%2CrefreshInterval%3A(pause%3A!t%2Cvalue%3A0)%2Ctime%3A(from%3Anow-24h%2Fh%2Cto%3Anow))|error messages>
- :elasticsearch: <https://00a4ef3362214c44a044feaa539b4686.us-central1.gcp.cloud.es.io:9243/app/monitoring#/overview?_g=(cluster_uuid:nkvJVBhsSwWfoqyHIA_raQ,refreshInterval:(pause:!f,value:10000),time:(from:now-15m,to:now))|prod elastic cluster> is :large_green_circle:
- :kibana: <https://cloud.elastic.co/deployments/f2f6ef644c28466cb6653383369d7160|prod elastic deployment> is :large_green_circle:
- :jigsaw: avg shard size per index

>```
34.82gb		gitlab-production-20240624-1635-reindex-1000118-0
5.4gb		gitlab-production-commits-20221104-1910-reindex-3-0
4.23gb		gitlab-production-merge_requests-20240205-1030-reindex-1000079-0
3.38gb		gitlab-production-notes-20240125-1800-reindex-1000076-0
1.0gb		gitlab-production-wikis-20240206-1030-reindex-1000080-0
0.89gb		gitlab-production-issues-20240130-1207-reindex-1000077-0
0.72gb		gitlab-production-projects-20240531-1650-reindex-1000113-0
0.69gb		gitlab-production-users-20231218-1500-reindex-1000074-0
0.07gb		gitlab-production-epics-20240201-1830-reindex-1000078-0
0.0gb		gitlab-production-work_items-20240606-1525
>```
- :mag_right: no pending migrations
Edited by Terri Chu

Merge request reports