Change shard size to report avg per index
what/why
This MR changes the shard size report in mr metrics man to always report average shard size for each index. This is in line with the alerts from the monitoring cluster.
before
Note: I had to add >
before the backticks so the code would display properly, not sure if there's a better way to have backticks within backticks
➜ DRY_RUN=1 ruby mr_metric_man.rb
> Running queries for this time frame:
- gte: 2024-06-26T12:00:00Z
- lte: 2024-06-27T12:10:30Z
*24 hour updates*
- :turtle: *3,279* <https://log.gprd.gitlab.net/app/discover#/view/00101480-4bf4-11ec-a012-eb2e5674cacf?_g=(filters%3A!()%2CrefreshInterval%3A(pause%3A!t%2Cvalue%3A0)%2Ctime%3A(from%3Anow-24h%2Fh%2Cto%3Anow))|searches with 5+s durations> Amount of traffic within SLO: `99.56%` :large_green_circle:
- :boom: *110* <https://log.gprd.gitlab.net/app/discover#/view/0573a930-b1d1-11ec-afaf-2bca15dfbf33?_g=(filters%3A!()%2CrefreshInterval%3A(pause%3A!t%2Cvalue%3A0)%2Ctime%3A(from%3Anow-24h%2Fh%2Cto%3Anow))|searches blew up> with these <https://log.gprd.gitlab.net/app/discover#/view/32dd16a0-4989-11ec-a012-eb2e5674cacf?_g=(filters%3A!()%2CrefreshInterval%3A(pause%3A!t%2Cvalue%3A0)%2Ctime%3A(from%3Anow-24h%2Fh%2Cto%3Anow))|error messages>
- :elasticsearch: <https://00a4ef3362214c44a044feaa539b4686.us-central1.gcp.cloud.es.io:9243/app/monitoring#/overview?_g=(cluster_uuid:nkvJVBhsSwWfoqyHIA_raQ,refreshInterval:(pause:!f,value:10000),time:(from:now-15m,to:now))|prod elastic cluster> is :large_green_circle:
- :kibana: <https://cloud.elastic.co/deployments/f2f6ef644c28466cb6653383369d7160|prod elastic deployment> is :large_green_circle:
- :jigsaw: shard size warnings :warning:
>```
too small 3.63gb gitlab-production-notes-20240125-1800-reindex-1000076-0
too small 0.08gb gitlab-production-epics-20240201-1830-reindex-1000078-0
too small 1.08gb gitlab-production-wikis-20240206-1030-reindex-1000080-0
too small 0.95gb gitlab-production-issues-20240130-1207-reindex-1000077-0
too small 0.77gb gitlab-production-projects-20240531-1650-reindex-1000113-0
too small 0.0gb gitlab-production-work_items-20240606-1525
too small 4.53gb gitlab-production-merge_requests-20240205-1030-reindex-1000079-0
too small 5.8gb gitlab-production-commits-20221104-1910-reindex-3-0
too small 0.74gb gitlab-production-users-20231218-1500-reindex-1000074-0
>```
- :mag_right: no pending migrations
after
➜ DRY_RUN=1 ruby mr_metric_man.rb
> Running queries for this time frame:
- gte: 2024-06-26T15:00:00Z
- lte: 2024-06-27T15:16:15Z
*24 hour updates*
- :turtle: *3,298* <https://log.gprd.gitlab.net/app/discover#/view/00101480-4bf4-11ec-a012-eb2e5674cacf?_g=(filters%3A!()%2CrefreshInterval%3A(pause%3A!t%2Cvalue%3A0)%2Ctime%3A(from%3Anow-24h%2Fh%2Cto%3Anow))|searches with 5+s durations> Amount of traffic within SLO: `99.54%` :large_green_circle:
- :boom: *121* <https://log.gprd.gitlab.net/app/discover#/view/0573a930-b1d1-11ec-afaf-2bca15dfbf33?_g=(filters%3A!()%2CrefreshInterval%3A(pause%3A!t%2Cvalue%3A0)%2Ctime%3A(from%3Anow-24h%2Fh%2Cto%3Anow))|searches blew up> with these <https://log.gprd.gitlab.net/app/discover#/view/32dd16a0-4989-11ec-a012-eb2e5674cacf?_g=(filters%3A!()%2CrefreshInterval%3A(pause%3A!t%2Cvalue%3A0)%2Ctime%3A(from%3Anow-24h%2Fh%2Cto%3Anow))|error messages>
- :elasticsearch: <https://00a4ef3362214c44a044feaa539b4686.us-central1.gcp.cloud.es.io:9243/app/monitoring#/overview?_g=(cluster_uuid:nkvJVBhsSwWfoqyHIA_raQ,refreshInterval:(pause:!f,value:10000),time:(from:now-15m,to:now))|prod elastic cluster> is :large_green_circle:
- :kibana: <https://cloud.elastic.co/deployments/f2f6ef644c28466cb6653383369d7160|prod elastic deployment> is :large_green_circle:
- :jigsaw: avg shard size per index
>```
34.82gb gitlab-production-20240624-1635-reindex-1000118-0
5.4gb gitlab-production-commits-20221104-1910-reindex-3-0
4.23gb gitlab-production-merge_requests-20240205-1030-reindex-1000079-0
3.38gb gitlab-production-notes-20240125-1800-reindex-1000076-0
1.0gb gitlab-production-wikis-20240206-1030-reindex-1000080-0
0.89gb gitlab-production-issues-20240130-1207-reindex-1000077-0
0.72gb gitlab-production-projects-20240531-1650-reindex-1000113-0
0.69gb gitlab-production-users-20231218-1500-reindex-1000074-0
0.07gb gitlab-production-epics-20240201-1830-reindex-1000078-0
0.0gb gitlab-production-work_items-20240606-1525
>```
- :mag_right: no pending migrations
Edited by Terri Chu