Further ideas to reduce cloud costs

• create/improve Grafana dashboard to visualize build job duration; use DW data if possible, and divide by rpm/upstream, rhel version, ...
• investigate ccache coverage, cache hit/miss penalty, and its effect on build time.
• investigate effect of CPU count, CPU frequency
• Determine the relationship between IOPS, throughput of storage, on duration of the build, experiment with placing sources into tmpfs.
• Change the EBS volume type from gp2 to gp3, to have 20% storage cost saving realized with increased read IOPS: https://gitlab.cee.redhat.com/cki-project/deployment-all/-/merge_requests/2405 . Launch templates updated, newly deployed instances will use the new volume type.
• Investigate possibility of launching builds in other EC2 regions, depending on spot price of the instance.
• Measure and optimize cost per unit (cost per build in case of kernel build).
• Install CloudWatch agent to better understand instance utilization, including disk utilization: solved via node-exporter
• investigate s3 transfer costs: pipeline-definition!1702 (merged)
• create EC2 machine reaper: cki-tools!741 (merged)
• create S3 artifact cleaner: #95 (closed), #107 (closed)
• tag shared cluster: deployment-all!1647 (merged)
• Change the frequency of data collection for AWS Config. As AWS Config is IT-managed, I requested the improvement in https://redhat-internal.slack.com/archives/C04LX6S77NE/p1705586996872539 .
• Removal of t2.micro instances from regions, where we don't run OpenShift workloads: https://issues.redhat.com/browse/OHSS-36926 (~$204/month saving)
• Removal of inactive NAT Gateways ($100/month saving): https://issues.redhat.com/browse/OHSS-36924