Pipeline Efficiency: Capacity planning and cost optimization with AI and Observability

Proposal

Pipeline efficiency is an ongoing topic. There are many ways to analyse, refactor, and optimize pipelines to reduce cost, and also leverage AI already to do forecast planning.

This proposal involves multiple ideas and MVCs

1. Collect CI/CD Observability data (storage, CPU, etc.) as base metrics, job logs, add tracing for CI/CD Observability (#338943)
   • Data from Runner fleet management grouprunner and groupobservability
2. Use AI for capacity planning and forecasting of CI/CD infrastructure
   • Based on the ideas of TAMland with FB Prophet and Prometheus metrics.
3. Use OpenCost models and AI to estimate operational costs for CI/CD
   • Corooot implements
4. Train AI models that help optimize pipelines with a focus on cost optimization
   • Fixing pipelines fast is covered in this proposal: Pipeline Efficiency: Provide AI assisted help t... (#386863 - closed)

Resources

1. @dnsmichi talk "Observability for Efficient DevSecOps Pipelines" https://go.gitlab.com/VDAvMw
   • Cloudland feedback https://gitlab.com/gitlab-com/marketing/corporate_marketing/corporate-marketing/-/issues/7506#note_1443211586
2. TAMland https://go.gitlab.com/etwJmr
   • PromCon EU 2022: How GitLab Uses Long-Term Monitoring Data For Capacity Forecasting https://go.gitlab.com/yO6NCN
3. Coroot Cost monitoring https://github.com/coroot/coroot/releases/tag/0.15.0
4. OpenCost https://www.opencost.io/

Suggested Tier

Paid feature, which helps with reducing costs and more efficient pipelines.

GitLab Ultimate