How can we get CI build pipelines down to ~10-15 mins
The long duration of our pipelines has also been brought up in the recent product team meeting.
We discuss setting an ambitious goal of reducing this down to 10-15 mins. As we scale the productivity of our Engineers and Product managers should be high on the priority list.
An inspirational read from @gweaver https://www.jamesshore.com/Agile-Book/ten_minute_build.html
Some suggestions from @joshlambert
Longer term, some product improvements here as well to consider. Container start, caching, other issues. Compare a comparable job to something like CircleCI, it’s slower. Capture better metrics around cache hit rate, job timings, etc?
We currently have a KR in Q3 to reduce the overall pipeline.
This issue serves as a discussion point on how to achieve this. Ideas:
- Optimize aggressively using the new DAG feature to skip unneeded jobs
- Implement change detection.
- Deduplicate tests further with the help from all of engineering.
- etc.