How can we get CI build pipelines down to ~10-15 mins
The long duration of our pipelines has also been brought up in the recent product team meeting. https://docs.google.com/document/d/1feF46WvqBuWJ-bYnA-BuCD4gq_J5UagupNHJYTBtII8/edit#bookmark=id.pzxm52fvm3ii We discuss setting an ambitious goal of reducing this down to 10-15 mins. As we scale the productivity of our Engineers and Product managers should be high on the priority list. An inspirational read from @gweaver https://www.jamesshore.com/Agile-Book/ten_minute_build.html Some suggestions from @joshlambert > Longer term, some product improvements here as well to consider. Container start, caching, other issues. Compare a comparable job to something like CircleCI, it’s slower. > Capture better metrics around cache hit rate, job timings, etc? We currently have a KR in Q3 to reduce the overall pipeline. This issue serves as a discussion point on how to achieve this. Ideas: * Optimize aggressively using the new DAG feature to skip unneeded jobs * Implement change detection. * Deduplicate tests further with the help from all of engineering. * etc. @gl-quality/eng-prod @jramsay
issue