How can we get CI build pipelines down to ~10-15 mins

The long duration of our pipelines has also been brought up in the recent product team meeting.

https://docs.google.com/document/d/1feF46WvqBuWJ-bYnA-BuCD4gq_J5UagupNHJYTBtII8/edit#bookmark=id.pzxm52fvm3ii

We discuss setting an ambitious goal of reducing this down to 10-15 mins. As we scale the productivity of our Engineers and Product managers should be high on the priority list.

An inspirational read from @gweaver https://www.jamesshore.com/Agile-Book/ten_minute_build.html

Some suggestions from @joshlambert

Longer term, some product improvements here as well to consider. Container start, caching, other issues. Compare a comparable job to something like CircleCI, it’s slower. Capture better metrics around cache hit rate, job timings, etc?

We currently have a KR in Q3 to reduce the overall pipeline.

This issue serves as a discussion point on how to achieve this. Ideas:

  • Optimize aggressively using the new DAG feature to skip unneeded jobs
  • Implement change detection.
  • Deduplicate tests further with the help from all of engineering.
  • etc.

@gl-quality/eng-prod @jramsay

Edited by Mek Stittri