Predictive tests improvements discussion

Use this issue to discuss improvements and challenges related to predictive tests.

Here are some related links

1. Runbook for predictive testing
2. Snowflake dashboards
   1. DX - Predictive tests
   2. DX - Predictive Pipelines
3. Running predictive tests check for GDK in local (feedback issue)

Questions

1. How to gather quality metrics here so we can use it together with failure tests data?
2. How do we store the failed test data?
3. Comparing with failure dashboards: What would provide great value for us here?
   1. Mappings have a scoring system to find legitimate failing tests
   2. Run some spec automatically in lefthook
   3. Enabling coverage strategy in tier-2

Things to look out for

1. Dynamic mapping must be precise, otherwise we might miss bugs and failed tests.
2. Master moves fast but the mapping is only created every 2 hours on master. Code changes might not match created mappings anymore.
3. Mappings may have changed because we run our pipelines as merged results.

Ideas

1. Use the stack traces to run only the fastest tests which would cover all the changes? 
2. Run some specs automatically in lefthook.
3. Mappings have a scoring system to find legitimate failing tests
4. Runbook mentions multiple strategies for improving the efficiency of crystalball mapping to cover all related tests for files.

Trends

1. Coverage based strategy has 10x more tests but still provides value because of that. It’s usable in CI, and will increase costs more instead of runtime due to parallelisation.