Experiment framework - options and analysis - improve, build, buy
High level comparison of three options for improving the experimentation capability at GitLab:
- Improve: improve the current framework #3 (moved) &/or #4 (closed)
- Dogfood: dogfood GitLab Feature Flag feature #5 (moved)
- Buy: purcahse an off the shelf solution for testing
Growth product requirements for experimentation: #1 (closed) define required capabilities for Bucketing, Measuring, Analysis, Data and privacy. Table provides a high level view of whether the solution does - or is expected to - meet the key requirements in the short term. Note that analysis of specific buy options has not been provided.
Time to value is a high level estimate of when the key requirements could be met by each of the three options.
- Now - requirements are already met
- Short term - likely within 3 months
- Medium term 3-6 months
- Unknown - definitely longer than 3 months based on what we currently know, more information required
| Improve | Dogfood | Buy | |
|---|---|---|---|
| Bucketing | |||
| Measuring | |||
| Analysis | |||
| Data/privacy | |||
| Time to value | Short to Medium term | Unknown | Unknown |
| Cost | Opex | Opex | Capex - Unknown |
| Risk | Changing requirements | Relies on other product teams priorities | Integration, data privacy, time to analyze requirements |