Update Growthbook implementation to ensure stability of variation assignment
Goal
The value of running experiments in Growthbook comes from being able to reliably infer causal relations from the statistical difference of behaviour between a control group and treatment group. For this to be reliable, the assignment of a given user to a control group or a treatment group should be as stable as possible. There is room improve this stability in the current Growthbook integration.
What needs to be done
- The first time an SDK call is made to select a user/device into a feature variation (in a logged-out state), save the selected variation to a dedicated local persistent cookie for that feature
- On all subsequent requests for a selection to a given feature (in logged-out state), first check to see whether a variation has been stored in a local persistent cookie. If so, return that value as the selection in the SDK call. If not, run the experiment assignment, and store the value as above
Dev notes
https://gitlab.com/minds/front/-/blob/master/src/app/modules/experiments/experiments.service.ts#L85
- Store as a JSON object with a single key
- Confirm that it is an experiment before using the cached assignment
QA
Ensure that for transitions between logged-in and logged-out states, and transitions to different devices and browser sessions that, as much as possible, the assignment of a variation for a given feature is stable for a given user.
Acceptance Criteria
-
Logged-out variation assignments are stable between browser sessions -
Logged-in variation assignments are consistent from session to session -
Logged-out and logged-in assignments are consistent as much as possible -
Rate-limiting of callbacks to Snowplow events, per-feature, are preserved as in the current implementation -
Ensure rate-limiting is as reliable as possible (in both directions: not blocking events that should go, and blocking events that shouldn't go)
Definition of Ready Checklist
-
Definition Of Done (DoD) -
Acceptance criteria -
Weighted -
QA
Edited by Ben