DRAFT: Backend: Update Growthbook implementation to ensure stability of variation assignment
Goal
The value of running experiments in Growthbook comes from being able to reliably infer causal relations from the statistical difference of behaviour between a control group and treatment group. For this to be reliable, the assignment of a given user to a control group or a treatment group should be as stable as possible. There is room improve this stability in the current Growthbook integration.
What needs to be done
- The first time an SDK call is made to select a user/device into a feature variation (in a logged-out state), save the selected variation to a dedicated local persistent cookie for that feature
- On all subsequent requests for a selection to a given feature (in logged-out state), first check to see whether a variation has been stored in a local persistent cookie. If so, return that value as the selection in the SDK call. If not, run the experiment assignment, and store the value as above
- When a user transitions from a logged out state to a logged in state, query the back-end config to see if assignments have been made to experiments for this account. If so, overwrite the local persistent cookie with the variation associated with the account. For any locally assigned and stored variations that are not stored on the back end account record, send the local assignment values to the back end for storage.
Dev notes
https://gitlab.com/minds/front/-/blob/master/src/app/modules/experiments/experiments.service.ts#L85
QA
Ensure that for transitions between logged-in and logged-out states, and transitions to different devices and browser sessions that, as much as possible, the assignment of a variation for a given feature is stable for a given user.
Acceptance Criteria
-
Logged-out variation assignments are stable between browser sessions -
Logged-in variation assignments are persisted with the user record -
Logged-in variation assignments are consistent from session to session -
Logged-in variation assignments are consistent across devices -
Logged-out and logged-in assignments are consistent as much as possible -
Rate-limiting of callbacks to Snowplow events, per-feature, are preserved as in the current implementation -
Ensure rate-limiting is as reliable as possible (in both directions: not blocking events that should go, and blocking events that shouldn't go)
Definition of Ready Checklist
-
Definition Of Done (DoD) -
Acceptance criteria -
Weighted -
QA
Edited by Mark Harding