Production system emulation in staging via Zuora
Problem
Once we have the Zuora production-copy environment on staging, we can then query Zuora objects from there. However, we won't be able to interact fully with them as we have no corresponding objects in the customers application DB.
Proposal
Replicate the production system in staging with anonymous data linked to the production-copy Zuora DB.
There are at least 2 ways of doing this:
- Copy the production DB (scrubbed) and link objects to Zuora production-copy (perhaps based on ID). We'll need Infra to deliver this and some sort of script to manually keep it updated. Automation could be tricky.
- Based on the Zuora objects, and querying the Zuora API in bulk, recreate the production DB with fake data. Some things can't be replicated exactly, but we can randomize those that we don't know for sure. We may have to query other APIs such as GitLab (staging) in order to manually create users/groups, etc...
I'm more inclined to the latter since it will just be an engineering effort that we could maintain.
Result
Have a staging environment that looks like production, with lots of users, subscriptions, and all anonymised data.
Next steps (if any)
Get https://gitlab.com/gitlab-com/gl-infra/infrastructure/-/issues/9779 done and https://gitlab.com/gitlab-org/customers-gitlab-com/-/merge_requests/1246 merged in order to get the production-copy environment set on staging.
Explore how we can populate the customer portal based on Zuora's API.
How will we measure success?
This will help engineering debug issues without touching prod, engineers in test do testing there, plus Support could also replicate things there without the need to run dangerous commands on production :)
cc @amandarueda