[Spike] Move away from Zuora callouts
Problem
Zuora callouts (having up to 3 retries) are not completely reliable and they may not reach CustomersDot for a number of reasons. While this is an edge case, there could be more reliable ways to receive them.
Proposal
Refactor Zsim so it is possible to forward the callouts without relying on Zuora.
This will work in a similar fashion as it does now for retrieving callouts in development environments, with some modifications:
- Exponential retries can be added to make sure CDot receives the callout if it fails several times
- A daemon mode can be added
- Could be hooked into our current Ansible deployment or deploy it somewhere else
- We could eventually add Prometheus metrics and alerts to Zsim to check the rate of success and detect any problems
Result
Reliable callouts with 100% success rate
Next steps (if any)
Break this issue down into smaller chunks
Edited by James Lopez