Feature flag/percentage rollout for using separate read connection for CI replicas
From our rollout plan &6160 (closed) in "Phase 3: Serve CI reads from CI standby cluster" we need to be able to configure an extra ci
connection in our config/database.yml
.
While it is likely safe for us to point CI reads to a different set of CI replicas (which are just like any other replicas but with slightly different configuration) it is still likely a good idea to have a safer rollout approach.
We should consider some safe way to use CI connection that we can easily enable and disable quickly at times that are suitable to run the experiment.
It could be a feature flag for:
- Percentage of users, projects, groups, traffic
Feature flag may be tricky because the connection may need to do something at Rails startup. We should try to keep it simple and pick the easiest option that allows us to quickly enable and disable without the need to re-deploy everything.
Alternatively, since the separate connections may be infrastructure configured, it may be ideal to roll it out to a subset of Rails nodes at a time but this may be too expensive to rollback.
Related to #341451 (closed).