Lightweight feature flags that don't need the database
Everyone can contribute. Help move this issue forward while earning points, leveling up and collecting rewards.
Problem
Our current feature flags are stored in the database and cached per thread.
This is problematic for two reasons:
- Feature flags in hot spots like sidekiq or rack middlewares causing a database call every time a thread starts, adding pressure to the database cluster.
- Code that executes before the database cluster is available, such as changes to Puma configuration or initializers that execute early in the chain, cannot use FFs, and have led to incidents in the past. We often use environment variables to switch these code paths instead, which is a highly manual process and requires close coordination between product and production engineers.
Moreover, in those places being able to scope feature flags to groups or projects often doesn't make sense, so perhaps we should have a way of storing these feature flags in the environment or in memory of the parent process rather than per thread.
Production Incidents
This was raised as a corrective action from an incident in February 2020. It would also have prevented an incident in January 2021 as well as December 2021.
In the most recent incident, a new Redis store for sessions required a feature flag check on every request which double the transactions per second on the database.
Solution
Provide a solution that allows us to switch code and configuration such that:
- The database is not on the critical path
- Developers can act autonomously when rolling these out, just as they do with ordinary FFs