Geo: Feature flag changes do not propagate immediately to secondaries
Problem
I am seeing weird behavior locally. When I toggle a feature flag
:geo_package_file_replication
on the primary, the cache invalidation event is sent and processed, but the feature flag does not return the updated value in Rails console or inrails-web
behavior (when I check the expected change in the UI). But when I exit and start Rails console it updates in console as well as in the UI (restartingrails-web
also worked). Any ideas?
I believe this happens because the Feature class loads the stored feature flags n the following order:
- Memoized: using
Gitlab::SafeRequestStore
or@flipper
- L1: using Process cache
- L2: using Redis cache
- DB: using a single SQL query
The feature flag does not return the updated value in the Rails console or in the UI because any of these caches are still valid since we only clear the Redis cache while processing the event on a Geo secondary node.
Proposal
CacheInvalidationEventStore
is only fired from https://gitlab.com/gitlab-org/gitlab/blob/master/ee/lib/ee/feature.rb#L39, so it's only used for feature flags, so we can do whatever we want, specific to feature flags, in https://gitlab.com/gitlab-org/gitlab/blob/master/ee/lib/gitlab/geo/log_cursor/events/cache_invalidation_event.rb#L18.