### Background During a [recent analysis](https://gitlab.com/gitlab-com/gl-infra/scalability/-/issues/2538) of memory usage in Redis Shared State, we discovered that [gitlab-experiment](https://gitlab.com/gitlab-org/ruby/gems/gitlab-experiment) was a large consumer of memory. By [auditing](https://gitlab.com/gitlab-com/gl-infra/scalability/-/issues/2538#note_1595224567) the set of keys, we were able to determine that we have many stale keys from concluded experiments that can be deleted. After deleting those keys [we reclaimed](https://gitlab.com/gitlab-com/gl-infra/scalability/-/issues/2538#note_1595373896) 6 GiB of space, 40% of the used memory:  ### Problem Our [experiment process](https://about.gitlab.com/handbook/engineering/development/growth/experimentation/) and [`gitlab-experiment`](https://gitlab.com/gitlab-org/ruby/gems/gitlab-experiment) in particular are leaving stale keys behind when we conclude experiments. These waste a lot of Redis memory. We should ensure they get cleaned up. ### Proposal We could augment the [Experiment Cleanup process](https://about.gitlab.com/handbook/engineering/development/growth/experimentation/#experiment-cleanup-issue) to also clear out experiment data from Redis once the experiment has concluded. #### Possible Solutions :construction: - :one: Amend [chatops cleanup](https://docs.gitlab.com/ee/development/feature_flags/controls.html#cleanup-chatops) to detect when it is `experiment` feature flag type(I believe in the api [here](https://gitlab.com/gitlab-org/gitlab/-/blob/e0e1ec15490e0f9c3eec4abe4d9c5c3eeb03f3ac/lib/api/features.rb#L166-166)) and also try to run a clear on the key using [this method](https://gitlab.com/gitlab-org/ruby/gems/gitlab-experiment/-/blob/948bc7aa07ec95d46702b3e6b0ff26ebac2bda3f/lib/gitlab/experiment/cache/redis_hash_store.rb#L23-23) - Pros: - no extra commands added to cleanup process - automatically handles this concern(no need to lint or make sure we remember during MR review to do something else) - maybe the behavior we want for stopping any experiment anyway...in cases other than cleanup(? not sure here, if it isn't we could ensure it only happens on 'delete' action) - would only target SaaS instances by default(no need to add `.com?`) logic anywhere else. - Cons: - too much magic - surprising behavior if we were to delete a feature flag for an experiment and we expected that re-enabling it would keep old assignments. - :two: Using a [post deployment migration](https://docs.gitlab.com/ee/development/database/post_deployment_migrations.html) run the clear command using [this method](https://gitlab.com/gitlab-org/ruby/gems/gitlab-experiment/-/blob/948bc7aa07ec95d46702b3e6b0ff26ebac2bda3f/lib/gitlab/experiment/cache/redis_hash_store.rb#L23-23) - Pros: - is very declarative in the code - Cons: - an extra step to the MR cleanup process that could be missed/might need to be linted on. - would need to probably guard the cleanup with a `.com?` check as experiments are only ran on SaaS An example of clearing cache can be seen in the tests in [1](https://gitlab.com/gitlab-org/gitlab/-/blob/e0e1ec15490e0f9c3eec4abe4d9c5c3eeb03f3ac/spec/experiments/application_experiment_spec.rb#L238-238) and [2](https://gitlab.com/gitlab-org/gitlab/-/blob/e0e1ec15490e0f9c3eec4abe4d9c5c3eeb03f3ac/spec/experiments/application_experiment_spec.rb#L283-283) with an example implementation looking like this for `tier_badge`: ```ruby Gitlab::Experiment::Configuration.cache.clear(key: 'tier_badge') # main assignment cache Gitlab::Experiment::Configuration.cache.clear(key: 'tier_badge_attrs') # any arbitrary attributes that were cached ```