Review SharedState workload for Redis Cluster compatibility
Following the discussion in gitlab-com/runbooks!5407 (comment 1280241770), the method of waiting for an elapsed period of time before switching read targets to the "warmed" data store will work for a majority of our use cases (cache, ratelimiting, db-load-balancing all use a TTL).
Gitlab::Redis::SharedState is slightly trickier since it points to redis-persistent and the workloads may expect a certain degree of persistence (which means lack of TTL). I've looked through our existing SharedState usage to pick out possible problematic areas should we decide to migrate them.
Pub/Sub & Workhorse
Note: To be done resolved by &1066
Compatibility with Redis Cluster
actioncable
Note: To be done resolved by &1066
eval scripts
Lua scripts taking in multiple cross-slot keys would not work in a Redis Cluster set-up. Also, Lua scripts that run commands on keys not passed into the script would be problematic (keys might not be on that node).
Eval scripts with more than 1 input keys
- lib/gitlab/counters/buffered_counter.rb -- but it uses hash tags
existing allowed cross-slot operations (pipelines vs transactions)
| file | operations | fix |
|---|---|---|
| app/models/ci/build_trace_chunks/redis_base.rb (via build_trace_chunks/redis.rb) | bulk del
|
convert to pipeline |
| ee/app/services/elastic/process_bookkeeping_service.rb | bulk unlink
|
convert to pipeline |
| ee/app/services/elastic/indexing_control_service.rb |
del with 2 keys |
gitlab-org/gitlab!125351 (merged) |
| lib/gitlab/manifest_import/metadata.rb |
multi withmanifest_import:metadata:user:#{user.id}:#{field} keys |
gitlab-org/gitlab!126446 (merged) |
| lib/gitlab/issues/rebalancing/state.rb |
multi with cross-slot keys |
gitlab-org/gitlab!126281 (merged) |
| app/models/active_session.rb | mget |
use cross-slot pipeline |
| lib/gitlab/sidekiq_status.rb | mget |
use cross-slot pipeline |
Cache warming
This is not a big problem considering we have a validator script. But it would be good to give us an idea on the proportion/volume of keys to be migrated.
Click to expand
Directories checked (on commit f2a60ab0bb59d1e83ba6255d6ee8b2902df81fca)
- app/
- lib/
- ee/lib/
- ee/app/
Files containing SharedState usage with expiry
| file | TTL? |
|---|---|
| ee/lib/ee/gitlab/exclusive_lease.rb | variable TTL |
| ee/lib/gitlab/auth/group_saml/response_store.rb | 5 minutes |
| ee/lib/gitlab/ci/minutes/cached_quota.rb | 10 minutes |
| ee/app/services/ci/minutes/update_project_and_namespace_usage_service.rb | 12 hours |
| ee/app/services/ci/minutes/track_live_consumption_service.rb | 5 minutes |
| app/helpers/users/callouts_helper.rb | 1 hour |
| app/services/database/consistency_check_service.rb | 7 days |
| app/models/concerns/web_hooks/has_web_hooks.rb | 1 hour |
| app/models/awareness_session.rb | 1 hour |
| app/models/jira_connect/public_key.rb | 5 minutes |
| app/models/ci/runner.rb | 1 hour |
| lib/api/helpers/internal_helpers.rb | NA |
| lib/gitlab/etag_caching/store.rb | 20 minutes |
| lib/gitlab/phabricator_import/worker_state.rb | 24 hours |
| lib/gitlab/issues/rebalancing/state.rb | 10 days |
| lib/gitlab/sidekiq_daemon/monitor.rb | 24 hours |
| lib/gitlab/workhorse.rb | 1 hour |
| lib/gitlab/setup_helper.rb | NA |
| lib/gitlab/manifest_import/metadata.rb | 1 week |
| lib/gitlab/search/recent_items.rb | 7 days |
| lib/gitlab/sidekiq_status.rb | 30 minutes |
| lib/gitlab/background_migration/fix_incoherent_packages_size_on_project_statistics.rb | NA |
| lib/gitlab/usage/metrics/aggregates/sources/postgres_hll.rb | 30 hours |
| lib/gitlab/exclusive_lease.rb | variable TTL |
| lib/gitlab/job_waiter.rb | 6 hours |
| lib/gitlab/ci/trace.rb | 10 seconds |
| lib/gitlab/checks/post_push_message.rb | 5 minutes |
| lib/gitlab/github_gists_import/status.rb | 24 hours |
| lib/gitlab/reference_counter.rb | 10 minutes |
| lib/gitlab/chat_name_token.rb | 10 minutes |
| lib/gitlab/web_hooks/recursion_detection.rb | 30 minutes |
| lib/gitlab/database/rename_reserved_paths_migration/v1/rename_base.rb | 2 weeks |
| lib/gitlab/usage_data_counters/hll_redis_counter.rb | 6 weeks |
Files which uses SharedState without expiring keys
- ee/lib/gitlab/mirror.rb
- ee/lib/gitlab/geo/event_gap_tracking.rb
- lib/gitlab/auth/unique_ips_limiter.rb
- lib/gitlab/counters/buffered_counter.rb
- lib/gitlab/inactive_projects_deletion_warning_tracker.rb
- app/models/hooks/project_hook.rb
- app/models/design_management/design_collection.rb
- app/models/concerns/awareness.rb
- app/workers/concerns/limited_capacity/job_tracker.rb
- app/models/concerns/repositories/can_housekeep_repository.rb
- ee/app/services/elastic/indexing_control_service.rb
- ee/app/services/elastic/process_bookkeeping_service.rb
- ee/app/models/geo/project_registry.rb
- ee/app/services/product_analytics/initialize_stack_service.rb