## Summary The Geo logcursor calls `perform_async` for every blob event regardless of the `sync_object_storage` setting. The worker discards these jobs immediately — but the logcursor has already executed 13 Sidekiq client middleware per event. Since ~99% of events are blob-related, this wastes ~99% of logcursor capacity on noops. On our secondary the backlog grew to 82 million events (~5 days). geo:status reported "Healthy" and "99% synced" throughout — a failover would have caused 5 days of data loss. ## Steps to reproduce 1. Deploy a Geo secondary where the primary generates >100 blob events/sec 2. Set `sync_object_storage: false` on the secondary (or use shared object storage) 3. Observe that logcursor throughput does not improve and event gap grows continuously ## What is the current bug behavior? The `sync_object_storage?` check happens inside Geo::EventWorker, after perform_async has paid the full middleware cost: 1. Primary always creates events — https://gitlab.com/gitlab-org/gitlab/-/blob/v18.9.1-ee/ee/lib/gitlab/geo/replicator.rb#L329 checks only `Gitlab::Geo.primary?` 2. Logcursor always enqueues — https://gitlab.com/gitlab-org/gitlab/-/blob/v18.9.1-ee/ee/lib/gitlab/geo/log_cursor/daemon.rb#L151 returns `true` for SSF events (project_id = nil — all blob events) 3. Filtering only inside worker — https://gitlab.com/gitlab-org/gitlab/-/blob/v18.9.1-ee/ee/app/models/concerns/geo/replicable_model.rb#L69-75 checks `sync_object_storage?` after enqueue ## Environment: GitLab 18.9.1-ee, multi-node secondary (3 Rails, 3 Sidekiq), Premium self-managed. Primary: ~164 events/sec sustained, ~99% blobs. Logcursor: ~90 events/sec (BATCH_SIZE = 50), single-leader (https://gitlab.com/gitlab-org/gitlab/-/blob/v18.9.1-ee/ee/lib/gitlab/geo/log_cursor/daemon.rb#L30). ## What is the expected correct behavior? When `sync_object_storage` is disabled, the logcursor should skip `perform_async` for blob events before entering the middleware chain. ## Possible fixes Add the `sync_object_storage?` check to https://gitlab.com/gitlab-org/gitlab/-/blob/v18.9.1-ee/ee/lib/gitlab/geo/log_cursor/daemon.rb#L151-158 in Daemon, which already handles selective sync filtering. Currently https://gitlab.com/gitlab-org/gitlab/-/blob/v18.9.1-ee/ee/lib/gitlab/geo/log_cursor/daemon.rb#L152 unconditionally returns `true` for all SSF events (`project_id.nil?`), including blob events that will be discarded by the worker when sync_object_storage is disabled. However this is prone to problems: `can_replay?` doesn't know the event type, and sits at the wrong layer of abstraction. We would rather move this check into the Event processing part, which has access to the event data. ## Implementation plan ### Skip logic We add the skip logic to [`Events::Event#process`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/lib/gitlab/geo/log_cursor/events/event.rb?ref_type=heads) under a feature flag. This allows users to opt-in to the fix, while everyone else gets no behaviour change in case the skip logic is buggy. ```ruby def process if skip_enqueue? log_event('Skipped enqueue (not in replicables for current secondary)') return end ::Geo::EventWorker.perform_async(event.replicable_name, event.event_name, event.payload) end private def skip_enqueue? return false unless Feature.enabled?(:geo_log_cursor_skip_object_storage_events) return false if event.event_name == 'deleted' # We always enqueue deleted events # We always enqueue events if the node is set to sync storage node = ::Gitlab::Geo.current_node return false if node&.sync_object_storage? # We always enqueue objects that are not storable model_class = ::Gitlab::Geo::Replicator.for_replicable_name(event.replicable_name).model return false unless model_class.object_storable? # At that point, we've established that objects are not synced externally and the model is storable. The last check is whether the actual object is stored locally. If so, we still want to sync it. model_record_id = event.payload['model_record_id'] return false unless model_record_id return false if model_class.with_files_stored_locally.where(id: model_record_id).exists? true end ``` ### Extend specs Add cases under [`describe "#process"`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/spec/lib/gitlab/geo/log_cursor/events/event_spec.rb?ref_type=heads) covering: | Scenario | FF | `sync_object_storage?` | event_name | replicator | storage | Expected | |---|---|---|---|---|---|---| | Flag disabled | off | false | created | blob | remote | enqueues | | Flag on, sync_object_storage true | on | true | created | blob | remote | enqueues | | Flag on, delete event | on | false | deleted | blob | remote | enqueues | | Flag on, non-object-storable replicator | on | false | created | repository | n/a | enqueues | | Flag on, blob in object storage | on | false | created | blob | remote | **skips** | | Flag on, blob stored locally | on | false | created | blob | local | enqueues | | Unknown replicable_name | on | false | created | unknown | n/a | raises NotImplementedError | Use `stub_feature_flags`, factory `create(:geo_event, :package_file, ...)`, and stub `::Gitlab::Geo.current_node` for node attributes.