2026-02-23: Sidekiq queueing apdex SLO violation on multiple shards

Sidekiq queueing apdex SLO violation on multiple shards (Severity 3)

Problem: Sidekiq jobs failed to meet queueing duration targets across the catchall, import-shared-storage, and elasticsearch shards, causing multiple apdex SLO violations and severe job processing delays.

Impact: Sidekiq queueing delays occurred on three shards:

• On the catchall shard, only 78.7% of jobs met queueing duration targets in the past hour, with apdex near zero from 14:05 to 14:45 UTC, causing severe job processing delays for import/export tasks.

• On the import-shared-storage shard, apdex dropped to 24.42% over the last hour, meaning most jobs experienced significant delays.

• On the elasticsearch shard, the apdex was 97.44% over 6 hours, falling below the required SLO.

Causes: A high volume of Projects::ImportExport::RelationExportWorker jobs overloaded multiple Sidekiq shards, including catchall, import-shared-storage, and elasticsearch. Planned changes to move these jobs to a different shard were not merged, so the jobs continued to run on their original shards.

Response strategy: No configuration changes or merges have been made. Merge requests to reroute import/export jobs remain open for consideration.

This ticket was created to track INC-7649, by incident.io 🔥