On-Call Handover 2022-02-24 07:00 UTC

On-Call Handover

Brought to you by woodhouse

• EOC egress: @f_santos
• EOC ingress: @alejandro
• IM egress: @sloyd
• IM ingress: kwanyangu jprovaznik @wortschi

📖 Summary:

What (if any) time-critical work is being handed over?

What contextual info may be useful for the next few on-call shifts?

🔴 Ongoing alerts/incidents:

GitLab

• production#6419 (closed) - severity3 2022-02-23: Service Desk no longer processing emails
• production#6417 (closed) - severity4 2022-02-23: unusual size of memory_bound sidekiq backlog
• production#6416 (closed) - severity4 2022-02-23: Logging visibility (workhorse) delayed
• production#6415 (closed) - severity3 2022-02-23: The Horizontal Pod Autoscaler Desired Replicas resource of the sidekiq service (main stage) has a saturation exceeding SLO and is close to its capacity limit
• production#6407 (closed) - severity3 2022-02-22: Cloudflare intermittently returning 521/520 errors during pipeline clones
• production#6381 (closed) - severity4 2022-02-18: Creation timestamp of container images shown as null in the UI/API
• production#6338 (closed) - severity3 2022-02-12: Increased latency from us-east1-d for GCS buckets

✅ Resolved alerts/incidents:

GitLab

• production#6420 (closed) - severity3 2022-02-23: The goserver SLI of the gitaly service on node file-43-stor-gprd.c.gitlab-production.internal has an apdex violating SLO

🔓 Change issues:

Closed

• production#6378 (closed) - Bump KAS to v14.8.1 in gprd

added SRE:On-Call label

assigned to @alejandro and @wortschi

closed