SSO not working
Please note: if the incident relates to sensitive data, or is security related consider labeling this issue with security and mark it confidential.
Summary
Enabling the feature enforced_sso_requires_session
was making groups inaccessible for members and pipelines fail for customers using SAML in groups. https://gitlab.com/gitlab-org/gitlab-ee/issues/11704
Service(s) affected: CI and all features for impacted Groups in the UI.
Team attribution: Manage
Minutes downtime or degradation: 243 minutes
Timeline
2019-05-22
- 06:39 UTC - deployer: John Jarvis is starting a deploy pipeline of 11.11.0-rc5.ee.0 on gprd
- 09:53 UTC - jedwardsjones
/chatops run feature set enforced_sso_requires_session true
- 10:16 UTC - first Zendesk issue opened by customer: https://gitlab.zendesk.com/agent/tickets/122122
- 11:51 UTC - deployer: John Jarvis finished a deploy of 11.11.0-rc5.ee.0 on gprd
- 12:49 UTC - gitlab-ee issue https://gitlab.com/gitlab-org/gitlab-ee/issues/11704 opened by support
- 13:23 UTC - e-group slack channel notified of customer issues: https://gitlab.slack.com/archives/C5W3VS1C4/p1558531417312100
- 13:25 UTC - zoom call with customer
- 13:51 UTC - customer reporting workaround for users by going through the SSO login process again (but this is not fixing the ci tokens): https://gitlab.com/gitlab-org/gitlab-ee/issues/11704#note_173307502
- 14:08 UTC - deployer: Yorick Peterse is starting a deploy pipeline of 11.11.0-ee.0 on gprd
- 14:09 UTC - issue reported by @markpundsack in incident-management slack channel:
- 14:13 UTC - incident created via imoc-bot by @markpundsack
- 14:15 UTC - @ahanselka paging on-call
- 14:16 UTC - @stanhu pointing at enabling
enforced_sso_requires_session
feature as possible reason - 14:19 UTC - alex
/chatops run feature set enforced_sso_requires_session false
- 14:21 UTC - @hphilipps creating incident issue manually as imoc-bot failed to do it
- 14:35 UTC - customer reporting issues to be fixed: https://gitlab.com/gitlab-org/gitlab-ee/issues/11704#note_173329786
- 14:39 UTC - status.io update about investigating issues with SSO on ci-runners
- 14:59 UTC - status.io update about feature being disabled
- 15:16 UTC - status.io resolved
- ...
Incident Doc
RCA
https://gitlab.com/gitlab-com/gl-infra/infrastructure/issues/6750