Threat Insights FCL for Incident 7127
Feature Change Lock
Reason for thisOn 2022-05-23 we introduced a change to our security configuration through !86639 (merged) which caused runtime errors for certain users on .com.
The issue was mitigated by rolling back the MR.
Team
- EM: @nmccorrison
- PM: @matt_wilson
- Team: @gitlab-org/secure/threat-insights-frontend-team
FCL time period
5 business days - 2022-06-01 to 2022-06-08
Resources
- Incident issue: gitlab-com/gl-infra/production#7127 (closed)
- Slack channel for FCL: https://gitlab.slack.com/archives/C03GYMK6FCN
- Slack channel for incident: https://gitlab.slack.com/archives/C03GCPDRP9D
- Reliability standup doc: https://docs.google.com/document/d/1vww0BfRzHtrGhMppTEw5Q27KSzD1e72dmJ3XoppxC-A/edit#heading=h.kg4wp2r1yf82
- FCL Handbook page: https://about.gitlab.com/handbook/engineering/#feature-change-locks
Focuses for FCL
- Root cause analysis on !86639 (merged)
- Re-visit solution for how to display tab conditionally against the user's license tier - #358183 (closed)
- Identify and update documentation on how to test different environment and user licenses locally.
- Revisit how we stub tests to ensure proper coverage of tests and usage of mock data.
- Evaluate E2E test coverage and opportunities to shift left on failed test pipelines.
- Evaluate escalation procedure and extending visibility to team on failures.
- Pipeline failure - #363313 (closed)
- Sentry error - #363267 (closed)
- Revisit, confirm and re-communicate our issue/MR verification policy. Handbook page
- Create a team-based PTO handoff policy where non-draft MRs are handed off to another team member.
- Prioritize frontend analysis for performance optimizations on our Vulnerability Report - #322795 (closed)
- Identify future reliability work
Tasks/Timeline
-
Complete Incident Review section - @nmccorrison -
Create FCL slack channel @nmccorrison -
Add members of slack channel per @nmccorrison -
FCL definition -
FCL kickoff -
Create engineering task issues (based on focuses above) and promote this issue to an epic -
Schedule synchronous closing ceremony @nmccorrison -
Conduct synchronous closing ceremony @nmccorrison
Edited by Neil McCorrison