Root Cause Analysis of Recent Web IDE Regression
Summary
Some high visible regressions (example) for the Web IDE were deployed to .com
for sometime and released in the initial 12.7.0
.
Inspired from our handbook, let's use this issue to capture the root cause of this and any high priority follow up action items.
Analysis
Why was the regression introduced?
- This MR !22364 (merged) which focused on MR diff UX, touched some components that are shared by the Web IDE. This relationship was unfortunately overlooked and the changes made broke some critical behaviors of the Web IDE.
- There were no specs on these specific Web IDE behaviors which would have caught this.
Why were there no specs for these specific Web IDE behaviors?
- This specific use case would is difficult to capture/anticipate in just a unit test.
- We don't have any pattern for FE integrated tests (see issue #26982 (closed))
- The features specs for the Web IDE are broken in the development environment so they've been neglected for some time (see issue #23579 (closed)).
Why was the relationship of these shared components overlooked?
- These components are very fragile and their tests are weak/focused on the happy path.
Why did this reach .com
and 12.7.0.
?
- The MR introducing the regression was merged on Thursday, but it didn't reach
.com
until Friday-ish when the regression was noticed #197493 (closed). Monday-ish (which was a US holiday) was the cutoff for the release😞
Action Items
- Fix feature specs in the Web IDE and increase feature coverage, probably with the help of the quality team (see issue #23579 (closed)).
- Establish integrated FE test pattern which will be extremely helpful for FE-heavy features like the Web IDE (see issue #26982 (closed)).
- Refactor the fragile components and back with proper specs (relevant comment #118442 (comment 275596278)).
Lessons Learned
What went well
- Revert of the MR went smoothly and quickly (many thanks to our glorious release manager @mayra-cabrera)!
What went wrong
- Unfortunate close time window of issue visible/identified and the release cut off. Possible remedy is to make sure there is sufficient business days between release and cut off.
References
- The culprit MR !22364 (merged)
- The revert !23428 (merged)
- #197493 (closed)
- #198002 (closed)
- #198029 (closed)
- #198057 (closed)
Edited by Paul Slaughter