DAST Full Scans fail to execute
Incident review: https://gitlab.com/gitlab-com/gl-infra/infrastructure/issues/8179
Summary
A recent release of DAST caused full scans to fail to execute. Baseline scans were not affected. Note that this issue in no way would have affected GitLab.com in general.
Service(s) affected : DAST (full scan)
Team attribution : Secure, DAST
Minutes downtime or degradation : 2027 minutes (1 day, 9 hours, 47 minutes)
Timeline
2019-10-08
- 16:44 UTC - commit Upgrade to Python 3 gitlab-org/security-products/dast@5100d514 is merged into DAST master branch
2019-10-13
- 04:30 UTC - scheduler runs and automatically deploys DAST https://gitlab.com/gitlab-org/security-products/dast/pipelines/88455397
- 23:40 UTC - DAST engineer(s) learn that DAST end to end tests have been silently failing, submit fixes
2019-10-14
- 03:21 UTC - DAST engineer(s) learn that recent release is affected
- 14:17 UTC - DAST engineer(s) test and release, resolving issue
Analysis
Considering this a S2 because less than 50% of DAST users will be affected (most DAST users will be using a Baseline scan). There was no workaround while the feature was broken.
/cc @sethgitlab
Edited by Cameron Swords