2018-08-02 11.2.0-rc1 Deploy errors

Summary

infrastructure~3760141

Timeline of events

08-02 12:20 UTC Deploy Started on GitLab.com of 11.2.0-rc1. As the deploy rolled out - we started getting alerts of higher rails Error rates: https://performance.gitlab.net/d/000000256/rails?orgId=1&from=1533222000000&to=1533234057279

08-02 17:07 UTC - started incident to track down elevated 5xx errors.

08-02 18:00 UTC - we identified errors related to deploy tokens and are making patches to the release and are rolling those out.

08-02 18:30 UTC - the patches have been rolled out and the error rates are back at normal levels.

Monitoring

Screen_Shot_2018-08-03_at_9.36.59_AM

Screen_Shot_2018-08-03_at_9.36.53_AM

Logs

Fixes were:

  • https://dev.gitlab.org/gitlab/post-deployment-patches/merge_requests/89
  • https://dev.gitlab.org/gitlab/post-deployment-patches/merge_requests/88
Edited Aug 03, 2018 by John Jarvis
Assignee Loading
Time tracking Loading