mini post mortem
A little postmortem for #1247 (Can't save project. Please try again later):
- 2015-03-18 05:00 (UTC) v7.9rc1 is released and deployed to gitlab.com
- 2015-03-18 10:00 (UTC) a user notices the bug and reports it to the community forum
- 2015-03-18 12:00 (UTC) @axil sees the forum entry, creates an issue and informs the core team
- 2016-03-18 13:00 (UTC) first attempts at locating the error are made by @haynes and @DouweM
- 2015-03-18 14:00 (UTC) @sytses wakes up and informs @marin
- 2015-03-18 14:40 (UTC) @marin starts looking into the issue
- 2015-03-18 14:48 (UTC) @marin fixes the issue on gitlab.com
In total the issue was active on gitlab.com for ~10 hours
and it took ~5 hours from the creation of the bug report to fixing it.
3 of those 5 hours were spend sleeping by half of the team because of the different time zones.
To ensure that the response time is faster in case something like this happens again, @DouweM now has an emergency mail address he can use to wake up @sytses and @dzaporozhets
It's still being investigated how the bug could pass the gitlab test suite without being noticed.