2020-03-09: File-27 was briefly inaccessible

Summary

file-27 was inaccessible for a brief period of time, causing requests the were interacting with the node to fail. We forced a node reboot to restore the situation.

More information will be added as we investigate the issue.

Timeline

All times UTC.

2020-03-09

• 09:28 - We're paged about file-27 being down
• 09:29 - We try to SSH into the node but the connection doesn't go through
• 09:31 - We connect to the node via serial port but no errors were observed
• 09:35 - We decide to reboot the node from GCP console to restore the service
• 09:38 - The alert is resolved

Resources

1. If the Situation Zoom room was utilised, recording will be automatically uploaded to Incident room Google Drive folder (private)