contributors app hasn't been updated for over a month

Summary

It was reported via internal slack that our contributors application has stopped updating.

https://gitlab.slack.com/archives/C02PF508L/p1542240066562400

• Service(s) affected : contributors
• Team attribution :
• Minutes downtime or degradation : nearly 30 days

Timeline

2018-11-15

• 00:01 UTC - reported that contributors application data is out of date

2018-12-05

• 13:56 UTC - thread was bumped to gain visibility
• 14:10 UTC - investigation concluded that space had run out on the server preventing the application from working properly
• 14:33 UTC - the production.log file is not rotated in this application, the app was stopped, log file was gzipped
• 14:34 UTC - the application was manually started

Improvements

• This is a one-off server, we have multiple de-prioritized issues to get this server into a decent state:
  • https://gitlab.com/gitlab-com/gl-infra/infrastructure/issues/4663
  • https://gitlab.com/gitlab-com/gl-infra/infrastructure/issues/1927

What did not go well

• Due to the above desired improvements we had zero notification that something was wrong except for an end user hitting us up and the maintainer manually pinging a member of the infrastructure team