Saturation growth predictions and reporting with Jupyter notebooks, juypter books, Facebook Prophet, integrated with Prometheus saturation metrics

This would deliver FY21-Q3 Infra KR: Improve runbooks experience using Jupyter notebooks

gitlab-com/www-gitlab-com#8276 (closed)

I've been working with GitLab's Jupyter integration for the past few days, and, at present, am not convinced that this can be used in critical and time-sensitive periods, such as during an incident - the times we tend to use runbooks the most.

As a first step, I propose that we use this functionality for something not runtime critical: capacity planning.

Proposal: Capacity Planning using Facebook Prophet and Prometheus Saturation Metrics

1. Integrate Juypter Notebooks, Jupyter Books, The Prometheus Pandas Library and Facebooks' Prophet library
2. Provide forward forecasts for up to 90 days, based on saturation metrics over the past year
3. Run this regularly in CI (on the ops instance) and notify if any saturation events are predicted
4. Persist the jupyter book in GitLab Pages
5. Allow operators to run the book interactively in Jupyter

As a proof-of-concept, I've already been experimenting with this, running Jupyter Notebooks locally.

Some of the graphs from this: