keep track of pending migrations
As a release manager, it will be very useful to know if we have pending migrations in each environment. It will be extremely useful in several scenarios discussed in #1978 (closed).
We have prometheus node-exporter running on our deploy boxes, they are already configured with textfile collector pointing to /opt/prometheus/node_exporter/metrics
.
We could extend the deployer migrations
and post-deployment-migrations
jobs to gather such information at the end of their current playbook.
- get the number of pending migrations with
gitlab-rails db:migrate:status | egrep "^ +down +" | wc -l
- write the information in a temporary file on the host using prometheus text file format:
# HELP delivery_metrics_pending_migrations_total number of pending migrations # TYPE delivery_metrics_pending_migrations_total gauge delivery_metrics_pending_migrations_total PUT_THE_NUMBER_HERE
- Move the temporary file to
/opt/prometheus/node_exporter/metric/delivery_metrics.prom
, this is a best practice as the node-exporter may read a partially written file, but moving it after it's already on the same disk will be an atomic operation.
We don't need extra labels as prometheus will correctly apply environment
and stage
labels.
Final status
These graphs show the pending migrations over time for the last week. The first panel is about the main
stage, the second one is for cny
.
NOTE: the DB is shared at the env
level (gprd
or gstg
) regardless of the stage (cny
or main
). But the values are different because pending migrations depend on the installed version.
NOTE: the cny
stage does not have a post-deployment migration job, because of that the values for the cny
stage are lagging behind. They only get updated when a new package is rolled out into the environment.