Add metric to track state of new packages

What does this MR do and why?

Describe in detail what your merge request does and why.

Content

Add a metric called delivery_auto_deploy_new_package_state which will track if there are packages newer than the one on gstg-cny. The metric will track packages in the running, success or failed states, i.e. packages that are being built, have been successfully built or have failed building.

The metric name (delivery_auto_deploy_new_package_state) isn't the best, so I'm open to suggestions.

The metric has a single label (pkg_state), which can have 3 possible values (running, success, failed) to track packages that are building, have been successfully built and have failed building. If there are one or more new packages (newer than what's on gstg-cny) being built, delivery_auto_deploy_new_package_state{pkg_state='running'} will be set to 1. If there are no new packages being built delivery_auto_deploy_new_package_state{pkg_state='running'} will be set to 0. Similarly for the success and failed states.

The metric is changed on the following events:

• When a new package starts being built, delivery_auto_deploy_new_package_state{pkg_state="running"} is set to 1.

• When a package completes building successfully, delivery_auto_deploy_new_package_state{pkg_state="success"} is set to 1.

  • And we check to see if delivery_auto_deploy_new_package_state{pkg_state="running"} and delivery_auto_deploy_new_package_state{pkg_state="failed"} should be set to 0. pkg_state="running" should be set to 0 if there are no new packages being built. pkg_state="failed" should be set to 0 if there are no new packages that have failed building. This is done by checking for packager pipelines in dev.gitlab.org that are for versions newer than what's on gstg-cny.

• When package building fails, delivery_auto_deploy_new_package_state{pkg_state="failed"} is set to 1.

  • And we check to see if delivery_auto_deploy_new_package_state{pkg_state="running"} should be set to 0.

• When a package is deployed to gstg-cny, we check to see if all 3 package states (success, failed, running) should be set to 0.

This metric can be used in the following Prometheus query to get the intersection of time when gstg-cny was idle and new packages were in different states.

(max by (pkg_state) (delivery_auto_deploy_new_package_state) > bool 0) * on() group_left max(delivery_auto_deploy_environment_state{target_env="gstg",target_stage="cny",env_state="ready"})

gitlab-com/gl-infra/delivery#20337 (closed)

Author Check-list

• Has documentation been updated?