Add Alerts for high rate of created but not started Sessions

Problem

In gitlab-com/gl-infra/production#22241 (closed) and other incidents related to Duo CLI updates before that, we had the case that sessions got created but then never properly started, since the CLI initialization failed. Our normal alerting on Duo Workflow Service cannot catch this other than through analyzing traffic patterns, since the connection to the service is never properly established. Alerts based on this are notoriously difficult to get right.

Suggested Solution

Establish Monitoring/alerting on the amount of workflows in "created" status. A higher amount of those would signify a problem like in the incident or related (e.g. runners not working correctly or similar).

Edited by 🤖 GitLab Bot 🤖