Post-deployment monitoring (continuous verification) MVC

Problem to Solve

Continuous deployment should be easy and boring. One thing that makes it more comfortable is to have monitoring to measure service-level objectives and the impact on those SLOs of an individual deploy. When doing an automatic incremental deploy (https://gitlab.com/gitlab-org/gitlab-ee/issues/1660) or canary deploy (https://gitlab.com/gitlab-org/gitlab-ee/issues/1659), we should be able to use these measurements to automatically halt a deploy and even revert/rollback.

Use Cases

Scenario: Incremental rollout, notices error rate exceeds SLO of 0.1%, aborts rollout at 1%, and reverts to last-known-good version.

Proposal

• We will use [pre-existing defined error rates] (https://docs.gitlab.com/ee/user/project/integrations/prometheus_library/nginx_ingress.html#metrics-supported)

Name	Query
Throughput (req/sec)	sum(label_replace(rate(nginx_ingress_controller_requests{namespace="%{kube_namespace}",ingress=~".%{ci_environment_slug}."}[2m]), "status_code", "${1}xx", "status", "(.)..")) by (status_code)
Latency (ms)	sum(rate(nginx_ingress_controller_ingress_upstream_latency_seconds_sum{namespace="%{kube_namespace}",ingress=~".%{ci_environment_slug}."}[2m])) / sum(rate(nginx_ingress_controller_ingress_upstream_latency_seconds_count{namespace="%{kube_namespace}",ingress=~".%{ci_environment_slug}."}[2m])) * 1000
HTTP Error Rate (%)	sum(rate(nginx_ingress_controller_requests{status=~"5.",namespace="%{kube_namespace}",ingress=~".%{ci_environment_slug}."}[2m])) / sum(rate(nginx_ingress_controller_requests{namespace="%{kube_namespace}",ingress=~".%{ci_environment_slug}.*"}[2m])) * 100

For the POC we will use HTTP Error Rate (%)

• Using the existing Prometheus API we will query the current threshold of error rates
• If an error threshold exceeds the defined threshold we will stop deployment (we need to check if we can leverage the existing trigger similar to the incident response issue creation)
• If the rollout was stopped due to exceeding threshold, On the deploy board there should be a notification of: "Rollout stopped due to high error rate"

We will present the error rate only in the environment page (deploy board) and we will make this very minimal - UX TBD As for Notifications - for the MVC we will use issue creation/email notifications that exist for incident response - TODOs/assignments will not be part of the MVC

Future UX

Links / references

Further details

• Prior art: https://harness.io/harness-continuous-delivery/secret-sauce/continuous-verification/

changed milestone to %11.8

changed title from Monitoring Feedback Loops to Post-deployment monitoring MVC

added UX + 1 deleted label

added to epic &494 (closed)

changed title from Post-deployment monitoring MVC to Post-deployment monitoring (continuous verification) MVC

added Category:Continuous Delivery + 1 deleted label

removed from epic &494 (closed)

changed milestone to %"2019 Q2"

added [deprecated] Accepting merge requests + 1 deleted label

changed the description

added 2 deleted labels and removed 2 deleted labels

mentioned in epic &494 (closed)

added GitLab Premium + 1 deleted label

marked #1661 (closed) as a duplicate of this issue

changed the description

changed milestone to %12.2

changed milestone to %12.0

added to epic &765 (closed)

changed milestone to %12.4

Updating items more than 3 months out to align to the quarter instead of a specific release. We are still targeting within the same time period, but it's unnecessary to apply to a specific release this far out.

changed milestone to %FY20 Q3

added maturitylovable + 1 deleted label

changed milestone to %Backlog

removed Product Vision FY20 + 1 deleted label

added Product Vision FY20 + 1 deleted label

large customer interested https://gitlab.my.salesforce.com/0016100001ABpAc?srPos=0&srKp=001

@jeffersonj Can you connect me to the customer to deepen the understanding of what they want/expect?

Customer shared two concerns

1. Would like the capability to incrementally rollout eg: 10% or 50% and have the ability to rollback that 10% or 50% released. (Not sure if its all or none today)
2. Have the capability to either auto rollback when a certain metric threshold has exceeded within the metrics gathered or have the ability in the gitlab-ci.yml to rollout to 10% and then have the ability to set a wait time (eg: 30 mins or some other arbitrary time ) before continuing the rollout. Since there is no automated rollback this would allow users the time to monitor the rollout and determine if the change is stable enough to continue the rollout.

We do have timed rollouts so that should solve for 2. Other than the ability to auto rollback when a certain metric threshold has been exceeded.

added customer success + 1 deleted label

changed milestone to %12.3

Set milestone to 12.3

changed milestone to %12.7

@darbyfrey Can you help me figure out where we can get the error rate info from?

@dosuken123 Do you have an idea how we can pull out this data?

@ogolowinski For taking metrics (e.g. error rate), we can consider using our Prometheus integration. You can ask devopsmonitor folks about the details.

A basic idea to approach this issue is:

1. Users setup prometheus instance per environment. GitLab continuously polls the metrics from each prometheus (background process) and persists the status summary in database.
2. If one of the performance factor exceeds a threshold, GitLab hooks the event and do a corrective action (e.g. rollback).

The questions are:

• Is this feature available on GKE integration only? Or configurable in any format?
• Do we want to dogfood this feature in our auto deploy flow? If so, please loop in Delivery folks. Since our auto deploy uses external CD tools, we cannot control rollback in GitLab side. We'd need a way to communicate with an external CD tools.

@dosuken123 thank you! this is very helpful. In keeping in mind our iteration value, I think I will split this ticket into the following:

1, Following https://docs.gitlab.com/ee/administration/monitoring/prometheus/, I assume we will need some kind of API to poll prometheus, also I am not sure that the rack_uncaught_errors_total metric is good enough for these kinds of errors, so we need to make sure we can poll SLO metrics such as defined here: https://engineering.bitnami.com/articles/implementing-slos-using-prometheus.html. These metrics will be set and cannot be configured for the first iteration

2.Provide a way to view the metrics - either by hovering over a pod or in a separate page or link to the Prometheus dashboard- UX TBD

3. Add a setting to allow user to halt deployment on reaching threshold. This will stop deployment to additional pods and notify the user - UX TBD

4. Add a setting to allow user to rollback deployment on reaching threshold. User must also configure last known version to rollback to. This will rollback the deployment to the last known good deployment and notify the user - UX TBD

5. Add audit log event for auto rollback, indicating the error and timestamp of the rollback

6. Allow user to configure custom metrics

I would start with Kubernetes first and then think about other deployments (with your help). What do you think of this plan?

@ogolowinski It seems we've already had the most of the parts you've described => https://docs.gitlab.com/ee/user/project/integrations/prometheus.html. Can we loopin someone from devopsmonitor about how we should hook an event (e.g. alerts) from prometheus and trigger a certain action in GitLab?

@dhershkovitch Can your team help us here?

@ogolowinski, right now the way to set alert on Prometheus metrics (e.g. error rate) is to go to the Metrics chart and manually setup the alert using the 3 dots button A user can then go to the Settings ->Operation tab, enable the incident Which creates an issue for each triggered alert

@dosuken123 so it looks like indeed most of this already exists up until triggered alert. If we can read this alert we could then halt/rollback. I would create an issue for halt and an issue for rollback. what else do you think we need here?

@ogolowinski I feel rollback is a strong action and it's not always the best way to mitigate the incident. For example, if there is an abuser and he does DoS attack to the webserver, rollback doesn't help the case. On the other hand, halting the environment seems good and practical idea. We often do the same in our auto deploy that if there is an on-going incident, we don't deploy on time and postpone it. So halting deployment (lock down or unlock environments) seems good first step.

Probably, we'd add another checkbox under "Incidents" section that "Prevent further deployments until manually unlocking the environments"? Documentation-wise, we'll likely add another section under Taking action on incidents. We don't have the lock-down feature yet, so that would need to be added in a separate issue.

By the way, this issue targets GitLab Premium, however, the prometheus alert actions are under GitLab Ultimate. We would need to consider adjusting the tiers.

@dosuken123 @dhershkovitch Can we get error rate data from Prometheus ?

@sarahwaldner Can we add a checkbox under "Incidents" section named "Prevent further deployments until manually unlocking the environments" Also is there an API that catches the event besides the email? I understand that it also automatically creates an issue. @dosuken123 Can we leverage the issue in any way?

@ogolowinski, Yes, Prometheus already collects this metric automatically

@ogolowinski I am catching up here.

Can we add a checkbox under "Incidents" section named "Prevent further deployments until manually unlocking the environments"

That does not seem like the right place to add that setting. The Incidents section within Settings > Operations is for managing Incidents and notifications. I think that a better place for that setting would in Environments or Repository. If you would like to discuss this further, let's get a call set up with us and our respective UX team members.

@sarahwaldner Is there an API that catches the event of exceeding threshold besides the email notification?

@ogolowinski

When a threshold is exceeded and an event (alert) is generated, a couple things can happen:

• Email is sent (as you mentioned above)
• Issue is created (IF the user has enabled their project to do this)

After these things happen, the alert is gone. We are not storing them anywhere so there will be no record of it. Does that answer your question?

@dosuken123 based on @sarahwaldner's answer in #8295 (comment 291754703) can that be used as a trigger, or do we need to create an SPI that will trigger the pipeline?

@ogolowinski I think it can be used for trigger.

After these things happen, the alert is gone. We are not storing them anywhere so there will be no record of it

Interesting. So we'd need to persist somewhere to present the error rate in deploy board.

added grouprelease [DEPRECATED] + 1 deleted label

added Enterprise Edition + 1 deleted label

Another competitor solution for comparison: https://www.weave.works/oss/flagger/

added 2 deleted labels

added cicdplanning + 1 deleted label

added workflowdesign + 1 deleted label

@ogolowinski reading through this issue it looks like it's still too vague for potential inclusion in %12.7, in my opinion. It's missing some key decisions around what will be monitored (and maybe how), and what behaviors will be implemented and when based on what's seen, how it could be configured (if it could be configured), etc. This is also a pretty big/interesting topic and might be good to do a validation flow on.

added GitLab Ultimate + 1 deleted label and removed GitLab Premium + 1 deleted label

changed milestone to %12.8

added Product Vision FY21 + 1 deleted label

@rverissimo This is a large issue and needs a lot of UX consideration and splitting down to smaller issues,

My thought is a. create an alert if error rate exceeds SLO of 0.1%, based on https://docs.gitlab.com/ee/user/project/integrations/prometheus.html#taking-action-on-incidents-ultimate

b. if the threshold appears - we need to indicate this on the deploy board on the problematic pod

c. Introduce a new setting "Prevent further deployments until manually unlocking the environments"? Documentation-wise, we'll likely add another section under Taking action on incidents. Following #8295 (comment 263372375) we need to think of the right place to put this

d. 1. Add a setting to allow user to halt deployment on reaching threshold. This will stop deployment to additional pods and notify the user - UX TBD e. Add a setting to allow user to rollback deployment on reaching threshold. User must also configure last known version to rollback to. This will rollback the deployment to the last known good deployment and notify the user - UX TBD f. Add audit log event for auto rollback, indicating the error and timestamp of the rollback

g. Allow user to configure the threshold or other metrics (based on Prometheus)

Since this needs a lot of preparation, I am moving this to 12,9 but we should add the UX (at least for the first 2 iterations) during 12.8 period

@mnichols1 can you work on UX for

b. if the threshold appears - we need to indicate this on the deploy board on the problematic pod

Similar to the mock in the description - show the error on the deploy board. Hitting Details should bring you to the monitor page for further investigation https://gitlab.com/gitlab-org/gitlab-ee/uploads/9abdbd1a84bd348085ac8e7c4f7e5b3e/service-level-objectives.png

@jyavorska pinging you to go over the plan for stopping rollout based on #8295 (comment 265610323). I have yet to update the proposal since I am waiting for a final answer on #8295 (comment 283720871) to move forward.

The basic idea is to leverage what we already have supported in devopsmonitor and try to leverage incidence response configuration that is already supported for exceeding the threshold to trigger to the pipeline to stop rollout.

@ogolowinski yeah that roughly makes sense. As you mention though it needs a lot more thinking through of how we're going to achieve that.

changed milestone to %12.9

changed epic to &2274

mentioned in issue #196508 (closed)

Meeting notes - 20 jan, 2020

@rverissimo @dosuken123 @nudalova @ogolowinski

• MVC: Inform the user of manual intervention when rolling out is not possible. Stop rolling out the deployment.
  • Automatically rollback is not part of the mvc
  • How do we create a trigger to our pipeline so it stops rolling out?
    • Shinya: If we create an issue, it should be possible to create events.
    • Incremental rollout is handled in Kubernetes, a different platform. The pipeline just tell K8 to start a deployment. We prob need a way to tell k8 to stop rolling out - not sure if it's possible, need to dig into k8 docs and help from ~"devops::configure".
    • Orit: We can tell the user to stop it manually before k8 is told to stop it.
    • It's good to show the notification in the environments dashboard
• Rayana: We need to think where else the users need to be notified from -- since it requires a manual action from them. Email notification. Does it create a todo for users?
  • Orit: Not familiar with it, it seems like it creates issues. Not sure if the issue gets assigned to someone. Once you have an alert, users can go to operation settings. We can add a todo, it should be possible.
  • Nadia: Can we do some investigation on how it's being used?
• Future: Automatically create issues, initiate with incident response (use that to trigger a pipeline)
• Overlaps with ~"devops::configure", devopsmonitor
  • Orit: First phase we shouldn't need the configure team.
  • Shinya: Deployment board depends on k8, it's only available because of it.
  • Rayana: Need to know what those teams have planned for the UX of the feature.
• Shinya: Need to investigate the feasibility.
• Rayana: UI of the MR also needs to to be altered.
  • Orit: Need to be using monitoring, need to go to incident settings and allow to create issue/todo, and then it goes back to the interface (error message).
• Orit: Also worth to talk to customers and show some mockups. Start recruiting people cc @loriewhitaker

As a first step we want to alert the user that there is a problem that requires manual stop of rollout. We are thinking of adding this to the deploy board. At the moment if a threshold is met, and IR is enabled in the settings , an issue is created. Can we add a notification - email and/or TODO? @sarahwaldner

We need to figure out . way to tell K8s to stop deployment - @dosuken123

Next phase automatic stop deployment will be handled

@danielgruesso @sarahwaldner We will need your help to get this in place.

Action items:

• @dosuken123 technical research for telling deployment to stop on K8s
• @ogolowinski talk to @sarahwaldner regarding settings once a th=reshold is met
• @rverissimo Create Mockups
• @rverissimo & @loriewhitaker UX Research with customers to understand wat they are doing today, introduce new mockups and understand where we need to add notifications/actions

Meeting Recording: https://gitlab.zoom.us/rec/share/5s9MEajipz9Lc7P87BiDWqA-EL7PX6a80yMWqaBbnh7X36mQzbpE2LNCh78w-kRi

We need to figure out . way to tell K8s to stop deployment

@dosuken123 @tkuah is there any way to stop deployment? I think the deployment controller takes one instruction at a time so in theory one could deploy and roll back? Not sure if there's a way to "stop".

is there any way to stop deployment?

This depends on how the "rollout" is setup. For Auto DevOps timed rollout, is there one or many Deployments involved ?

https://kubernetes.io/docs/concepts/workloads/controllers/deployment/ lists all the things you can do with one Deployment (undo aka rollback, pause). I think generally, you just rollback to stop an update of the Deployment (usually if it's erroring / crash loop).

@tkuah if we are using incremental rollouts/canary - could we halt the rollout?

@ogolowinski Canary rollout, and incremental rollout relies on manual action to play the job.

For timed incremental rollout (https://docs.gitlab.com/ee/topics/autodevops/#timed-incremental-rollout-to-production-premium), I guess it's possible to halt by cancelling the pipeline. We will probably then have to undo the deployment in Kubernetes (possibly with kubectl rollout undo...)