## Background Triage is a process of detecting and identifying service or application problems, intending to understand the root cause quickly and accurately. To conduct fast and effective troubleshooting, it is required to collect all relevant information and provide meaningful insights using deep visibility into the segments of the application. We plan to provide a streamline triage experience to allows our users to quickly identify and effectively troubleshoot an application problem ## Proposed user journey - the steps of this user journey are captured as linked epics 1. As a user I would like to get notified when a metric was breached 2. As a user I would like to view the metrics chart in the incident 3. As a user I would like to drill into the log explorer while preserving the same context and look for the root cause analysis 4. As a user I would like to drill from log viewer into traces to view the request that cause the metric breach ## User Journey We can use our existing metrics that we collect today. We can use NGINX error rate metrics as an example. 1. User is notified by an alert (e.g. NGINX error rate over 5%) * Users can be notified via [email](https://docs.gitlab.com/ee/user/project/integrations/prometheus.html#taking-action-on-incidents-ultimate) or [Slack notifications](https://docs.gitlab.com/ee/user/project/integrations/slack.html#slack-notifications-service) about incidents today * User can manually [create alerts from the metrics dashboard](https://docs.gitlab.com/ee/user/project/integrations/prometheus.html#setting-up-alerts-for-prometheus-metrics-ultimate) today * We will explore the option of doing this using Auto-DevOps 2. The alert automatically opens an issue with the metrics chart showing the **current status** of the metrics embedded (possible today) and a link to the log explorer * [Prometheus alerts can automatically create incidents (GitLab issues)](https://docs.gitlab.com/ee/user/project/integrations/prometheus.html#taking-action-on-incidents-ultimate) today * Users can [embed metrics charts in issues](https://docs.gitlab.com/ee/user/project/integrations/prometheus.html#embedding-gitlab-managed-kubernetes-metrics) today * We need to add the ability to link to the log explorer - once we completed the [MVC integration with ElasticSearch](https://gitlab.com/groups/gitlab-org/-/epics/2114) this will be possible 3. User drills into the log explorer preserving the same time stamp * [Drill down to the log explorer focusing on a specific timeframe](https://gitlab.com/gitlab-org/gitlab/issues/198937) * [Filter log view by time](https://gitlab.com/gitlab-org/gitlab/issues/37276) 4. User searches logs to find the root cause of the problem * [Epic capturing all work involved in investigating a list of logs](https://gitlab.com/gitlab-org/gitlab/issues/37425) 5. Navigate to and drill into associated stack trace to see transaction that generated the problem to evaluate user impact * [Drill down from Log explorer to Tracing UI](https://gitlab.com/gitlab-org/gitlab/issues/37441) ## Additional comments * The user can either manually create the alert through the metrics dashboard or auto-define an alert using auto-devops * Logs would need to be viable to allow: * Drill down to a specific time span * Search through logs * On the logging to tracing transition we'll provide a drill down to the current tracing UI in Gitlab.