## Overview Once a monitoring system provides an indication that requires attention (alert, incident, customer complaint), the user would need to quickly diagnose the root cause by making the system observable via the monitoring data they'd collected. ## Our goal Users are able to hone-in on the root cause of a problem quickly across multiple systems and tools without losing context. ## How * A user should start the **highest-level system alert** that is showing symptoms, reviewing its work and resource metrics and any associated events. * For each monitored system the user gets a **pre-configured dashboard** which displays all its key metrics, with relevant events overlaid * The user can create an [**ad-hoc data visualization**](https://docs.datadoghq.com/graphing/metrics/explorer/) to quickly build time-series based visualizations based * The user would have the ability to quickly dive into the relevant **logs and traces** for live tailing or ad-hoc search ## User Journeys The following are high-level, visionary journeys we want to enable in GitLab. Each step in these journeys could be a detailed workflow. Additionally, these journeys are ever evolving as we learn more about our users and identify new opportunities. ## Workflow * This workflow represents a completed integrated Observability solution, our strategy isn't necessarily to evolve each solution to complete, but to leverage Gitlab as a single UI to consume all data * A user might stop at any point during this workflow and the drill-downs between solutions (explorers, high-level dashboards) should easily be by-passed if direct known information is desired ``` mermaid graph LR; subgraph Alert on Top Level Metrics A(Alert Generated)-->B[Business Metric out of SLO]; A-->C(Tracing error rates out of SLO); end subgraph Metrics B-->D(High level dashboard view of critical metrics); C-->D; D --> E[View Per Tech including App]; D --> F[Metrics view across infra]; E --> G[Ad Hoc Metrics Explorer]; F --> G; end subgraph Logs and Traces G --> H[Log Explorer] G--> I[Traces Explorer] end ``` [Triaging Workflow (Mural)](https://app.mural.co/invitation/mural/gitlab2474/1570086270242?sender=dhershkovitch7730&key=c69d4e76-7d7f-486e-8657-56fcfd29a569) ## Problem Validation [Incident Automation Opportunity Canvas](https://docs.google.com/document/d/1cecU42zAZpqzuFmRljPTFMbspDvBRhwO5axBw36IxV0/edit#heading=h.4mt5fmtn0ax4) [Alerts Opportunity Canvas](https://docs.google.com/document/d/1-sJoVFr3ND_-N3EtXUNYF6oVROfhE4rcjVqpnQd5N6U/edit)