## Summary GitLab is a large application integrating many components. We should: 1. Unify our logging outputs to a single, well defined structured format 1. Document these log formats so that users of GitLab can use their preferred application (Fluentd, Logstash, Splunk, etc) to ingest the logs with a well known and defined ingestion ruleset 1. Prevents log-proliferation across the filesystem and makes it easier to troubleshoot application issues 1. Aligns the logging between GitLab Omnibus, GitLab Charts, GitLab OpenShift so that switching between them is frictionless [Opportunity Canvas](https://docs.google.com/document/d/1kNYmKMgP6yWxt9K_2EXy-cdGnLwnC6SCM2D5ShdgkXE/edit?usp=sharing) ## Points of Contact - Distribution: @rmarshall - Delivery: _**TBD**_ - Infrastructure/SRE: @mwasilewski-gitlab - Product: @dorrino - Support: @dstanley Delivery and Support are awaiting confirmation from the respective managers. ## Where the work will need to happen 1. Application components will need to be aligned to the format per each so that our integrated pieces are consumers, and not trying to maintain a fragile connection between the two 1. Third party components will need to be considered by Distribution to align how they can fit into the overall logging story 1. Distribution and Production will need to align on logging formats so that we are supporting them well while maintaining something generic for our customers to use in their own on-prem deployments 1. Support should be deeply embedded with the effort to ensure we are aligning to ways we can make their lives easier ## Background In the course of working on the [Charts unified logging epic](https://gitlab.com/groups/gitlab-org/charts/-/epics/17) it's come into focus that we do not have a single story around our logging features. Case in point, our rails application sends information to a host of logs across the filesystem. We have done some prelimary work with **JSON** structured logging and made it the default in Omnibus where possible, starting with [this merge request in May, 2019](https://gitlab.com/gitlab-org/omnibus-gitlab/-/merge_requests/3292). GitLab Charts and GitLab Omnibus are currently on a slightly divergent path that, over time, will become problematic. We already see some of this as we are moving .COM to Charts and trying to improve logging. The move from GitLab Omnibus to GitLab charts should not have logging/observability as a sticking point. ## Technical Specifics 1. Logging presents a structured format where each shares traits: 1. Epoch TimeStamp 1. Component Name (Workhorse, Rails, Gitaly, etc) 1. MessageType 1. Payload (depends on component) 1. Identify the most popular/well supported logging packages and provide 1. supported ingestion ruleset for our standard components 1. community contributor section for non-guaranteed ingestion rules that are community supported 1. Standard Logging Facilities 1. Allow redirection to `rsyslog` remote 1. All logs to be directly sent to ingestion services (to be selected) ## Before and After ```mermaid graph LR gc1-->gc2 gc3-->gc4 gc5-->gc6 subgraph GitLab Components subgraph Before gc1[Mixed JSON and Unstructured] gc3[Multiple File Locations] gc5[Externally Standardized] end subgraph After gc2[Structured Format] gc4[Single Location] gc6[Internally Standardized] end end ``` ```mermaid graph LR is1-->is2 is3-->is4 is5-->is6 subgraph Integrated Services subgraph Before is1[JSON and Custom] is3[Multiple File Locations] is5[.COM and Self-Managed Divergent] end subgraph After is2[Structured Format] is4[Single Location] is6[.COM and Self-Managed Aligned] end end ``` ```mermaid graph LR gt1-->gt2 gt3-->gt4 gt5-->gt6 subgraph GitLab Tasks subgraph Before gt1[Unstructured] gt3[STDOUT and Files] gt5[No Standardization] end subgraph After gt2[Structured Format] gt4[Single Location] gt6[Published Standard] end end ```