### Problem to solve The log files use a variety of different timestamp formats. This inconsistency complicates parsing (and sometimes even reading) the logfiles. For example: - Human date, military time - application.log: May 23, 2018 09:56 - ISO 8601 - Gitaly: 2018-05-23T13:41:10Z - Pages: 2018-05-23T11:14:50Z - ISO 8601.3 - api_json: 2018-05-23T15:47:07.471Z - geo: 2018-05-23T16:04:23.624Z - production_json: 2018-05-23T15:24:23.915Z - ISO 8601.6 - sidekiq: 2018-05-23T14:40:00.808Z - unicorn_stderr: 2015-02-13T06:14:46.680381 - ISO 8601+timezone - gitlab-shell: 2018-05-23T15:45:56+00:00 ### Intended users * [Delaney (Development Team Lead)](https://about.gitlab.com/handbook/marketing/product-marketing/roles-personas/#delaney-development-team-lead) * [Sasha (Software Developer)](https://about.gitlab.com/handbook/marketing/product-marketing/roles-personas/#sasha-software-developer) * [Devon (DevOps Engineer)](https://about.gitlab.com/handbook/marketing/product-marketing/roles-personas/#devon-devops-engineer) * [Sidney (Systems Administrator)](https://about.gitlab.com/handbook/marketing/product-marketing/roles-personas/#sidney-systems-administrator) * [Sam (Security Analyst)](https://about.gitlab.com/handbook/marketing/product-marketing/roles-personas/#sam-security-analyst) ### Further details - This was extracted from [the issue to implement JSON logging for application.log](https://gitlab.com/gitlab-org/gitlab/issues/22812). - This falls within GitLab's ongoing [effort to implement structured logging](https://gitlab.com/groups/gitlab-org/-/epics/469). ### Proposal All log files should use the "ISO 8601.3" format. This is already the most widely used format, by intention. ([@stanhu said](https://gitlab.com/gitlab-org/gitlab/issues/22812#note_215168499): "I've been using `timestamp.utc.iso8601(3)` in all the logs that I've touched (production_json.log, api_json.log, geo.log, Sidekiq, etc.)") The following logs should be adjusted to use "ISO 8601.3" format: - [x] application.log (MR gitlab!22793) - [x] gitlab-shell (MR gitlab-shell!361) - [x] Gitaly (Issue gitaly#2432) - [ ] Pages (Issue https://gitlab.com/gitlab-org/gitlab-pages/-/issues/149) - [x] sidekiq (MR gitlab!21835) - [x] unicorn_stderr.log (https://gitlab.com/gitlab-org/gitlab/issues/204749) - [x] CNG (MR https://gitlab.com/gitlab-org/build/CNG/-/merge_requests/380) - [x] Fluentd (MR https://gitlab.com/gitlab-cookbooks/gitlab_fluentd/-/merge_requests/108) - [x] GitLab (MR https://gitlab.com/gitlab-org/gitlab/-/merge_requests/24612) - [x] Omnibus (MR https://gitlab.com/gitlab-org/omnibus-gitlab/-/merge_requests/3926#note_288254500) ### Testing Changing the timestamp format could disrupt log-monitoring tools that have been set up on self-hosted instances. This is especially true for structured log files, which are more likely to be monitored. ### Related issues - [Implement json logging for application.log](https://gitlab.com/gitlab-org/gitlab/issues/22812) - [Consider dropping unstructured rails logs](https://gitlab.com/gitlab-org/gitlab/issues/23070)