Standardize timestamp format in log files
Problem to solve
The log files use a variety of different timestamp formats. This inconsistency complicates parsing (and sometimes even reading) the logfiles. For example:
- Human date, military time
- application.log: May 23, 2018 09:56
- ISO 8601
- Gitaly: 2018-05-23T13:41:10Z
- Pages: 2018-05-23T11:14:50Z
- ISO 8601.3
- api_json: 2018-05-23T15:47:07.471Z
- geo: 2018-05-23T16:04:23.624Z
- production_json: 2018-05-23T15:24:23.915Z
- ISO 8601.6
- sidekiq: 2018-05-23T14:40:00.808Z
- unicorn_stderr: 2015-02-13T06:14:46.680381
- ISO 8601+timezone
- gitlab-shell: 2018-05-23T15:45:56+00:00
Intended users
- Delaney (Development Team Lead)
- Sasha (Software Developer)
- Devon (DevOps Engineer)
- Sidney (Systems Administrator)
- Sam (Security Analyst)
Further details
- This was extracted from the issue to implement JSON logging for application.log.
- This falls within GitLab's ongoing effort to implement structured logging.
Proposal
All log files should use the "ISO 8601.3" format. This is already the most widely used format, by intention. (@stanhu said: "I've been using timestamp.utc.iso8601(3) in all the logs that I've touched (production_json.log, api_json.log, geo.log, Sidekiq, etc.)") The following logs should be adjusted to use "ISO 8601.3" format:
-
application.log (MR !22793 (merged)) -
gitlab-shell -
Gitaly -
Pages -
sidekiq (MR !21835 (merged)) -
unicorn_stderr.log
Testing
Changing the timestamp format could disrupt log-monitoring tools that have been set up on self-hosted instances. This is especially true for structured log files, which are more likely to be monitored.
Related issues
Edited by Dan Jensen