Use structured logging for GitLab.com Registry
In gitlab-org/omnibus-gitlab!2911 (merged), @alejandro added the option to allow structured logging for docker/distribution aka GitLab Registry logs.
Presently, the logs in Kibana are not yet JSON formatted, and appear as follows:
{"message":"10.216.5.3 - - [13/Mar/2019:11:05:44 +0000] \"GET /v2/ HTTP/1.1\" 401 87 \"\" \"docker/17.03.2-ce go/go1.10.3 git-commit/f5ec1e2 kernel/4.14.65+ os/linux arch/amd64 UpstreamClient(Go-http-client/1.1)\"","tag":"registry","environment":"gprd","hostname":"registry-01-sv-gprd","fqdn":"registry-01-sv-gprd.c.gitlab-production.internal"}
This is difficult to parse and difficult to do analysis on.
Since gitlab-org/omnibus-gitlab!2911 (merged) makes this easy, we should:
- Switch from
texttojsonfor the log formatter. Details in https://github.com/docker/distribution/blob/master/docs/configuration.md - Parse the logs as JSON and not text (in fluentd/tdagent) and send to ELK as such
Once we've done that this will open up options for
- Analysis on Cloud Spend by individual projects and accounts
- Abuse reporting
- Diagnostics on the regular registry outages