improve the process of applying changes to the logging schema (currently defined in runbooks as ES static mappings)
With this change: gitlab-com/runbooks!2284 (merged) we switched to static mappings in ES. Static mappings enforce a pre-defined logging schema. It was part of a wider effort: https://gitlab.com/gitlab-com/gl-infra/infrastructure/-/issues/10094 that aimed at reducing cpu load on the ES masters (updates to index mappings are generating CPU load on the masters, if done in massive quantities they contribute to saturation which was the immediate problem we needed to address). Dynamic mappings were also resulting in logs being rejected (a single log entry could break an entire log stream): production#2090 (closed)
Using a pre-defined logging schema means that changes to the application that involve changes to the schema need to be reflected in the static mappings defined in the runbooks repo, for example, here's the mapping definition for rails: https://gitlab.com/gitlab-com/runbooks/-/blob/master/elastic/managed-objects/lib/index_mappings/rails.jsonnet
Due to the fragmentation of the logging frameworks across our stack, it's currently not feasible to enforce the logging schema next to the application code.
Short-term, we will need a way to facilitate changes to the schema, for example:
- Development team proposes/makes a logging change
- Team opens an issue in Production project (via a template eliciting questions about the change)
- Change is reviewed by infra and security.
Long-term, there was a logging working group in the past, part of its objective was to unify logging frameworks, todos:
-
determine what's the status of the logging working group -
make logging schema enforcement part of the logging framework