Prevent application from emitting malformed log events
Problem
CustomersDot's logger has no safe-guards currently to catch log events that will end up being dropped by Kibana because they violate its index schema: gitlab-com/gl-infra/production-engineering#24966 (moved)
An example is emitting an error
field that's a string instead of hash/object.
Currently, the application will run and emit logs fine, but as soon as we try to move them into Kibana from GCP, ingestion fails and developers will never know (symptom is missing events in Kibana).
Proposal
On the application side we should make log context resilient against developers using common log context incorrectly or overwriting standard context. We can look at gitlab-labkit and gitlab-rails for reference; for example, labkit rejects well-known keys that it knows would overwrite important log data: https://gitlab.com/gitlab-org/ruby/gems/labkit-ruby/-/blob/29bba8df8b1190b01a2de213cf11ae4f3b47b0a3/lib/labkit/logging/json_logger.rb#L54
Result
Next steps (if any)
How will we measure success?
Log events are never silently dropped. Ideally we catch malformed events before deploying the application.