Do not permit use of reserved log labels

In gitlab-org/gitlab!111285 (merged) we found that there is some surprising behavior in our logging stack: if the application logs events using "reserved" labels such as type (which IIUC gets injected by fluentd after the event leaves the rails application), then we quietly accept these and they will be overwritten later on.

There are two problems here:

  • Clients using generic label names such as type with context-specific semantics (in this case CI artifact type)
  • External services overwriting certain "reserved" labels

I think we should do one or both of:

  • Not allowing user applied labels to appear at the top-level label "namespace", e.g. by always moving them under an extra.* prefix. This would have resolved the clash since there would have been type and extra.type
  • Fail fast i.e. reject labels we know have existing meaning such as type. It would also help enforce better naming policies since in this example, I think ci_artifact_type would have been a much better label name to begin with.

PS: I am not sure this is solvable in labkit-ruby alone since we apply gitlab-rails specific logic and use many different loggers too.