Manage metadata propagation for go services in labkit-go

While working on gitlab-org/gitlab#232645 (closed) I ran into a recurring theme:

We often have several metadata values that we want to propagate between services. In my case I was introducing remote_ip and username. This required propagating those values separately in many places. And the already existing propagation doesn't always use the same naming (username vs meta.user, context vs meta, etc).

On the ruby side we already have a mechanism to propagate such values, Labkit::Context.

We should port this concept to labkit-go as well. This might also be an opportunity to change a few things:

  • Naming: context is a very overloaded term in go. We may want to go with something like "baggage" or "meta".
  • Single opaque field: We currently pass one header per field. With some exceptions (X-Forwarded-For is the standard for propagating remote_ip over HTTP) we may want to combine those into a single value. That will also allow propagation without decoding/re-encoding, allowing new fields to be introduced without having to upgrade all middleware services.
  • Versioning: With the above point, we can also include a version field just in case we ever need to migrate the data format in the future.
  • Logging configuration: As we introduce more fields, we'll want to be selective about which ones are logged in order to manage log volume. A LABKIT_LOG_META_FIELDS (or such) environment variable, that includes a comma-separated list of fields can help manage this via configuration.
  • Central logic for encoding this data over HTTP and GRPC for use with rails, workhorse, and gitaly.

There are several other ongoing efforts that will benefit from having this in place (propagating metadata back from rails to workhorse, propagating metadata to gitaly).

Edited by Igor