Add product usage event log

What does this MR do and why?

Adds logging of events collected for Product Usage Data. Logs to the file log/product_usage_data.log

The main motivation for logging events is to give customers full visibility into the data being sent.

Log lines look like this:

{"severity":"INFO","time":"2025-04-09T13:43:40.254Z","message":"sending event","payload":"{\"e\":\"se\",\"se_ca\":\"projects:merge_requests:diffs\",\"se_ac\":\"i_code_review_user_searches_diff\",\"cx\":\"eyJzY2hlbWEiOiJpZ2x1OmNvbS5zbm93cGxvd2FuYWx5dGljcy5zbm93cGxvdy9jb250ZXh0cy9qc29uc2NoZW1hLzEtMC0xIiwiZGF0YSI6W3sic2NoZW1hIjoiaWdsdTpjb20uZ2l0bGFiL2dpdGxhYl9zdGFuZGFyZC9qc29uc2NoZW1hLzEtMS0xIiwiZGF0YSI6eyJlbnZpcm9ubWVudCI6ImRldmVsb3BtZW50Iiwic291cmNlIjoiZ2l0bGFiLXJhaWxzIiwiY29ycmVsYXRpb25faWQiOiJlNDk2NzNjNWI2MGQ5ODc0M2U4YWI0MjZiMTZmMTkxMiIsInBsYW4iOiJkZWZhdWx0IiwiZXh0cmEiOnt9LCJ1c2VyX2lkIjpudWxsLCJnbG9iYWxfdXNlcl9pZCI6bnVsbCwiaXNfZ2l0bGFiX3RlYW1fbWVtYmVyIjpudWxsLCJuYW1lc3BhY2VfaWQiOjMxLCJwcm9qZWN0X2lkIjo2LCJmZWF0dXJlX2VuYWJsZWRfYnlfbmFtZXNwYWNlX2lkcyI6bnVsbCwicmVhbG0iOiJzZWxmLW1hbmFnZWQiLCJpbnN0YW5jZV9pZCI6IjJkMDg1NzBkLWNmZGItNDFmMy1iODllLWM3MTM5YmFjZTI3NSIsImhvc3RfbmFtZSI6ImpsYXJzZW4tLTIwMjIxMjE0LVBWWTY5IiwiaW5zdGFuY2VfdmVyc2lvbiI6IjE3LjExLjAiLCJjb250ZXh0X2dlbmVyYXRlZF9hdCI6IjIwMjUtMDQtMDkgMTM6NDM6NDAgVVRDIn19LHsic2NoZW1hIjoiaWdsdTpjb20uZ2l0bGFiL2dpdGxhYl9zZXJ2aWNlX3BpbmcvanNvbnNjaGVtYS8xLTAtMSIsImRhdGEiOnsiZGF0YV9zb3VyY2UiOiJyZWRpc19obGwiLCJldmVudF9uYW1lIjoiaV9jb2RlX3Jldmlld191c2VyX3NlYXJjaGVzX2RpZmYifX1dfQ==\",\"p\":\"srv\",\"dtm\":\"1744206220253\",\"tna\":\"gl\",\"tv\":\"rb-0.8.0\",\"eid\":\"4f067989-d10d-40b0-9312-ad9d7355be7f\"}"}

When sending events, lines like this will be logged (newlines added for reability). payload is the exact event that is being sent to the Snowplow collector.

{
  "severity": "INFO",
  "time": "2025-04-09T13:43:40.254Z",
  "message": "sending event",
  "payload": "{\"e\":\"se\",\"se_ca\":\"projects:merge_requests:diffs\",\"se_ac\":\"i_code_review_user_searches_diff\",\"cx\":\"eyJzY2hlbWEiOiJpZ2x1OmNvbS5zbm93cGxvd2FuYWx5dGljcy5zbm93cGxvdy9jb250ZXh0cy9qc29uc2NoZW1hLzEtMC0xIiwiZGF0YSI6W3sic2NoZW1hIjoiaWdsdTpjb20uZ2l0bGFiL2dpdGxhYl9zdGFuZGFyZC9qc29uc2NoZW1hLzEtMS0xIiwiZGF0YSI6eyJlbnZpcm9ubWVudCI6ImRldmVsb3BtZW50Iiwic291cmNlIjoiZ2l0bGFiLXJhaWxzIiwiY29ycmVsYXRpb25faWQiOiJlNDk2NzNjNWI2MGQ5ODc0M2U4YWI0MjZiMTZmMTkxMiIsInBsYW4iOiJkZWZhdWx0IiwiZXh0cmEiOnt9LCJ1c2VyX2lkIjpudWxsLCJnbG9iYWxfdXNlcl9pZCI6bnVsbCwiaXNfZ2l0bGFiX3RlYW1fbWVtYmVyIjpudWxsLCJuYW1lc3BhY2VfaWQiOjMxLCJwcm9qZWN0X2lkIjo2LCJmZWF0dXJlX2VuYWJsZWRfYnlfbmFtZXNwYWNlX2lkcyI6bnVsbCwicmVhbG0iOiJzZWxmLW1hbmFnZWQiLCJpbnN0YW5jZV9pZCI6IjJkMDg1NzBkLWNmZGItNDFmMy1iODllLWM3MTM5YmFjZTI3NSIsImhvc3RfbmFtZSI6ImpsYXJzZW4tLTIwMjIxMjE0LVBWWTY5IiwiaW5zdGFuY2VfdmVyc2lvbiI6IjE3LjExLjAiLCJjb250ZXh0X2dlbmVyYXRlZF9hdCI6IjIwMjUtMDQtMDkgMTM6NDM6NDAgVVRDIn19LHsic2NoZW1hIjoiaWdsdTpjb20uZ2l0bGFiL2dpdGxhYl9zZXJ2aWNlX3BpbmcvanNvbnNjaGVtYS8xLTAtMSIsImRhdGEiOnsiZGF0YV9zb3VyY2UiOiJyZWRpc19obGwiLCJldmVudF9uYW1lIjoiaV9jb2RlX3Jldmlld191c2VyX3NlYXJjaGVzX2RpZmYifX1dfQ==\",\"p\":\"srv\",\"dtm\":\"1744206220253\",\"tna\":\"gl\",\"tv\":\"rb-0.8.0\",\"eid\":\"4f067989-d10d-40b0-9312-ad9d7355be7f\"}"
}

Configuration

Logging can be disabled by setting the environment variable GITLAB_DISABLE_PRODUCT_USAGE_EVENT_LOGGING.

Product usage data event viewer script

The log file is not very readable. Especially the context (cx) property that is base64 encoded is inaccessible.

This MR introduces a command line tool to give user a better chance for inspecting the logged data. The encoded contexts are decoded displayed in a pretty printed fashion.

The script can we used in two ways. Either like this:

scripts/product_usage_data_event_formatter.rb log/product_usage_data.log

Or like this:

tail -f log/product_usage_data.log | scripts/product_usage_data_event_formatter.rb

The script formats the above event like this:

{
  "severity": "INFO",
  "time": "2025-04-09T13:43:40.254Z",
  "message": "sending event",
  "payload": {
    "e": "se",
    "se_ca": "projects:merge_requests:diffs",
    "se_ac": "i_code_review_user_searches_diff",
    "cx": {
      "schema": "iglu:com.snowplowanalytics.snowplow/contexts/jsonschema/1-0-1",
      "data": [
        {
          "schema": "iglu:com.gitlab/gitlab_standard/jsonschema/1-1-1",
          "data": {
            "environment": "development",
            "source": "gitlab-rails",
            "correlation_id": "e49673c5b60d98743e8ab426b16f1912",
            "plan": "default",
            "extra": {},
            "user_id": null,
            "global_user_id": null,
            "is_gitlab_team_member": null,
            "namespace_id": 31,
            "project_id": 6,
            "feature_enabled_by_namespace_ids": null,
            "realm": "self-managed",
            "instance_id": "2d08570d-cfdb-41f3-b89e-c7139bace275",
            "host_name": "jlarsen--20221214-PVY69",
            "instance_version": "17.11.0",
            "context_generated_at": "2025-04-09 13:43:40 UTC"
          }
        },
        {
          "schema": "iglu:com.gitlab/gitlab_service_ping/jsonschema/1-0-1",
          "data": {
            "data_source": "redis_hll",
            "event_name": "i_code_review_user_searches_diff"
          }
        }
      ]
    },
    "p": "srv",
    "dtm": "1744206220253",
    "tna": "gl",
    "tv": "rb-0.8.0",
    "eid": "4f067989-d10d-40b0-9312-ad9d7355be7f"
  }
}

References

Screenshots or screen recordings

Before	After

How to set up and validate locally

MR acceptance checklist

Evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.

Related to #521081