[Feature flag] gitaly_always_log_feature_flags

Rollout feature flag for calculating repo size with cat-file

What

Enable the :gitaly_always_log_feature_flag feature flag for Gitaly. This flag always add feature_flags field in gRPC logs. By default, to reduce the logging volume, we only log FF in failure cases only.

Owners

• Team: Gitaly
• Most appropriate slack channel to reach out to: #g_create_gitaly
• Best individual to reach out to: @qmnguyen0711

Expectations

What release does this feature occur in first?

15.6

What are we expecting to happen?

After this flag is enabled, we expect to see Gitaly gRPC logs start to attach feature_flags field. This field's value is the concatenation of enabled feature flags.

What might happen if this goes wrong?

The logging should not affect gRPC processing. In addition, after enabled, Elasticsearch should be in a healthy state. We used to bring down Elasticsearch cluster before due to huge log volume.

What can we monitor to detect problems with this?

• Logs: https://log.gprd.gitlab.net/goto/735094d0-649c-11ed-85ed-e7557b0a598c
• Elasticsearch health: https://dashboards.gitlab.net/d/search-main/search-overview?orgId=1

Roll Out Steps

• Enable on staging
  • Is the required code deployed on staging? (howto)
  • Enable on staging (howto)
  • Add featureflagstaging to this issue (howto)
  • Test on staging (howto)
  • Verify the feature flag was used by checking Prometheus metric gitaly_feature_flag_checks_total
• Enable on production
  • Is the required code deployed on production? (howto)
  • Enable on production in #production (howto)
  • Add featureflagproduction to this issue
  • Verify the feature flag was used by checking Prometheus metric gitaly_feature_flag_checks_total
• Default-enable the feature flag (optional, only required if backwards-compatibility concerns exist)
  • Wait for release containg default-disabled feature flag.
  • Change the feature flag to default-enabled (howto)
  • Wait for release containing default-enabled feature flag.
• Remove feature flag
  • Remove the feature flag and the pre-feature-flag code (howto)
  • Remove the feature flag via chatops (howto)
  • Close this issue

Please refer to the documentation of feature flags for further information.