Skip to content

RedisHLLMetric property_name handling

Michał Wielich requested to merge michold-redis-metric-property into master

What does this MR do and why?

Related to #415139 (closed)

We are already saving RedisHLL counters separately for each property name. Now, let's make it so that we also read those new counters in RedisHLLMetric.

RedisHLLMetrics that have been created for internal events have an events.unique option defined. This option correlates to what is called a property_name inside the RedisHLLCounter class. In this MR, we will be passing this option into the RedisHLLCounter class, so that the counter class can use it for reading the data.

MR acceptance checklist

Please evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.

Screenshots or screen recordings

Screenshots are required for UI changes, and strongly recommended for all other merge requests.

Before After

How to set up and validate locally

  1. Enable the feature flag: Feature.enable(:redis_hll_value_name_tracking)

A. Test RedisHLLMetric:

  1. Create a new metric or modify a new one [for example: this one], making it so that it uses a new unique value for its events.unique property [in the given example: we can just change user.id to project.id]
  2. Trigger the metric a few times, sending the events.unique property's pre-dot part as property_name and time set to last week [for example: Gitlab::UsageDataCounters::HLLRedisCounter.track_event('g_project_management_issue_weight_changed', values: SecureRandom.uuid, property_name: 'project', time: 1.week.ago)]
  3. Check the current value for this event, for example: Gitlab::UsageDataCounters::HLLRedisCounter.unique_events(event_names: 'g_project_management_issue_weight_changed', end_date: Date.today, start_date: 3.week.ago, property_name: 'project.id')
  4. Generate a new Service Ping: sp = Gitlab::Usage::ServicePingReport.for(output: :all_metrics_values)
  5. Check the metric's key path [example: sp['redis_hll_counters']['issues_edit']['g_project_management_issue_weight_changed_monthly']]. It should have the same value as the one we saw using the HLLRedisCounter class.

B. Make sure we haven't affected existing metrics

Something that we want to be fully sure about is that we're not affecting any metrics with this code. To check that, we can:

  1. Enable the gdk and perform some actions: for example, create an issue, add comments, create an epic, add an emote reaction
  2. Checkout the master branch and generate the next week's Service Ping in the rails console:
require 'active_support/testing/time_helpers'
include ActiveSupport::Testing::TimeHelpers
sp = travel_to(Date.today + 7.days) { Gitlab::Usage::ServicePingReport.for(output: :all_metrics_values) }
  1. Checkout this MR's branch and go back to the rails console:
reload!
sp2 = travel_to(Date.today + 7.days) { Gitlab::Usage::ServicePingReport.for(output: :all_metrics_values) }
  1. Select all the keys that are different between these two service pings:
def find_different_keys(h1, h2, result, current_key = '')
  raise StandardError if h1.keys != h2.keys

  diffs = h1.keys.each do |k|
    if h1[k].is_a?(Hash)
      result = find_different_keys(h1[k], h2[k], result, current_key + "#{k}.")
    else
      (h1[k] != h2[k]) ? result += [current_key + "#{k}"] : nil
    end
  end

  result
end

find_different_keys(sp, sp2, [])

This should only list metrics that are not calculated using RedisHLL. For example, in my case, it was:

["elasticsearch_enabled",
 "geo_enabled",
 "gitlab_pages.enabled",
 "database.flavor",
 "object_store.artifacts.enabled",
 "object_store.artifacts.object_store.enabled",
 "object_store.artifacts.object_store.direct_upload",
 "object_store.artifacts.object_store.background_upload",
 "object_store.external_diffs.enabled",
 "object_store.external_diffs.object_store.enabled",
 "object_store.external_diffs.object_store.direct_upload",
 "object_store.external_diffs.object_store.background_upload",
 "object_store.lfs.enabled",
 "object_store.lfs.object_store.enabled",
 "object_store.lfs.object_store.direct_upload",
 "object_store.lfs.object_store.background_upload",
 "object_store.uploads.enabled",
 "object_store.uploads.object_store.enabled",
 "object_store.uploads.object_store.direct_upload",
 "object_store.uploads.object_store.background_upload",
 "object_store.packages.enabled",
 "object_store.packages.object_store.enabled",
 "object_store.packages.object_store.direct_upload",
 "object_store.packages.object_store.background_upload",
 "topology.duration_s"]

If you got any other keys, they can be checked by searching for the key in https://metrics.gitlab.com/ [for exmaple: here]. After loading a row, make sure that the source column for it is not redis_hll or internal_events. For the given example, the source is system.

Edited by Michał Wielich

Merge request reports