Skip to content

Move PackageEventCounter from UsageData to metrics definitions

Michał Wielich requested to merge refactor-redis-counters-v1 into master

What does this MR do and why?

Related to #339445 (closed)

We want to move metrics-related code from our chunky UsageData class into a system of yml files including metrics definitions.

This is the bash script that has been used to update the yml files:

affected_events=("i_package_composer_delete_package" "i_package_composer_pull_package" "i_package_composer_push_package" "i_package_conan_delete_package" "i_package_conan_pull_package" "i_package_conan_push_package" "i_package_debian_delete_package" "i_package_debian_pull_package" "i_package_delete_package" "i_package_delete_package_by_deploy_token" "i_package_delete_package_by_guest" "i_package_delete_package_by_user" "i_package_generic_delete_package" "i_package_generic_pull_package" "i_package_generic_push_package" "i_package_golang_delete_package" "i_package_golang_pull_package" "i_package_golang_push_package" "i_package_helm_pull_package" "i_package_helm_push_package" "i_package_maven_delete_package" "i_package_maven_pull_package" "i_package_maven_push_package" "i_package_npm_delete_package" "i_package_npm_pull_package" "i_package_npm_push_package" "i_package_nuget_delete_package" "i_package_nuget_pull_package" "i_package_nuget_push_package" "i_package_nuget_pull_symbol_package" "i_package_nuget_push_symbol_package" "i_package_pull_package" "i_package_pull_package_by_deploy_token" "i_package_pull_package_by_guest" "i_package_pull_package_by_user" "i_package_pull_symbol_package" "i_package_pull_symbol_package_by_deploy_token" "i_package_pull_symbol_package_by_guest" "i_package_pull_symbol_package_by_user" "i_package_push_package" "i_package_push_package_by_deploy_token" "i_package_push_package_by_guest" "i_package_push_package_by_user" "i_package_push_symbol_package" "i_package_push_symbol_package_by_deploy_token" "i_package_push_symbol_package_by_guest" "i_package_push_symbol_package_by_user" "i_package_pypi_delete_package" "i_package_pypi_pull_package" "i_package_pypi_push_package" "i_package_rubygems_delete_package" "i_package_rubygems_pull_package" "i_package_rubygems_push_package" "i_package_terraform_module_delete_package" "i_package_terraform_module_pull_package" "i_package_terraform_module_push_package")
for event_name in "${affected_events[@]}"
do
  find "/gdk_directory/gitlab-development-kit/gitlab/config/metrics/counts_all" -type f -name "*${event_name}\.yml" | while read FILE ; do
    echo $event_name
    sed -i '' "s/data_source: redis/data_source: redis\ninstrumentation_class: RedisMetric\noptions:\n  prefix: package_events\n  event: $event_name/" "$FILE"
  done
done

Screenshots or screen recordings

Screenshots are required for UI changes, and strongly recommended for all other merge requests.

How to set up and validate locally

This test should make sure that we didn't miss any metrics [eg, it is failing after removing the instrumentation_class: clause from any of the yml files].

To test that the actual values for events are also counted correctly:

  1. Checkout master branch
  2. Trigger some package-related events: for example publish and download a package using the curl commands
  3. Run this in the console:
counter_class = Gitlab::UsageDataCounters::PackageEventCounter
raw_events = counter_class::KNOWN_EVENTS; nil
events = raw_events.map { |ev| "#{counter_class::PREFIX}_#{ev}"}; nil
service_ping = Gitlab::Usage::ServicePingReport.for(output: :all_metrics_values)

values1 = service_ping[:counts].values_at(*events)
  • This will save the event statistics under the values1 variable. The values1 array should include some non-zero values if the package events have been successfully triggered. Now what we will need to do is compare it with the result we get with this MR:
  1. Check out this MR's branch
  2. Run this in the console:
reload!
service_ping = Gitlab::Usage::ServicePingReport.for(output: :all_metrics_values)
values2 = service_ping[:counts].values_at(*events)
  1. Now, the values1 and values2 should have the same value. This will prove that the events have been added to Service Ping data correctly.
values2 == values1
=> true

MR acceptance checklist

This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.

Edited by Michał Wielich

Merge request reports