Filter usage_data payload considering metric category
Compute regular usage ping and filter data before sending it to versions app considering the instance configuration.
Scenarios
CE / EE | Instance Service Ping Enabled | License Service Ping Enabled | Optional | Operational | Standard | Subscription | Comment | |
---|---|---|---|---|---|---|---|---|
1 | CE | No | No | No | No | No | No | Nothing enabled, nothing gets sent |
2 | CE | Yes | No (not possible) | Yes | Yes | Yes | Yes (partial( | Everything gets sent, Subscription data only for metrics that are available for CE |
3 | EE | No | No | No | No | No | No | Nothing enabled, nothing gets sent |
4 | EE | No | Yes | No | Yes | Yes | Yes | Enabled via TAM service, everything but optional |
5 | EE | Yes | No | Yes | Yes | Yes | Yes | Instance Setting overrules everything, we want to send everything |
6 | EE | Yes | Yes | Yes | Yes | Yes | Yes | Everything is enabled, everything gets sent |
Given the following metrics with the information in their YAML definitions
uuid: Standard
hostname: Standard
license_subscription_id: Standard
license_md5: Standard
counts.license_management_jobs: Subscription
counts.merge_requests: Operational
counts.epics: Operational
counts.keys: Optional
counts.groups: Optional
Scenario 1
If there is no single users consent we send no data User.single_user&.requires_usage_stats_consent? == true
def requires_usage_stats_consent?
self.admin? && 7.days.ago > self.created_at && !has_current_license? && User.single_user? && !consented_usage_stats?
end
Scenario 1
For a free user, when License.current
is blank when usage ping enabled we send
Standard + Subscription + Optional + Operational
{
uuid: 1, # Standard
hostname: 'name', # Standard
license_subscription_id: 1, # Standard
license_md5: 'md5', # Standard
counts: {
license_management_jobs: 100, # Subscription
merge_requests: 100, # Operational
epics: 100, # Operational
keys: 100, # Optional
groups: 100 # Optional
}
}
Scenario 2
For a free user, when License.current
is blank when usage ping is disabled we don't send any data
Scenario 3
For a customer, when License.current
is not blank. When usage ping enabled and operational data enabled(License.current.usage_ping? ==. true
) we send
Standard + Subscription + Optional + Operational
{
uuid: 1, # Standard
hostname: 'name', # Standard
license_subscription_id: 1, # Standard
license_md5: 'md5', # Standard
counts: {
license_management_jobs: 100, # Subscription
merge_requests: 100, # Operational
epics: 100, # Operational
keys: 100, # Optional
groups: 100 # Optional
}
}
Scenario 4
For a customer, when License.current
is not blank. When usage ping enabled and operational data disabled(License.current.usage_ping? == false
) we send
Standard + Subscription + Optional + Operational
{
uuid: 1, # Standard
hostname: 'name', # Standard
license_subscription_id: 1, # Standard
license_md5: 'md5', # Standard
counts: {
license_management_jobs: 100, # Subscription
merge_requests: 100, # Operational
epics: 100, # Operational
keys: 100, # Optional
groups: 100 # Optional
}
}
Scenario 5
For a customer, when License.current
is not blank. When usage ping disabled and operational data enabled(License.current.usage_ping? ==. true
) we send
Standard + Subscription + Operational
{
uuid: 1, # Standard
hostname: 'name', # Standard
license_subscription_id: 1, # Standard
license_md5: 'md5', # Standard
counts: {
counts.license_management_jobs: 100# Subscription
merge_requests: 100, # Operational
epics: 100, # Operational
}
}
Scenario 6
For a customer, when License.current
is not blank. When usage ping disabled and operational data disabled(License.current.usage_ping? == false
) we don't send any data.
Notes
-
We should calculate Devops score only when sending all types of data [Standard, Subscription, Operational, Optional]
-
When previewing usage going in admin setting we should filter the data considering instance configuration.
app/controllers/admin/application_settings_controller.rb
-
For instance review page with the redirect on CustomerDot should be prefilled with data after filtering usage ping
app/controllers/admin/instance_review_controller.rb
according to the instance configuration -
The data we send should be reflected to collected_data_types metric we add to usage ping
-
With this changes, there should be tests that we can successfully post all types of payload to VersionsApp and values are successfully saved in
raw_usage_data
table
Implementation proposal
This could be one way to implemented it, there might be other better ways. Maybe having a separate class that takes care of deciding what data is selected, up to the developer picking up the issue to decide on implementation.
https://gitlab.com/gitlab-org/gitlab/-/blob/master/app/services/submit_usage_ping_service.rb#L19
This is pseudocode, please refer to initial implementation
def execute
# This is part of the old logic, please confirm we should check this right at the begining
return if User.single_user&.requires_usage_stats_consent?
usage_data = filter_usage_data(Gitlab::UsageData.data(force_refresh: true))
# save the filtered data.
raw_usage_data = save_raw_usage_data(usage_data)
response = post_to_versions_app(usage_data)
raw_usage_data.update_version_metadata!(usage_data_id: response.version_usage_data_id)
# We would. have devops score to save only when we send to VersionsApp all data.
store_metrics(response)
end
private
# This could be done probably in a better way
def filter_usage_data(usage_data)
if License.current.present?
if Gitlab::CurrentSettings.usage_ping_enabled? && License.current.usage_ping?
# we can calculate devops scopre with this set
select_metrics(data_type: ['Standard', 'Subscription', 'Operational', 'Optional'])
elsif Gitlab::CurrentSettings.usage_ping_enabled?
select_metrics(data_type: ['Standard', `Subscription`, 'Optional'])
elsif License.current.usage_ping?
select_metrics(data_type: ['Standard', 'Operational'])
end
else
if Gitlab::CurrentSettings.usage_ping_enabled?
# we can calculate devops scopre with this set
select_metrics(data_type: ['Standard', 'Subscription', 'Operational', 'Optional'])
end
end
end
def select_metrics(data_type:)
# TODO
end
# conv index deptends on operational and optional data. In order to compute it we should have both available.
def store_metrics(response)
metrics = response['conv_index'] || response['dev_ops_score'] # leaving dev_ops_score here, as the response data comes from the gitlab-version-com
return unless metrics.present?
DevOpsReport::Metric.create!(
metrics.slice(*METRICS)
)
end