Redis/RedisHLL events don't get triggered
Problem
Redis(HLL) metrics are not getting updated with events from the frontend from 15th February onwards on Gitlab.com.
original from when the issue was created:
1. The `user_visited_dashboard` event doesn't seem to get triggered when entering an analytics dashboard: for example, here, after opening the `network` in Google Dev Tools and clicking on a dashboard, there's no `/track_event` call visible. 2. The same is true for the `user_viewed_dashboard_list` event, which should get triggered when entering the dashboard list view.Judging by our metrics, it looks like this bug started some time during the last 2 weeks, probably at the same point in time for both of the metrics:
For the current week, we have 0 triggers for user_visited_dashboard
and one for user_viewed_dashboard_list
- tested on production console:
[ gprd ] production> Gitlab::UsageDataCounters::HLLRedisCounter.un
ique_events(event_names: 'user_visited_dashboard', start_date: Dat
e.current, end_date: Date.current + 1.week, property_name: :user)
=> 0
[ gprd ] production> Gitlab::UsageDataCounters::HLLRedisCounter.un
ique_events(event_names: 'user_viewed_dashboard_list', start_date:
Date.current, end_date: Date.current + 1.week, property_name: :us
er)
=> 1
(please note that start_date: Date.current
actually checks data for the whole week, not just for today)
Detection
@michold detected while implementing another feature. Automatic detection only happened later through https://getmontecarlo.com/incidents/26977dc9-8572-4af2-a53f-e61bc0a82f3d?utm_source=slack
Impact
The following event counters for Gitlab.com did not get incremented from the frontend during the period from 2024-02-15
and 2024-02-27
Redis
diff_searches
users_clicking_registration_features_offer
users_visiting_testing_license_compliance_full_report
users_visiting_testing_manage_license_compliance
users_clicking_license_testing_visiting_external_website
RedisHLL
i_code_review_user_searches_diff
users_visiting_testing_license_compliance_full_report
users_visiting_testing_manage_license_compliance
users_clicking_license_testing_visiting_external_website
i_testing_metrics_report_widget_total
i_testing_load_performance_widget_total
incident_management_issuable_resource_link_visited
i_testing_full_code_quality_report_total
users_visiting_pipeline_security
i_testing_group_code_coverage_project_click_total
i_testing_group_code_coverage_visit_total
g_analytics_ci_cd_release_statistics
g_analytics_ci_cd_deployment_frequency
g_analytics_ci_cd_lead_time
g_analytics_ci_cd_time_to_restore_service
g_analytics_ci_cd_change_failure_rate
i_code_review_post_merge_click_cherry_pick
i_code_review_post_merge_click_revert
i_code_review_post_merge_delete_branch
users_visiting_security_configuration_threat_management
p_analytics_ci_cd_time_to_restore_service
p_analytics_ci_cd_change_failure_rate
i_code_review_user_create_mr_from_issue
i_code_review_click_diff_view_setting
i_code_review_diff_view_inline
i_code_review_diff_view_parallel
i_code_review_click_file_browser_setting
i_code_review_file_browser_tree_view
i_code_review_file_browser_list_view
i_code_review_click_whitespace_setting
i_code_review_diff_show_whitespace
i_code_review_diff_hide_whitespace
i_code_review_click_single_file_mode_setting
i_code_review_diff_single_file
i_code_review_diff_multiple_files
design_action
users_expanding_testing_license_compliance_report
InternalEvents
user_viewed_cluster_configuration
i_code_review_saved_replies_create
merge_request_click_start_review_on_changes_tab
merge_request_click_add_to_review_on_changes_tab
p_analytics_ci_cd_pipelines
p_analytics_ci_cd_deployment_frequency
p_analytics_ci_cd_lead_time
i_code_review_saved_replies_use
i_code_review_saved_replies_use_in_mr
i_code_review_saved_replies_use_in_other
user_viewed_dashboard
user_created_custom_dashboard
user_edited_custom_dashboard
user_viewed_dashboard_designer
user_viewed_custom_dashboard
user_viewed_builtin_dashboard
user_viewed_visualization_designer
user_created_custom_visualization
exclude_anonymised_users
user_viewed_dashboard_list
value_streams_dashboard_metric_link_clicked
value_streams_dashboard_change_failure_rate_link_clicked
value_streams_dashboard_contributor_count_link_clicked
value_streams_dashboard_cycle_time_link_clicked
value_streams_dashboard_deployment_frequency_link_clicked
value_streams_dashboard_deploys_link_clicked
value_streams_dashboard_issues_completed_link_clicked
value_streams_dashboard_issues_link_clicked
value_streams_dashboard_lead_time_for_changes_link_clicked
value_streams_dashboard_lead_time_link_clicked
value_streams_dashboard_merge_request_throughput_link_clicked
value_streams_dashboard_time_to_restore_service_link_clicked
value_streams_dashboard_vulnerability_critical_link_clicked
value_streams_dashboard_vulnerability_high_link_clicked
i_analytics_dev_ops_adoption
i_analytics_dev_ops_score
insights_chart_item_clicked
insights_issue_chart_item_clicked
insights_merge_request_chart_item_clicked
user_viewed_instrumentation_directions
g_project_management_users_epic_issue_added_from_epic
click_create_confidential_mr_issues_list
click_create_mr_issues_list
click_new_merge_request_list
click_new_merge_request_empty_list
user_edited_cluster_configuration
7d metrics dependent on those won't be correct until 2024-03-05
, and 28d metrics won't be correct until 2024-03-26
. This is only the case if the metric is dependent on the Redis value and does not have a Snowplow equivalent.
Additional information
Checklist
-
Assigned severity tags based on this guidance -
Assigned to PM and EM of groupanalytics instrumentation -
Posted link to incident in g_analyze_analytics_instrumentation
and tagged both PM and EM of the group
<---- TO BE FILLED BY ASSIGNEE / RESOLUTION DRI---->
Root Cause
The issue was caused by the removal of the feature flag usage_data_api. It was removed from the backend in this MR, but it was not removed from the frontend. As a result, this was blocking Redis/RedisHLL and InternalEvents API requests from being sent. As part of the fix,
Resolution
@ankit.panchal has removed feature flag from the frontend as well in !145690 (merged)