Redis/RedisHLL events don't get triggered

Problem

Redis(HLL) metrics are not getting updated with events from the frontend from 15th February onwards on Gitlab.com.

original from when the issue was created:

1. The `user_visited_dashboard` event doesn't seem to get triggered when entering an analytics dashboard: for example, here, after opening the `network` in Google Dev Tools and clicking on a dashboard, there's no `/track_event` call visible. 2. The same is true for the `user_viewed_dashboard_list` event, which should get triggered when entering the dashboard list view.

Judging by our metrics, it looks like this bug started some time during the last 2 weeks, probably at the same point in time for both of the metrics:

For the current week, we have 0 triggers for user_visited_dashboard and one for user_viewed_dashboard_list - tested on production console:

[ gprd ] production> Gitlab::UsageDataCounters::HLLRedisCounter.un
ique_events(event_names: 'user_visited_dashboard', start_date: Dat
e.current, end_date: Date.current + 1.week, property_name: :user)
=> 0

[ gprd ] production> Gitlab::UsageDataCounters::HLLRedisCounter.un
ique_events(event_names: 'user_viewed_dashboard_list', start_date:
 Date.current, end_date: Date.current + 1.week, property_name: :us
er)
=> 1

(please note that start_date: Date.current actually checks data for the whole week, not just for today)

Detection

@michold detected while implementing another feature. Automatic detection only happened later through https://getmontecarlo.com/incidents/26977dc9-8572-4af2-a53f-e61bc0a82f3d?utm_source=slack

Impact

The following event counters for Gitlab.com did not get incremented from the frontend during the period from 2024-02-15 and 2024-02-27

Redis

• diff_searches
• users_clicking_registration_features_offer
• users_visiting_testing_license_compliance_full_report
• users_visiting_testing_manage_license_compliance
• users_clicking_license_testing_visiting_external_website

RedisHLL

• i_code_review_user_searches_diff
• users_visiting_testing_license_compliance_full_report
• users_visiting_testing_manage_license_compliance
• users_clicking_license_testing_visiting_external_website
• i_testing_metrics_report_widget_total
• i_testing_load_performance_widget_total
• incident_management_issuable_resource_link_visited
• i_testing_full_code_quality_report_total
• users_visiting_pipeline_security
• i_testing_group_code_coverage_project_click_total
• i_testing_group_code_coverage_visit_total
• g_analytics_ci_cd_release_statistics
• g_analytics_ci_cd_deployment_frequency
• g_analytics_ci_cd_lead_time
• g_analytics_ci_cd_time_to_restore_service
• g_analytics_ci_cd_change_failure_rate
• i_code_review_post_merge_click_cherry_pick
• i_code_review_post_merge_click_revert
• i_code_review_post_merge_delete_branch
• users_visiting_security_configuration_threat_management
• p_analytics_ci_cd_time_to_restore_service
• p_analytics_ci_cd_change_failure_rate
• i_code_review_user_create_mr_from_issue
• i_code_review_click_diff_view_setting
• i_code_review_diff_view_inline
• i_code_review_diff_view_parallel
• i_code_review_click_file_browser_setting
• i_code_review_file_browser_tree_view
• i_code_review_file_browser_list_view
• i_code_review_click_whitespace_setting
• i_code_review_diff_show_whitespace
• i_code_review_diff_hide_whitespace
• i_code_review_click_single_file_mode_setting
• i_code_review_diff_single_file
• i_code_review_diff_multiple_files
• design_action
• users_expanding_testing_license_compliance_report

InternalEvents

• user_viewed_cluster_configuration
• i_code_review_saved_replies_create
• merge_request_click_start_review_on_changes_tab
• merge_request_click_add_to_review_on_changes_tab
• p_analytics_ci_cd_pipelines
• p_analytics_ci_cd_deployment_frequency
• p_analytics_ci_cd_lead_time
• i_code_review_saved_replies_use
• i_code_review_saved_replies_use_in_mr
• i_code_review_saved_replies_use_in_other
• user_viewed_dashboard
• user_created_custom_dashboard
• user_edited_custom_dashboard
• user_viewed_dashboard_designer
• user_viewed_custom_dashboard
• user_viewed_builtin_dashboard
• user_viewed_visualization_designer
• user_created_custom_visualization
• exclude_anonymised_users
• user_viewed_dashboard_list
• value_streams_dashboard_metric_link_clicked
• value_streams_dashboard_change_failure_rate_link_clicked
• value_streams_dashboard_contributor_count_link_clicked
• value_streams_dashboard_cycle_time_link_clicked
• value_streams_dashboard_deployment_frequency_link_clicked
• value_streams_dashboard_deploys_link_clicked
• value_streams_dashboard_issues_completed_link_clicked
• value_streams_dashboard_issues_link_clicked
• value_streams_dashboard_lead_time_for_changes_link_clicked
• value_streams_dashboard_lead_time_link_clicked
• value_streams_dashboard_merge_request_throughput_link_clicked
• value_streams_dashboard_time_to_restore_service_link_clicked
• value_streams_dashboard_vulnerability_critical_link_clicked
• value_streams_dashboard_vulnerability_high_link_clicked
• i_analytics_dev_ops_adoption
• i_analytics_dev_ops_score
• insights_chart_item_clicked
• insights_issue_chart_item_clicked
• insights_merge_request_chart_item_clicked
• user_viewed_instrumentation_directions
• g_project_management_users_epic_issue_added_from_epic
• click_create_confidential_mr_issues_list
• click_create_mr_issues_list
• click_new_merge_request_list
• click_new_merge_request_empty_list
• user_edited_cluster_configuration

7d metrics dependent on those won't be correct until 2024-03-05, and 28d metrics won't be correct until 2024-03-26. This is only the case if the metric is dependent on the Redis value and does not have a Snowplow equivalent.

Additional information

Checklist

• Assigned severity tags based on this guidance
• Assigned to PM and EM of groupanalytics instrumentation
• Posted link to incident in g_analyze_analytics_instrumentation and tagged both PM and EM of the group

<---- TO BE FILLED BY ASSIGNEE / RESOLUTION DRI---->

Root Cause

The issue was caused by the removal of the feature flag usage_data_api. It was removed from the backend in this MR, but it was not removed from the frontend. As a result, this was blocking Redis/RedisHLL and InternalEvents API requests from being sent. As part of the fix,

Resolution

@ankit.panchal has removed feature flag from the frontend as well in !145690 (merged)