Draft: [skip ci] PoC: group prevented-package stats with ClickHouse reads

Arpit Gogiarequested to merge
631890-group-prevented-stats-poc into master

This PoC is superseded by a 7-part stack starting at !261477 (merged), with the stack table in each part's description. The stack changes one design point: groups are refreshed by a deduplicated Sidekiq job per group (concurrency limit 5) instead of the Redis set and 5-minute cron. This MR stays open as a reference and will not be merged.

What does this MR do and why?

This is a proof of concept for https://gitlab.com/gitlab-org/gitlab/-/work_items/631890: prevented-package stats for group dependency firewall dashboards, plus ClickHouse reads for projects and groups. It is not for merge. Once it has been tested, it will be split into smaller MRs, and CI is skipped on purpose.

The MR implements approach B from the issue's design proposal. PostgreSQL group reads come from precomputed summaries rather than query-time distinct counts, because computing distinct counts across all of a group's projects at query time exceeds the query guidelines at gitlab-org scale.

The branch diff touches 46 files in 16 commits. About 60% of the added lines are specs.

How it works

Piece What it does
Summary table dependency_firewall_namespace_prevented_stats 2 rows per group (7 and 30 day windows), sharding key namespace_id
GroupRefreshService Recomputes a group in batches of 500 projects from dependency_firewall_prevented_packages, reading from replicas, merging package sets so counts equal today's query-time group counts (parity spec)
RefreshQueue (Redis set) Groups waiting for a refresh; the existing flush worker marks the ancestor groups of flushed projects, and a dashboard read miss marks the group
Workers Refresh marked groups every 5 minutes; re-mark all groups with rows daily; an EventStore subscriber marks groups on project and group transfers and deletions
DataCollector Reads ClickHouse (Siphon replica) when analytics ClickHouse is enabled, dependency_firewall_clickhouse_reads is on and Siphon replicates the table; otherwise reads PostgreSQL, with projects computed query-time and groups from the summary table (7 and 30 day windows, UTC dates, rows at most one day old)
Group page Shows the new dashboard when dependency_firewall_dashboard_v2 and dependency_firewall_group_prevented_stats are both on; policies defined in a project or subgroup below the group show "Defined in" with a link
Flags dependency_firewall_group_prevented_stats (beta; enables group numbers on any backend) and dependency_firewall_clickhouse_reads (beta; rollout #631122)

How to set up and validate locally

  1. Enable dependency_firewall_phase1, dependency_firewall_dashboard_v2 and dependency_firewall_group_prevented_stats for the top-level group.
  2. PostgreSQL path: run Security::DependencyFirewall::PreventedStats::GroupRefreshService.new(group).execute in bin/rails console (or wait for the 5-minute cron), then query group(fullPath: ...) { dependencyFirewallActivitySummary(from: <7 or 30 days ago>) { ... } } in GraphQL explorer. The group's Secure > Dependency firewall page shows the same numbers.
  3. ClickHouse path: enable "Use ClickHouse for analytics", run Siphon in GDK (replicates dependency_firewall_prevented_packages), enable dependency_firewall_clickhouse_reads; groups then accept any from.
  4. Specs: bin/rspec ee/spec/lib/security/dependency_firewall/ ee/spec/services/security/dependency_firewall/ ee/spec/workers/security/dependency_firewall/ ee/spec/models/security/dependency_firewall_namespace_prevented_stat_spec.rb ee/spec/requests/api/graphql/group/dependency_firewall_rule_activity_spec.rb ee/spec/requests/api/graphql/project/dependency_firewall_rule_activity_spec.rb (534 examples passing locally).

Not done yet

  • Database Lab timing of a refresh batch and of a full gitlab-org refresh; memory of the merged package set.
  • Wrap the read-miss Redis call in a rescue; check the group flag before the ClickHouse query.
  • Fill introduced_by_url in the two flag files and db/docs.

MR acceptance checklist

Evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.

References

Edited by Arpit Gogia

Merge request reports