Loading
Draft: [skip ci] Serve prevented-package stats from ClickHouse where available
What does this MR do and why?
Part 6 of 7 for https://gitlab.com/gitlab-org/gitlab/-/work_items/631890.
This MR adds a ClickHouse-backed alternative to the PostgreSQL read path from part 5, for installations that have ClickHouse analytics configured.
ClickHouseCollector reads the Siphon replica siphon_dependency_firewall_prevented_packages for both projects and groups, for any window, not just 7 or 30 days. It deduplicates replicated row versions and skips deleted rows, since the replica can hold multiple versions of the same logical row.
DataCollector uses ClickHouseCollector only when analytics ClickHouse is enabled, the new flag dependency_firewall_clickhouse_reads (beta, default off) is on, and Siphon is actually replicating the table; otherwise the PostgreSQL paths from part 5 are unchanged. For groups, dependency_firewall_group_prevented_stats is also required and is checked before any ClickHouse lookup is attempted, so the ClickHouse path never runs ahead of the feature it augments.
Parity specs assert that ClickHouse counts equal PostgreSQL counts for the same data, for both projects and groups.
Stack
| Part | MR | Targets |
|---|---|---|
| 1 | Summary table and flag !261477 (merged) | master |
| 2 | Paged group refresh service and refresh job !261478 | part 1 |
| 3 | Affected-group lookup and refresh scheduling on flush and transfers !261479 | part 2 |
| 4 | Daily refresh !261485 | part 2 |
| 5 | Group read path !261481 | part 2 |
| 6 | ClickHouse reads !261486 | part 5 |
| 7 | Group page !261483 | part 1 |
How to set up and validate locally
Enable "Use ClickHouse for analytics", run Siphon in GDK (it replicates dependency_firewall_prevented_packages), then enable dependency_firewall_clickhouse_reads. Groups will then accept any from window. Run:
bin/rspec ee/spec/lib/security/dependency_firewall/prevented_stats/MR acceptance checklist
Evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.
References
- Issue: https://gitlab.com/gitlab-org/gitlab/-/work_items/631890
- PoC (all parts together, tested end to end): !260820
- Flag rollout: #631122 (closed)
Edited by Arpit Gogia