Draft: [skip ci] Mark groups for a prevented-stats refresh
What does this MR do and why?
Part 3 of 7 for https://gitlab.com/gitlab-org/gitlab/-/work_items/631890.
This MR adds the triggers that schedule a refresh of a group's precomputed prevented-package summary.
It also adds AffectedGroups.for_projects(project_ids) and .for_namespace(namespace_id), which return the ids of the groups a project or namespace rolls up to, only for top-level groups that have the dependency_firewall_group_prevented_stats flag on.
The stats table's flush worker, FlushPreventedPackagesWriteBufferCronWorker, now schedules RefreshGroupPreventedStatsWorker for the affected groups of the projects it just flushed, via AffectedGroups, after the write commits. This is best effort: errors are tracked and never affect the flush itself, so a scheduling failure cannot break the underlying write path.
A new EventStore subscriber, MarkGroupsForPreventedStatsRefreshWorker, schedules RefreshGroupPreventedStatsWorker for the affected groups (via AffectedGroups) on project transfers and deletions, and on group transfers and deletions. This keeps summaries eligible for recomputation whenever a project moves between groups, or a group's hierarchy otherwise changes, since those events can change which projects roll up into a group's totals.
Stack
| Part | MR | Targets |
|---|---|---|
| 1 | Summary table and flag !261477 (merged) | master |
| 2 | Paged group refresh service and refresh job !261478 | part 1 |
| 3 | Affected-group lookup and refresh scheduling on flush and transfers !261479 | part 2 |
| 4 | Daily refresh !261485 | part 2 |
| 5 | Group read path !261481 | part 2 |
| 6 | ClickHouse reads !261486 | part 5 |
| 7 | Group page !261483 | part 1 |
Database
AffectedGroups runs three lookups on existing tables: projects joined to namespaces by project id (at most 1,000 ids per call, the flush batch size), namespaces by id and type, and the root groups by id. The raw SQL and Database Lab plans are still to be added here.
How to set up and validate locally
Run:
bin/rspec ee/spec/workers/security/dependency_firewall/flush_prevented_packages_write_buffer_cron_worker_spec.rb ee/spec/workers/security/dependency_firewall/mark_groups_for_prevented_stats_refresh_worker_spec.rb ee/spec/lib/security/dependency_firewall/prevented_stats/affected_groups_spec.rbMR acceptance checklist
Evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.
References
- Issue: https://gitlab.com/gitlab-org/gitlab/-/work_items/631890
- PoC (all parts together, tested end to end): !260820