[SPIKE] Investigate feasibility of asynchronous metric aggregation and caching (Sidekiq+Redis/Postgres)
As a follow up to Spike: Investigate options for searching vulner... (#352665 - closed) • Gregory Havenga (is on PTO from 15 December 2025 to 02 January 2026) • 15.9, a possible alternative strategy to provide richer analytical information for our users is to aggregate and cache these metrics asynchronously to provide to users on demand?
As all this information is already present in GitLab, it should be possible to generate these metrics and cache them, either in Redis or Postgres itself to be requested on demand. The primary concern is what kind of load this style of generation might put on our Postgres DB, what kind of computation expense we might see in Sidekiq doing this, and depending on both of those facts, how often might we be willing to provide this kind of metric information? All of these being a function of the quantity of data being analysed and aggregated, which we could feasibly cap by date range?
Expected Outcomes
-
What kind of performance can we expect when performing metric aggregation queries for a respective quantity of data. (For example, time spent in each state (detected, confirmed, resolved), average time to resolve, count by fields other than severity (currently the only counts in the security dashboard) etc). -
What sort of query times would we expect to see to determine these values against the GitLab project itself for example? -
Could these metrics feasibly be generated asynchronously without putting too much strain on our Postgres resources? If so, how often? And Can we attenuate generation frequency automatically according to query times? (So projects with more data that take longer are processed less often? -
How might we store these pre-calculated values? Postgres? Or Redis Caching? -
Should we automatically generate these values, or only trigger their processing when an attempt is made to access them? -
What will the impact be on our self-managed users?