Automatically enable group hierarch optimization
Group hierarchy optimization can improve the performance of group-level database queries by asynchronously caching the descendant group or project ids.
This optimization is currently enabled for gitlab-org and gitlab-com groups manually for months now.
Enabling the optimization globally makes no sense because 99% of the groups would not see benefit of it.
Idea: enable the optimization selectively, for the groups with the most descendants.
Implementation
Implement a periodical worker that scans all groups in the database. For each group, count the descendants (subgroups and projects). When a certain count (500) is reached, enable the optimization for the group.
By enabling the optimization, these groups would automatically start invoking optimized group hierarchy queries.
The worker should have runtime limitation and it should continue the processing from the previously finished group id. To do that, implement a cursor mechanism using Redis.
When the last group id was reached in the database, start the iteration from the beginning.