Optimistic lock for namespace descendants
What does this MR do and why?
Problem:
When we update namespace_descendants row for a given Group, the group and the cache record is locked for writing. For a large namespace hierarchy, this locking can take long time which affects group and project creation/update.
- Related issue: https://gitlab.com/gitlab-org/gitlab/-/issues/534760+
- Short doc about the optimization: https://docs.gitlab.com/development/database/group_hierarchy_optimization/
Solution:
This MR introduces a lock-free approach using optimistic locking.
-
Namespaces::Descendantrecord has a field calledoutdated_at. - This field is always bumped when the cache is marked outdated.
- In the
Namespaces::UpdateDenormalizedDescendantsServicewe note down the currentoutdated_atvalue. - We calculate the cached values without locks.
- When updating the
Namespaces::Descendantrecord with the new cache values we compare the oldoutdated_atvalue with the currentoutdated_atvalue from the DB. - If the timestamp differ, the cache stays in outdated state so the next run can attempt caching it again.
- If the timestamp is the same that means that the group hierarchy did not change during the cache calculation. The record will be marked up to date.
The UPDATE query uses low lock timeout to avoid lock contention.
Screenshots or screen recordings
| Before | After |
|---|---|
How to set up and validate locally
MR acceptance checklist
Evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.
Edited by Adam Hegyi