Implement a namespace hierarchy iterator library
What does this MR do and why?
This change implements an efficient iterator over the namespace hierarchy graph in SQL based on the PoC documented here: https://docs.gitlab.com/ee/development/database/poc_tree_iterator.html
Note: the query is quite complex so I tried to not overdo Arel and used raw SQL when it made sense.
Queries:
- First batch: https://console.postgres.ai/gitlab/gitlab-production-tunnel-pg12/sessions/24853/commands/78974
- Second batch: https://console.postgres.ai/gitlab/gitlab-production-tunnel-pg12/sessions/24853/commands/78975
How to set up and validate locally
Snippet for listing the tree for a group:
group = Group.first
cursor = { current_id: group.id, depth: [group.id] }
iterator = iterator = Gitlab::Database::NamespaceEachBatch.new(namespace_class: Namespace, cursor: cursor)
iterator.each_batch(of: 100) do |ids|
puts ids
end
MR acceptance checklist
This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.
-
I have evaluated the MR acceptance checklist for this MR.
Related to #433479 (closed)
Edited by Adam Hegyi