[Group Hierarchy] Implement job and service for persisting denormalized data
Persisting the de-normalied hierarchy when change happens should be fast and efficient. The service should be able to consistently handle updates without timeout (batching).
Job definition
The job should be scheduled (cron) for every two minutes and it would be allowed to run for 60 seconds. See the worker implementation from the PoC: !134499 (diffs)
The job loads N Namespaces::Descendants
rows and invokes the service.
Service
The service should work on one Namespaces::Descendants
model. Use the following algorithm:
- Start a transaction.
- Acquire lock on the
Namespaces::Descendants
record. - Invoke the iterator. (implemented: #433479 (closed)).
- For each batch, load the actual namespace record.
- Collect the ids separately: if Group ->
group_ids
, if Project ->project_ids
- Invoke an upsert query.
- Transaction done.
Note: Iterating in a transaction is generally not a good idea. Unfortunately, in this case we don't have a choice. For a large group hierarchy the transaction might be open for a 1-2 seconds.
Edited by Adam Hegyi