Improve query to identify up to date storages for reads distribution
Benefit from versioning to retrieve up to date storages that can server read operation for the repository in order to distribute reads across all healthy storages of the virtual storage.
Closes: #2944 (closed)
Merge request reports
Activity
changed milestone to %13.2
added Category:Gitaly devopscreate groupgitaly workflowready for review + 1 deleted label
added 1 commit
- 1fed0790 - Improve query to identify up to date storages for reads distribution
added bugperformance typebug labels
- Resolved by Pavlo Strokov
assigned to @samihiltunen and @pks-t
mentioned in issue gitlab#227215 (closed)
- Resolved by Toon Claes
@8bitlife @zj-gitlab I really wonder if we should pull in database experts to do reviews on such changes. I wouldn't consider myself qualified enough to reason about database plans and optimization of queries.
mentioned in issue #2951 (closed)
- Resolved by Pavlo Strokov
- Resolved by Pavlo Strokov
- Resolved by Pavlo Strokov
- Resolved by Pavlo Strokov
assigned to @8bitlife and unassigned @samihiltunen
added 1 commit
- c1613448 - Improve query to identify up to date storages for reads distribution
assigned to @samihiltunen and unassigned @8bitlife
@samihiltunen @pks-t the approach was slightly changed and now we have an additional table to track the state of the queue per repository(more details in the desc).
We could consider it as a transition solution until we have !2317 (closed) in place to benefit fromRepository Versions
or until we lend #2941 (closed).- Resolved by Pavlo Strokov
- Resolved by Patrick Steinhardt
- Resolved by Pavlo Strokov
added 13 commits
-
c1613448...1fae8b36 - 8 commits from branch
master
- debe150e - Improve query to identify up to date storages for reads distribution
- c90fa43c - Apply 1 suggestion(s) to 1 file(s)
- 31fd2b86 - Improve query to identify up to date storages for reads distribution
- 7e04cc40 - Praefect: handling of stale replication jobs
- 32e1b629 - Praefect: handling of stale replication jobs
Toggle commit list-
c1613448...1fae8b36 - 8 commits from branch
added Deliverable label
unassigned @pks-t
- Resolved by Pavlo Strokov
- Resolved by Pavlo Strokov
The migration might have a problem where one node performs the migration but older nodes manage to write to the replication queue before they are taken down. Those writes would then not be in present in the empty queue table.
As far as I understand the solution, we are referencing the latest queued event for a given virtual-storage/storage/relative-path in the empty queue table and then using it to determine whether the queue is fully empty.
This should be fine as a transitionary solution I think. I'm just wondering if it is worth it to put it in just now as I'll have the work from !2317 (closed) ready for review today and we could base distributed reads on it this week if nothing else comes up.
assigned to @8bitlife
added 1 commit
- e8894b0f - Praefect: handling of stale replication jobs
added typemaintenance label
changed milestone to %13.3
added missed-deliverable missed:13.2 labels
added 105 commits
-
e8894b0f...43e8389d - 104 commits from branch
master
- ed0a59b9 - Improve query to identify up to date storages for reads distribution
-
e8894b0f...43e8389d - 104 commits from branch
unassigned @8bitlife
assigned to @pks-t
added 1 commit
- cf17c84a - Improve query to identify up to date storages for reads distribution
added typefeature label
@pks-t @samihiltunen could you please take another round of review as implementation was drastically changed and now it relies on the versioning of repositories.