Skip to content
Snippets Groups Projects

Improve query to identify up to date storages for reads distribution

Merged Pavlo Strokov requested to merge ps-improve-reads-dist into master

Benefit from versioning to retrieve up to date storages that can server read operation for the repository in order to distribute reads across all healthy storages of the virtual storage.

Closes: #2944 (closed)

Edited by Pavlo Strokov

Merge request reports

Loading
Loading

Activity

Filter activity
  • Approvals
  • Assignees & reviewers
  • Comments (from bots)
  • Comments (from users)
  • Commits & branches
  • Edits
  • Labels
  • Lock status
  • Mentions
  • Merge request status
  • Tracking
  • Pavlo Strokov resolved all threads

    resolved all threads

  • Pavlo Strokov added 1 commit

    added 1 commit

    • 8a2e01dc - Apply 1 suggestion(s) to 1 file(s)

    Compare with previous version

  • mentioned in issue #2951 (closed)

  • Sami Hiltunen assigned to @8bitlife and unassigned @samihiltunen

    assigned to @8bitlife and unassigned @samihiltunen

  • Pavlo Strokov marked as a Work In Progress

    marked as a Work In Progress

  • Pavlo Strokov added 1 commit

    added 1 commit

    • c1613448 - Improve query to identify up to date storages for reads distribution

    Compare with previous version

  • Pavlo Strokov unmarked as a Work In Progress

    unmarked as a Work In Progress

  • Pavlo Strokov changed the description

    changed the description

  • Pavlo Strokov assigned to @samihiltunen and unassigned @8bitlife

    assigned to @samihiltunen and unassigned @8bitlife

  • Author Contributor

    @samihiltunen @pks-t the approach was slightly changed and now we have an additional table to track the state of the queue per repository(more details in the desc).
    We could consider it as a transition solution until we have !2317 (closed) in place to benefit from Repository Versions or until we lend #2941 (closed).

  • Pavlo Strokov resolved all threads

    resolved all threads

  • Patrick Steinhardt assigned to @8bitlife and unassigned @pks-t

    assigned to @8bitlife and unassigned @pks-t

  • Pavlo Strokov added 13 commits

    added 13 commits

    • c1613448...1fae8b36 - 8 commits from branch master
    • debe150e - Improve query to identify up to date storages for reads distribution
    • c90fa43c - Apply 1 suggestion(s) to 1 file(s)
    • 31fd2b86 - Improve query to identify up to date storages for reads distribution
    • 7e04cc40 - Praefect: handling of stale replication jobs
    • 32e1b629 - Praefect: handling of stale replication jobs

    Compare with previous version

  • added Deliverable label

  • Pavlo Strokov assigned to @pks-t and unassigned @8bitlife

    assigned to @pks-t and unassigned @8bitlife

  • Patrick Steinhardt resolved all threads

    resolved all threads

  • Patrick Steinhardt approved this merge request

    approved this merge request

    • Resolved by Pavlo Strokov

      The migration might have a problem where one node performs the migration but older nodes manage to write to the replication queue before they are taken down. Those writes would then not be in present in the empty queue table.

      As far as I understand the solution, we are referencing the latest queued event for a given virtual-storage/storage/relative-path in the empty queue table and then using it to determine whether the queue is fully empty.

      This should be fine as a transitionary solution I think. I'm just wondering if it is worth it to put it in just now as I'll have the work from !2317 (closed) ready for review today and we could base distributed reads on it this week if nothing else comes up.

  • assigned to @8bitlife

  • Pavlo Strokov added 1 commit

    added 1 commit

    • e8894b0f - Praefect: handling of stale replication jobs

    Compare with previous version

  • 🤖 GitLab Bot 🤖 changed milestone to %13.3

    changed milestone to %13.3

  • Pavlo Strokov added 105 commits

    added 105 commits

    Compare with previous version

  • Pavlo Strokov changed the description

    changed the description

  • assigned to @pks-t

  • Pavlo Strokov added 1 commit

    added 1 commit

    • cf17c84a - Improve query to identify up to date storages for reads distribution

    Compare with previous version

  • added typefeature label

  • Author Contributor

    @pks-t @samihiltunen could you please take another round of review as implementation was drastically changed and now it relies on the versioning of repositories.

  • Pavlo Strokov resolved all threads

    resolved all threads

  • Loading
  • Loading
  • Loading
  • Loading
  • Loading
  • Loading
  • Loading
  • Loading
  • Loading
  • Loading
  • Please register or sign in to reply
    Loading