Materialize valid_primaries view (14.4)
The dataloss query is extremely slow for bigger datasets. The problem is that for each row that the data loss query is returning, Postgres computes the full result of the valid_primaries view only to filter down to the correct record. This results in an o(n2) complexity which kills the performance as soon as the dataset size increases. It's not clear why the join parameters are not pushed down in to the view in the query.
This optimizes the query by materializing the valid_primaries view. This ensures Postgres computes the full view only once and joins with the pre-computed result.
RepositoryStoreCollector gathers metrics on repositories which don't have a valid primary candidates available. This indicates the repository is unavailable as the current primary is not valid and ther are no valid candidates to failover to. The query is currently extremely inefficient on some versions of Postgres as it ends up computing the full valid_primaries view for each of the rows it checks. This doesn't seem to occur on all versions of Postgres, namely 12.6 at least manages to push down the search criteria inside the view. This commit fixes the situation by materializing the valid_primaries view prior to querying it. This ensures the full view isn't computed for all of the rows but rather Postgres just uses the pre-computed result.
See #3744 (comment 735797239) for explanation and query plans.
Part of #3744 (closed)
Merge request reports
Activity
changed milestone to %14.4
added Category:Gitaly devopscreate groupgitaly + 1 deleted label
assigned to @samihiltunen
3 Warnings df6b165f: The commit body should not contain more than 72 characters per line. For more information, take a look at our Commit message guidelines. 57bef779: The commit body should not contain more than 72 characters per line. For more information, take a look at our Commit message guidelines. 6d569bb6: The commit body should not contain more than 72 characters per line. For more information, take a look at our Commit message guidelines. Reviewer roulette
Changes that require review have been detected! A merge request is normally reviewed by both a reviewer and a maintainer in its primary category and by a maintainer in all other categories.
To spread load more evenly across eligible reviewers, Danger has picked a candidate for each review slot. Feel free to override these selections if you think someone else would be better-suited or use the GitLab Review Workload Dashboard to find other available reviewers.
To read more on how to use the reviewer roulette, please take a look at the Engineering workflow and code review guidelines. Please consider assigning a reviewer or maintainer who is a domain expert in the area of the merge request.
Once you've decided who will review this merge request, mention them as you normally would! Danger does not automatically notify them for you.
Category Reviewer Maintainer ~ Ævar Arnfjörð Bjarmason ( @avar
) (UTC+1, 1 hour behind@samihiltunen
)Patrick Steinhardt ( @pks-t
) (UTC+1, 1 hour behind@samihiltunen
)If needed, you can retry the
danger-review
job that generated this comment.Generated by
DangerEdited by GitalyBotadded 1 commit
- d5147eb8 - Materialize valid_primaries view in RepositoryStoreCollector
added sectiondev label
changed milestone to %14.6
added missed:14.4 label
removed review request for @8bitlife
changed milestone to %14.4
added auto updated label
added 1 commit
- eccffd92 - Materialize valid_primaries view in RepositoryStoreCollector
added 1 commit
- 4b95882c - Materialize valid_primaries view in RepositoryStoreCollector
enabled automatic add to merge train when the pipeline for 8d1f7a64 succeeds
mentioned in commit c1cf3752
mentioned in issue #3931 (closed)