Skip to content
Snippets Groups Projects

Compare revisions

Changes are shown as if the source revision was being merged into the target revision. Learn more about comparing revisions.

Source

Select target project
No results found

Target

Select target project
  • limansky/gitaly
  • cmingxu/gitaly
  • bugrargn/gitaly
  • asedge/gitaly
  • softwareplumber/gitaly
  • gitlab-org/gitaly
  • armbiant/gitlab-gitaly
  • sinwav/gitaly
  • Mattlk13/gitaly
  • Ruby-and-Friends/gitaly
  • kutelev/gitaly
  • bbodenmiller/gitaly
  • alejandro/gitaly
  • lukeshu/gitaly
  • eddiet/gitaly
  • mpoornima/gitaly
  • dturner_ts/gitaly
  • zj-gitlab/gitaly
  • nadiavu/gitaly
  • balasankarc/gitaly
  • mpszone/gitaly
  • sysu-liyanliang/gitaly
  • frenkel/gitaly
  • lfauts/gitaly
  • yang-zaixin/gitaly
  • mydigitalself/gitaly
  • litongdeng/gitaly
  • toon/gitaly
  • axot/gitaly
  • josecordaz/gitaly
  • sj14/gitaly
  • MaykonOliveira/gitaly
  • zjkyz8/gitaly
  • svankmajer/gitaly
  • qipa/gitaly
  • sptr/gitaly
  • kiameisomabes/gitaly
  • bradobro/gitaly
  • fcgravalos/gitaly
  • reprazent/gitaly
  • tbuida/gitaly
  • arjunmayilvaganan/gitaly
  • b1n/gitaly
  • timniederhausen/gitaly
  • maksim-paskal/gitaly
  • zhutiankai/gitaly
  • maxmati/gitaly
  • 358141721/gitaly
  • despinosa/gitaly
  • c.bianchi/gitaly
  • tamilselvan10021998/gitaly
  • Randomneo/gitaly
  • yemanaung/gitaly
  • siemens/gitaly
  • avar/gitaly
  • SeananXu/gitaly
  • smusavi/gitaly
  • johncai/gitaly
  • utkarsh2102/gitaly
  • jhenkens/gitaly
  • ba2014sheer/gitaly
  • jacobvosmaer-gitlab/gitaly
  • Esteban_Carnuccio/gitaly
  • d0c-s4vage/gitaly
  • thetradedesk/gitaly
  • AfolabiOlaoluwa/gitaly
  • njkevlani/gitaly
  • chriscool/gitaly
  • ethan.gitlab/gitaly
  • GeniusLearner/gitaly
  • wasphin/gitaly
  • CauhxMilloy/gitaly
  • nnelson/gitaly
  • simchatop/gitaly
  • htkaslan/gitaly
  • hemanthdev/gitaly
  • shovanmaity/gitaly
  • Temtaime/gitaly
  • Ash560/gitaly
  • gitlab-bot-readonly/gitaly
  • tylerthetester/gitaly
  • sushovan/gitaly
  • shy-Xu/gitaly
  • thmr1/gitaly
  • pravi/gitaly
  • EphremDms/gitaly
  • sluongng/gitaly
  • nejc/gitaly
  • ksashikumar/gitaly
  • motogami/gitaly
  • bs223617/gitaly
  • eagle_fly_sky/gitaly
  • hashworks/gitaly
  • anatol.pomozov/gitaly
  • T4cC0re/gitaly
  • ethan.reesor/contrib/gitaly
  • czh592/gitaly
  • wchandler/gitaly
  • diogo_nicoleti_ifood/gitaly
  • bulemka12345poczta/gitaly
  • taroguru/gitaly
  • dyrone/gitaly
  • inguin/gitaly
  • mathieuusoyan/gitaly
  • mfechner/gitaly
  • wwwicbd/icbd-gitaly
  • blanet/gitaly
  • paulsd2020/gitaly
  • harry-hov/gitaly
  • pranav/gitaly
  • abhitidarbar/gitaly
  • heygirlhey356/gitaly
  • maoqiang/gitaly
  • yyexplore/gitaly
  • cgsyam/gitaly
  • behrmann/gitaly
  • nfishe/gitaly
  • g4s8/gitaly
  • brett.higgins/gitaly
  • test11042/gitaly
  • jeffzhu503/gitaly
  • TheSide/gitaly
  • nanmu42/gitaly
  • van4elotti/gitaly
  • Cactusinhand/gitaly
  • yamini.bhaskar007/gitaly
  • icbd/gitaly
  • ooulwluoo/gitaly
  • nagypeter/gitaly
  • Abhilashsiyer/gitaly
  • jiangxin/gitaly
  • KimJi55/gitaly
  • flycutter/gitaly
  • leonard-adleman-sec/gitaly
  • ryan-qianchen/gitaly
  • feistel/gitaly
  • snowcrystall/gitaly
  • edith007/gitaly
  • zhanglinjie/gitaly
  • luminarrr/gitaly
  • stephankirsten/gitaly
  • etanot/gitaly
  • nikovega21/gitaly
  • m0rosan/gitaly
  • HaroldKnowlden/gitaly
  • iotcl/test/gitaly
  • okenwaonyebuchi103/gitaly
  • simpleclickers/gitaly
  • kdaudt/gitaly
  • YushuaiLI/gitaly
  • davebarr/gitaly
  • dgsdoug/gitaly
  • trakos/gitaly
  • XciD/gitaly
  • imskr/gitaly
  • mj.bruijns/gitaly
  • markoke256/gitaly
  • Sahil162/gitaly
  • Kartik1397/gitaly
  • akumar1503/gitaly
  • bill2022/gitaly
  • ncu_gsoc/gitaly
  • SURA907/gitaly
  • hongeinh/gitaly
  • joshua.sickmeyer/gitaly
  • adlternative/gitaly
  • amstal93/gitaly
  • gitlab-renovate-forks/gitaly
  • xiaowenxia/gitaly
  • checkscale-gitlab/gitaly
  • KyleFromKitware/gitaly
  • n0h4ppy/gitaly
  • pwn3/gitaly
  • jokerpwn/gitaly
  • salamacha7389/gitaly
  • L11R/gitaly
  • hanmingliang/gitaly
  • Isengart1/gitaly
  • cbj/gitaly
  • gitlab-community/gitaly
  • shamanthb90/gitaly
  • mjkalasky2/gitaly
  • VladPetriv/gitaly
  • arkn98/gitaly
  • scwang18/gitaly
  • sebicioacata/gitaly
  • edsonmichaque/gitaly
  • Kexin2000/gitaly
  • johnwparent/gitaly
  • kevin.rojas/wr-gitaly
  • marc.ratombotsoa/gitaly
  • igor.drozdov/gitaly
  • radityasurya1911/gitaly
  • CS2Us/gitaly
  • RryLee/gitaly
  • liruixin-coding/gitaly
  • gerardo/gitaly
  • josephburnett/gitaly
  • b.s.strelnikov_tinkoff/gitaly
  • OK_MF/gitaly-fork
  • zeb0x01/gitaly
  • 8bitlife/gitaly
  • nraj0408/gitaly
  • troyfox758/gitaly
  • ouladsine.saloua/gitaly
  • qeesung/gitaly
  • dzcdada123/gitaly
  • ashmckenzie/gitaly
  • Freedisch/gitaly
  • xsizxenjin/gitaly
  • eric.p.ju/gitaly
  • zzhzero/gitaly
  • lenghan1991/gitaly
  • jhammer101/gitaly
  • wheredidfranciscogo1/gitaly
  • zakiir206/gitaly
  • jonas.hogman/gitaly
  • sauravchanda9/gitaly
  • lzampier/gitaly
  • fkhe/gitaly
  • liaoxingju/gitaly
  • bufdev/gitaly
  • a.songer/gitaly
  • bcooksley/gitaly
  • Wangyadong1108/gitaly
  • MashyBasker/gitaly
  • nickaldwin/gitaly
  • avevlad/gitaly
  • sat-h/gitaly
  • wingred96399/gitaly-mine
  • chen_feng/gitaly
  • grootwang/gitaly
  • bhatianikhil551/gitaly
  • schuam_alice/gitaly
  • armbiant/hive-gitaly
  • ollevche/gitaly
  • HassanAkbar/gitaly
  • echui-gitlab/gitaly
238 results
Show changes
Commits on Source (4)
  • Sami Hiltunen's avatar
    Materialize valid_primaries view in dataloss query · 6d569bb6
    Sami Hiltunen authored
    The dataloss query is extremely slow for bigger datasets. The problem
    is that for each row that the data loss query is returning,
    Postgres computes the full result of the valid_primaries view only to
    filter down to the correct record. This results in an o(n2) complexity
    which kills the performance as soon as the dataset size increases. It's
    not clear why the join parameters are not pushed down in to the view in
    the query.
    
    This commit optimizes the query by materializing the valid_primaries view.
    This ensures Postgres computes the full view only once and joins with the
    pre-computed result.
    
    Changelog: performance
    6d569bb6
  • Sami Hiltunen's avatar
    Get the latest generation from repositories instead of a view · 57bef779
    Sami Hiltunen authored
    Dataloss query is currently getting the latest generation of a repository
    from a view that takes the max generation from storage_repositories. This
    is unnecessary as the repositories table already contains the latest generation
    and we can take it from there instead. This commit reads it from the repositories
    table instead.
    
    Changelog: performance
    57bef779
  • Sami Hiltunen's avatar
    Materialize valid_primaries view in RepositoryStoreCollector · df6b165f
    Sami Hiltunen authored
    RepositoryStoreCollector gathers metrics on repositories which don't
    have a valid primary candidates available. This indicates the repository
    is unavailable as the current primary is not valid and ther are no valid
    candidates to failover to. The query is currently extremely inefficient
    on some versions of Postgres as it ends up computing the full valid_primaries
    view for each of the rows it checks. This doesn't seem to occur on all versions
    of Postgres, namely 12.6 at least manages to push down the search criteria
    inside the view. This commit fixes the situation by materializing the
    valid_primaries view prior to querying it. This ensures the full view isn't
    computed for all of the rows but rather Postgres just uses the pre-computed
    result.
    
    Changelog: performance
    df6b165f
  • Toon Claes's avatar
    Merge branch 'smh-optimize-dataloss-query-14-4' into '14-4-stable' · c1cf3752
    Toon Claes authored
    Materialize valid_primaries view (14.4)
    
    See merge request !4090
    c1cf3752
......@@ -72,6 +72,11 @@ func (c *RepositoryStoreCollector) Collect(ch chan<- prometheus.Metric) {
// they are either unhealthy or out of date.
func (c *RepositoryStoreCollector) queryMetrics(ctx context.Context) (map[string]int, error) {
rows, err := c.db.QueryContext(ctx, `
WITH valid_primaries AS MATERIALIZED (
SELECT virtual_storage, relative_path
FROM valid_primaries
)
SELECT virtual_storage, COUNT(*)
FROM repositories
WHERE NOT EXISTS (
......
......@@ -651,6 +651,10 @@ func (rs *PostgresRepositoryStore) GetPartiallyAvailableRepositories(ctx context
// than the assigned ones.
//
rows, err := rs.db.QueryContext(ctx, `
WITH valid_primaries AS MATERIALIZED (
SELECT * FROM valid_primaries
)
SELECT
json_build_object (
'RelativePath', relative_path,
......@@ -670,7 +674,7 @@ FROM (
relative_path,
repositories.primary,
storage,
repository_generations.generation - COALESCE(storage_repositories.generation, -1) AS behind_by,
repositories.generation - COALESCE(storage_repositories.generation, -1) AS behind_by,
repository_assignments.storage IS NOT NULL AS assigned,
healthy_storages.storage IS NOT NULL AS healthy,
valid_primaries.storage IS NOT NULL AS valid_primary
......@@ -688,7 +692,6 @@ FROM (
)
) AS repository_assignments USING (virtual_storage, relative_path, storage)
JOIN repositories USING (virtual_storage, relative_path)
JOIN repository_generations USING (virtual_storage, relative_path)
LEFT JOIN healthy_storages USING (virtual_storage, storage)
LEFT JOIN valid_primaries USING (virtual_storage, relative_path, storage)
WHERE virtual_storage = $1
......
......@@ -1321,10 +1321,17 @@ func TestPostgresRepositoryStore_GetPartiallyAvailableRepositories(t *testing.T)
})
if !tc.nonExistentRepository {
var maxGeneration int
for _, generation := range tc.existingGenerations {
if generation > maxGeneration {
maxGeneration = generation
}
}
_, err := tx.ExecContext(ctx, `
INSERT INTO repositories (virtual_storage, relative_path, "primary")
VALUES ('virtual-storage', 'relative-path', 'repository-primary')
`)
INSERT INTO repositories (virtual_storage, relative_path, "primary", generation)
VALUES ('virtual-storage', 'relative-path', 'repository-primary', $1)
`, maxGeneration)
require.NoError(t, err)
}
......