Skip to content

Resolve "Add a migration to add project permissions to commit documents"

What does this MR do and why?

This backfills the visibility_level and repository_access_level properties for commit documents in the blob index. Historically, those properties were only on project documents and not commits. This change is part of a bigger process to ultimately migrate commit documents to their own index.

This migration iterates over the permutations of visibility_level and repository_access_level for all projects and updates commit documents with those levels. Based on this discussion, these updates will be performed in batches of 1,500 commits at a time.

Screenshots or screen recordings

To give a sense of scale, there are 1,211,113 projects and 684,716,921 commits and the following number of projects:

Spreadsheet that includes my math on how many migration runs will be required to update all commits: https://docs.google.com/spreadsheets/d/1QW2O_oJ4Q9m6Tp0SfBySUFpmjPsnh5KrFlsRKzuGP5Y/edit?usp=sharing

Data generated by this Elasticsearch query (as of 05/16/2022)

Click to expand Elastic query
GET gitlab-production/_search
{
  "size": 0, 
    "aggs" : {
        "my_buckets": {
            "composite" : {
                "sources" : [
                    { "visibility_level": { "terms" : { "field": "visibility_level" } } },
                    { "repository_access_level": { "terms" : { "field": "repository_access_level" } } }
                ]
            }
        }
     }
     , "query": {
       "term": {
         "type": {
           "value": "project"
         }
       }
     }
}

image

How to set up and validate locally

Numbered steps to set up and validate the change are strongly suggested.

MR acceptance checklist

This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.

Related to #344459 (closed)

Edited by John Mason

Merge request reports