Resolve "Add a migration to add project permissions to commit documents"
What does this MR do and why?
This backfills the visibility_level
and repository_access_level
properties for commit documents in the blob index. Historically, those properties were only on project documents and not commits. This change is part of a bigger process to ultimately migrate commit documents to their own index.
This migration iterates over the permutations of visibility_level
and repository_access_level
for all projects and updates commit documents with those levels. Based on this discussion, these updates will be performed in batches of 1,500
commits at a time.
Screenshots or screen recordings
To give a sense of scale, there are 1,211,113
projects and 684,716,921
commits and the following number of projects:
Spreadsheet that includes my math on how many migration runs will be required to update all commits: https://docs.google.com/spreadsheets/d/1QW2O_oJ4Q9m6Tp0SfBySUFpmjPsnh5KrFlsRKzuGP5Y/edit?usp=sharing
Data generated by this Elasticsearch query (as of 05/16/2022)
Click to expand Elastic query
GET gitlab-production/_search
{
"size": 0,
"aggs" : {
"my_buckets": {
"composite" : {
"sources" : [
{ "visibility_level": { "terms" : { "field": "visibility_level" } } },
{ "repository_access_level": { "terms" : { "field": "repository_access_level" } } }
]
}
}
}
, "query": {
"term": {
"type": {
"value": "project"
}
}
}
}
How to set up and validate locally
Numbered steps to set up and validate the change are strongly suggested.
MR acceptance checklist
This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.
-
I have evaluated the MR acceptance checklist for this MR.
Related to #344459 (closed)