Skip to content

Remove issue project joins for global scoped searches

The issue data stored in Elasticsearch will now contain access_level field to determine permissions.

This should allow global scoped searches to use the access_level field instead of doing a join on the parent field.

We expect this to reduce the complexity and improve performance.

Example globally scoped issues Elasticsearch query generated
{
  "query": {
    "bool": {
      "must": [
        {
          "simple_query_string": {
            "_name": "issue:match:search_terms",
            "fields": [
              "title^2",
              "description"
            ],
            "query": "test",
            "default_operator": "and"
          }
        }
      ],
      "filter": [
        {
          "term": {
            "type": {
              "_name": "doc:is_a:issue",
              "value": "issue"
            }
          }
        },
        {
          "has_parent": {
            "_name": "issue:authorized:project",
            "parent_type": "project",
            "query": {
              "bool": {
                "should": [
                  {
                    "bool": {
                      "filter": [
                        {
                          "term": {
                            "visibility_level": {
                              "_name": "issue:authorized:project:any",
                              "value": 0
                            }
                          }
                        },
                        {
                          "terms": {
                            "_name": "issue:authorized:project:issues:enabled_or_private",
                            "issues_access_level": [
                              20,
                              10
                            ]
                          }
                        }
                      ]
                    }
                  },
                  {
                    "bool": {
                      "_name": "issue:authorized:project:visibility:10:issues:access_level",
                      "filter": [
                        {
                          "term": {
                            "visibility_level": {
                              "_name": "issue:authorized:project:visibility:10",
                              "value": 10
                            }
                          }
                        },
                        {
                          "terms": {
                            "_name": "issue:authorized:project:visibility:10:issues:access_level:enabled_or_private",
                            "issues_access_level": [
                              20,
                              10
                            ]
                          }
                        }
                      ]
                    }
                  },
                  {
                    "bool": {
                      "_name": "issue:authorized:project:visibility:20:issues:access_level",
                      "filter": [
                        {
                          "term": {
                            "visibility_level": {
                              "_name": "issue:authorized:project:visibility:20",
                              "value": 20
                            }
                          }
                        },
                        {
                          "terms": {
                            "_name": "issue:authorized:project:visibility:20:issues:access_level:enabled_or_private",
                            "issues_access_level": [
                              20,
                              10
                            ]
                          }
                        }
                      ]
                    }
                  }
                ]
              }
            }
          }
        }
      ]
    }
  },
  "highlight": {
    "fields": {
      "title": {},
      "description": {}
    },
    "number_of_fragments": 0,
    "pre_tags": [
      "gitlabelasticsearch→"
    ],
    "post_tags": [
      "←gitlabelasticsearch"
    ]
  }
}

Epic order

  1. Remove joins from the Elasticsearch query for project/group scoped issues search
  2. De-normalize the issue permission data. <-- will include migration and can be done in multiple MRs
  3. Remove joins from the Elasticsearch query for globally scoped issues search
  4. Migration to copy issue data to new index
  5. Migration to remove issue data from combined index **
  6. Rake task changes to support new index **
  7. Backend changes for supporting new indexes during Zero Downtime Re-indexing **
  8. Admin UI changes to support new index (any time after new index created/data migrated to it)

** must be released in conjunction with the Migration to copy issue data to new index

Edited by Terri Chu