Skip to content

De-normalize project visibility_level into issues Elasticsearch documents

What does this MR do?

As part of a long sequence of changes to our Elasticsearch index we are moving issues to their own Elasticsearch index, primarily for performance reasons &4697 (closed) . Since you cannot join across indices in Elasticsearch we will need to denormalize (ie. add project level fields to the issue document in Elasticsearch) any project level settings that impact our issue searches so that our queries can use this permission data to filter the correct results for the user #273234 (closed) .

There are 2 parts to storing project level permissions in the issue document:

  1. When the issue document is indexed we need to look up these project level values and save them with the document.
  2. In order to keep this in sync we need to ensure the issue is updated in the index every time the project changes (for the fields that are actually stored in the issue document).

This work was started with the issues_access_level in !47007 (merged) and !46916 (merged) but we later learnt that you also need to use the visibility_level of the project to determine when a project is "Internal". Based solely on the issues_access_level there is no way to distinguish an "Internal" project with issues visible to everyone logged in and a "Public" project with issues visible also to anonymous users.

We do also get some benefit merging these changes now before we start using them in searches as this will effectively start the process of backfilling some permissions in the meantime.

Important note about ordering/backfilling

Previously I was thinking we could separate this change into 2 MRs like we did with !46916 (merged) and !47007 (merged) but that leaves open the possible risk that we end up with some issues that have an out of date visibility_level set. We are later going to use the absence of the visibility_level in the index to determine which documents require backfill. Therefore it is important that we are also triggering updates for projects when the visibility_level changes as soon as we're indexing the visibility_level.

Screenshots (strongly suggested)

Does this MR meet the acceptance criteria?

Conformity

Availability and Testing

Security

If this MR contains changes to processing or storing of credentials or tokens, authorization and authentication methods and other items described in the security review guidelines:

  • Label as security and @ mention @gitlab-com/gl-security/appsec
  • The MR includes necessary changes to maintain consistency between UI, API, email, or other methods
  • Security reports checked/validated by a reviewer from the AppSec team

Related to #273234 (closed)

Merge request reports