Skip to content

Migration to add project and issue permissions to issues in index

Terri Chu requested to merge 273234-add-new-data-to-issues-documents into master

What does this MR do?

Related to #273234 (closed)

Part of an overall larger effort to migration issues documents to a new index in Elasticsearch. This MR contains a new migration which will back fill the issues_access_level and visibility_level fields on all issues documents within the current index. Added specs for the new migration options class and the migration itself.

Note: Only checking for issues missing visibility_level to determine if the issue needs to be re-indexed, due to the order that the previous MRs were merged, there could be issues in the index with issues_access_level that would be incorrect

To support this work, a few changes to the Elastic::MigrationWorker were required:

  • adding MigrationOptions class and instance methods to allow setting options used by the migration worker. Currently two options are supported: batched! and throttle_delay. The throttle_delay is defaulted to 5 minutes
  • support for running batched migrations, the worker will re-enqueue itself after running a migration set to use batched mode
  • updated documentation on how to create migrations for Elasticsearch - the new migration options are described

Max time estimated to complete: 84 hours for GitLab.com

  • ~5,000,000 issues in GitLab.com
  • run in batches of 5,000 every 5 minutes
  • complete 60,000 an hour

Not user facing so I did not add a changelog

Screenshots (strongly suggested)

Does this MR meet the acceptance criteria?

Conformity

Availability and Testing

Security

If this MR contains changes to processing or storing of credentials or tokens, authorization and authentication methods and other items described in the security review guidelines:

  • [-] Label as security and @ mention @gitlab-com/gl-security/appsec
  • [-] The MR includes necessary changes to maintain consistency between UI, API, email, or other methods
  • [-] Security reports checked/validated by a reviewer from the AppSec team

Related to #273234 (closed)

Edited by Dylan Griffith

Merge request reports