Skip to content

Add upvotes to merge requests documents

What does this MR do?

Related to #263365

Add an Advanced Search migration which adds a new field upvotes to the merge request documents. This includes:

  • a new migration options to better support batched migrations (specs and documentation updates to go with them)
  • adding some helper methods to make adding new fields easier in the future (and using them in this MR to make sure they work)
  • a migration to create the new data (and a spec for it)
  • Add a new MergeRequests Preloader to preload related data + lazily load the upvotes_count (using BatchLoader) during indexing to reduce DB calls
  • kicks off indexing if upvotes get added or removed so the merge request document gets updated
  • re-order when indexes get created for testing, switch to the migrations index being created before the data indexes

How to test

Note: The initial index should be created on the default branch prior to testing out this MR.

  • Get your gdk setup to work with Elasticsearch and create your index from scratch on the default branch
  • switch to this branch: git checkout 263365-add-upvotes-field-to-merge-requests-index
  • open a rails console and start the Elastic::MigrationWorker job: Elastic::MigrationWorker.new.perform
  • follow along in the log/elasticsearch.log file
An example of the migration running from my gdk
{"severity":"INFO","time":"2021-07-23T20:22:49.866Z","correlation_id":null,"message":"MigrationWorker: migration[AddUpvotesToMergeRequests] executing migrate method"}
{"severity":"INFO","time":"2021-07-23T20:22:50.477Z","correlation_id":null,"message":"[Elastic::Migration: 20210722112600] Checking if there are documents without upvotes field: 70 documents left"}
{"severity":"INFO","time":"2021-07-23T20:22:50.477Z","correlation_id":null,"message":"[Elastic::Migration: 20210722112600] Adding upvotes field to merge requests documents for batch of 5000 documents"}
{"severity":"DEBUG","time":"2021-07-23T20:22:50.524Z","correlation_id":null,"class":"Elastic::ProcessInitialBookkeepingService","redis_set":"elastic:bulk:initial:2:zset","message":"track_items","count":6,"tracked_items_encoded":"[[1,\"MergeRequest 30 merge_request_30 project_7\"],[2,\"MergeRequest 16 merge_request_16 project_3\"],[3,\"MergeRequest 18 merge_request_18 project_3\"],[4,\"MergeRequest 42 merge_request_42 project_4\"],[5,\"MergeRequest 28 merge_request_28 project_6\"],[6,\"MergeRequest 37 merge_request_37 project_2\"]]"}
{"severity":"DEBUG","time":"2021-07-23T20:22:50.526Z","correlation_id":null,"class":"Elastic::ProcessInitialBookkeepingService","redis_set":"elastic:bulk:initial:4:zset","message":"track_items","count":5,"tracked_items_encoded":"[[1,\"MergeRequest 61 merge_request_61 project_14\"],[2,\"MergeRequest 44 merge_request_44 project_5\"],[3,\"MergeRequest 1 merge_request_1 project_1\"],[4,\"MergeRequest 34 merge_request_34 project_1\"],[5,\"MergeRequest 12 merge_request_12 project_2\"]]"}
{"severity":"DEBUG","time":"2021-07-23T20:22:50.527Z","correlation_id":null,"class":"Elastic::ProcessInitialBookkeepingService","redis_set":"elastic:bulk:initial:5:zset","message":"track_items","count":4,"tracked_items_encoded":"[[1,\"MergeRequest 47 merge_request_47 project_7\"],[2,\"MergeRequest 4 merge_request_4 project_1\"],[3,\"MergeRequest 9 merge_request_9 project_2\"],[4,\"MergeRequest 56 merge_request_56 project_11\"]]"}
{"severity":"DEBUG","time":"2021-07-23T20:22:50.527Z","correlation_id":null,"class":"Elastic::ProcessInitialBookkeepingService","redis_set":"elastic:bulk:initial:10:zset","message":"track_items","count":4,"tracked_items_encoded":"[[1,\"MergeRequest 62 merge_request_62 project_14\"],[2,\"MergeRequest 43 merge_request_43 project_5\"],[3,\"MergeRequest 65 merge_request_65 project_16\"],[4,\"MergeRequest 6 merge_request_6 project_1\"]]"}
{"severity":"DEBUG","time":"2021-07-23T20:22:50.528Z","correlation_id":null,"class":"Elastic::ProcessInitialBookkeepingService","redis_set":"elastic:bulk:initial:11:zset","message":"track_items","count":5,"tracked_items_encoded":"[[1,\"MergeRequest 48 merge_request_48 project_7\"],[2,\"MergeRequest 52 merge_request_52 project_9\"],[3,\"MergeRequest 59 merge_request_59 project_13\"],[4,\"MergeRequest 68 merge_request_68 project_17\"],[5,\"MergeRequest 38 merge_request_38 project_2\"]]"}
{"severity":"DEBUG","time":"2021-07-23T20:22:50.529Z","correlation_id":null,"class":"Elastic::ProcessInitialBookkeepingService","redis_set":"elastic:bulk:initial:12:zset","message":"track_items","count":5,"tracked_items_encoded":"[[1,\"MergeRequest 29 merge_request_29 project_7\"],[2,\"MergeRequest 23 merge_request_23 project_5\"],[3,\"MergeRequest 51 merge_request_51 project_9\"],[4,\"MergeRequest 5 merge_request_5 project_1\"],[5,\"MergeRequest 63 merge_request_63 project_15\"]]"}
{"severity":"DEBUG","time":"2021-07-23T20:22:50.530Z","correlation_id":null,"class":"Elastic::ProcessInitialBookkeepingService","redis_set":"elastic:bulk:initial:3:zset","message":"track_items","count":4,"tracked_items_encoded":"[[1,\"MergeRequest 54 merge_request_54 project_10\"],[2,\"MergeRequest 19 merge_request_19 project_3\"],[3,\"MergeRequest 32 merge_request_32 project_8\"],[4,\"MergeRequest 50 merge_request_50 project_8\"]]"}
{"severity":"DEBUG","time":"2021-07-23T20:22:50.531Z","correlation_id":null,"class":"Elastic::ProcessInitialBookkeepingService","redis_set":"elastic:bulk:initial:6:zset","message":"track_items","count":3,"tracked_items_encoded":"[[1,\"MergeRequest 24 merge_request_24 project_5\"],[2,\"MergeRequest 46 merge_request_46 project_6\"],[3,\"MergeRequest 70 merge_request_70 project_18\"]]"}
{"severity":"DEBUG","time":"2021-07-23T20:22:50.531Z","correlation_id":null,"class":"Elastic::ProcessInitialBookkeepingService","redis_set":"elastic:bulk:initial:9:zset","message":"track_items","count":7,"tracked_items_encoded":"[[1,\"MergeRequest 22 merge_request_22 project_5\"],[2,\"MergeRequest 53 merge_request_53 project_10\"],[3,\"MergeRequest 39 merge_request_39 project_3\"],[4,\"MergeRequest 45 merge_request_45 project_6\"],[5,\"MergeRequest 35 merge_request_35 project_1\"],[6,\"MergeRequest 8 merge_request_8 project_2\"],[7,\"MergeRequest 10 merge_request_10 project_2\"]]"}
{"severity":"DEBUG","time":"2021-07-23T20:22:50.532Z","correlation_id":null,"class":"Elastic::ProcessInitialBookkeepingService","redis_set":"elastic:bulk:initial:0:zset","message":"track_items","count":1,"tracked_items_encoded":"[[1,\"MergeRequest 20 merge_request_20 project_3\"]]"}
{"severity":"DEBUG","time":"2021-07-23T20:22:50.533Z","correlation_id":null,"class":"Elastic::ProcessInitialBookkeepingService","redis_set":"elastic:bulk:initial:1:zset","message":"track_items","count":3,"tracked_items_encoded":"[[1,\"MergeRequest 17 merge_request_17 project_3\"],[2,\"MergeRequest 25 merge_request_25 project_6\"],[3,\"MergeRequest 14 merge_request_14 project_2\"]]"}
{"severity":"DEBUG","time":"2021-07-23T20:22:50.534Z","correlation_id":null,"class":"Elastic::ProcessInitialBookkeepingService","redis_set":"elastic:bulk:initial:7:zset","message":"track_items","count":4,"tracked_items_encoded":"[[1,\"MergeRequest 15 merge_request_15 project_3\"],[2,\"MergeRequest 27 merge_request_27 project_6\"],[3,\"MergeRequest 13 merge_request_13 project_2\"],[4,\"MergeRequest 31 merge_request_31 project_8\"]]"}
{"severity":"DEBUG","time":"2021-07-23T20:22:50.534Z","correlation_id":null,"class":"Elastic::ProcessInitialBookkeepingService","redis_set":"elastic:bulk:initial:8:zset","message":"track_items","count":4,"tracked_items_encoded":"[[1,\"MergeRequest 21 merge_request_21 project_3\"],[2,\"MergeRequest 26 merge_request_26 project_6\"],[3,\"MergeRequest 33 merge_request_33 project_1\"],[4,\"MergeRequest 11 merge_request_11 project_2\"]]"}
{"severity":"DEBUG","time":"2021-07-23T20:22:50.535Z","correlation_id":null,"class":"Elastic::ProcessInitialBookkeepingService","redis_set":"elastic:bulk:initial:13:zset","message":"track_items","count":2,"tracked_items_encoded":"[[1,\"MergeRequest 40 merge_request_40 project_3\"],[2,\"MergeRequest 49 merge_request_49 project_8\"]]"}
{"severity":"DEBUG","time":"2021-07-23T20:22:50.536Z","correlation_id":null,"class":"Elastic::ProcessInitialBookkeepingService","redis_set":"elastic:bulk:initial:15:zset","message":"track_items","count":6,"tracked_items_encoded":"[[1,\"MergeRequest 41 merge_request_41 project_4\"],[2,\"MergeRequest 60 merge_request_60 project_13\"],[3,\"MergeRequest 67 merge_request_67 project_17\"],[4,\"MergeRequest 69 merge_request_69 project_18\"],[5,\"MergeRequest 58 merge_request_58 project_12\"],[6,\"MergeRequest 64 merge_request_64 project_15\"]]"}
{"severity":"DEBUG","time":"2021-07-23T20:22:50.537Z","correlation_id":null,"class":"Elastic::ProcessInitialBookkeepingService","redis_set":"elastic:bulk:initial:14:zset","message":"track_items","count":7,"tracked_items_encoded":"[[1,\"MergeRequest 57 merge_request_57 project_12\"],[2,\"MergeRequest 66 merge_request_66 project_16\"],[3,\"MergeRequest 2 merge_request_2 project_1\"],[4,\"MergeRequest 3 merge_request_3 project_1\"],[5,\"MergeRequest 7 merge_request_7 project_1\"],[6,\"MergeRequest 36 merge_request_36 project_1\"],[7,\"MergeRequest 55 merge_request_55 project_11\"]]"}
{"severity":"INFO","time":"2021-07-23T20:22:50.537Z","correlation_id":null,"message":"[Elastic::Migration: 20210722112600] Adding upvotes field to merge requests documents is completed for batch of 70 documents"}
{"severity":"INFO","time":"2021-07-23T20:22:50.570Z","correlation_id":null,"message":"[Elastic::Migration: 20210722112600] Checking if there are documents without upvotes field: 70 documents left"}
{"severity":"INFO","time":"2021-07-23T20:22:50.570Z","correlation_id":null,"message":"MigrationWorker: migration[AddUpvotesToMergeRequests] kicking off next migration batch"}
{"severity":"INFO","time":"2021-07-23T20:22:50.597Z","correlation_id":null,"message":"[Elastic::Migration: 20210722112600] Checking if there are documents without upvotes field: 70 documents left"}
{"severity":"INFO","time":"2021-07-23T20:22:50.597Z","correlation_id":null,"message":"MigrationWorker: migration[AddUpvotesToMergeRequests] updating with completed: false"}
{"severity":"ERROR","time":"2021-07-23T20:22:50.613Z","correlation_id":null,"message":"[Elastic::MigrationRecord]: Elasticsearch::Transport::Transport::Errors::NotFound: [404] {\"_index\":\"gitlab-development-migrations\",\"_type\":\"_doc\",\"_id\":\"20210722112600\",\"found\":false}"}
{"severity":"ERROR","time":"2021-07-23T20:22:50.621Z","correlation_id":null,"message":"[Elastic::MigrationRecord]: Elasticsearch::Transport::Transport::Errors::NotFound: [404] {\"_index\":\"gitlab-development-migrations\",\"_type\":\"_doc\",\"_id\":\"20210722112600\",\"found\":false}"}
{"severity":"INFO","time":"2021-07-23T20:23:04.854Z","correlation_id":"39d80b413bfec098a08e387737b70046","message":"bulk_indexing_start","redis_set":"elastic:bulk:initial:0:zset","records_count":1,"first_score":1.0,"last_score":1.0}
{"severity":"INFO","time":"2021-07-23T20:23:04.855Z","correlation_id":"39d80b413bfec098a08e387737b70046","message":"bulk_indexing_start","redis_set":"elastic:bulk:initial:1:zset","records_count":3,"first_score":1.0,"last_score":3.0}
{"severity":"INFO","time":"2021-07-23T20:23:04.856Z","correlation_id":"39d80b413bfec098a08e387737b70046","message":"bulk_indexing_start","redis_set":"elastic:bulk:initial:2:zset","records_count":6,"first_score":1.0,"last_score":6.0}
{"severity":"INFO","time":"2021-07-23T20:23:04.856Z","correlation_id":"39d80b413bfec098a08e387737b70046","message":"bulk_indexing_start","redis_set":"elastic:bulk:initial:3:zset","records_count":4,"first_score":1.0,"last_score":4.0}
{"severity":"INFO","time":"2021-07-23T20:23:04.857Z","correlation_id":"39d80b413bfec098a08e387737b70046","message":"bulk_indexing_start","redis_set":"elastic:bulk:initial:4:zset","records_count":5,"first_score":1.0,"last_score":5.0}
{"severity":"INFO","time":"2021-07-23T20:23:04.857Z","correlation_id":"39d80b413bfec098a08e387737b70046","message":"bulk_indexing_start","redis_set":"elastic:bulk:initial:5:zset","records_count":4,"first_score":1.0,"last_score":4.0}
{"severity":"INFO","time":"2021-07-23T20:23:04.857Z","correlation_id":"39d80b413bfec098a08e387737b70046","message":"bulk_indexing_start","redis_set":"elastic:bulk:initial:6:zset","records_count":3,"first_score":1.0,"last_score":3.0}
{"severity":"INFO","time":"2021-07-23T20:23:04.858Z","correlation_id":"39d80b413bfec098a08e387737b70046","message":"bulk_indexing_start","redis_set":"elastic:bulk:initial:7:zset","records_count":4,"first_score":1.0,"last_score":4.0}
{"severity":"INFO","time":"2021-07-23T20:23:04.858Z","correlation_id":"39d80b413bfec098a08e387737b70046","message":"bulk_indexing_start","redis_set":"elastic:bulk:initial:8:zset","records_count":4,"first_score":1.0,"last_score":4.0}
{"severity":"INFO","time":"2021-07-23T20:23:04.859Z","correlation_id":"39d80b413bfec098a08e387737b70046","message":"bulk_indexing_start","redis_set":"elastic:bulk:initial:9:zset","records_count":7,"first_score":1.0,"last_score":7.0}
{"severity":"INFO","time":"2021-07-23T20:23:04.859Z","correlation_id":"39d80b413bfec098a08e387737b70046","message":"bulk_indexing_start","redis_set":"elastic:bulk:initial:10:zset","records_count":4,"first_score":1.0,"last_score":4.0}
{"severity":"INFO","time":"2021-07-23T20:23:04.860Z","correlation_id":"39d80b413bfec098a08e387737b70046","message":"bulk_indexing_start","redis_set":"elastic:bulk:initial:11:zset","records_count":5,"first_score":1.0,"last_score":5.0}
{"severity":"INFO","time":"2021-07-23T20:23:04.860Z","correlation_id":"39d80b413bfec098a08e387737b70046","message":"bulk_indexing_start","redis_set":"elastic:bulk:initial:12:zset","records_count":5,"first_score":1.0,"last_score":5.0}
{"severity":"INFO","time":"2021-07-23T20:23:04.860Z","correlation_id":"39d80b413bfec098a08e387737b70046","message":"bulk_indexing_start","redis_set":"elastic:bulk:initial:13:zset","records_count":2,"first_score":1.0,"last_score":2.0}
{"severity":"INFO","time":"2021-07-23T20:23:04.861Z","correlation_id":"39d80b413bfec098a08e387737b70046","message":"bulk_indexing_start","redis_set":"elastic:bulk:initial:14:zset","records_count":7,"first_score":1.0,"last_score":7.0}
{"severity":"INFO","time":"2021-07-23T20:23:04.861Z","correlation_id":"39d80b413bfec098a08e387737b70046","message":"bulk_indexing_start","redis_set":"elastic:bulk:initial:15:zset","records_count":6,"first_score":1.0,"last_score":6.0}
{"severity":"INFO","time":"2021-07-23T20:23:05.875Z","correlation_id":"39d80b413bfec098a08e387737b70046","message":"bulk_submitted","body_size_bytes":46945,"bulk_count":70,"errors_count":0}
{"severity":"INFO","time":"2021-07-23T20:23:05.891Z","correlation_id":"39d80b413bfec098a08e387737b70046","message":"bulk_indexing_end","redis_set":"elastic:bulk:initial:0:zset","records_count":1,"first_score":1.0,"last_score":1.0,"failures_count":0,"bulk_execution_duration_s":1.746009}
{"severity":"INFO","time":"2021-07-23T20:23:05.911Z","correlation_id":"39d80b413bfec098a08e387737b70046","message":"bulk_indexing_end","redis_set":"elastic:bulk:initial:1:zset","records_count":3,"first_score":1.0,"last_score":3.0,"failures_count":0,"bulk_execution_duration_s":1.765355}
{"severity":"INFO","time":"2021-07-23T20:23:05.942Z","correlation_id":"39d80b413bfec098a08e387737b70046","message":"bulk_indexing_end","redis_set":"elastic:bulk:initial:2:zset","records_count":6,"first_score":1.0,"last_score":6.0,"failures_count":0,"bulk_execution_duration_s":1.79716}
{"severity":"INFO","time":"2021-07-23T20:23:05.947Z","correlation_id":"39d80b413bfec098a08e387737b70046","message":"bulk_indexing_end","redis_set":"elastic:bulk:initial:3:zset","records_count":4,"first_score":1.0,"last_score":4.0,"failures_count":0,"bulk_execution_duration_s":1.802239}
{"severity":"INFO","time":"2021-07-23T20:23:05.948Z","correlation_id":"39d80b413bfec098a08e387737b70046","message":"bulk_indexing_end","redis_set":"elastic:bulk:initial:4:zset","records_count":5,"first_score":1.0,"last_score":5.0,"failures_count":0,"bulk_execution_duration_s":1.802795}
{"severity":"INFO","time":"2021-07-23T20:23:05.948Z","correlation_id":"39d80b413bfec098a08e387737b70046","message":"bulk_indexing_end","redis_set":"elastic:bulk:initial:5:zset","records_count":4,"first_score":1.0,"last_score":4.0,"failures_count":0,"bulk_execution_duration_s":1.803297}
{"severity":"INFO","time":"2021-07-23T20:23:05.949Z","correlation_id":"39d80b413bfec098a08e387737b70046","message":"bulk_indexing_end","redis_set":"elastic:bulk:initial:6:zset","records_count":3,"first_score":1.0,"last_score":3.0,"failures_count":0,"bulk_execution_duration_s":1.803727}
{"severity":"INFO","time":"2021-07-23T20:23:05.949Z","correlation_id":"39d80b413bfec098a08e387737b70046","message":"bulk_indexing_end","redis_set":"elastic:bulk:initial:7:zset","records_count":4,"first_score":1.0,"last_score":4.0,"failures_count":0,"bulk_execution_duration_s":1.804171}
{"severity":"INFO","time":"2021-07-23T20:23:05.950Z","correlation_id":"39d80b413bfec098a08e387737b70046","message":"bulk_indexing_end","redis_set":"elastic:bulk:initial:8:zset","records_count":4,"first_score":1.0,"last_score":4.0,"failures_count":0,"bulk_execution_duration_s":1.804709}
{"severity":"INFO","time":"2021-07-23T20:23:05.950Z","correlation_id":"39d80b413bfec098a08e387737b70046","message":"bulk_indexing_end","redis_set":"elastic:bulk:initial:9:zset","records_count":7,"first_score":1.0,"last_score":7.0,"failures_count":0,"bulk_execution_duration_s":1.805179}
{"severity":"INFO","time":"2021-07-23T20:23:05.951Z","correlation_id":"39d80b413bfec098a08e387737b70046","message":"bulk_indexing_end","redis_set":"elastic:bulk:initial:10:zset","records_count":4,"first_score":1.0,"last_score":4.0,"failures_count":0,"bulk_execution_duration_s":1.805648}
{"severity":"INFO","time":"2021-07-23T20:23:05.951Z","correlation_id":"39d80b413bfec098a08e387737b70046","message":"bulk_indexing_end","redis_set":"elastic:bulk:initial:11:zset","records_count":5,"first_score":1.0,"last_score":5.0,"failures_count":0,"bulk_execution_duration_s":1.8062}
{"severity":"INFO","time":"2021-07-23T20:23:05.952Z","correlation_id":"39d80b413bfec098a08e387737b70046","message":"bulk_indexing_end","redis_set":"elastic:bulk:initial:12:zset","records_count":5,"first_score":1.0,"last_score":5.0,"failures_count":0,"bulk_execution_duration_s":1.806663}
{"severity":"INFO","time":"2021-07-23T20:23:05.952Z","correlation_id":"39d80b413bfec098a08e387737b70046","message":"bulk_indexing_end","redis_set":"elastic:bulk:initial:13:zset","records_count":2,"first_score":1.0,"last_score":2.0,"failures_count":0,"bulk_execution_duration_s":1.807212}
{"severity":"INFO","time":"2021-07-23T20:23:05.953Z","correlation_id":"39d80b413bfec098a08e387737b70046","message":"bulk_indexing_end","redis_set":"elastic:bulk:initial:14:zset","records_count":7,"first_score":1.0,"last_score":7.0,"failures_count":0,"bulk_execution_duration_s":1.807776}
{"severity":"INFO","time":"2021-07-23T20:23:05.954Z","correlation_id":"39d80b413bfec098a08e387737b70046","message":"bulk_indexing_end","redis_set":"elastic:bulk:initial:15:zset","records_count":6,"first_score":1.0,"last_score":6.0,"failures_count":0,"bulk_execution_duration_s":1.808368}
{"severity":"INFO","time":"2021-07-23T20:23:14.458Z","correlation_id":null,"message":"MigrationWorker: migration[AddUpvotesToMergeRequests] executing migrate method"}
{"severity":"INFO","time":"2021-07-23T20:23:14.478Z","correlation_id":null,"message":"[Elastic::Migration: 20210722112600] Checking if there are documents without upvotes field: 0 documents left"}
{"severity":"INFO","time":"2021-07-23T20:23:14.478Z","correlation_id":null,"message":"[Elastic::Migration: 20210722112600] Skipping adding upvotes field to merge requests documents migration since it is already applied"}
{"severity":"INFO","time":"2021-07-23T20:23:14.485Z","correlation_id":null,"message":"[Elastic::Migration: 20210722112600] Checking if there are documents without upvotes field: 0 documents left"}
{"severity":"INFO","time":"2021-07-23T20:23:14.492Z","correlation_id":null,"message":"[Elastic::Migration: 20210722112600] Checking if there are documents without upvotes field: 0 documents left"}
{"severity":"INFO","time":"2021-07-23T20:23:14.492Z","correlation_id":null,"message":"MigrationWorker: migration[AddUpvotesToMergeRequests] updating with completed: true"}

Screenshots or Screencasts (strongly suggested)

Non-UI change

Does this MR meet the acceptance criteria?

Conformity

Availability and Testing

Security

Does this MR contain changes to processing or storing of credentials or tokens, authorization and authentication methods or other items described in the security review guidelines? If not, then delete this Security section.

  • Label as security and @ mention @gitlab-com/gl-security/appsec
  • The MR includes necessary changes to maintain consistency between UI, API, email, or other methods
  • Security reports checked/validated by a reviewer from the AppSec team
Edited by Terri Chu

Merge request reports