BackfillProjectPermissionsInBlobs has been halted

Summary

The BackfillProjectPermissionsInBlobs migration has been halted

!117224 (merged)

Steps to reproduce

What is the current bug behavior?

The migration has failed on production and has been marked as halted

What is the expected correct behavior?

It should successfully complete

Relevant logs and/or screenshots

Click to expand

[ gprd ] production> Elastic::MigrationRecord.current_migration
=>
#<Elastic::MigrationRecord:0x00007f7053fad2a8
 @filename=
  "/opt/gitlab/embedded/service/gitlab-rails/ee/elastic/migrate/20230411064300_backfill_project_permissions_in_blobs.rb",
 @migration=nil,
 @name="BackfillProjectPermissionsInBlobs",
 @version=20230411064300>
[ gprd ] production> Elastic::MigrationRecord.current_migration.load_state
=>
{"projects_in_progress"=>
  [{"task_id"=>"0C6ijkHVTw2PKKvTZbGbbQ:551556926", "project_id"=>20874998},
   {"task_id"=>"rczdV00SReGFkOT-sWcO-Q:519343745", "project_id"=>41751234},
   {"task_id"=>"bVr8EQeTSG2dv7Pyae4JYQ:490972147", "project_id"=>21201171},
   {"task_id"=>"4Nz2d3PoTueC-nXdXLKnkA:540530825", "project_id"=>8094352},
   {"task_id"=>"8L5Q12DaSbmiNc3-6fN-FA:486402471", "project_id"=>39850717},
   {"task_id"=>"8L5Q12DaSbmiNc3-6fN-FA:486402491", "project_id"=>22496430},
   {"task_id"=>"8L5Q12DaSbmiNc3-6fN-FA:486402495", "project_id"=>40174028},
   {"task_id"=>"4Nz2d3PoTueC-nXdXLKnkA:540530870", "project_id"=>26950433},
   {"task_id"=>"rczdV00SReGFkOT-sWcO-Q:519343827", "project_id"=>21201200},
   {"task_id"=>"8L5Q12DaSbmiNc3-6fN-FA:486402545", "project_id"=>26812405},
   {"task_id"=>"4Nz2d3PoTueC-nXdXLKnkA:540530902", "project_id"=>15963925},
   {"task_id"=>"4Nz2d3PoTueC-nXdXLKnkA:540530915", "project_id"=>29643596},
   {"task_id"=>"gOUN-D5LRcOxobqE_9tHYA:471072881", "project_id"=>29586073},
   {"task_id"=>"4Nz2d3PoTueC-nXdXLKnkA:540530930", "project_id"=>23223441},
   {"task_id"=>"bVr8EQeTSG2dv7Pyae4JYQ:490972286", "project_id"=>41688356},
   {"task_id"=>"bVr8EQeTSG2dv7Pyae4JYQ:490972293", "project_id"=>41772875},
   {"task_id"=>"4Nz2d3PoTueC-nXdXLKnkA:540530952", "project_id"=>21201328},
   {"task_id"=>"8L5Q12DaSbmiNc3-6fN-FA:486402601", "project_id"=>21201303},
   {"task_id"=>"rczdV00SReGFkOT-sWcO-Q:519343896", "project_id"=>21201011},
   {"task_id"=>"gOUN-D5LRcOxobqE_9tHYA:471072909", "project_id"=>33928193},
   {"task_id"=>"8L5Q12DaSbmiNc3-6fN-FA:486402613", "project_id"=>21201099},
   {"task_id"=>"rczdV00SReGFkOT-sWcO-Q:519343908", "project_id"=>14936198},
   {"task_id"=>"0C6ijkHVTw2PKKvTZbGbbQ:551557129", "project_id"=>44006831},
   {"task_id"=>"4Nz2d3PoTueC-nXdXLKnkA:540530984", "project_id"=>20651566},
   {"task_id"=>"4Nz2d3PoTueC-nXdXLKnkA:540531002", "project_id"=>16456533},
   {"task_id"=>"0C6ijkHVTw2PKKvTZbGbbQ:551557176", "project_id"=>4288352},
   {"task_id"=>"bVr8EQeTSG2dv7Pyae4JYQ:490972387", "project_id"=>36526808},
   {"task_id"=>"4Nz2d3PoTueC-nXdXLKnkA:540531025", "project_id"=>24360210},
   {"task_id"=>"8L5Q12DaSbmiNc3-6fN-FA:486402742", "project_id"=>26977995},
   {"task_id"=>"rczdV00SReGFkOT-sWcO-Q:519343993", "project_id"=>20763437},
   {"task_id"=>"4Nz2d3PoTueC-nXdXLKnkA:540531070", "project_id"=>6563721},
   {"task_id"=>"4Nz2d3PoTueC-nXdXLKnkA:540531072", "project_id"=>35770966},
   {"task_id"=>"8L5Q12DaSbmiNc3-6fN-FA:486402773", "project_id"=>28938680},
   {"task_id"=>"bVr8EQeTSG2dv7Pyae4JYQ:490972474", "project_id"=>38815778},
   {"task_id"=>"gOUN-D5LRcOxobqE_9tHYA:471073081", "project_id"=>33283873},
   {"task_id"=>"4Nz2d3PoTueC-nXdXLKnkA:540531134", "project_id"=>45133306},
   {"task_id"=>"4Nz2d3PoTueC-nXdXLKnkA:540531170", "project_id"=>12009279}],
 "remaining_count"=>948904619,
 "previous_attempts"=>29,
 "halted"=>true,
 "halted_indexing_unpaused"=>false,
 "message"=>
  "MigrationWorker: migration has failed with NoMethodError:undefined method `<<' for nil:NilClass, no retries left",
 "failed"=>true}

Possible fixes