BackfillProjectPermissionsInBlobs has been halted
Summary
The BackfillProjectPermissionsInBlobs
migration has been halted
Steps to reproduce
What is the current bug behavior?
The migration has failed on production and has been marked as halted
What is the expected correct behavior?
It should successfully complete
Relevant logs and/or screenshots
Click to expand
[ gprd ] production> Elastic::MigrationRecord.current_migration
=>
#<Elastic::MigrationRecord:0x00007f7053fad2a8
@filename=
"/opt/gitlab/embedded/service/gitlab-rails/ee/elastic/migrate/20230411064300_backfill_project_permissions_in_blobs.rb",
@migration=nil,
@name="BackfillProjectPermissionsInBlobs",
@version=20230411064300>
[ gprd ] production> Elastic::MigrationRecord.current_migration.load_state
=>
{"projects_in_progress"=>
[{"task_id"=>"0C6ijkHVTw2PKKvTZbGbbQ:551556926", "project_id"=>20874998},
{"task_id"=>"rczdV00SReGFkOT-sWcO-Q:519343745", "project_id"=>41751234},
{"task_id"=>"bVr8EQeTSG2dv7Pyae4JYQ:490972147", "project_id"=>21201171},
{"task_id"=>"4Nz2d3PoTueC-nXdXLKnkA:540530825", "project_id"=>8094352},
{"task_id"=>"8L5Q12DaSbmiNc3-6fN-FA:486402471", "project_id"=>39850717},
{"task_id"=>"8L5Q12DaSbmiNc3-6fN-FA:486402491", "project_id"=>22496430},
{"task_id"=>"8L5Q12DaSbmiNc3-6fN-FA:486402495", "project_id"=>40174028},
{"task_id"=>"4Nz2d3PoTueC-nXdXLKnkA:540530870", "project_id"=>26950433},
{"task_id"=>"rczdV00SReGFkOT-sWcO-Q:519343827", "project_id"=>21201200},
{"task_id"=>"8L5Q12DaSbmiNc3-6fN-FA:486402545", "project_id"=>26812405},
{"task_id"=>"4Nz2d3PoTueC-nXdXLKnkA:540530902", "project_id"=>15963925},
{"task_id"=>"4Nz2d3PoTueC-nXdXLKnkA:540530915", "project_id"=>29643596},
{"task_id"=>"gOUN-D5LRcOxobqE_9tHYA:471072881", "project_id"=>29586073},
{"task_id"=>"4Nz2d3PoTueC-nXdXLKnkA:540530930", "project_id"=>23223441},
{"task_id"=>"bVr8EQeTSG2dv7Pyae4JYQ:490972286", "project_id"=>41688356},
{"task_id"=>"bVr8EQeTSG2dv7Pyae4JYQ:490972293", "project_id"=>41772875},
{"task_id"=>"4Nz2d3PoTueC-nXdXLKnkA:540530952", "project_id"=>21201328},
{"task_id"=>"8L5Q12DaSbmiNc3-6fN-FA:486402601", "project_id"=>21201303},
{"task_id"=>"rczdV00SReGFkOT-sWcO-Q:519343896", "project_id"=>21201011},
{"task_id"=>"gOUN-D5LRcOxobqE_9tHYA:471072909", "project_id"=>33928193},
{"task_id"=>"8L5Q12DaSbmiNc3-6fN-FA:486402613", "project_id"=>21201099},
{"task_id"=>"rczdV00SReGFkOT-sWcO-Q:519343908", "project_id"=>14936198},
{"task_id"=>"0C6ijkHVTw2PKKvTZbGbbQ:551557129", "project_id"=>44006831},
{"task_id"=>"4Nz2d3PoTueC-nXdXLKnkA:540530984", "project_id"=>20651566},
{"task_id"=>"4Nz2d3PoTueC-nXdXLKnkA:540531002", "project_id"=>16456533},
{"task_id"=>"0C6ijkHVTw2PKKvTZbGbbQ:551557176", "project_id"=>4288352},
{"task_id"=>"bVr8EQeTSG2dv7Pyae4JYQ:490972387", "project_id"=>36526808},
{"task_id"=>"4Nz2d3PoTueC-nXdXLKnkA:540531025", "project_id"=>24360210},
{"task_id"=>"8L5Q12DaSbmiNc3-6fN-FA:486402742", "project_id"=>26977995},
{"task_id"=>"rczdV00SReGFkOT-sWcO-Q:519343993", "project_id"=>20763437},
{"task_id"=>"4Nz2d3PoTueC-nXdXLKnkA:540531070", "project_id"=>6563721},
{"task_id"=>"4Nz2d3PoTueC-nXdXLKnkA:540531072", "project_id"=>35770966},
{"task_id"=>"8L5Q12DaSbmiNc3-6fN-FA:486402773", "project_id"=>28938680},
{"task_id"=>"bVr8EQeTSG2dv7Pyae4JYQ:490972474", "project_id"=>38815778},
{"task_id"=>"gOUN-D5LRcOxobqE_9tHYA:471073081", "project_id"=>33283873},
{"task_id"=>"4Nz2d3PoTueC-nXdXLKnkA:540531134", "project_id"=>45133306},
{"task_id"=>"4Nz2d3PoTueC-nXdXLKnkA:540531170", "project_id"=>12009279}],
"remaining_count"=>948904619,
"previous_attempts"=>29,
"halted"=>true,
"halted_indexing_unpaused"=>false,
"message"=>
"MigrationWorker: migration has failed with NoMethodError:undefined method `<<' for nil:NilClass, no retries left",
"failed"=>true}
Possible fixes
Edited by Dmitry Gruzd