Repository mirroring from GitHub not updating

A customer has reported that the pull mirror from GitHub is not updating the repository and it still shows the last updated. Status is was showing last updated: 20 hours ago. It didn't fail either.

Errors we have observed.
What we have tried.
  • created a new branch on GitHub and check if it's updating. But it didn't.
  • tested removing the mirroring and re-adding it to the same project but that did not work.
  • manually mirrored the GH repo to a new group in GL -> this worked perfectly.
  • manual API Pull curl command. No effect.
  • manual git clone showed only 7-8 seconds to clone.
  • We forced the mirror update manually. No impact.
What worked.
  • We first checked the import state and saw what' the current status. It was marked as scheduled.
[ gprd ] production> pp _
#<ProjectImportState:0x00007f17d8371210
 project_id: SOMEID,
 retry_count: 0,
 last_update_started_at: Tue, 05 Oct 2021 07:41:41.371181000 UTC +00:00,
 last_update_scheduled_at: Tue, 05 Oct 2021 08:12:14.572174000 UTC +00:00,
 next_execution_timestamp: Wed, 06 Oct 2021 04:05:28.557560000 UTC +00:00,
 id: 3057250,
 status: "scheduled",
 jid: "REDACTED",
 last_error: nil,
 last_update_at: Tue, 05 Oct 2021 07:41:42.365357000 UTC +00:00,
 last_successful_update_at: Tue, 05 Oct 2021 07:41:42.365357000 UTC +00:00,
 correlation_id_value: "f2214e89f8437ec5408ac8c360235754">
  • Next we forced the import to fail from the backend and marked the import as failed (unsuccessful).
 p.import_state.mark_as_failed("Failed manually through console.")

#<ProjectImportState:0x00007f17d8371210
 project_id: SOMEID,
 retry_count: 0,
 last_update_started_at: Tue, 05 Oct 2021 07:41:41.371181000 UTC +00:00,
 last_update_scheduled_at: Tue, 05 Oct 2021 08:12:14.572174000 UTC +00:00,
 next_execution_timestamp: Wed, 06 Oct 2021 05:02:54.085365553 UTC +00:00,
 id: 3057250,
 status: "failed",
 jid: "REDACTED",
 last_error: "Failed manually through console.",
 last_update_at: Wed, 06 Oct 2021 05:02:54.083505767 UTC +00:00,
 last_successful_update_at: Tue, 05 Oct 2021 07:41:42.365357000 UTC +00:00,
 correlation_id_value: "f2214e89f8437ec5408ac8c360235754">
  • Then we triggered the mirror again and it started working. It also triggered the pipelines.
 p.import_state.force_import_job!
  • We have also reloaded the project state to check and it's started showing correct values as successful
[ gprd ] production> pp _
#<ProjectImportState:0x00007f17d8371210
 project_id: SOMEID,
 retry_count: 0,
 last_update_started_at: Wed, 06 Oct 2021 06:16:20.228570000 UTC +00:00,
 last_update_scheduled_at: Wed, 06 Oct 2021 06:16:20.104082000 UTC +00:00,
 next_execution_timestamp: Wed, 06 Oct 2021 06:46:59.362241000 UTC +00:00,
 id: REDACTED,
 status: "finished",
 jid: "REDACTED",
 last_error: nil,
 last_update_at: Wed, 06 Oct 2021 06:16:21.362191000 UTC +00:00,
 last_successful_update_at: Wed, 06 Oct 2021 06:16:21.362191000 UTC +00:00,
 correlation_id_value: "f9ed58c1fb1e43753e0524dbf7f0cb71">

I have also tested that how many other projects have import status is scheduled and the number is constantly increasing

production> ProjectImportState.with_status([:scheduled]).with_jid.count
D, [2021-10-06T11:14:52.456413 #28364] DEBUG -- :    (4.4ms)  /*application:console,db_config_name:main*/ SELECT COUNT(*) FROM "project_mirror_data" WHERE ("project_mirror_data"."status" IN ('scheduled')) AND "project_mirror_data"."jid" IS NOT NULL
=> 54
Similar issues:

cc// @nick.thomas @tkuah what do you think?

Edited by Alexander Strachan