gitlab-org/gitlab project returning 404

Please note: if the incident relates to sensitive data, or is security related consider labeling this issue with security and mark it confidential.

---

Summary

Accidental delete fo the gitlab-org/gitlab project - working on restoring the project.

Service(s) affected : gitlab-org/gitlab project Team attribution : Minutes downtime or degradation :

Timeline

2019-09-24

• 00:10:53 UTC - accidental api request for delete
• 00:12 UTC - reports of 404 on gitlab-org/gitlab
• 00:13 UTC - identified the issues- a team member accidentally marked the project for delete when trying to delete another artifact using the API
• 00:30 UTC - found project in DB as marked for delete- stopping all background processing temporarily
• 00:31 UTC - found project on file server in moved/marked for delete state - backing up
• 00:34 UTC - Looking for the job that would do the delete to make sure it does not process the delete.
• 00:51 UTC - Removing the job that would do the delete
• 00:54 UTC - gitlab-org/gitlab is back with error "This project was scheduled for deletion, but failed with the following message: Sidekiq::Shutdown" due to the actions at 00:30 UTC
• 01:00 UTC - we removed the primary delete job.
• 01:10 UTC - restarted all but best-effort sidekiq nodes (which process deletes)
• 01:10 UTC - working to remove all other delete jobs from the sidekiq queues
• 01:20 UTC - restarted best-effort sidekiq nodes
• 01:45 UTC - the job retried in sidekiq and the project is again 404ing
• 01:50 UTC - stopped all sidekiq jobs and changing user permissions for the job attempting the delete.
• 01:59 UTC - moved back git files on the file stor node and project is not 404 ing
• 02:00 UTC - restarting sidekiq, with permissions changed for user that would perform the delete
• 02:40 UTC - all pending jobs appear to have been cleared, object pools and banners look okay. marking as resovled.

added incident severity1 labels

changed the description

Log entry of API call to delete project: https://log.gitlab.net/app/kibana#/doc/AWOSvARQwig0Nc2UGcr2/pubsub-rails-inf-gprd-2019.09.24-000001/doc?id=AW1gm7mjt-NbMNWQGu81&_g=h@44136fa

For the record:

"message": "{\"severity\":\"INFO\",\"time\":\"2019-09-24T00:10:53.745Z\",\"class\":\"ProjectDestroyWorker\",\"args\":[\"278964\",\"3403222\",\"{}\"],\"retry\":3,\"queue\":\"project_destroy\",\"backtrace\":5,\"jid\":\"aef63c6130d26f4fab394159\",\"created_at\":\"2019-09-24T00:10:53.743Z\",\"correlation_id\":\"DZw8LMRTJ79\",\"enqueued_at\":\"2019-09-24T00:10:53.743Z\",\"pid\":19907,\"message\":\"ProjectDestroyWorker JID-aef63c6130d26f4fab394159: start\",\"job_status\":\"start\",\"scheduling_latency_s\":0.003,\"tag\":\"sidekiq\",\"environment\":\"gprd\",\"hostname\":\"sidekiq-besteffort-01-sv-gprd\",\"fqdn\":\"sidekiq-besteffort-01-sv-gprd.c.gitlab-production.internal\"}"```

changed the description

changed the description

changed the description

Hi @dawsmith,

This incident issue does not have any service attribution. Please add one or more of the appropriate service label that are prefixed with Service:.

Thanks for your help!

---

You are welcome to help improve this comment.

changed the description

changed the description

changed the description

closed

Zoom notes and other slack notes from group: https://docs.google.com/document/d/173bPRhsm7lviP5rtFJItA7J5PXUE495XsXkzm7Lp7oA/edit

mentioned in issue #1184 (closed)

mentioned in issue gitlab-org/gitlab#32592 (closed)

Gonna reopen this, the fork network is still broken.

[ gprd ] production> proj = Project.find_by_full_path('gitlab-org/gitlab')
=> #<Project id:278964 gitlab-org/gitlab>
[ gprd ] production> proj.fork_network
=> #<ForkNetwork id: 184665, root_project_id: 278964, deleted_root_project_name: nil>
[ gprd ] production> nick = Project.find_by_full_path('nick.thomas/gitlab-ee')
=> #<Project id:1471537 nick.thomas/gitlab-ee>
[ gprd ] production> nick.fork_network
=> #<ForkNetwork id: 2949, root_project_id: nil, deleted_root_project_name: "GitLab.org / GitLab">
[ gprd ] production> repra = Project.find_by_full_path('reprazent/gitlab-ee')
=> #<Project id:5776166 reprazent/gitlab-ee>
[ gprd ] production> repra.fork_network
=> #<ForkNetwork id: 2949, root_project_id: nil, deleted_root_project_name: "GitLab.org / GitLab">

reopened

The following snippet was suggested by @reprazent to recover the situation:

original_fork_network = ForkNetwork.find(2949)
new_fork_network = ForkNetwork.find(184665)

new_fork_network.fork_network_members.update_all(fork_network_id: original_fork_network.id)
original_fork_network.update(root_project_id: 278964, deleted_root_project_name: nil)

# Check if all the new forks have joined the old network
new_fork_network.reload.fork_network_members.size

# if ok
new_fork_network.destroy

@T4cC0re are you okay with this change ?

Looking at it now

@toon

Do I read it correctly?
It would update all projects, where the forknetwork is 184665 to 2949, then set the root_project_id of the 2949 fork network to 278964 lastly check and delete the 184665 fork network?

@T4cC0re It would update all ForkNetworkMember rows for which the fork network is is 184665. There currently are 3 of them: 2 forks and the root (gitlab-org/gitlab). It would make those projects "join" the old ForkNetwork (2949).

Then it would update the "old forknetwork" to have gitlab-org/gitlab as its root, as that was marked as deleted before.

Then we check the newly created fork network, it should now be empty. If it's empty, we can safely delete it, if it isn't, someone forked gitlab-org/gitlab again while we were doing this. So we'd need to step through again.

@reprazent Looks good to me, but wouldn't it make more sense to update the "old forknetwork" to have gitlab-org/gitlab as its root first?

Anyways, who will execute this? I can do it, but I am not really that deep into rails. So I'd prefer if someone who knows what they are doing could execute this.

but wouldn't it make more sense to update the "old forknetwork" to have gitlab-org/gitlab as its root first?

The "root" needs to be a member of the fork network before we can do that.

Anyways, who will execute this? I can do it, but I am not really that deep into rails. So I'd prefer if someone who knows what they are doing could execute this.

I don't have production access, and I wrote the snippet. So I'm not the best person either, we could ask @toon who helped me debug this, or @nick.thomas who discovered it. But they're both in calls right now.

mentioned in issue gitlab-org/quality/triage-ops#332 (closed)

mentioned in issue gitlab-org/gitlab#32647

I opened up gitlab-org/gitlab#32647 to evaluate the possibility to disable the DELETE method on a project on a per project basis.

I ran the commands as suggested: #1183 (comment 221094063)

@reprazent confirmed his fork is working again.

closed

marked this issue as related to gitlab-org/gitlab#32647

mentioned in issue delivery#350

marked this issue as related to #1187 (closed)

changed the description

mentioned in issue gitlab-org/gitlab#33085 (closed)

mentioned in issue gitlab-org/gitlab#32935 (closed)

added ServiceGitaly label

added ServiceGit label and removed ServiceGitaly label

This incident was closed before the IncidentResolved label was applied applied—or after a review was started by not completed. The issue is being reopened so that it will appear on the Production Incidents board. Please mark it as either IncidentResolved or IncidentReview-Completed before closing.

Please review the Incident Workflow section on the Incident Management handbook page for more information.

/open

changed the description

added auto updated label

added review-requested label

added IncidentReview-Completed label