Unable to Delete or Rename Projects in GitLab (“Internal Error 500”)
Summary
Currently I have two GitLab VMs: one is hosted by a third-party, and the other is installed on a Proxmox VE Server on a server I have hardware access to. I am in the transition on migrating the Projects on the Third-Party GitLab VM to the Server GitLab VM. One big issue I had to resolve early on is that the Server is home to other VMs, and they all require Reverse Proxying through an Nginx VM. I set up Certbot SSL Keys, so the networking side SEEMS fine (but I want to mention this in case its relevant to my current issues).
Right now on the GitLab Server VM, I am unable to delete or rename projects: I receive a "500 - Whoops, something went wrong on our end." in the browser when I try to delete/rename projects in the GitLab GUI. I can import projects just fine (admittedly I'm having a large project of GBs in size to import, and the last time I tried to import is missing files; why I want to delete the current project I imported and try importing again, but obviously the error problems).
When I try to delete projects via GitLab API, running the following command:
curl --request DELETE --header "PRIVATE-TOKEN: <api-suo Personal Access Token>" https://git.domainname.com/api/v4/projects/3
(3 happens to be the ID for a project I want to delete)
I receive a {"message":"500 Internal Server Error"}
back
Here are the contents of my /etc/gitlab/gitlab.rb
file (modified for privacy's sake):
external_url 'https://git.domainname.com'
registry_nginx['redirect_http_to_https'] = true
mattermost_nginx['redirect_http_to_https'] = true
nginx['redirect_http_to_https_port'] = 80
nginx['redirect_http_to_https'] = true
nginx['listen_port'] = 80
nginx['listen_https'] = false
letsencrypt['enable'] = true # If I set this to false, I get an nginx error ... otherwise I would set to false, since Nginx is supposed to handle the SSL certs
letsencrypt['contact_emails'] = ['admin@domainname.com']
letsencrypt['auto_renew'] = true
letsencrypt['auto_renew_hour'] = 12
letsencrypt['auto_renew_minute'] = 30
letsencrypt['auto_renew_day_of_month'] = "*/7"
At the end of the day, I just want to be able to delete and rename projects, and import my project from the Third-Party GitLab VM successfully.
For more details, see the topic I created in the GitLab Forum community: https://forum.gitlab.com/t/gitlab-backup-restore-through-gui/27498 (the posts further down is where I ran into these current problems).
Steps to reproduce
I'm not sure how you could reproduce what has occurred, unless you mirror closely my setup:
- Install an Ubuntu VM with GitLab v11.10.1 (861fe405) (with native Proxmox, vmWare, or some other VM Management OS).
- Setup the GitLab VM to reverse proxy through an Nginx (or Apache) VM
- Import a large project of +20 GB in size from another GitLab (also v11.10.1 (861fe405), because I was told that to import a project from one GitLab to another requires both GitLabs be the same version), but the old GitLab is installed on native Ubuntu Server OS, to the VM with GitLab.
- Attempt to delete and rename the project.
Example Project
Given the size of the project to import, and the unique networking setup with this GitLab on my server, I don't think that's possible.
What is the current bug behavior?
Explained in the Summary.
What is the expected correct behavior?
That files will delete and rename without a 500 Internal Error
. Also, that I am able to import my +20 GB project from the old GitLab I've imported my project from to the new GitLab VM I've set up.
Relevant logs and/or screenshots
Here are the Production Logs when I search for "error" and "Error" (modified for privacy's sake):
root@gitlab:~# less /var/log/gitlab/gitlab-rails/production_json.log | grep error
{"method":"GET","path":"/admin/runners","format":"html","controller":"Admin::RunnersController","action":"index","status":500,"error":"ActionView::Template::Error: ","duration":121.42,"view":0.0,"db":7.32,"time":"2019-08-07T01:42:51.642Z","params":[],"remote_ip":"1.2.3.4","user_id":4,"username":"naupe","ua":"Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:68.0) Gecko/20100101 Firefox/68.0","queue_duration":null,"correlation_id":"hIca7O7LTW5"}
{"method":"GET","path":"/admin/runners","format":"html","controller":"Admin::RunnersController","action":"index","status":500,"error":"ActionView::Template::Error: ","duration":115.21,"view":0.0,"db":7.27,"time":"2019-08-07T01:42:54.503Z","params":[],"remote_ip":"1.2.3.4","user_id":4,"username":"naupe","ua":"Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:68.0) Gecko/20100101 Firefox/68.0","queue_duration":null,"correlation_id":"HAivIvWG7S8"}
{"method":"DELETE","path":"/eg-leaders/Leads_Repository","format":"html","controller":"ProjectsController","action":"destroy","status":500,"error":"OpenSSL::Cipher::CipherError: ","duration":31.56,"view":0.0,"db":3.37,"time":"2019-08-07T01:46:24.697Z","params":[{"key":"_method","value":"delete"},{"key":"authenticity_token","value":"[FILTERED]"},{"key":"projectName","value":"Leads_Repository"},{"key":"namespace_id","value":"eg-leaders"},{"key":"id","value":"Leads_Repository"}],"remote_ip":"1.2.3.4","user_id":4,"username":"naupe","ua":"Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:68.0) Gecko/20100101 Firefox/68.0","queue_duration":null,"correlation_id":"ydplROYmLt4"}
{"method":"DELETE","path":"/eg-leaders/Leads_Repository","format":"html","controller":"ProjectsController","action":"destroy","status":500,"error":"OpenSSL::Cipher::CipherError: ","duration":32.27,"view":0.0,"db":4.92,"time":"2019-08-07T02:00:16.685Z","params":[{"key":"_method","value":"delete"},{"key":"authenticity_token","value":"[FILTERED]"},{"key":"projectName","value":"Leads_Repository"},{"key":"namespace_id","value":"eg-leaders"},{"key":"id","value":"Leads_Repository"}],"remote_ip":"1.2.3.4","user_id":4,"username":"naupe","ua":"Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:68.0) Gecko/20100101 Firefox/68.0","queue_duration":null,"correlation_id":"4yrWRjCyXI5"}
{"method":"DELETE","path":"/eg-leaders/Leads_Repository","format":"html","controller":"ProjectsController","action":"destroy","status":500,"error":"OpenSSL::Cipher::CipherError: ","duration":32.77,"view":0.0,"db":4.88,"time":"2019-08-07T02:03:04.759Z","params":[{"key":"_method","value":"delete"},{"key":"authenticity_token","value":"[FILTERED]"},{"key":"projectName","value":"Leads_Repository"},{"key":"namespace_id","value":"eg-leaders"},{"key":"id","value":"Leads_Repository"}],"remote_ip":"1.2.3.4","user_id":4,"username":"naupe","ua":"Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:68.0) Gecko/20100101 Firefox/68.0","queue_duration":null,"correlation_id":"Ru1dgp3V2b1"}
{"method":"DELETE","path":"/EJH/death-screen","format":"html","controller":"ProjectsController","action":"destroy","status":500,"error":"OpenSSL::Cipher::CipherError: ","duration":32.42,"view":0.0,"db":4.69,"time":"2019-08-07T02:03:12.164Z","params":[{"key":"_method","value":"delete"},{"key":"authenticity_token","value":"[FILTERED]"},{"key":"projectName","value":"death-screen"},{"key":"namespace_id","value":"EJH"},{"key":"id","value":"death-screen"}],"remote_ip":"1.2.3.4","user_id":4,"username":"naupe","ua":"Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:68.0) Gecko/20100101 Firefox/68.0","queue_duration":null,"correlation_id":"3uELfUESUi9"}
{"method":"DELETE","path":"/EJH/death-screen","format":"html","controller":"ProjectsController","action":"destroy","status":500,"error":"OpenSSL::Cipher::CipherError: ","duration":30.03,"view":0.0,"db":3.35,"time":"2019-08-07T02:29:25.226Z","params":[{"key":"_method","value":"delete"},{"key":"authenticity_token","value":"[FILTERED]"},{"key":"projectName","value":"death-screen"},{"key":"namespace_id","value":"EJH"},{"key":"id","value":"death-screen"}],"remote_ip":"1.2.3.4","user_id":4,"username":"naupe","ua":"Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:68.0) Gecko/20100101 Firefox/68.0","queue_duration":null,"correlation_id":"Gz9Be9SwfY2"}
{"method":"DELETE","path":"/company-name/ProjectName","format":"html","controller":"ProjectsController","action":"destroy","status":500,"error":"OpenSSL::Cipher::CipherError: ","duration":36.17,"view":0.0,"db":5.5,"time":"2019-08-07T02:29:45.279Z","params":[{"key":"_method","value":"delete"},{"key":"authenticity_token","value":"[FILTERED]"},{"key":"projectName","value":"ProjectName"},{"key":"namespace_id","value":"epoch-games"},{"key":"id","value":"ProjectName"}],"remote_ip":"1.2.3.4","user_id":4,"username":"naupe","ua":"Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:68.0) Gecko/20100101 Firefox/68.0","queue_duration":null,"correlation_id":"S5mi1jkR1F1"}
{"method":"GET","path":"/company-name/ProjectName/settings/ci_cd","format":"html","controller":"Projects::Settings::CiCdController","action":"show","status":500,"error":"ActionView::Template::Error: ","duration":658.05,"view":0.0,"db":35.62,"time":"2019-08-07T02:30:27.939Z","params":[{"key":"namespace_id","value":"company-name"},{"key":"project_id","value":"ProjectName"}],"remote_ip":"1.2.3.4","user_id":4,"username":"naupe","ua":"Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:68.0) Gecko/20100101 Firefox/68.0","queue_duration":null,"gitaly_calls":1,"gitaly_duration":6.16,"correlation_id":"IaMsGvtXi07"}
{"method":"GET","path":"/company-name/ProjectName/settings/ci_cd","format":"html","controller":"Projects::Settings::CiCdController","action":"show","status":500,"error":"ActionView::Template::Error: ","duration":771.09,"view":0.0,"db":36.14,"time":"2019-08-07T02:30:39.404Z","params":[{"key":"namespace_id","value":"company-name"},{"key":"project_id","value":"ProjectName"}],"remote_ip":"1.2.3.4","user_id":4,"username":"naupe","ua":"Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:68.0) Gecko/20100101 Firefox/68.0","queue_duration":null,"gitaly_calls":1,"gitaly_duration":7.46,"correlation_id":"C5gWIR2wY96"}
{"method":"DELETE","path":"/EJH/death-screen","format":"html","controller":"ProjectsController","action":"destroy","status":500,"error":"OpenSSL::Cipher::CipherError: ","duration":30.94,"view":0.0,"db":3.34,"time":"2019-08-07T02:31:28.846Z","params":[{"key":"utf8","value":"✓"},{"key":"_method","value":"delete"},{"key":"authenticity_token","value":"[FILTERED]"},{"key":"namespace_id","value":"EJH"},{"key":"id","value":"death-screen"}],"remote_ip":"1.2.3.4","user_id":4,"username":"naupe","ua":"Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:68.0) Gecko/20100101 Firefox/68.0","queue_duration":null,"correlation_id":"crvA0uih4r1"}
{"method":"DELETE","path":"/EJH/death-screen","format":"html","controller":"ProjectsController","action":"destroy","status":500,"error":"OpenSSL::Cipher::CipherError: ","duration":32.06,"view":0.0,"db":5.31,"time":"2019-08-07T02:31:34.835Z","params":[{"key":"utf8","value":"✓"},{"key":"_method","value":"delete"},{"key":"authenticity_token","value":"[FILTERED]"},{"key":"namespace_id","value":"EJH"},{"key":"id","value":"death-screen"}],"remote_ip":"1.2.3.4","user_id":4,"username":"naupe","ua":"Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:68.0) Gecko/20100101 Firefox/68.0","queue_duration":null,"correlation_id":"wcWlJp5EiA"}
{"method":"DELETE","path":"/EJH/death-screen","format":"html","controller":"ProjectsController","action":"destroy","status":500,"error":"OpenSSL::Cipher::CipherError: ","duration":33.74,"view":0.0,"db":4.36,"time":"2019-08-07T02:32:47.818Z","params":[{"key":"utf8","value":"✓"},{"key":"_method","value":"delete"},{"key":"authenticity_token","value":"[FILTERED]"},{"key":"namespace_id","value":"EJH"},{"key":"id","value":"death-screen"}],"remote_ip":"1.2.3.4","user_id":4,"username":"naupe","ua":"Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:68.0) Gecko/20100101 Firefox/68.0","queue_duration":null,"correlation_id":"OAloisb56u"}
root@gitlab:~# less /var/log/gitlab/gitlab-rails/production.log | grep Error
Completed 500 Internal Server Error in 120ms (ActiveRecord: 7.3ms)
ActionView::Template::Error ():
Completed 500 Internal Server Error in 113ms (ActiveRecord: 7.3ms)
ActionView::Template::Error ():
Completed 500 Internal Server Error in 30ms (ActiveRecord: 3.4ms)
OpenSSL::Cipher::CipherError ():
Completed 500 Internal Server Error in 31ms (ActiveRecord: 4.9ms)
OpenSSL::Cipher::CipherError ():
Completed 500 Internal Server Error in 31ms (ActiveRecord: 4.9ms)
OpenSSL::Cipher::CipherError ():
Completed 500 Internal Server Error in 31ms (ActiveRecord: 4.7ms)
OpenSSL::Cipher::CipherError ():
OpenSSL::Cipher::CipherError ():
Completed 500 Internal Server Error in 28ms (ActiveRecord: 3.4ms)
OpenSSL::Cipher::CipherError ():
Completed 500 Internal Server Error in 34ms (ActiveRecord: 5.5ms)
OpenSSL::Cipher::CipherError ():
Completed 500 Internal Server Error in 654ms (ActiveRecord: 35.6ms)
ActionView::Template::Error ():
Completed 500 Internal Server Error in 769ms (ActiveRecord: 36.1ms)
ActionView::Template::Error ():
Completed 500 Internal Server Error in 29ms (ActiveRecord: 3.3ms)
OpenSSL::Cipher::CipherError ():
Completed 500 Internal Server Error in 30ms (ActiveRecord: 5.3ms)
OpenSSL::Cipher::CipherError ():
Completed 500 Internal Server Error in 32ms (ActiveRecord: 4.4ms)
OpenSSL::Cipher::CipherError ():
(I have added generic company
, company-name
, ProjectName
, EJH
and 1.2.3.4
for privacy’s sake)
The logs seem to be reporting ActionView::Template::Error
and OpenSSL::Cipher::CipherError
. What could this mean? The latter is obviously related to SSL (ugh), but the former? I have no idea.
Output of checks
(If you are reporting a bug on GitLab.com, write: This bug happens on GitLab.com)
Results of GitLab environment info
I installed GitLab using these steps. Considering GitLab is installed on Ubuntu Server 18.04, I thought I technically installed from source, but running the bundle command had me realize ... nope, command not found
, so must be the Omnibus Package.
Here is the output:
root@gitlab:~# sudo gitlab-rake gitlab:env:info
System information
System: Ubuntu 18.04
Current User: git
Using RVM: no
Ruby Version: 2.5.3p105
Gem Version: 2.7.6
Bundler Version:1.17.3
Rake Version: 12.3.2
Redis Version: 3.2.12
Git Version: 2.18.1
Sidekiq Version:5.2.5
Go Version: unknown
GitLab information
Version: 11.10.1
Revision: 861fe405571
Directory: /opt/gitlab/embedded/service/gitlab-rails
DB Adapter: PostgreSQL
DB Version: 9.6.11
URL: https://git.domainames.com
HTTP Clone URL: https://git.domainames.com/some-group/some-project.git
SSH Clone URL: git@git.domainames.com:some-group/some-project.git
Using LDAP: no
Using Omniauth: yes
Omniauth Providers:
GitLab Shell
Version: 9.0.0
Repository storage paths:
- default: /var/opt/gitlab/git-data/repositories
GitLab Shell path: /opt/gitlab/embedded/service/gitlab-shell
Git: /opt/gitlab/embedded/bin/git
I've obviously not given the actual URL, although some-group/some-project.git
is actually output.
Results of GitLab application Check
root@gitlab:~# sudo gitlab-rake gitlab:check SANITIZE=true
Checking GitLab subtasks ...
Checking GitLab Shell ...
GitLab Shell: ... GitLab Shell version >= 9.0.0 ? ... OK (9.0.0)
Running /opt/gitlab/embedded/service/gitlab-shell/bin/check
Check GitLab API access: OK
Redis available via internal API: OK
Access to /var/opt/gitlab/.ssh/authorized_keys: OK
gitlab-shell self-check successful
Checking GitLab Shell ... Finished
Checking Gitaly ...
Gitaly: ... default ... OK
Checking Gitaly ... Finished
Checking Sidekiq ...
Sidekiq: ... Running? ... yes
Number of Sidekiq processes ... 1
Checking Sidekiq ... Finished
Checking Incoming Email ...
Incoming Email: ... Reply by email is disabled in config/gitlab.yml
Checking Incoming Email ... Finished
Checking LDAP ...
LDAP: ... LDAP is disabled in config/gitlab.yml
Checking LDAP ... Finished
Checking GitLab App ...
Git configured correctly? ... yes
Database config exists? ... yes
All migrations up? ... yes
Database contains orphaned GroupMembers? ... no
GitLab config exists? ... yes
GitLab config up to date? ... yes
Log directory writable? ... yes
Tmp directory writable? ... yes
Uploads directory exists? ... yes
Uploads directory has correct permissions? ... yes
Uploads directory tmp has correct permissions? ... yes
Init script exists? ... skipped (omnibus-gitlab has no init script)
Init script up-to-date? ... skipped (omnibus-gitlab has no init script)
Projects have namespace: ...
7/3 ... yes
7/4 ... yes
53/7 ... yes
Redis version >= 2.8.0? ... yes
Ruby version >= 2.5.3 ? ... yes (2.5.3)
Git version >= 2.18.0 ? ... yes (2.18.1)
Git user has default SSH configuration? ... yes
Active users: ... 90
Checking GitLab App ... Finished
Checking GitLab subtasks ... Finished
I've obviously not given the actual URL again.
Possible fixes
Right now I'm not sure. I originally thought getting rid of the syntactical errors in /etc/gitlab/gitlab.rb
would be enough, given that GitLab boots up just fine and I see no more