Gitlab-ce v13.9.3 (ea359c58) Error 500 (Internal Server Error) when trying to access CID Settings or User Profile
Summary
Error 500 through several actions: Occurences (presumably not limited to)
- Access CID Settings in Admin Area
- Access User Profile using top bar avatar menu.
- Build server does not pick up any jobs
- git push triggers error 500
🚯
remote: GitLab: http post to gitlab api /post_receive endpoint: 500 Internal Server Error
Steps to reproduce
- I upgraded Gitlab-CE using #apt install -y gitlab-ce
- Noticed that the 500 Internal error occurred.
- Made a backup of my instance
- Did a full clean install of Gitlab-CE version 13.9.3
- restored the backup
- issue persists
Example Project
N/A
What is the current bug behavior?
I get an error 500 when trying to access CID Settings from the Admin area or when trying to access my User Profile from the top bar avatar menu
What is the expected correct behavior?
I see the relevant pages
Relevant logs and/or screenshots
- gitlab-ctl tail nginx
5.173.209.137 - - [14/Mar/2021:11:27:47 +0000] "GET /raf HTTP/2.0" 500 2926 "https://git.macula.io/admin" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:86.0) Gecko/20100101 Firefox/86.0" -
5.173.209.137 - - [14/Mar/2021:11:29:03 +0000] "GET /admin/application_settings/ci_cd HTTP/2.0" 500 2926 "https://git.macula.io/admin" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:86.0) Gecko/20100101 Firefox/86.0" -
- gitlab-ctl tail postgresql
==> /var/log/gitlab/postgresql/current <==
2021-03-14_11:26:31.95965 ERROR: column user_statuses.clear_status_at does not exist at character 44
2021-03-14_11:26:31.95982 STATEMENT: SELECT 1 AS one FROM "user_statuses" WHERE "user_statuses"."clear_status_at" <= '2021-03-14 11:26:31.958555' LIMIT 1
2021-03-14_11:27:13.39940 ERROR: column user_statuses.clear_status_at does not exist at character 44
2021-03-14_11:27:13.39959 STATEMENT: SELECT 1 AS one FROM "user_statuses" WHERE "user_statuses"."clear_status_at" <= '2021-03-14 11:27:13.398621' LIMIT 1
2021-03-14_11:27:47.85690 ERROR: relation "user_follow_users" does not exist at character 41
2021-03-14_11:27:47.85693 STATEMENT: SELECT COUNT(*) FROM "users" INNER JOIN "user_follow_users" ON "users"."id" = "user_follow_users"."follower_id" WHERE "user_follow_users"."followee_id" = 12
2021-03-14_11:28:01.19865 ERROR: column user_statuses.clear_status_at does not exist at character 44
2021-03-14_11:28:01.19867 STATEMENT: SELECT 1 AS one FROM "user_statuses" WHERE "user_statuses"."clear_status_at" <= '2021-03-14 11:28:01.197900' LIMIT 1
2021-03-14_11:29:36.86107 ERROR: column user_statuses.clear_status_at does not exist at character 44
2021-03-14_11:29:36.86111 STATEMENT: SELECT 1 AS one FROM "user_statuses" WHERE "user_statuses"."clear_status_at" <= '2021-03-14 11:29:36.859954' LIMIT 1
==> /var/log/gitlab/postgresql/state <==
==> /var/log/gitlab/postgresql/current <==
2021-03-14_11:30:09.48218 ERROR: column user_statuses.clear_status_at does not exist at character 44
2021-03-14_11:30:09.48221 STATEMENT: SELECT 1 AS one FROM "user_statuses" WHERE "user_statuses"."clear_status_at" <= '2021-03-14 11:30:09.481268' LIMIT 1
2021-03-14_11:30:09.49724 ERROR: relation "packages_composer_cache_files" does not exist at character 454
2021-03-14_11:30:09.49727 STATEMENT: SELECT a.attname, format_type(a.atttypid, a.atttypmod),
2021-03-14_11:30:09.49728 pg_get_expr(d.adbin, d.adrelid), a.attnotnull, a.atttypid, a.atttypmod,
2021-03-14_11:30:09.49729 c.collname, col_description(a.attrelid, a.attnum) AS comment
2021-03-14_11:30:09.49729 FROM pg_attribute a
2021-03-14_11:30:09.49730 LEFT JOIN pg_attrdef d ON a.attrelid = d.adrelid AND a.attnum = d.adnum
2021-03-14_11:30:09.49730 LEFT JOIN pg_type t ON a.atttypid = t.oid
2021-03-14_11:30:09.49730 LEFT JOIN pg_collation c ON a.attcollation = c.oid AND a.attcollation <> t.typcollation
2021-03-14_11:30:09.49731 WHERE a.attrelid = '"packages_composer_cache_files"'::regclass
2021-03-14_11:30:09.49731 AND a.attnum > 0 AND NOT a.attisdropped
2021-03-14_11:30:09.49731 ORDER BY a.attnum
2021-03-14_11:30:09.49731
Output of checks
N/A
Results of GitLab environment info
- gitlab-ctl gitlab:env:info
System information
System: Debian 9.13
Current User: git
Using RVM: no
Ruby Version: 2.7.2p137
Gem Version: 3.1.4
Bundler Version:2.1.4
Rake Version: 13.0.3
Redis Version: 6.0.10
Git Version: 2.29.0
Sidekiq Version:5.2.9
Go Version: unknown
GitLab information
Version: 13.9.3
Revision: ea359c58edb
Directory: /opt/gitlab/embedded/service/gitlab-rails
DB Adapter: PostgreSQL
DB Version: 12.5
URL: https://git.macula.io
HTTP Clone URL: https://git.macula.io/some-group/some-project.git
SSH Clone URL: git@git.macula.io:some-group/some-project.git
Using LDAP: no
Using Omniauth: no
GitLab Shell
Version: 13.17.0
Repository storage paths:
- default: /var/opt/gitlab/git-data/repositories
GitLab Shell path: /opt/gitlab/embedded/service/gitlab-shell
Git: /opt/gitlab/embedded/bin/git
``
#### Results of GitLab application Check
``
Checking GitLab subtasks ...
Checking GitLab Shell ...
GitLab Shell: ... GitLab Shell version >= 13.17.0 ? ... OK (13.17.0)
Running /opt/gitlab/embedded/service/gitlab-shell/bin/check
Internal API available: OK
Redis available via internal API: OK
gitlab-shell self-check successful
Checking GitLab Shell ... Finished
Checking Gitaly ...
Gitaly: ... default ... OK
Checking Gitaly ... Finished
Checking Sidekiq ...
Sidekiq: ... Running? ... yes
Number of Sidekiq processes ... 1
Checking Sidekiq ... Finished
Checking Incoming Email ...
Incoming Email: ... Checking Reply by email ...
IMAP server credentials are correct? ... Checking macula@discomco.pl
no
Try fixing it:
An error occurred: OpenSSL::SSL::SSLError: SSL_connect returned=1 errno=0 state=error: wrong version number
Check that the information in config/gitlab.yml is correct
For more information see:
doc/administration/reply_by_email.md
Please fix the error above and rerun the checks.
Init.d configured correctly? ... skipped
MailRoom running? ... skipped
Checking Reply by email ... Finished
Checking Incoming Email ... Finished
Checking LDAP ...
LDAP: ... LDAP is disabled in config/gitlab.yml
Checking LDAP ... Finished
Checking GitLab App ...
Git configured correctly? ... yes
Database config exists? ... yes
All migrations up? ... yes
Database contains orphaned GroupMembers? ... no
GitLab config exists? ... yes
GitLab config up to date? ... yes
Log directory writable? ... yes
Tmp directory writable? ... yes
Uploads directory exists? ... yes
Uploads directory has correct permissions? ... yes
Uploads directory tmp has correct permissions? ... yes
Init script exists? ... skipped (omnibus-gitlab has no init script)
Init script up-to-date? ... skipped (omnibus-gitlab has no init script)
Projects have namespace: ...
7/3 ... yes
2/4 ... yes
9/5 ... yes
11/6 ... yes
13/7 ... yes
15/8 ... yes
23/10 ... yes
23/11 ... yes
9/12 ... yes
4/13 ... yes
4/14 ... yes
4/15 ... yes
25/17 ... yes
23/18 ... yes
26/20 ... yes
26/23 ... yes
26/24 ... yes
26/25 ... yes
25/26 ... yes
26/27 ... yes
23/29 ... yes
26/30 ... yes
4/32 ... yes
25/33 ... yes
4/34 ... yes
26/35 ... yes
26/36 ... yes
4/37 ... yes
4/39 ... yes
26/43 ... yes
26/44 ... yes
4/45 ... yes
26/46 ... yes
4/49 ... yes
26/50 ... yes
26/52 ... yes
26/54 ... yes
25/55 ... yes
26/56 ... yes
30/58 ... yes
32/59 ... yes
32/60 ... yes
32/62 ... yes
32/64 ... yes
36/71 ... yes
36/72 ... yes
36/75 ... yes
36/77 ... yes
30/78 ... yes
36/79 ... yes
32/80 ... yes
32/81 ... yes
32/82 ... yes
32/83 ... yes
23/85 ... yes
39/86 ... yes
32/87 ... yes
Redis version >= 4.0.0? ... yes
Ruby version >= 2.7.2 ? ... yes (2.7.2)
Git version >= 2.29.0 ? ... yes (2.29.0)
Git user has default SSH configuration? ... yes
Active users: ... 4
Is authorized keys file accessible? ... yes
GitLab configured to store new projects in hashed storage? ... yes
All projects are in hashed storage? ... yes
Checking GitLab App ... Finished
Checking GitLab subtasks ... Finished
Possible fixes
I think something is wrong with the Postgres DB Schema (see log above)
Edited by R Lefever