500 Internal Server Error ActionView::Template::Error PG::NotNullViolation: ERROR
Summary
Some users get 500 Internal Server Error on some of the pages. The error is always the same though: config/initializers/active_record_locking.rb:14:in '_create_record'
Work around
Postgres Sequence is missing.
$ gitlab-psql
# CREATE SEQUENCE user_preferences_id_seq START WITH 302;
# ALTER SEQUENCE user_preferences_id_seq OWNER TO gitlab;
# ALTER SEQUENCE user_preferences_id_seq OWNED BY user_preferences.id;
# ALTER TABLE user_preferences ALTER COLUMN id SET default nextval('user_preferences_id_seq'::regclass);Steps to reproduce
Error 500 when user visiting these example URLs:
Work fine when the user is visiting:
- https://gitlab.example.com/profile
- https://gitlab.example.com/explore
- https://gitlab.example.com/projects/new
Going to the "new project page" works, but the user gets a 500 error when gitlab redirects to the newly created project. The project was however created successfully it seems.
What is the current bug behavior?
500 Whoops, something went wrong on our end.
What is the expected correct behavior?
No 500 Error
Relevant logs and/or screenshots
==> /var/log/gitlab/gitlab-rails/production.log <==
Started GET "/profile/personal_access_tokens" for 10.64.14.247 at 2019-04-25 08:45:01 +0200
Processing by Profiles::PersonalAccessTokensController#index as HTML
Completed 500 Internal Server Error in 412ms (ActiveRecord: 15.7ms | Elasticsearch: 0.0ms)
ActionView::Template::Error (PG::NotNullViolation: ERROR: null value in column "id" violates not-null constraint
DETAIL: Failing row contains (null, 1339, 0, 0, 2019-04-25 06:45:02.273844+00, 2019-04-25 06:45:02.273844+00, null, null, null, null, null, 0, null).
: INSERT INTO "user_preferences" ("user_id", "created_at", "updated_at") VALUES (1339, '2019-04-25 06:45:02.273844', '2019-04-25 06:45:02.273844') RETURNING "id"):
31: It cannot be used to access any other data.
32: .col-lg-8.feed-token-reset
33: = label_tag :feed_token, 'Feed token', class: "label-bold"
34: = text_field_tag :feed_token, current_user.feed_token, class: 'form-control', readonly: true, onclick: 'this.select()'
35: %p.form-text.text-muted
36: Keep this token secret. Anyone who gets ahold of it can read activity and issue RSS feeds or your calendar feed as if they were you.
37: You should
config/initializers/active_record_locking.rb:14:in `_create_record'
app/models/concerns/token_authenticatable_strategies/base.rb:39:in `reset_token!'
app/models/concerns/token_authenticatable_strategies/base.rb:32:in `ensure_token!'
app/models/concerns/token_authenticatable.rb:45:in `block in add_authentication_token_field'
app/models/user.rb:1389:in `feed_token'
app/views/profiles/personal_access_tokens/index.html.haml:34:in `_app_views_profiles_personal_access_tokens_index_html_haml__4202348645333370037_70106710352260'
app/controllers/application_controller.rb:116:in `render'
lib/gitlab/i18n.rb:55:in `with_locale'
lib/gitlab/i18n.rb:61:in `with_user_locale'
app/controllers/application_controller.rb:434:in `set_locale'
lib/gitlab/middleware/rails_queue_duration.rb:27:in `call'
lib/gitlab/metrics/rack_middleware.rb:17:in `block in call'
lib/gitlab/metrics/transaction.rb:57:in `run'
lib/gitlab/metrics/rack_middleware.rb:17:in `call'
lib/gitlab/middleware/multipart.rb:103:in `call'
lib/gitlab/request_profiler/middleware.rb:16:in `call'
ee/lib/gitlab/jira/middleware.rb:17:in `call'
lib/gitlab/middleware/go.rb:20:in `call'
lib/gitlab/etag_caching/middleware.rb:13:in `call'
lib/gitlab/middleware/correlation_id.rb:16:in `block in call'
lib/gitlab/correlation_id.rb:15:in `use_id'
lib/gitlab/middleware/correlation_id.rb:15:in `call'
lib/gitlab/middleware/read_only/controller.rb:42:in `call'
lib/gitlab/middleware/read_only.rb:18:in `call'
lib/gitlab/middleware/basic_health_check.rb:25:in `call'
lib/gitlab/request_context.rb:26:in `call'
lib/gitlab/metrics/requests_rack_middleware.rb:29:in `call'
lib/gitlab/middleware/release_env.rb:13:in `call'
==> /var/log/gitlab/gitlab-rails/production_json.log <==
{"method":"GET","path":"/profile/personal_access_tokens","format":"html","controller":"Profiles::PersonalAccessTokensController","action":"index","status":500,"error":"ActionView::Template::Error: PG::NotNullViolation: ERROR: null value in column \"id\" violates not-null constraint\nDETAIL: Failing row contains (null, 1339, 0, 0, 2019-04-25 06:45:02.273844+00, 2019-04-25 06:45:02.273844+00, null, null, null, null, null, 0, null).\n: INSERT INTO \"user_preferences\" (\"user_id\", \"created_at\", \"updated_at\") VALUES (1339, '2019-04-25 06:45:02.273844', '2019-04-25 06:45:02.273844') RETURNING \"id\"","duration":412.99,"view":0.0,"db":15.7,"time":"2019-04-25T06:45:01.866Z","params":[],"remote_ip":"10.64.14.247","user_id":1339,"username":"cristian.perez-molin","ua":"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_4) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/12.1 Safari/605.1.15","queue_duration":87.33,"correlation_id":"NjTh1DCvny5"}
==> /var/log/gitlab/postgresql/current <==
2019-04-25_06:45:02.27473 ERROR: null value in column "id" violates not-null constraint
2019-04-25_06:45:02.27477 DETAIL: Failing row contains (null, 1339, 0, 0, 2019-04-25 06:45:02.273844+00, 2019-04-25 06:45:02.273844+00, null, null, null, null, null, 0, null).
2019-04-25_06:45:02.27477 STATEMENT: INSERT INTO "user_preferences" ("user_id", "created_at", "updated_at") VALUES (1339, '2019-04-25 06:45:02.273844', '2019-04-25 06:45:02.273844') RETURNING "id"Output of checks
(If you are reporting a bug on GitLab.com, write: This bug happens on GitLab.com)
Results of GitLab environment info
Expand for output related to GitLab environment info
System information System: Proxy: no Current User: git Using RVM: no Ruby Version: 2.5.3p105 Gem Version: 2.7.6 Bundler Version:1.17.3 Rake Version: 12.3.2 Redis Version: 3.2.12 Git Version: 2.18.1 Sidekiq Version:5.2.5 Go Version: unknown GitLab information Version: 11.10.1-ee Revision: 2cd7cedf96c Directory: /opt/gitlab/embedded/service/gitlab-rails DB Adapter: PostgreSQL DB Version: 9.6.11 URL: https://gitlab.example.com HTTP Clone URL: https://gitlab.example.com/some-group/some-project.git SSH Clone URL: git@gitlab.example.com:some-group/some-project.git Elasticsearch: no Geo: no Using LDAP: yes Using Omniauth: no GitLab Shell Version: 9.0.0 Repository storage paths: - default: /gitlab-data/git-data/repositories GitLab Shell path: /opt/gitlab/embedded/service/gitlab-shell Git: /opt/gitlab/embedded/bin/git
Results of GitLab application Check
Expand for output related to the GitLab application check
Checking GitLab subtasks ... Checking GitLab Shell ... GitLab Shell: ... GitLab Shell version >= 9.0.0 ? ... OK (9.0.0) Running /opt/gitlab/embedded/service/gitlab-shell/bin/check Check GitLab API access: OK Redis available via internal API: OK Access to /var/opt/gitlab/.ssh/authorized_keys: OK gitlab-shell self-check successful Checking GitLab Shell ... Finished Checking Gitaly ... Gitaly: ... default ... OK Checking Gitaly ... Finished Checking Sidekiq ... Sidekiq: ... Running? ... yes Number of Sidekiq processes ... 1 Checking Sidekiq ... Finished Checking Incoming Email ... Incoming Email: ... Reply by email is disabled in config/gitlab.yml Checking Incoming Email ... Finished Checking LDAP ... LDAP: ... Server: ldapmain LDAP authentication... Success LDAP users with access to your GitLab server (only showing the first 100 results) Checking LDAP ... Finished Checking GitLab App ... Git configured correctly? ... yes Database config exists? ... yes All migrations up? ... yes Database contains orphaned GroupMembers? ... no GitLab config exists? ... yes GitLab config up to date? ... yes Log directory writable? ... yes Tmp directory writable? ... yes Uploads directory exists? ... yes Uploads directory has correct permissions? ... yes Uploads directory tmp has correct permissions? ... skipped (no tmp uploads folder yet) Init script exists? ... skipped (omnibus-gitlab has no init script) Init script up-to-date? ... skipped (omnibus-gitlab has no init script) Projects have namespace: ... all ... yes Redis version >= 2.8.0? ... yes Ruby version >= 2.5.3 ? ... yes (2.5.3) Git version >= 2.18.0 ? ... yes (2.18.1) Git user has default SSH configuration? ... yes Active users: ... 411 Elasticsearch version 5.6 - 6.x? ... skipped (elasticsearch is disabled) Checking GitLab App ... Finished Checking GitLab subtasks ... Finished
Possible fixes
$ gitlab-psql
# CREATE SEQUENCE user_preferences_id_seq START WITH 302;
# ALTER SEQUENCE user_preferences_id_seq OWNER TO gitlab;
# ALTER SEQUENCE user_preferences_id_seq OWNED BY user_preferences.id;
# ALTER TABLE user_preferences ALTER COLUMN id SET default nextval('user_preferences_id_seq'::regclass);