Using personal API token to show project gives 500 internal error.

After upgrading from Gitlab 11.6.2 to 11.9.1 CE from source (Ruby 2.5.5), It seems when I use my personal access token to query project, I get a "500 Internal Server Error" message. I tried creating a new personal token and it seems to still give 500 internal server error.

curl -s --header "PRIVATE-TOKEN: <my personal api token>" --header "Content-Type:text/json" "https://mygitwebsite.com/api/v4/projects/mysite"
{"message":"500 Internal Server Error"}

When I look at my production.log file,

Started GET "/api/v4/projects/mysite" for 1.2.3.4 at 2019-04-02 20:26:52 -0500

OpenSSL::Cipher::CipherError ():
  /home/git/.gem/ruby/2.5.0/gems/encryptor-3.0.0/lib/encryptor.rb:98:in `final'
  /home/git/.gem/ruby/2.5.0/gems/encryptor-3.0.0/lib/encryptor.rb:98:in `crypt'
  /home/git/.gem/ruby/2.5.0/gems/encryptor-3.0.0/lib/encryptor.rb:49:in `decrypt'
  /opt/gitlab/gitlab-11.9.1/lib/gitlab/crypto_helper.rb:27:in `aes256_gcm_decrypt'
  /opt/gitlab/gitlab-11.9.1/app/models/concerns/token_authenticatable_strategies/encrypted.rb:45:in `get_token'
  /opt/gitlab/gitlab-11.9.1/app/models/concerns/token_authenticatable_strategies/base.rb:33:in `ensure_token!'
  /opt/gitlab/gitlab-11.9.1/app/models/concerns/token_authenticatable.rb:43:in `block in add_authentication_token_field'
  /opt/gitlab/gitlab-11.9.1/app/models/project.rb:1543:in `runners_token'
  /home/git/.gem/ruby/2.5.0/gems/grape-entity-0.7.1/lib/grape_entity/delegator/plain_object.rb:8:in `delegate'
  /home/git/.gem/ruby/2.5.0/gems/grape-entity-0.7.1/lib/grape_entity/entity.rb:512:in `delegate_attribute'
  /home/git/.gem/ruby/2.5.0/gems/grape-entity-0.7.1/lib/grape_entity/exposure/delegator_exposure.rb:8:in `value'
  /home/git/.gem/ruby/2.5.0/gems/grape-entity-0.7.1/lib/grape_entity/exposure/base.rb:82:in `valid_value'
  /home/git/.gem/ruby/2.5.0/gems/grape-entity-0.7.1/lib/grape_entity/exposure/base.rb:66:in `serializable_value'
  /home/git/.gem/ruby/2.5.0/gems/grape-entity-0.7.1/lib/grape_entity/exposure/nesting_exposure.rb:41:in `block in serializable_value'
  /home/git/.gem/ruby/2.5.0/gems/grape-entity-0.7.1/lib/grape_entity/exposure/nesting_exposure.rb:120:in `block (2 levels) in map_entity_exposures'
  /home/git/.gem/ruby/2.5.0/gems/grape-entity-0.7.1/lib/grape_entity/exposure/base.rb:116:in `block in with_attr_path'
  /home/git/.gem/ruby/2.5.0/gems/grape-entity-0.7.1/lib/grape_entity/options.rb:90:in `with_attr_path'
  /home/git/.gem/ruby/2.5.0/gems/grape-entity-0.7.1/lib/grape_entity/exposure/base.rb:115:in `with_attr_path'
  /home/git/.gem/ruby/2.5.0/gems/grape-entity-0.7.1/lib/grape_entity/exposure/nesting_exposure.rb:119:in `block in map_entity_exposures'
  /home/git/.gem/ruby/2.5.0/gems/grape-entity-0.7.1/lib/grape_entity/exposure/nesting_exposure.rb:118:in `each'
  /home/git/.gem/ruby/2.5.0/gems/grape-entity-0.7.1/lib/grape_entity/exposure/nesting_exposure.rb:118:in `each_with_object'
  /home/git/.gem/ruby/2.5.0/gems/grape-entity-0.7.1/lib/grape_entity/exposure/nesting_exposure.rb:118:in `map_entity_exposures'
  /home/git/.gem/ruby/2.5.0/gems/grape-entity-0.7.1/lib/grape_entity/exposure/nesting_exposure.rb:40:in `serializable_value'
  /home/git/.gem/ruby/2.5.0/gems/grape-entity-0.7.1/lib/grape_entity/entity.rb:489:in `serializable_hash'
  /home/git/.gem/ruby/2.5.0/gems/grape-entity-0.7.1/lib/grape_entity/entity.rb:520:in `to_json'
  /home/git/.gem/ruby/2.5.0/gems/grape-1.1.0/lib/grape/formatter/json.rb:6:in `call'
  /home/git/.gem/ruby/2.5.0/gems/grape-1.1.0/lib/grape/middleware/formatter.rb:45:in `block (2 levels) in build_formatted_response'
  /home/git/.gem/ruby/2.5.0/gems/grape-1.1.0/lib/grape/middleware/formatter.rb:45:in `collect'
  /home/git/.gem/ruby/2.5.0/gems/grape-1.1.0/lib/grape/middleware/formatter.rb:45:in `block in build_formatted_response'
  /home/git/.gem/ruby/2.5.0/gems/activesupport-5.0.7.1/lib/active_support/notifications.rb:166:in `instrument'
  /home/git/.gem/ruby/2.5.0/gems/grape-1.1.0/lib/grape/middleware/formatter.rb:44:in `build_formatted_response'
  /home/git/.gem/ruby/2.5.0/gems/grape-1.1.0/lib/grape/middleware/formatter.rb:28:in `after'
  /home/git/.gem/ruby/2.5.0/gems/grape-1.1.0/lib/grape/middleware/base.rb:34:in `call!'
  /home/git/.gem/ruby/2.5.0/gems/grape-1.1.0/lib/grape/middleware/base.rb:24:in `call'
  /home/git/.gem/ruby/2.5.0/gems/grape-1.1.0/lib/grape/middleware/base.rb:31:in `call!'
  /home/git/.gem/ruby/2.5.0/gems/grape-1.1.0/lib/grape/middleware/base.rb:24:in `call'
  /home/git/.gem/ruby/2.5.0/gems/rack-oauth2-1.2.3/lib/rack/oauth2/server/resource.rb:20:in `call'
  /home/git/.gem/ruby/2.5.0/gems/rack-oauth2-1.2.3/lib/rack/oauth2/server/resource/bearer.rb:8:in `call'
  /home/git/.gem/ruby/2.5.0/gems/grape-1.1.0/lib/grape/middleware/error.rb:38:in `block in call!'
  /home/git/.gem/ruby/2.5.0/gems/grape-1.1.0/lib/grape/middleware/error.rb:37:in `catch'
  /home/git/.gem/ruby/2.5.0/gems/grape-1.1.0/lib/grape/middleware/error.rb:37:in `call!'
  /home/git/.gem/ruby/2.5.0/gems/grape-1.1.0/lib/grape/middleware/base.rb:24:in `call'
  /home/git/.gem/ruby/2.5.0/gems/grape_logging-1.7.0/lib/grape_logging/middleware/request_logger.rb:60:in `block in call!'
  /home/git/.gem/ruby/2.5.0/gems/grape_logging-1.7.0/lib/grape_logging/middleware/request_logger.rb:58:in `catch'
  /home/git/.gem/ruby/2.5.0/gems/grape_logging-1.7.0/lib/grape_logging/middleware/request_logger.rb:58:in `call!'
  /home/git/.gem/ruby/2.5.0/gems/grape-1.1.0/lib/grape/middleware/base.rb:24:in `call'
  /home/git/.gem/ruby/2.5.0/gems/rack-2.0.6/lib/rack/head.rb:12:in `call'
  /home/git/.gem/ruby/2.5.0/gems/grape-1.1.0/lib/grape/endpoint.rb:227:in `call!'
  /home/git/.gem/ruby/2.5.0/gems/grape-1.1.0/lib/grape/endpoint.rb:221:in `call'
  /home/git/.gem/ruby/2.5.0/gems/grape-1.1.0/lib/grape/router/route.rb:72:in `exec'
  /home/git/.gem/ruby/2.5.0/gems/grape-1.1.0/lib/grape/router.rb:121:in `process_route'
  /home/git/.gem/ruby/2.5.0/gems/grape-1.1.0/lib/grape/router.rb:74:in `block in identity'
  /home/git/.gem/ruby/2.5.0/gems/grape-1.1.0/lib/grape/router.rb:93:in `transaction'
  /home/git/.gem/ruby/2.5.0/gems/grape-1.1.0/lib/grape/router.rb:72:in `identity'
  /home/git/.gem/ruby/2.5.0/gems/grape-1.1.0/lib/grape/router.rb:57:in `block in call'
  /home/git/.gem/ruby/2.5.0/gems/grape-1.1.0/lib/grape/router.rb:137:in `with_optimization'
  /home/git/.gem/ruby/2.5.0/gems/grape-1.1.0/lib/grape/router.rb:56:in `call'
  /home/git/.gem/ruby/2.5.0/gems/grape-1.1.0/lib/grape/api.rb:119:in `call'
  /home/git/.gem/ruby/2.5.0/gems/grape-1.1.0/lib/grape/api.rb:45:in `call!'
  /home/git/.gem/ruby/2.5.0/gems/grape-1.1.0/lib/grape/api.rb:40:in `call'
  /home/git/.gem/ruby/2.5.0/gems/actionpack-5.0.7.1/lib/action_dispatch/routing/mapper.rb:17:in `block in <class:Constraints>'
  /home/git/.gem/ruby/2.5.0/gems/actionpack-5.0.7.1/lib/action_dispatch/routing/mapper.rb:46:in `serve'
  /home/git/.gem/ruby/2.5.0/gems/actionpack-5.0.7.1/lib/action_dispatch/journey/router.rb:39:in `block in serve'
  /home/git/.gem/ruby/2.5.0/gems/actionpack-5.0.7.1/lib/action_dispatch/journey/router.rb:26:in `each'
  /home/git/.gem/ruby/2.5.0/gems/actionpack-5.0.7.1/lib/action_dispatch/journey/router.rb:26:in `serve'
  /home/git/.gem/ruby/2.5.0/gems/actionpack-5.0.7.1/lib/action_dispatch/routing/route_set.rb:727:in `call'
  /home/git/.gem/ruby/2.5.0/gems/omniauth-1.9.0/lib/omniauth/strategy.rb:192:in `call!'
  /home/git/.gem/ruby/2.5.0/gems/omniauth-1.9.0/lib/omniauth/strategy.rb:169:in `call'
  /opt/gitlab/gitlab-11.9.1/lib/gitlab/middleware/multipart.rb:103:in `call'
  /opt/gitlab/gitlab-11.9.1/lib/gitlab/request_profiler/middleware.rb:16:in `call'
  /opt/gitlab/gitlab-11.9.1/lib/gitlab/middleware/go.rb:20:in `call'
  /opt/gitlab/gitlab-11.9.1/lib/gitlab/etag_caching/middleware.rb:13:in `call'
  /opt/gitlab/gitlab-11.9.1/lib/gitlab/middleware/correlation_id.rb:16:in `block in call'
  /opt/gitlab/gitlab-11.9.1/lib/gitlab/correlation_id.rb:15:in `use_id'
  /opt/gitlab/gitlab-11.9.1/lib/gitlab/middleware/correlation_id.rb:15:in `call'
  /home/git/.gem/ruby/2.5.0/gems/batch-loader-1.2.2/lib/batch_loader/middleware.rb:11:in `call'
  /home/git/.gem/ruby/2.5.0/gems/rack-attack-4.4.1/lib/rack/attack.rb:107:in `call'
  /home/git/.gem/ruby/2.5.0/gems/warden-1.2.7/lib/warden/manager.rb:36:in `block in call'
  /home/git/.gem/ruby/2.5.0/gems/warden-1.2.7/lib/warden/manager.rb:35:in `catch'
  /home/git/.gem/ruby/2.5.0/gems/warden-1.2.7/lib/warden/manager.rb:35:in `call'
  /home/git/.gem/ruby/2.5.0/gems/rack-cors-1.0.2/lib/rack/cors.rb:97:in `call'
  /home/git/.gem/ruby/2.5.0/gems/rack-2.0.6/lib/rack/etag.rb:25:in `call'
  /home/git/.gem/ruby/2.5.0/gems/rack-2.0.6/lib/rack/conditional_get.rb:25:in `call'
  /home/git/.gem/ruby/2.5.0/gems/rack-2.0.6/lib/rack/head.rb:12:in `call'
  /opt/gitlab/gitlab-11.9.1/lib/gitlab/middleware/read_only/controller.rb:40:in `call'
  /opt/gitlab/gitlab-11.9.1/lib/gitlab/middleware/read_only.rb:18:in `call'
  /home/git/.gem/ruby/2.5.0/gems/rack-2.0.6/lib/rack/session/abstract/id.rb:232:in `context'
  /home/git/.gem/ruby/2.5.0/gems/rack-2.0.6/lib/rack/session/abstract/id.rb:226:in `call'
  /home/git/.gem/ruby/2.5.0/gems/actionpack-5.0.7.1/lib/action_dispatch/middleware/cookies.rb:613:in `call'
  /home/git/.gem/ruby/2.5.0/gems/actionpack-5.0.7.1/lib/action_dispatch/middleware/callbacks.rb:38:in `block in call'
  /home/git/.gem/ruby/2.5.0/gems/activesupport-5.0.7.1/lib/active_support/callbacks.rb:97:in `__run_callbacks__'
  /home/git/.gem/ruby/2.5.0/gems/activesupport-5.0.7.1/lib/active_support/callbacks.rb:750:in `_run_call_callbacks'
  /home/git/.gem/ruby/2.5.0/gems/activesupport-5.0.7.1/lib/active_support/callbacks.rb:90:in `run_callbacks'
  /home/git/.gem/ruby/2.5.0/gems/actionpack-5.0.7.1/lib/action_dispatch/middleware/callbacks.rb:36:in `call'
  /home/git/.gem/ruby/2.5.0/gems/actionpack-5.0.7.1/lib/action_dispatch/middleware/remote_ip.rb:79:in `call'
  /home/git/.gem/ruby/2.5.0/gems/actionpack-5.0.7.1/lib/action_dispatch/middleware/debug_exceptions.rb:49:in `call'
  /home/git/.gem/ruby/2.5.0/gems/actionpack-5.0.7.1/lib/action_dispatch/middleware/show_exceptions.rb:31:in `call'
  /opt/gitlab/gitlab-11.9.1/lib/gitlab/middleware/basic_health_check.rb:25:in `call'
  /home/git/.gem/ruby/2.5.0/gems/railties-5.0.7.1/lib/rails/rack/logger.rb:36:in `call_app'
  /home/git/.gem/ruby/2.5.0/gems/railties-5.0.7.1/lib/rails/rack/logger.rb:24:in `block in call'
  /home/git/.gem/ruby/2.5.0/gems/activesupport-5.0.7.1/lib/active_support/tagged_logging.rb:69:in `block in tagged'
  /home/git/.gem/ruby/2.5.0/gems/activesupport-5.0.7.1/lib/active_support/tagged_logging.rb:26:in `tagged'
  /home/git/.gem/ruby/2.5.0/gems/activesupport-5.0.7.1/lib/active_support/tagged_logging.rb:69:in `tagged'
  /home/git/.gem/ruby/2.5.0/gems/railties-5.0.7.1/lib/rails/rack/logger.rb:24:in `call'
  /opt/gitlab/gitlab-11.9.1/lib/gitlab/request_context.rb:26:in `call'
  /home/git/.gem/ruby/2.5.0/gems/request_store-1.3.1/lib/request_store/middleware.rb:9:in `call'
  /home/git/.gem/ruby/2.5.0/gems/actionpack-5.0.7.1/lib/action_dispatch/middleware/request_id.rb:24:in `call'
  /home/git/.gem/ruby/2.5.0/gems/rack-2.0.6/lib/rack/method_override.rb:22:in `call'
  /home/git/.gem/ruby/2.5.0/gems/rack-2.0.6/lib/rack/runtime.rb:22:in `call'
  /home/git/.gem/ruby/2.5.0/gems/actionpack-5.0.7.1/lib/action_dispatch/middleware/executor.rb:12:in `call'
  /home/git/.gem/ruby/2.5.0/gems/rack-2.0.6/lib/rack/sendfile.rb:111:in `call'
  /opt/gitlab/gitlab-11.9.1/lib/gitlab/metrics/requests_rack_middleware.rb:29:in `call'
  /home/git/.gem/ruby/2.5.0/gems/sentry-raven-2.7.4/lib/raven/integrations/rack.rb:51:in `call'
  /home/git/.gem/ruby/2.5.0/gems/railties-5.0.7.1/lib/rails/engine.rb:522:in `call'
  /home/git/.gem/ruby/2.5.0/gems/railties-5.0.7.1/lib/rails/railtie.rb:193:in `public_send'
  /home/git/.gem/ruby/2.5.0/gems/railties-5.0.7.1/lib/rails/railtie.rb:193:in `method_missing'
  /opt/gitlab/gitlab-11.9.1/lib/gitlab/middleware/release_env.rb:13:in `call'
  /home/git/.gem/ruby/2.5.0/gems/rack-2.0.6/lib/rack/urlmap.rb:68:in `block in call'
  /home/git/.gem/ruby/2.5.0/gems/rack-2.0.6/lib/rack/urlmap.rb:53:in `each'
  /home/git/.gem/ruby/2.5.0/gems/rack-2.0.6/lib/rack/urlmap.rb:53:in `call'
  /home/git/.gem/ruby/2.5.0/gems/unicorn-5.4.1/lib/unicorn/http_server.rb:606:in `process_client'
  /home/git/.gem/ruby/2.5.0/gems/unicorn-worker-killer-0.4.4/lib/unicorn/worker_killer.rb:52:in `process_client'
  /home/git/.gem/ruby/2.5.0/gems/unicorn-5.4.1/lib/unicorn/http_server.rb:701:in `worker_loop'
  /home/git/.gem/ruby/2.5.0/gems/unicorn-5.4.1/lib/unicorn/http_server.rb:549:in `spawn_missing_workers'
  /home/git/.gem/ruby/2.5.0/gems/unicorn-5.4.1/lib/unicorn/http_server.rb:142:in `start'
  /home/git/.gem/ruby/2.5.0/gems/unicorn-5.4.1/bin/unicorn_rails:209:in `<top (required)>'
  /home/git/.gem/ruby/2.5.0/bin/unicorn_rails:23:in `load'
  /home/git/.gem/ruby/2.5.0/bin/unicorn_rails:23:in `<main>'

Additional output [sudo -u git -H bundle exec rake gitlab:env:info RAILS_ENV=production]

------------------------------------------
System information
System:
Current User:   git
Using RVM:      no
Ruby Version:   2.5.5p157
Gem Version:    2.7.6.2
Bundler Version:1.17.3
Rake Version:   12.3.2
Redis Version:  5.0.4
Git Version:    2.19.2
Sidekiq Version:5.2.5
Go Version:     go1.10.3 linux/amd64

GitLab information
Version:        11.9.1
Revision:       86f0b5dd4d0
Directory:      /opt/gitlab/gitlab-11.9.1
DB Adapter:     mysql2
URL:            https://mygitwebsite.com
HTTP Clone URL: https://mygitwebsite.com/some-group/some-project.git
SSH Clone URL:  git@mygitwebsite.com:some-group/some-project.git
Using LDAP:     yes
Using Omniauth: no

GitLab Shell
Version:        8.7.1
Repository storage paths:
- default:      /opt/gitlab/repositories
GitLab Shell path:              /opt/gitlab/gitlab-shell
Git:            /usr/bin/git
Checking GitLab subtasks ...

Checking GitLab Shell ...

GitLab Shell: ... GitLab Shell version >= 8.7.1 ? ... OK (8.7.1)
Running /opt/gitlab/gitlab-shell/bin/check
Check GitLab API access: OK
Redis available via internal API: OK

Access to /home/git/.ssh/authorized_keys: OK
gitlab-shell self-check successful

Checking GitLab Shell ... Finished

Checking Gitaly ...

Gitaly: ... default ... OK

Checking Gitaly ... Finished

Checking Sidekiq ...

Sidekiq: ... Running? ... yes
Number of Sidekiq processes ... 1

Checking Sidekiq ... Finished

Checking Incoming Email ...

Incoming Email: ... Reply by email is disabled in config/gitlab.yml

Checking Incoming Email ... Finished

Checking LDAP ...

LDAP: ... Server: ldapmain
LDAP authentication... Success
LDAP users with access to your GitLab server (only showing the first 100 results)
<hiding the names output>
Checking LDAP ... Finished

Checking GitLab App ...

Git configured correctly? ... yes
Database config exists? ... yes
All migrations up? ... yes
Database contains orphaned GroupMembers? ... no
GitLab config exists? ... yes
GitLab config up to date? ... yes
Log directory writable? ... yes
Tmp directory writable? ... yes
Uploads directory exists? ... yes
Uploads directory has correct permissions? ... yes
Uploads directory tmp has correct permissions? ... yes

Projects have namespace: ...
/ myproject ... yes
Redis version >= 2.8.0? ... yes
Ruby version >= 2.3.5 ? ... yes (2.5.5)
Git version >= 2.18.0 ? ... yes (2.19.2)
Git user has default SSH configuration? ... yes
Active users: ... 120

Checking GitLab App ... Finished


Checking GitLab subtasks ... Finished
------------------------------------------

Is their anyway to fix this? Is this a missing/broken encrypted tokens issue? If so, how can I reset the database.

Edited Nov 13, 2020 by Nicole Schwartz
Assignee Loading
Time tracking Loading