Skip to content

500 when container registry key is not RSA e.g. ECDSA: OpenSSL Pkey RESAError Neither PUB key nor PRIV key: nested asn1 error

Everyone can contribute. Help move this issue forward while earning points, leveling up and collecting rewards.

Summary

If GitLab's container registry is configured with an ECDSA (secp384r1) key (gitlab.yml), an internal error 500 happens when opening the container registry page on the web UI.

Steps to reproduce

Configure the container registry with an ECDSA private key on the GitLab side. for source installation that would be:

registry:
  ...
  key: /foobar/ecdsa-privkey.pem
  ...

Such a key has a format of:

-----BEGIN EC PARAMETERS-----
...
-----END EC PARAMETERS-----
-----BEGIN EC PRIVATE KEY-----
...
-----END EC PRIVATE KEY-----

Example Project

N/A

What is the current bug behavior?

Error page 500.

What is the expected correct behavior?

Container registry page works.

Relevant logs and/or screenshots

From production.log:

Started GET "/template/c/container_registry" for REDACTED at 2019-09-16 20:50:16 +0200
Processing by Projects::Registry::RepositoriesController#index as HTML
  Parameters: {"namespace_id"=>"template", "project_id"=>"c"}
Completed 500 Internal Server Error in 21ms (ActiveRecord: 3.1ms)
  
OpenSSL::PKey::RSAError (Neither PUB key nor PRIV key: nested asn1 error):
  
lib/json_web_token/rsa_token.rb:27:in `initialize'
lib/json_web_token/rsa_token.rb:27:in `new'
lib/json_web_token/rsa_token.rb:27:in `key'
lib/json_web_token/rsa_token.rb:31:in `public_key'
lib/json_web_token/rsa_token.rb:36:in `kid'
lib/json_web_token/rsa_token.rb:14:in `encoded'
app/services/auth/container_registry_authentication_service.rb:39:in `access_token'
app/services/auth/container_registry_authentication_service.rb:20:in `full_access_token'
app/models/container_repository.rb:18:in `registry'
app/models/container_repository.rb:11:in `client'
app/models/container_repository.rb:42:in `manifest'
app/models/container_repository.rb:46:in `tags'
app/models/container_repository.rb:60:in `has_tags?'
app/controllers/projects/registry/repositories_controller.rb:46:in `block (2 levels) in ensure_root_container_repository!'
app/controllers/projects/registry/repositories_controller.rb:45:in `tap'
app/controllers/projects/registry/repositories_controller.rb:45:in `block in ensure_root_container_repository!'
app/controllers/projects/registry/repositories_controller.rb:42:in `tap'
app/controllers/projects/registry/repositories_controller.rb:42:in `ensure_root_container_repository!'
lib/gitlab/session.rb:11:in `with_session'
app/controllers/application_controller.rb:450:in `set_session_storage'
lib/gitlab/i18n.rb:55:in `with_locale'
lib/gitlab/i18n.rb:61:in `with_user_locale'
app/controllers/application_controller.rb:444:in `set_locale'
lib/gitlab/middleware/multipart.rb:103:in `call'
lib/gitlab/request_profiler/middleware.rb:17:in `call'
lib/gitlab/middleware/go.rb:20:in `call'
lib/gitlab/etag_caching/middleware.rb:13:in `call'
lib/gitlab/middleware/correlation_id.rb:16:in `block in call'
lib/gitlab/middleware/correlation_id.rb:15:in `call'
lib/gitlab/middleware/read_only/controller.rb:40:in `call'
lib/gitlab/middleware/read_only.rb:18:in `call'
lib/gitlab/middleware/basic_health_check.rb:25:in `call'
lib/gitlab/request_context.rb:26:in `call'
lib/gitlab/metrics/requests_rack_middleware.rb:29:in `call'
lib/gitlab/middleware/release_env.rb:12:in `call'

Output of checks

See below.

Results of GitLab environment info

Expand for output related to GitLab environment info 
System information
System:         Debian 10
Current User:   gitlab
Using RVM:      no
Ruby Version:   2.6.4p104
Gem Version:    3.0.3
Bundler Version:1.17.3
Rake Version:   12.3.2
Redis Version:  5.0.3
Git Version:    2.23.0
Sidekiq Version:5.2.7
Go Version:     go1.11.6 linux/amd64

GitLab information
Version:        12.2.5
Revision:       09f8edbc29a
Directory:      /var/opt/gitlab/gitlab
DB Adapter:     PostgreSQL
DB Version:     11.5
URL:            https://git.mel.vin
HTTP Clone URL: https://git.mel.vin/some-group/some-project.git
SSH Clone URL:  gitlab@git.mel.vin:some-group/some-project.git
Using LDAP:     no
Using Omniauth: yes
Omniauth Providers:


GitLab Shell
Version:        9.3.0
Repository storage paths:
  • default: /var/opt/gitlab/repositories GitLab Shell path: /var/opt/gitlab/gitlab-shell Git: /usr/bin/git

Results of GitLab application Check

Expand for output related to the GitLab application check 
Checking GitLab subtasks ...

Checking GitLab Shell ...


GitLab Shell: ... GitLab Shell version >= 9.3.0 ? ... OK (9.3.0)
Running /var/opt/gitlab/gitlab-shell/bin/check
Check GitLab API access: OK
Redis available via internal API: OK


Access to /var/opt/gitlab/.ssh/authorized_keys: OK
gitlab-shell self-check successful


Checking GitLab Shell ... Finished


Checking Gitaly ...


Gitaly: ... default ... OK


Checking Gitaly ... Finished


Checking Sidekiq ...


Sidekiq: ... Running? ... yes
Number of Sidekiq processes ... 1


Checking Sidekiq ... Finished


Checking Incoming Email ...


Incoming Email: ... Checking Reply by email ...


IMAP server credentials are correct? ... yes
Init.d configured correctly? ... yes
MailRoom running? ... yes


Checking Reply by email ... Finished


Checking Incoming Email ... Finished


Checking LDAP ...


LDAP: ... LDAP is disabled in config/gitlab.yml


Checking LDAP ... Finished


Checking GitLab App ...


Git configured correctly? ... yes
Database config exists? ... yes
All migrations up? ... yes
Database contains orphaned GroupMembers? ... no
GitLab config exists? ... yes
GitLab config up to date? ... yes
Log directory writable? ... yes
Tmp directory writable? ... yes
Uploads directory exists? ... yes
Uploads directory has correct permissions? ... yes
Uploads directory tmp has correct permissions? ... yes
Init script exists? ... yes
Init script up-to-date? ... yes
Projects have namespace: ...
84/1 ... yes
*** stripped a lot of entries, all are yes ***
102/224 ... yes
Redis version >= 2.8.0? ... yes
Ruby version >= 2.5.3 ? ... yes (2.6.4)
Git version >= 2.22.0 ? ... yes (2.23.0)
Git user has default SSH configuration? ... yes
Active users: ... 15


Checking GitLab App ... Finished


Checking GitLab subtasks ... Finished

Possible fixes

I suspect the current logic is hardcoded for RSA keys. It should instead let OpenSSL handle everything directly. This is just a guess though, I could be wrong.

Workaround: generate a RSA key specifically for gitlab.yml and the docker-registry back-end. The nginx proxy still uses the ECDSA variant. This instantly resolves the issue on F5, no reload or restart is needed.

Edited by 🤖 GitLab Bot 🤖