Skip to content

Neither PUB key nor PRIV key: nested asn1 error when gitlab-runner checking for jobs

Summary

After updated to 12.10.1, when gitlab-runner try to checking for jobs, it's fail (HTTP 500 error) due to an exception in the /var/log/gitlab/gitlab-rails/api_json.log log file (see below).

Steps to reproduce

  • Update to 12.10.1
  • Trying to start a new job
  • Job seems to be inactive
  • /var/log/gitlab/nginx/gitlab_access.log have an 500 error response
  • /var/log/gitlab/gitlab-rails/api_json.log have an exception Neither PUB key nor PRIV key: nested asn1 error (like #32097 maybe?)

What is the current bug behavior?

It's seems that jwt.rb doesn't read the Rails.application.secrets.openid_connect_signing_key option correctly in the lib/gitlab/ci/jwt.rb file, line 63.

What is the expected correct behavior?

Read the Rails.application.secrets.openid_connect_signing_key option correctly

Relevant logs and/or screenshots

In /var/log/gitlab/gitlab-rails/api_json.log file:

{"time":"2020-04-27T13:27:21.573Z","severity":"INFO","duration_s":0.24,"db_duration_s":0.1,"view_duration_s":0.14,"status":500,"method":"POST","path":"/api/v4/jobs/request","params":[{"key":"info","value":{"name":"gitlab-runner","version":"12.10.1","revision":"ce065b93","platform":"linux","architecture":"amd64","executor":"docker","shell":"bash","features":{"variables":"[FILTERED]","image":null,"services":null,"artifacts":null,"cache":null,"shared":null,"upload_multiple_artifacts":null,"upload_raw_artifacts":null,"session":null,"terminal":null,"refspecs":null,"masking":null,"proxy":null,"raw_variables":"[FILTERED]"}}},{"key":"token","value":"[FILTERED]"},{"key":"last_update","value":"6d94cba2f539cfa618b5038915bb35f4"}],"host":"[PRIVATE]","remote_ip":"[PRIVATE], [PRIVATE]","ua":"gitlab-runner 12.10.1 (12-10-stable; go1.13.8; linux/amd64)","route":"/api/:version/jobs/request","exception.class":"OpenSSL::PKey::RSAError","exception.message":"Neither PUB key nor PRIV key: nested asn1 error","exception.backtrace":["lib/gitlab/ci/jwt.rb:63:in `initialize'","lib/gitlab/ci/jwt.rb:63:in `new'","lib/gitlab/ci/jwt.rb:63:in `key'","lib/gitlab/ci/jwt.rb:67:in `public_key'","lib/gitlab/ci/jwt.rb:71:in `kid'","lib/gitlab/ci/jwt.rb:23:in `encoded'","lib/gitlab/ci/jwt.rb:10:in `for_build'","app/models/ci/build.rb:983:in `block in job_jwt_variables'","app/models/ci/build.rb:980:in `tap'","app/models/ci/build.rb:980:in `job_jwt_variables'","app/models/ci/build.rb:528:in `block in variables'","lib/gitlab/utils/strong_memoize.rb:30:in `strong_memoize'","app/models/ci/build.rb:525:in `variables'","app/presenters/ci/build_runner_presenter.rb:121:in `block in git_depth_variable'","lib/gitlab/utils/strong_memoize.rb:30:in `strong_memoize'","app/presenters/ci/build_runner_presenter.rb:120:in `git_depth_variable'","app/presenters/ci/build_runner_presenter.rb:28:in `git_depth'","app/presenters/ci/build_runner_presenter.rb:40:in `refspecs'"],"queue_duration_s":0.01,"gitaly_calls":2,"gitaly_duration_s":0,"redis_calls":9,"redis_duration_s":0.01,"correlation_id":"ERgArTStN83"}

Output of checks

Results of GitLab environment info

Expand for output related to GitLab environment info 
System information
System:		Debian 9.12
Current User:	git
Using RVM:	no
Ruby Version:	2.6.5p114
Gem Version:	2.7.10
Bundler Version:1.17.3
Rake Version:	12.3.3
Redis Version:	5.0.7
Git Version:	2.26.2
Sidekiq Version:5.2.7
Go Version:	unknown

GitLab information
Version:	12.10.1
Revision:	e658772bd63
Directory:	/opt/gitlab/embedded/service/gitlab-rails
DB Adapter:	PostgreSQL
DB Version:	11.7
URL:		[DOMAIN]
HTTP Clone URL:	https://[DOMAIN]/some-group/some-project.git
SSH Clone URL:	ssh://git@[DOMAIN]/some-group/some-project.git
Using LDAP:	no
Using Omniauth:	yes
Omniauth Providers: 

GitLab Shell
Version:	12.2.0
Repository storage paths:
- default: 	/var/opt/gitlab/git-data/repositories
GitLab Shell path:		/opt/gitlab/embedded/service/gitlab-shell
Git:		/opt/gitlab/embedded/bin/git

Results of GitLab application Check

Expand for output related to the GitLab application check 
Checking GitLab subtasks ...

Checking GitLab Shell ...


GitLab Shell: ... GitLab Shell version >= 12.2.0 ? ... OK (12.2.0)
Running /opt/gitlab/embedded/service/gitlab-shell/bin/check
Internal API available: OK
Redis available via internal API: OK
gitlab-shell self-check successful


Checking GitLab Shell ... Finished


Checking Gitaly ...


Gitaly: ... default ... OK


Checking Gitaly ... Finished


Checking Sidekiq ...


Sidekiq: ... Running? ... yes
Number of Sidekiq processes ... 1


Checking Sidekiq ... Finished


Checking Incoming Email ...


Incoming Email: ... Reply by email is disabled in config/gitlab.yml


Checking Incoming Email ... Finished


Checking LDAP ...


LDAP: ... LDAP is disabled in config/gitlab.yml


Checking LDAP ... Finished


Checking GitLab App ...


Git configured correctly? ... yes
Database config exists? ... yes
All migrations up? ... yes
Database contains orphaned GroupMembers? ... no
GitLab config exists? ... yes
GitLab config up to date? ... yes
Log directory writable? ... yes
Tmp directory writable? ... yes
Uploads directory exists? ... yes
Uploads directory has correct permissions? ... yes
Uploads directory tmp has correct permissions? ... yes
Init script exists? ... skipped (omnibus-gitlab has no init script)
Init script up-to-date? ... skipped (omnibus-gitlab has no init script)
Projects have namespace: ... yes
Redis version >= 4.0.0? ... yes
Ruby version >= 2.5.3 ? ... yes (2.6.5)
Git version >= 2.22.0 ? ... yes (2.26.2)
Git user has default SSH configuration? ... yes
Active users: ... 15
Is authorized keys file accessible? ... yes


Checking GitLab App ... Finished


Checking GitLab subtasks ... Finished

Possible fixes

When I copy the gitlab_rails.openid_connect_signing_key from the gitlab-secrets.json file to directly in the /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/ci/jwt.rb file, it's works.

Replacing:

@key ||= OpenSSL::PKey::RSA.new(Rails.application.secrets.openid_connect_signing_key)

by:

@key ||= OpenSSL::PKey::RSA.new("-----BEGIN RSA PRIVATE KEY-----\n<MY_RSA_PRIVATE_KEY_CONTENT>\n-----END RSA PRIVATE KEY-----\n")