After update, cannot login to registry - 400 bad request
Over the weekend I upgraded from 11.5.0 to 11.7.3. No configs were changed and no network changes occurred. Now when I run my pipelines none of them can log into my registry.
$ docker login -u $CI_REGISTRY_USER -p $CI_JOB_TOKEN $CI_REGISTRY
WARNING! Using --password via the CLI is insecure. Use --password-stdin.
time="2019-02-03T11:39:20Z" level=info msg="Error logging in to v2 endpoint, trying next endpoint: login attempt to https://gitlab.domain.net:6565/v2/ failed with status: 400 Bad Request"
Login: <html>
<head><title>400 The plain HTTP request was sent to HTTPS port</title></head>
<body bgcolor="white">
<center><h1>400 Bad Request</h1></center>
<center>The plain HTTP request was sent to HTTPS port</center>
<hr><center>nginx</center>
</body>
</html>
(Code: 400; Headers: map[Content-Length:[264] Server:[nginx] Date:[Sun, 03 Feb 2019 11:39:20 GMT] Content-Type:[text/html]])
ERROR: Job failed: exit code 1What i have in gitlab.rb which has always worked up until now:
###Container Registry
gitlab_rails['registry_enabled'] = true
gitlab_rails['registry_host'] = "gitlab.domain.net"
gitlab_rails['registry_port'] = "6565"
registry['registry_http_addr'] = "gitlab.domain.net:6565"
registry['log_level'] = "debug"
gitlab_rails['registry_issuer'] = "omnibus-gitlab-issuer"
registry['enable'] =true
registry['debug_addr'] = "localhost:5001"
registry_external_url 'https://gitlab.domain.net:6565'
registry_nginx['ssl_certificate'] = '/etc/gitlab/ssl/gitlab.domain.net.pem'
registry_nginx['ssl_certificate_key'] = '/etc/gitlab/ssl/gitlab.domain.net.key'
registry['storage'] = {
's3' => {
'accesskey' => 'ACCESS_KEY',
'secretkey' => 'SECRET_KEY',
'bucket' => 'bucket_name',
'region' => 'us-east-1'
}
}I also get this if i try to log in from my local which also did work prior to the upgrade:
$ docker login gitlab.domain.net:6565
Username: sfloyd
Password:
Error response from daemon: login attempt to https://gitlab.domain.net:6565/v2/ failed with status: 400 Bad RequestHere are the last few lines of gitlab_registry_access.log My gitlab_registry_error.log is empty.
10.X.X.X - - [03/Feb/2019:13:15:21 +0000] "GET /v2/ HTTP/1.1" 400 264 "" "Docker-Client/18.09.1 (linux)"
10.X.X.X - - [03/Feb/2019:13:15:21 +0000] "GET /v2/ HTTP/1.1" 400 264 "" "Docker-Client/18.09.1 (linux)"
10.X.X.X - - [03/Feb/2019:13:15:21 +0000] "GET /v2/ HTTP/1.1" 400 264 "" "Docker-Client/18.09.1 (linux)"
10.X.X.X - - [03/Feb/2019:13:15:21 +0000] "GET /v2/ HTTP/1.1" 400 264 "" "Docker-Client/18.09.1 (linux)"
10.X.X.X - gitlab-ci-token [03/Feb/2019:13:15:21 +0000] "GET /v1/users/ HTTP/1.1" 400 264 "" "Docker-Client/18.09.1 (linux)"
10.X.X.X - gitlab-ci-token [03/Feb/2019:13:15:21 +0000] "GET /v1/users/ HTTP/1.1" 400 264 "" "Docker-Client/18.09.1 (linux)"
10.X.X.X - - [03/Feb/2019:13:21:12 +0000] "GET /v2/ HTTP/1.1" 400 264 "" "docker/18.09.1 go/go1.10.6 git-commit/4c52b90 kernel/4.9.125-linuxkit os/linux arch/amd64 UpstreamClient(Docker-Client/18.09.1 \x5C(darwin\x5C))"
52.X.X.X - - [03/Feb/2019:13:21:12 +0000] "GET /v2/ HTTP/1.1" 400 264 "" "docker/18.09.1 go/go1.10.6 git-commit/4c52b90 kernel/4.9.125-linuxkit os/linux arch/amd64 UpstreamClient(Docker-Client/18.09.1 \x5C(darwin\x5C))"
10.X.X.X - - [03/Feb/2019:13:21:12 +0000] "GET /v2/ HTTP/1.1" 400 264 "" "docker/18.09.1 go/go1.10.6 git-commit/4c52b90 kernel/4.9.125-linuxkit os/linux arch/amd64 UpstreamClient(Docker-Client/18.09.1 \x5C(darwin\x5C))"
52.X.X.X - - [03/Feb/2019:13:21:12 +0000] "GET /v2/ HTTP/1.1" 400 264 "" "docker/18.09.1 go/go1.10.6 git-commit/4c52b90 kernel/4.9.125-linuxkit os/linux arch/amd64 UpstreamClient(Docker-Client/18.09.1 \x5C(darwin\x5C))"
10.X.X.X - - [03/Feb/2019:13:42:17 +0000] "GET /v2/ HTTP/1.1" 400 264 "" "docker/18.09.1 go/go1.10.6 git-commit/4c52b90 kernel/4.9.125-linuxkit os/linux arch/amd64 UpstreamClient(Docker-Client/18.09.1 \x5C(darwin\x5C))"
52.X.X.X - - [03/Feb/2019:13:42:17 +0000] "GET /v2/ HTTP/1.1" 400 264 "" "docker/18.09.1 go/go1.10.6 git-commit/4c52b90 kernel/4.9.125-linuxkit os/linux arch/amd64 UpstreamClient(Docker-Client/18.09.1 \x5C(darwin\x5C))"
10.X.X.X - - [03/Feb/2019:13:42:17 +0000] "GET /v2/ HTTP/1.1" 400 264 "" "docker/18.09.1 go/go1.10.6 git-commit/4c52b90 kernel/4.9.125-linuxkit os/linux arch/amd64 UpstreamClient(Docker-Client/18.09.1 \x5C(darwin\x5C))"
52.X.X.X - - [03/Feb/2019:13:42:17 +0000] "GET /v2/ HTTP/1.1" 400 264 "" "docker/18.09.1 go/go1.10.6 git-commit/4c52b90 kernel/4.9.125-linuxkit os/linux arch/amd64 UpstreamClient(Docker-Client/18.09.1 \x5C(darwin\x5C))"Ive run gitlab-ctl reconfigure a few times and restarted the services. Ive also restarted both my gitlab and gitlab-runner boxes and still get the same error. Not really sure what else to try.