Auto DevOps deploy fails on ensure_namespace - "x509: certificate signed by unknown authority"
Summary
Running Kubernetes on Google Cloud via Google Container Engine. Clusters setup on GC use self-signed certificates, and typically use the GC IAM service accounts for authentication. This is likely causing the issue.
Steps to reproduce
Run an auto devops deployment on Google Cloud wired up to GC Container Engine (kubernetes).
What is the current bug behavior?
What is the expected correct behavior?
The deployment succeeds.
Relevant logs and/or screenshots
deployment step output:
Running with gitlab-runner 10.0.2 (a9a76a50)
on server-optic-nexus (21590677)
Using Kubernetes namespace: gitlab
Using Kubernetes executor with image alpine:latest ...
Waiting for pod gitlab/runner-21590677-project-56-concurrent-0r2q3v to be running, status is Pending
Waiting for pod gitlab/runner-21590677-project-56-concurrent-0r2q3v to be running, status is Pending
Running on runner-21590677-project-56-concurrent-0r2q3v via server-optic-nexus...
Cloning repository...
Cloning into '/chris.eaton/test'...
Checking out 0240bfd2 as master...
Skipping Git submodules setup
Downloading artifacts for codequality (252)...
Downloading artifacts from coordinator... ok id=252 responseStatus=200 OK token=CzdnWp28
$ # Auto DevOps variables and functions # collapsed multi-line command
$ check_kube_domain
$ install_dependencies
fetch http://dl-cdn.alpinelinux.org/alpine/v3.6/main/x86_64/APKINDEX.tar.gz
fetch http://dl-cdn.alpinelinux.org/alpine/v3.6/community/x86_64/APKINDEX.tar.gz
(1/17) Installing ncurses-terminfo-base (6.0_p20170930-r0)
(2/17) Installing ncurses-terminfo (6.0_p20170930-r0)
(3/17) Installing ncurses-libs (6.0_p20170930-r0)
(4/17) Installing readline (6.3.008-r5)
(5/17) Installing bash (4.3.48-r1)
Executing bash-4.3.48-r1.post-install
(6/17) Installing ca-certificates (20161130-r2)
(7/17) Installing libssh2 (1.8.0-r1)
(8/17) Installing libcurl (7.56.0-r0)
(9/17) Installing curl (7.56.0-r0)
(10/17) Installing expat (2.2.0-r1)
(11/17) Installing pcre (8.41-r0)
(12/17) Installing git (2.13.5-r0)
(13/17) Installing gzip (1.8-r0)
(14/17) Installing libcrypto1.0 (1.0.2k-r0)
(15/17) Installing libssl1.0 (1.0.2k-r0)
(16/17) Installing openssl (1.0.2k-r0)
(17/17) Installing tar (1.29-r1)
Executing busybox-1.26.2-r5.trigger
Executing ca-certificates-20161130-r2.trigger
OK: 38 MiB in 28 packages
Connecting to github.com (192.30.255.113:443)
Connecting to github-production-release-asset-2e65be.s3.amazonaws.com (52.216.17.160:443)
glibc-2.23-r3.apk 9% |** | 271k 0:00:09 ETA
glibc-2.23-r3.apk 55% |***************** | 1609k 0:00:01 ETA
glibc-2.23-r3.apk 100% |*******************************| 2874k 0:00:00 ETA
(1/1) Installing glibc (2.23-r3)
OK: 42 MiB in 29 packages
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0
9 15.5M 9 1582k 0 0 1582k 0 0:00:10 --:--:-- 0:00:10 4548k
100 15.5M 100 15.5M 0 0 15.5M 0 0:00:01 0:00:01 --:--:-- 14.5M
Client: &version.Version{SemVer:"v2.6.1", GitCommit:"bbc1f71dc03afc5f00c6ac84b9308f8ecb4f39ac", GitTreeState:"clean"}
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0
22 49.8M 22 11.3M 0 0 11.3M 0 0:00:04 --:--:-- 0:00:04 33.6M
100 49.8M 100 49.8M 0 0 49.8M 0 0:00:01 --:--:-- 0:00:01 88.7M
Client Version: version.Info{Major:"1", Minor:"8", GitVersion:"v1.8.1", GitCommit:"f38e43b221d08850172a9a4ea785a86a3ffa3b3a", GitTreeState:"clean", BuildDate:"2017-10-11T23:27:35Z", GoVersion:"go1.8.3", Compiler:"gc", Platform:"linux/amd64"}
$ download_chart
Creating /root/.helm
Creating /root/.helm/repository
Creating /root/.helm/repository/cache
Creating /root/.helm/repository/local
Creating /root/.helm/plugins
Creating /root/.helm/starters
Creating /root/.helm/cache/archive
Creating /root/.helm/repository/repositories.yaml
$HELM_HOME has been configured at /root/.helm.
Not installing Tiller due to 'client-only' flag having been set
Happy Helming!
"gitlab" has been added to your repositories
Hang tight while we grab the latest from your chart repositories...
...Unable to get an update from the "local" chart repository (http://127.0.0.1:8879/charts):
Get http://127.0.0.1:8879/charts/index.yaml: dial tcp 127.0.0.1:8879: getsockopt: connection refused
...Successfully got an update from the "stable" chart repository
...Successfully got an update from the "gitlab" chart repository
Update Complete. ⎈Happy Helming!⎈
Saving 1 charts
Downloading postgresql from repo https://kubernetes-charts.storage.googleapis.com/
Deleting outdated charts
Hang tight while we grab the latest from your chart repositories...
...Unable to get an update from the "local" chart repository (http://127.0.0.1:8879/charts):
Get http://127.0.0.1:8879/charts/index.yaml: dial tcp 127.0.0.1:8879: getsockopt: connection refused
...Successfully got an update from the "stable" chart repository
...Successfully got an update from the "gitlab" chart repository
Update Complete. ⎈Happy Helming!⎈
Saving 1 charts
Downloading postgresql from repo https://kubernetes-charts.storage.googleapis.com/
Deleting outdated charts
$ ensure_namespace
Unable to connect to the server: x509: certificate signed by unknown authority
Unable to connect to the server: x509: certificate signed by unknown authority
ERROR: Job failed: error executing remote command: command terminated with non-zero exit code: Error executing in Docker Container: 1Results of GitLab environment info
Expand for output related to GitLab environment info
System information System: Debian 8.9 Proxy: no Current User: git Using RVM: no Ruby Version: 2.3.5p376 Gem Version: 2.6.13 Bundler Version:1.13.7 Rake Version: 12.0.0 Redis Version: 3.2.5 Git Version: 2.13.5 Sidekiq Version:5.0.4 Go Version: unknownGitLab information Version: 10.0.3-ee Revision: eff7821 Directory: /opt/gitlab/embedded/service/gitlab-rails DB Adapter: postgresql DB Version: 9.6.3 URL: omitted HTTP Clone URL: omitted/some-group/some-project.git SSH Clone URL: git@omitted:some-group/some-project.git Elasticsearch: no Geo: no Using LDAP: no Using Omniauth: yes Omniauth Providers: google_oauth2
GitLab Shell Version: 5.9.0 Repository storage paths:
- default: /var/opt/gitlab/git-data/repositories Hooks: /opt/gitlab/embedded/service/gitlab-shell/hooks Git: /opt/gitlab/embedded/bin/git
Results of GitLab application Check
Expand for output related to the GitLab application check
Checking GitLab Shell ...GitLab Shell version >= 5.9.0 ? ... OK (5.9.0) Repo base directory exists? default... yes Repo storage directories are symlinks? default... no Repo paths owned by git:root, or git:git? default... yes Repo paths access is drwxrws---? default... yes hooks directories in repos are links: ... 17/1 ... ok 4/2 ... ok 17/3 ... ok 2/4 ... ok 2/5 ... ok 2/6 ... ok 16/8 ... ok 15/9 ... ok 16/14 ... ok 16/17 ... ok 13/18 ... ok 16/19 ... ok 2/20 ... ok 4/23 ... ok 16/24 ... repository is empty 2/25 ... ok 22/26 ... ok 2/30 ... ok 13/31 ... ok 15/32 ... repository is empty 21/35 ... ok 21/37 ... ok 2/41 ... ok 21/42 ... ok 21/43 ... ok Running /opt/gitlab/embedded/service/gitlab-shell/bin/check Check GitLab API access: OK Redis available via internal API: OK
Access to /var/opt/gitlab/.ssh/authorized_keys: OK gitlab-shell self-check successful
Checking GitLab Shell ... Finished
Checking Sidekiq ...
Running? ... yes Number of Sidekiq processes ... 1
Checking Sidekiq ... Finished
Reply by email is disabled in config/gitlab.yml Checking LDAP ...
LDAP is disabled in config/gitlab.yml
Checking LDAP ... Finished
Checking GitLab ...
Git configured correctly? ... yes Database config exists? ... yes All migrations up? ... yes Database contains orphaned GroupMembers? ... no GitLab config exists? ... yes GitLab config up to date? ... yes Log directory writable? ... yes Tmp directory writable? ... yes Uploads directory exists? ... yes Uploads directory has correct permissions? ... yes Uploads directory tmp has correct permissions? ... yes Init script exists? ... skipped (omnibus-gitlab has no init script) Init script up-to-date? ... skipped (omnibus-gitlab has no init script) Projects have namespace: ... 17/1 ... yes 4/2 ... yes 17/3 ... yes 2/4 ... yes 2/5 ... yes 2/6 ... yes 16/8 ... yes 15/9 ... yes 16/14 ... yes 16/17 ... yes 13/18 ... yes Init script up-to-date? ... skipped (omnibus-gitlab has no init script) 16/19 ... yes 2/20 ... yes 4/23 ... yes 16/24 ... yes 2/25 ... yes 22/26 ... yes 2/30 ... yes 13/31 ... yes 15/32 ... yes 21/35 ... yes 21/37 ... yes 2/41 ... yes 21/42 ... yes 21/43 ... yes Redis version >= 2.8.0? ... yes Ruby version >= 2.3.3 ? ... yes (2.3.5) Git version >= 2.7.3 ? ... yes (2.13.5)