Internal API unreachable after Helm upgrade (4.2.3 - 4.2.4 or from 4.2.2 - 4.2.3)
Summary
Installing Gitlab on on-prem K8s using Helm chart gitlab/gitlab version 4.2.3 and then upgrading using Operator results in Internal API unreachable errors when committing changes to a project.
On-prem K8s is build with Kubespray.
Steps to reproduce
Deploy
# Default values for gitlab/gitlab chart
## NOTICE
# Due to the scope and complexity of this chart, all possible values are
# not documented in this file. Extensive documentation for these values
# and more can be found at https://gitlab.com/gitlab-org/charts/gitlab/
## Advanced Configuration
# Documentation for advanced configuration can be found under doc/advanced
# - external PostgreSQL
# - external Gitaly
# - external Redis
# - external NGINX
# - PersistentVolume configuration
# - external Object Storage providers
## The global properties are used to configure multiple charts at once.
## Extended documenation at doc/charts/globals.md
global:
## GitLab operator is Alpha. Not for production use.
operator:
enabled: True
rollout:
# Enables automatic pause for deployment rollout. This must be set to `true` to fix
# Helm's issue with 3-way merge. See:
# https://gitlab.com/gitlab-org/charts/gitlab/issues/1262
# https://github.com/helm/helm/issues/3805
autoPause: true
## doc/installation/deployment.md#deploy-the-community-edition
edition: ce
## doc/charts/globals.md#gitlab-version
# gitlabVersion: master
## doc/charts/globals.md#application-resource
application:
create: false
links: []
allowClusterRoles: true
## doc/charts/globals.md#configure-host-settings
hosts:
domain: demo.local
hostSuffix:
https: True
externalIP:
ssh: ~
gitlab:
name: gitlab.demo.local
minio:
name: minio.demo.local
registry:
name: registry.demo.local
tls: {}
## doc/charts/globals.md#configure-ingress-settings
ingress:
configureCertmanager: False
labels:
app: gitlab
traffic: external
annotations:
kubernetes.io/ingress.class: nginx
cert-manager.io/issuer: "ca-issuer"
enabled: True
tls:
enabled: True
## Initial root password for this GitLab installation
## Secret created according to doc/installation/secrets.md#initial-root-password
## If allowing shared-secrets generation, this is OPTIONAL.
initialRootPassword: {}
# secret: RELEASE-gitlab-initial-root-password
# key: password
## doc/charts/globals.md#configure-postgresql-settings
psql:
password:
secret: gitlab-dbuser-secret
key: dbuser-password
host: gitlab-psql-postgresql.automation-framework
port: 5432
username: gitlab
database: gitlabhq_production
# pool: 10
# preparedStatements: false
## doc/charts/globals.md#configure-redis-settings
redis:
password:
enabled: true
# secret:
# key:
# host: redis.hostedsomewhere.else
# port: 6379
# sentinels:
# - host:
# port:
persistence:
storageClass: standard
size: 5Gi
image:
tag: 5.0.7-debian-10-r32
## doc/charts/globals.md#configure-gitaly-settings
gitaly:
enabled: true
authToken: {}
# secret:
# key:
# serviceName:
internal:
names: ['default']
external: []
tls:
enabled: false
# secretName:
persistence:
storageClass: standard
size: 50Gi
## doc/charts/globals.md#configure-minio-settings
minio:
enabled: true
credentials: {}
# secret:
persistence:
storageClass: standard
size: 10Gi
ingress:
enabled: true
tls:
enabled: true
secretName: gitlab-all-tls
## doc/charts/globals.md#configure-grafana-integration
grafana:
enabled: false
## doc/charts/globals.md#configure-appconfig-settings
## Rails based portions of this chart share many settings
appConfig:
## doc/charts/globals.md#general-application-settings
enableUsagePing: true
enableSeatLink: true
enableImpersonation:
defaultCanCreateGroup: true
usernameChangingEnabled: true
issueClosingPattern:
defaultTheme:
defaultProjectsFeatures:
issues: true
mergeRequests: true
wiki: true
snippets: true
builds: true
webhookTimeout:
maxRequestDurationSeconds:
## doc/charts/globals.md#cron-jobs-related-settings
cron_jobs: {}
## Flag stuck CI builds as failed
# stuck_ci_jobs_worker:
# cron: "0 * * * *"
## Schedule pipelines in the near future
# pipeline_schedule_worker:
# cron: "19 * * * *"
## Remove expired build artifacts
# expire_build_artifacts_worker:
# cron: "50 * * * *"
## Periodically run 'git fsck' on all repositories.
# repository_check_worker:
# cron: "20 * * * *"
## Send admin emails once a week
# admin_email_worker:
# cron: "0 0 * * 0"
## Remove outdated repository archives
# repository_archive_cache_worker:
# cron: "0 * * * *"
## Verify custom GitLab Pages domains
# pages_domain_verification_cron_worker:
# cron: "*/15 * * * *"
## Export pseudonymized data
# pseudonymizer_worker:
# cron: "0 * * * *"
# schedule_migrate_external_diffs_worker:
# cron: "15 * * * *"
### GitLab Geo
# Geo Primary only!
# geo_prune_event_log_worker:
# cron: "*/5 * * * *"
## GitLab Geo repository sync worker
# geo_repository_sync_worker:
# cron: "*/5 * * * *"
## GitLab Geo file download dispatch worker
# geo_file_download_dispatch_worker:
# cron: "*/10 * * * *"
## GitLab Geo repository verification primary batch worker
# geo_repository_verification_primary_batch_worker:
# cron: "*/5 * * * *"
## GitLab Geo repository verification secondary scheduler worker
# geo_repository_verification_secondary_scheduler_worker:
# cron: "*/5 * * * *"
## GitLab Geo migrated local files clean up worker
# geo_migrated_local_files_clean_up_worker:
# cron: "15 */6 * * *"
### LDAP
# ldap_sync_worker:
# cron: "30 1 * * *"
# ldap_group_sync_worker:
# cron: "0 * * * *"
### Snapshot active user statistics
# historical_data_worker:
# cron: "0 12 * * *"
## doc/charts/globals.md#incoming-email-settings
## doc/installation/deployment.md#incoming-email
incomingEmail:
enabled: false
address: ""
host: "imap.gmail.com"
port: 993
ssl: true
startTls: false
user: ""
password:
secret: ""
key: password
expungeDeleted: false
logger:
logPath: "/dev/stdout"
mailbox: inbox
idleTimeout: 60
## doc/charts/globals.md#ldap
ldap:
# prevent the use of LDAP for sign-in via web.
preventSignin: false
servers:
## 'main' is the GitLab 'provider ID' of this LDAP server
main:
label: demolab
host: ldap.demo.local
port: 636 #389
uid: 'sAMAccountName'
bind_dn: CN=svc-af-vault,OU=Service Accounts,OU=demo Lab,DC=demolab,DC=local
password:
secret: binddn-secret
key: pass
active_directory: true
verify_certificates: False
allow_username_or_email_login: true
block_auto_created_users: false
encryption: 'simple_tls'
ssl_version: 'TLSv1_2'
base: DC=demolab,DC=local
group_base: OU=Groups,OU=demo,DC=demolab,DC=local
admin_group: admins
userfilter: ''
# ca_file: '/etc/ssl/certs/ca-cert-my-custom-ca.pem'
attributes:
username: ['sAMAccountName']
email: ['userPrincipleName']
name: 'displayName'
first_name: 'givenName'
last_name: 'sn'
## End of global.appConfig
## doc/charts/globals.md#configure-registry-settings
registry:
bucket: registry
certificate: {}
# secret:
httpSecret: {}
# secret:
# key:
ingress:
tls:
enabled: true
secretName: gitlab-all-tls
## doc/installation/deployment.md#outgoing-email
## Outgoing email server settings
smtp:
enabled: True
address: mailak.company.com
port: 25
user_name: ""
## doc/installation/secrets.md#smtp-password
password:
secret: ""
key: password
# domain:
authentication: ""
starttls_auto: false
openssl_verify_mode: none
## doc/installation/deployment.md#outgoing-email
## Email persona used in email sent by GitLab
email:
from: gitlab-demo@company.com
display_name: Gitlab-demo
reply_to: ''
subject_suffix: ''
smime:
enabled: false
secretName: ""
keyName: "tls.key"
certName: "tls.crt"
## Timezone for containers.
time_zone: UTC
## Global Service Annotations
service:
annotations: {}
## Global Deployment Annotations
deployment:
annotations: {}
antiAffinity: soft
## doc/installation/secrets.md#gitlab-workhorse-secret
workhorse: {}
# secret:
# key:
## doc/charts/globals.md#configure-webservice
webservice:
workerTimeout: 60
# ingress:
# tls:
# secretName: gitlab-gitlab-tls
## End of global
upgradeCheck:
enabled: true
image: {}
# repository:
# tag:
securityContext:
# in alpine/debian/busybox based images, this is `nobody:nogroup`
runAsUser: 65534
fsGroup: 65534
tolerations: []
resources:
requests:
cpu: 50m
## Settings to for the Let's Encrypt ACME Issuer
certmanager-issuer: {}
# ## The email address to register certificates requested from Let's Encrypt.
# ## Required if using Let's Encrypt.
# email: dummy@email.com
## Installation & configuration of jetstack/cert-manager
## See requirements.yaml for current version
certmanager:
createCustomResource: true
nameOverride: cert-manager
# Install cert-manager chart. Set to false if you already have cert-manager
# installed or if you are not using cert-manager.
install: False
# Other cert-manager configurations from upstream
# See https://github.com/jetstack/cert-manager/blob/master/deploy/charts/cert-manager/README.md#configuration
rbac:
create: true
webhook:
enabled: false
## doc/charts/nginx/index.md
## doc/architecture/decisions.md#nginx-ingress
## Installation & configuration of charts/nginx
nginx-ingress:
enabled: False
tcpExternalConfig: "true"
controller:
config:
hsts-include-subdomains: "false"
server-name-hash-bucket-size: "256"
enable-vts-status: "true"
use-http2: "false"
ssl-ciphers: "ECDHE-RSA-AES256-GCM-SHA384:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-RSA-AES256-SHA384:ECDHE-RSA-AES128-SHA256:ECDHE-RSA-AES256-SHA:ECDHE-RSA-AES128-SHA:AES256-GCM-SHA384:AES128-GCM-SHA256:AES256-SHA256:AES128-SHA256:AES256-SHA:AES128-SHA:!aNULL:!eNULL:!EXPORT:!DES:!MD5:!PSK:!RC4"
ssl-protocols: "TLSv1.3 TLSv1.2"
server-tokens: "false"
extraArgs:
force-namespace-isolation: ""
service:
externalTrafficPolicy: "Local"
resources:
requests:
cpu: 100m
memory: 100Mi
publishService:
enabled: true
replicaCount: 3
minAvailable: 2
scope:
enabled: true
stats:
enabled: true
metrics:
enabled: true
service:
annotations:
prometheus.io/scrape: "true"
prometheus.io/port: "10254"
defaultBackend:
minAvailable: 1
replicaCount: 2
resources:
requests:
cpu: 5m
memory: 5Mi
rbac:
create: true
serviceAccount:
create: true
## Installation & configuration of stable/prometheus
## See requirements.yaml for current version
prometheus:
install: false
rbac:
create: true
alertmanager:
enabled: false
alertmanagerFiles:
alertmanager.yml: {}
kubeStateMetrics:
enabled: false
nodeExporter:
enabled: false
pushgateway:
enabled: false
server:
retention: 15d
## Configuration of Redis
## doc/architecture/decisions.md#redis
## doc/charts/redis
redis:
install: true
existingSecret: gitlab-redis-secret
existingSecretKey: redis-password
usePasswordFile: true
cluster:
enabled: false
metrics:
enabled: true
## Instllation & configuration of stable/prostgresql
## See requirements.yaml for current version
postgresql:
postgresqlUsername: gitlab
# This just needs to be set. It will use a second entry in existingSecret for postgresql-postgres-password
postgresqlPostgresPassword: bogus
install: False
postgresqlDatabase: gitlabhq_production
image:
tag: 11.7.0
usePasswordFile: true
existingSecret: 'bogus'
initdbScriptsConfigMap: 'bogus'
metrics:
enabled: true
## Optionally define additional custom metrics
## ref: https://github.com/wrouesnel/postgres_exporter#adding-new-metrics-via-a-config-file
## Installation & configuration charts/registry
## doc/architecture/decisions.md#registry
## doc/charts/registry/
registry:
enabled: true
## Automatic shared secret generation
## doc/installation/secrets.md
## doc/charts/shared-secrets
shared-secrets:
enabled: true
rbac:
create: true
## Installation & configuration of gitlab/gitlab-runner
## See requirements.yaml for current version
gitlab-runner:
install: False
rbac:
create: true
runners:
locked: false
cache:
cacheType: s3
s3BucketName: runner-cache
cacheShared: true
s3BucketLocation: us-east-1
s3CachePath: gitlab-runner
s3CacheInsecure: false
## Settings for individual sub-charts under GitLab
## Note: Many of these settings are configurable via globals
gitlab:
## doc/charts/gitlab/task-runner
task-runner:
replicas: 1
## doc/charts/gitlab/migrations
# migrations:
# enabled: false
## doc/charts/gitlab/webservice
# webservice:
webservice:
ingress:
tls:
secretName: gitlab-all-tls
# enabled: false
## doc/charts/gitlab/sidekiq
# sidekiq:
# enabled: false
## doc/charts/gitlab/gitaly
# gitaly:
## doc/charts/gitlab/gitlab-shell
# gitlab-shell:
# enabled: false
## doc/charts/gitlab/gitlab-grafana
# gitlab-grafana:
What is the current bug behavior?
Any commits to projects result in Internal API unreachable errors
What is the expected correct behavior?
Successful commit
Relevant logs and/or screenshots
*** /var/log/gitlab/exceptions_json.log ***
{"severity":"ERROR","time":"2020-08-13T19:06:46.448Z","correlation_id":"RMlejdkwtK3","tags.correlation_id":"RMlejdkwtK3","tags.locale":"en","user.id":2,"user.email":"tmckenzie@demo.local","user.username":"tmckenzie","exception.class":"Gitlab::Git::PreReceiveError","exception.message":"Internal API unreachable","exception.backtrace":["lib/gitlab/gitaly_client/operation_service.rb:342:in `user_commit_files'","lib/gitlab/git/repository.rb:881:in `block in multi_action'","lib/gitlab/git/wraps_gitaly_errors.rb:7:in `wrapped_gitaly_errors'","lib/gitlab/git/repository.rb:880:in `multi_action'","app/models/repository.rb:822:in `block in multi_action'","app/models/repository.rb:805:in `with_cache_hooks'","app/models/repository.rb:822:in `multi_action'","app/models/repository.rb:795:in `update_file'","app/services/files/update_service.rb:6:in `create_commit!'","app/services/commits/create_service.rb:29:in `execute'","app/controllers/concerns/creates_commit.rb:25:in `create_commit'","app/controllers/projects/blob_controller.rb:76:in `update'","app/controllers/application_controller.rb:491:in `set_current_admin'","lib/gitlab/session.rb:11:in `with_session'","app/controllers/application_controller.rb:482:in `set_session_storage'","app/controllers/application_controller.rb:476:in `set_locale'","lib/gitlab/error_tracking.rb:50:in `with_context'","app/controllers/application_controller.rb:541:in `sentry_context'","app/controllers/application_controller.rb:469:in `block in set_current_context'","lib/gitlab/application_context.rb:52:in `block in use'","lib/gitlab/application_context.rb:52:in `use'","lib/gitlab/application_context.rb:20:in `with_context'","app/controllers/application_controller.rb:462:in `set_current_context'"]}
{"duration_ms":3007,"error":"Post http://gitlab-webservice:8181//api/v4/internal/allowed: dial tcp 10.233.30.45:8181: connect: no route to host","level":"error","method":"POST","msg":"Internal API unreachable","time":"2020-08-16T19:33:50.539Z","url":"http://gitlab-webservice:8181//api/v4/internal/allowed"}Output of checks
(If you are reporting a bug on GitLab.com, write: This bug happens on GitLab.com)
Results of GitLab environment info
Expand for output related to GitLab environment info
System information System: Current User: git Using RVM: no Ruby Version: 2.6.6p146 Gem Version: 2.7.10 Bundler Version:1.17.3 Rake Version: 12.3.3 Redis Version: unknown Git Version: unknown Sidekiq Version:5.2.9 Go Version: unknown GitLab information Version: 13.2.4 Revision: Directory: /srv/gitlab DB Adapter: PostgreSQL DB Version: 11.7 URL: https://gitlab.demo.local HTTP Clone URL: https://gitlab.demo.local/some-group/some-project.git SSH Clone URL: git@gitlab.demo.local:some-group/some-project.git Using LDAP: yes Using Omniauth: no GitLab Shell Version: 13.3.0 Repository storage paths: - default: /var/opt/gitlab/repo GitLab Shell path: /home/git/gitlab-shell Git: /usr/bin/git
Results of GitLab application Check
Expand for output related to the GitLab application check
(For installations with omnibus-gitlab package run and paste the output of:
sudo gitlab-rake gitlab:check SANITIZE=true) $ gitlab-rake gitlab:check SANITIZE=true Checking GitLab subtasks ...Checking GitLab Shell ...
GitLab Shell: ... GitLab Shell version >= 13.3.0 ? ... OK (13.3.0) Running /home/git/gitlab-shell/bin/check gitlab-shell self-check failed Try fixing it: Make sure GitLab is running; Check the gitlab-shell configuration file: sudo -u git -H editor /home/git/gitlab-shell/config.yml Please fix the error above and rerun the checks.
Checking GitLab Shell ... Finished
Checking Gitaly ...
Gitaly: ... default ... OK
Checking Gitaly ... Finished
Checking Sidekiq ...
Sidekiq: ... Running? ... no Try fixing it: sudo -u git -H RAILS_ENV=production bin/background_jobs start For more information see: doc/install/installation.md in section "Install Init Script" see log/sidekiq.log for possible errors Please fix the error above and rerun the checks.
Checking Sidekiq ... Finished
Checking Incoming Email ...
Incoming Email: ... Reply by email is disabled in config/gitlab.yml
Checking Incoming Email ... Finished
Checking LDAP ...
LDAP: ... Server: ldapmain not verifying SSL hostname of LDAPS server 'ldap.demo.local:636' LDAP authentication... Success LDAP users with access to your GitLab server (only showing the first 100 results) User output sanitized. Found 100 users of 100 limit.
Checking LDAP ... Finished
Checking GitLab App ...
Git configured correctly? ... no Trying to fix error automatically. ...Failed Try fixing it: sudo -u git -H "/usr/bin/git" config --global core.autocrlf "input" For more information see: doc/install/installation.md in section "GitLab" Database config exists? ... yes All migrations up? ... yes Database contains orphaned GroupMembers? ... no GitLab config exists? ... yes GitLab config up to date? ... yes Log directory writable? ... yes Tmp directory writable? ... yes Uploads directory exists? ... yes Uploads directory has correct permissions? ... yes Uploads directory tmp has correct permissions? ... skipped (no tmp uploads folder yet) Init script exists? ... no Try fixing it: Install the init script For more information see: doc/install/installation.md in section "Install Init Script" Please fix the error above and rerun the checks. Init script up-to-date? ... can't check because of previous errors Projects have namespace: ... 2/1 ... yes 2/2 ... yes Redis version >= 4.0.0? ... yes Ruby version >= 2.5.3 ? ... yes (2.6.6) Git version >= 2.22.0 ? ... no Your git bin path is "/usr/bin/git" Try fixing it: Update your git to a version >= 2.22.0 from Unknown Please fix the error above and rerun the checks. Git user has default SSH configuration? ... yes Active users: ... 2 Is authorized keys file accessible? ... skipped (authorized keys not enabled) GitLab configured to store new projects in hashed storage? ... yes All projects are in hashed storage? ... yes
Checking GitLab App ... Finished
Checking GitLab subtasks ... Finished
Possible fixes
Edited by Trenthani