(Resolv::ResolvError) error on S3 bucket DNS solving when creating merge request and upload artifacts to package registry
Summary
Hello!
We currently have a self-managed Gitlab instance running on our Kubernetes cluster using a helm chart. Recently, we updated our Gitlab instance version from 14.2.1 (chart 5.2.1) to 14.10.2 (chart 5.10.2) and now we are facing some problems to create merge requests and upload artifacts to package registry.
The problem pointed on our logs is with the S3 bucket DNS solving, but the only buckets with this problem are the ones related to ExternalDiffs and Packages. We have buckets for other storages on our instance and they are working fine.
One of the exceptions thrown on Webservice is put below on the logs session
We checked the connectivity to the bucket from the pods and it's accessible, but these 2 features keep throwing errors.
Can you help us with this problem?
Best regards! Guilherme Ferreira
Steps to reproduce
Update Gitlab instance version from 14.2.1 (chart 5.2.1) to 14.10.2 (chart 5.10.2)
Configuration used
global:
email:
display_name: Gitlab Visagio
from: Example@Example.com
reply_to: noreply@Example.com
smtp:
enabled: true
address: email-smtp.us-east-1.amazonaws.com
authentication: login
port: 587
starttls_auto: true
user_name: Example
password:
secret: gitlab-mail-credentials
registry:
bucket: visagio-gitlab-registry-storage
appConfig:
defaultCanCreateGroup: false
usernameChangingEnabled: false
omniauth:
enabled: true
allowSingleSignOn: ["openid_connect"]
autoSignInWithProvider: "openid_connect"
blockAutoCreatedUsers: false
syncProfileFromProvider: ["openid_connect"]
syncProfileAttributes: ["email", "name"]
providers:
- secret: gitlab-auth-credentials
key: provider
backups:
bucket: visagio-gitlab-backups-storage
tmpBucket: visagio-gitlab-tmp-storage
lfs:
bucket: visagio-gitlab-lfs-storage
connection:
secret: gitlab-buckets-credentials
key: lfs.yaml
artifacts:
bucket: visagio-gitlab-artifacts-storage
connection:
secret: gitlab-buckets-credentials
key: artifacts.yaml
uploads:
bucket: visagio-gitlab-uploads-storage
connection:
secret: gitlab-buckets-credentials
key: uploads.yaml
packages:
bucket: visagio-gitlab-packages-storage
connection:
secret: gitlab-buckets-credentials
key: packages.yaml
externalDiffs:
enabled: true
bucket: visagio-gitlab-externaldiffs-storage
connection:
secret: gitlab-buckets-credentials
key: externalDiffs.yaml
terraformState:
bucket: visagio-gitlab-terraformstate-storage
connection:
secret: gitlab-buckets-credentials
key: terraformState.yaml
pseudonymizer:
bucket: visagio-gitlab-pseudonymizer-storage
connection:
secret: gitlab-buckets-credentials
key: pseudonymizer.yaml
psql:
host: visagio-internal-pg-2.clxbcprct36q.us-east-1.rds.amazonaws.com
database: gitlab
password:
secret: gitlab-postgresql-password
key: password
hosts:
domain: visagio.com
registry:
name: gitlab-registry.visagio.com
ingress:
class: nginx
configureCertmanager: false
annotations:
cert-manager.io/cluster-issuer: "letsencrypt-prod"
minio:
enabled: false
gitlab:
gitlab-exporter:
resources:
requests:
memory: 300M
limits:
memory: 315M
gitlab-shell:
resources:
requests:
memory: 400M
limits:
memory: 500M
sidekiq:
minReplicas: 1
resources:
requests:
memory: 1.2G
limits:
memory: 1.22G
webservice:
ingress:
tls:
secretName: gitlab-visagio-tls-cert
replicaCount: 1
workerProcesses: 1
resources:
requests:
memory: 2.0G
limits:
memory: 2.22G
toolbox:
resources:
requests:
cpu: 50m
memory: 250M
limits:
memory: 265M
backups:
objectStorage:
backend: s3
config:
secret: gitlab-backup-storage-config
key: storage-config.yaml
gitaly:
resources:
requests:
memory: 550M
limits:
memory: 650M
nodeSelector:
topology.kubernetes.io/zone: us-east-1b
registry:
ingress:
tls:
secretName: registry-visagio-tls-cert
hpa:
minReplicas: 2
maxReplicas: 2
resources:
requests:
memory: 700M
limits:
memory: 750M
storage:
secret: gitlab-registry-storage
key: registry-storage.yaml
prometheus:
install: false
server:
persistentVolume:
storageClass: efs-tls
redis:
resources:
requests:
memory: 100M
limits:
memory: 105M
cluster:
enabled: false
global:
storageClass: efs-tls
nginx-ingress:
enabled: false
certmanager:
install: false
postgresql:
install: false
gitlab-runner:
install: false
Current behavior
Merge requests and upload of artifacts to package registry not working due to Resolv::ResolvError
no address for visagio-gitlab-packages-storage.s3.amazonaws.com (Resolv::ResolvError)
Expected behavior
Merge requests and upload of artifacts to package registry should work
Versions
-
Chart: 14.10.2
-
Platform:
- Cloud: EKS
-
Kubernetes: (
kubectl version
)- Client: Client Version: version.Info{Major:"1", Minor:"23", GitVersion:"v1.23.0", GitCommit:"ab69524f795c42094a6630298ff53f3c3ebab7f4", GitTreeState:"clean", BuildDate:"2021-12-07T18:16:20Z", GoVersion:"go1.17.3", Compiler:"gc", Platform:"linux/amd64"}
- Server: Server Version: version.Info{Major:"1", Minor:"19+", GitVersion:"v1.19.16-eks-25803e", GitCommit:"25803e8d008d5fa99b8a35a77d99b705722c0c8c", GitTreeState:"clean", BuildDate:"2022-02-16T23:37:16Z", GoVersion:"go1.15.15", Compiler:"gc", Platform:"linux/amd64"} WARNING: version difference between client (1.23) and server (1.19) exceeds the supported minor version skew of +/-1
-
Helm: (
helm version
)- Client: version.BuildInfo{Version:"v3.8.2", GitCommit:"6e3701edea09e5d55a8ca2aae03a68917630e91b", GitTreeState:"clean", GoVersion:"go1.17.5"}
Relevant logs
"severity": "ERROR",
"time": "2022-05-06T00:38:26.693Z",
"correlation_id": "01G2BCVEEDVMPN39GZYQB4H0V4",
"exception.class": "Excon::Error::Socket",
"exception.message": "no address for visagio-gitlab-packages-storage.s3.amazonaws.com (Resolv::ResolvError)",
"exception.backtrace": [
"app/uploaders/object_storage.rb:404:in `block in cache_remote_file!'",
"app/uploaders/object_storage.rb:403:in `tap'",
"app/uploaders/object_storage.rb:403:in `cache_remote_file!'",
"app/uploaders/object_storage.rb:367:in `cache!'",
"app/services/packages/create_package_file_service.rb:12:in `execute'",
"lib/api/maven_packages.rb:276:in `block (2 levels) in \u003cclass:MavenPackages\u003e'",
"ee/lib/gitlab/middleware/ip_restrictor.rb:14:in `block in call'",
"ee/lib/gitlab/ip_address_state.rb:10:in `with'",
"ee/lib/gitlab/middleware/ip_restrictor.rb:13:in `call'",
"lib/api/api_guard.rb:213:in `call'",
"lib/gitlab/metrics/elasticsearch_rack_middleware.rb:16:in `call'",
"lib/gitlab/middleware/rails_queue_duration.rb:33:in `call'",
"lib/gitlab/middleware/memory_report.rb:13:in `call'",
"lib/gitlab/middleware/speedscope.rb:13:in `call'",
"lib/gitlab/request_profiler/middleware.rb:17:in `call'",
"lib/gitlab/database/load_balancing/rack_middleware.rb:23:in `call'",
"lib/gitlab/metrics/rack_middleware.rb:16:in `block in call'",
"lib/gitlab/metrics/web_transaction.rb:46:in `run'",
"lib/gitlab/metrics/rack_middleware.rb:16:in `call'",
"lib/gitlab/jira/middleware.rb:19:in `call'",
"lib/gitlab/middleware/go.rb:20:in `call'",
"lib/gitlab/etag_caching/middleware.rb:21:in `call'",
"lib/gitlab/middleware/multipart.rb:178:in `block in call'",
"lib/gitlab/middleware/multipart.rb:63:in `with_open_files'",
"lib/gitlab/middleware/multipart.rb:177:in `call'",
"lib/gitlab/middleware/read_only/controller.rb:50:in `call'",
"lib/gitlab/middleware/read_only.rb:18:in `call'",
"lib/gitlab/middleware/same_site_cookies.rb:27:in `call'",
"lib/gitlab/middleware/handle_malformed_strings.rb:21:in `call'",
"lib/gitlab/middleware/basic_health_check.rb:25:in `call'",
"lib/gitlab/middleware/handle_ip_spoof_attack_error.rb:25:in `call'",
"lib/gitlab/middleware/request_context.rb:21:in `call'",
"lib/gitlab/middleware/webhook_recursion_detection.rb:15:in `call'",
"config/initializers/fix_local_cache_middleware.rb:11:in `call'",
"lib/gitlab/middleware/compressed_json.rb:26:in `call'",
"lib/gitlab/middleware/rack_multipart_tempfile_factory.rb:19:in `call'",
"lib/gitlab/middleware/sidekiq_web_static.rb:20:in `call'",
"lib/gitlab/metrics/requests_rack_middleware.rb:77:in `call'",
"lib/gitlab/middleware/release_env.rb:13:in `call'"
],