Skip to content

(Resolv::ResolvError) error on S3 bucket DNS solving when creating merge request and upload artifacts to package registry

Summary

Hello!

We currently have a self-managed Gitlab instance running on our Kubernetes cluster using a helm chart. Recently, we updated our Gitlab instance version from 14.2.1 (chart 5.2.1) to 14.10.2 (chart 5.10.2) and now we are facing some problems to create merge requests and upload artifacts to package registry.

The problem pointed on our logs is with the S3 bucket DNS solving, but the only buckets with this problem are the ones related to ExternalDiffs and Packages. We have buckets for other storages on our instance and they are working fine.

One of the exceptions thrown on Webservice is put below on the logs session

We checked the connectivity to the bucket from the pods and it's accessible, but these 2 features keep throwing errors.

Can you help us with this problem?

Best regards! Guilherme Ferreira

Steps to reproduce

Update Gitlab instance version from 14.2.1 (chart 5.2.1) to 14.10.2 (chart 5.10.2)

Configuration used

global:
  email:
    display_name: Gitlab Visagio
    from: Example@Example.com
    reply_to: noreply@Example.com
  smtp:
    enabled: true
    address: email-smtp.us-east-1.amazonaws.com
    authentication: login
    port: 587
    starttls_auto: true
    user_name: Example
    password:
      secret: gitlab-mail-credentials
  registry:
    bucket: visagio-gitlab-registry-storage
  appConfig:
    defaultCanCreateGroup: false
    usernameChangingEnabled: false
    omniauth:
      enabled: true
      allowSingleSignOn: ["openid_connect"]
      autoSignInWithProvider: "openid_connect"
      blockAutoCreatedUsers: false
      syncProfileFromProvider: ["openid_connect"]
      syncProfileAttributes: ["email", "name"]
      providers:
        - secret: gitlab-auth-credentials
          key: provider
    backups:
      bucket: visagio-gitlab-backups-storage
      tmpBucket: visagio-gitlab-tmp-storage
    lfs:
      bucket: visagio-gitlab-lfs-storage
      connection:
        secret: gitlab-buckets-credentials
        key: lfs.yaml
    artifacts:
      bucket: visagio-gitlab-artifacts-storage
      connection:
        secret: gitlab-buckets-credentials
        key: artifacts.yaml
    uploads:
      bucket: visagio-gitlab-uploads-storage
      connection:
        secret: gitlab-buckets-credentials
        key: uploads.yaml
    packages:
      bucket: visagio-gitlab-packages-storage
      connection:
        secret: gitlab-buckets-credentials
        key: packages.yaml
    externalDiffs:
      enabled: true
      bucket: visagio-gitlab-externaldiffs-storage
      connection:
        secret: gitlab-buckets-credentials
        key: externalDiffs.yaml
    terraformState:
      bucket: visagio-gitlab-terraformstate-storage
      connection:
        secret: gitlab-buckets-credentials
        key: terraformState.yaml
    pseudonymizer:
      bucket: visagio-gitlab-pseudonymizer-storage
      connection:
        secret: gitlab-buckets-credentials
        key: pseudonymizer.yaml
  psql:
    host: visagio-internal-pg-2.clxbcprct36q.us-east-1.rds.amazonaws.com
    database: gitlab
    password:
      secret: gitlab-postgresql-password
      key: password
  hosts:
    domain: visagio.com
    registry:
      name: gitlab-registry.visagio.com
  ingress:
    class: nginx
    configureCertmanager: false
    annotations:
      cert-manager.io/cluster-issuer: "letsencrypt-prod"
  minio:
    enabled: false

gitlab:
  gitlab-exporter:
    resources:
      requests:
        memory: 300M
      limits:
        memory: 315M
  gitlab-shell:
    resources:
      requests:
        memory: 400M
      limits:
        memory: 500M
  sidekiq:
    minReplicas: 1
    resources:
      requests:
        memory: 1.2G
      limits:
        memory: 1.22G
  webservice:
    ingress:
      tls:
        secretName: gitlab-visagio-tls-cert
    replicaCount: 1
    workerProcesses: 1
    resources:
      requests:
        memory: 2.0G
      limits:
        memory: 2.22G
  toolbox:
    resources:
      requests:
        cpu: 50m
        memory: 250M
      limits:
        memory: 265M
    backups:
      objectStorage:
        backend: s3
        config:
          secret: gitlab-backup-storage-config
          key: storage-config.yaml
  gitaly:
    resources:
      requests:
        memory: 550M
      limits:
        memory: 650M
    nodeSelector:
      topology.kubernetes.io/zone: us-east-1b
registry:
  ingress:
    tls:
      secretName: registry-visagio-tls-cert
  hpa:
    minReplicas: 2
    maxReplicas: 2
  resources:
    requests:
      memory: 700M
    limits:
      memory: 750M
  storage:
    secret: gitlab-registry-storage
    key: registry-storage.yaml
prometheus:
  install: false
  server:
    persistentVolume:
      storageClass: efs-tls
redis:
  resources:
    requests:
      memory: 100M
    limits:
      memory: 105M
  cluster:
    enabled: false
  global:
    storageClass: efs-tls
nginx-ingress:
  enabled: false
certmanager:
  install: false
postgresql:
  install: false
gitlab-runner:
  install: false

Current behavior

Merge requests and upload of artifacts to package registry not working due to Resolv::ResolvError

no address for visagio-gitlab-packages-storage.s3.amazonaws.com (Resolv::ResolvError)

Expected behavior

Merge requests and upload of artifacts to package registry should work

Versions

  • Chart: 14.10.2

  • Platform:

    • Cloud: EKS

  • Kubernetes: (kubectl version)

    • Client: Client Version: version.Info{Major:"1", Minor:"23", GitVersion:"v1.23.0", GitCommit:"ab69524f795c42094a6630298ff53f3c3ebab7f4", GitTreeState:"clean", BuildDate:"2021-12-07T18:16:20Z", GoVersion:"go1.17.3", Compiler:"gc", Platform:"linux/amd64"}
    • Server: Server Version: version.Info{Major:"1", Minor:"19+", GitVersion:"v1.19.16-eks-25803e", GitCommit:"25803e8d008d5fa99b8a35a77d99b705722c0c8c", GitTreeState:"clean", BuildDate:"2022-02-16T23:37:16Z", GoVersion:"go1.15.15", Compiler:"gc", Platform:"linux/amd64"} WARNING: version difference between client (1.23) and server (1.19) exceeds the supported minor version skew of +/-1

  • Helm: (helm version)

    • Client: version.BuildInfo{Version:"v3.8.2", GitCommit:"6e3701edea09e5d55a8ca2aae03a68917630e91b", GitTreeState:"clean", GoVersion:"go1.17.5"}

Relevant logs

"severity": "ERROR",
"time": "2022-05-06T00:38:26.693Z",
"correlation_id": "01G2BCVEEDVMPN39GZYQB4H0V4",
"exception.class": "Excon::Error::Socket",
"exception.message": "no address for visagio-gitlab-packages-storage.s3.amazonaws.com (Resolv::ResolvError)",
"exception.backtrace": [
        "app/uploaders/object_storage.rb:404:in `block in cache_remote_file!'",
        "app/uploaders/object_storage.rb:403:in `tap'",
        "app/uploaders/object_storage.rb:403:in `cache_remote_file!'",
        "app/uploaders/object_storage.rb:367:in `cache!'",
        "app/services/packages/create_package_file_service.rb:12:in `execute'",
        "lib/api/maven_packages.rb:276:in `block (2 levels) in \u003cclass:MavenPackages\u003e'",
        "ee/lib/gitlab/middleware/ip_restrictor.rb:14:in `block in call'",
        "ee/lib/gitlab/ip_address_state.rb:10:in `with'",
        "ee/lib/gitlab/middleware/ip_restrictor.rb:13:in `call'",
        "lib/api/api_guard.rb:213:in `call'",
        "lib/gitlab/metrics/elasticsearch_rack_middleware.rb:16:in `call'",
        "lib/gitlab/middleware/rails_queue_duration.rb:33:in `call'",
        "lib/gitlab/middleware/memory_report.rb:13:in `call'",
        "lib/gitlab/middleware/speedscope.rb:13:in `call'",
        "lib/gitlab/request_profiler/middleware.rb:17:in `call'",
        "lib/gitlab/database/load_balancing/rack_middleware.rb:23:in `call'",
        "lib/gitlab/metrics/rack_middleware.rb:16:in `block in call'",
        "lib/gitlab/metrics/web_transaction.rb:46:in `run'",
        "lib/gitlab/metrics/rack_middleware.rb:16:in `call'",
        "lib/gitlab/jira/middleware.rb:19:in `call'",
        "lib/gitlab/middleware/go.rb:20:in `call'",
        "lib/gitlab/etag_caching/middleware.rb:21:in `call'",
        "lib/gitlab/middleware/multipart.rb:178:in `block in call'",
        "lib/gitlab/middleware/multipart.rb:63:in `with_open_files'",
        "lib/gitlab/middleware/multipart.rb:177:in `call'",
        "lib/gitlab/middleware/read_only/controller.rb:50:in `call'",
        "lib/gitlab/middleware/read_only.rb:18:in `call'",
        "lib/gitlab/middleware/same_site_cookies.rb:27:in `call'",
        "lib/gitlab/middleware/handle_malformed_strings.rb:21:in `call'",
        "lib/gitlab/middleware/basic_health_check.rb:25:in `call'",
        "lib/gitlab/middleware/handle_ip_spoof_attack_error.rb:25:in `call'",
        "lib/gitlab/middleware/request_context.rb:21:in `call'",
        "lib/gitlab/middleware/webhook_recursion_detection.rb:15:in `call'",
        "config/initializers/fix_local_cache_middleware.rb:11:in `call'",
        "lib/gitlab/middleware/compressed_json.rb:26:in `call'",
        "lib/gitlab/middleware/rack_multipart_tempfile_factory.rb:19:in `call'",
        "lib/gitlab/middleware/sidekiq_web_static.rb:20:in `call'",
        "lib/gitlab/metrics/requests_rack_middleware.rb:77:in `call'",
        "lib/gitlab/middleware/release_env.rb:13:in `call'"
    ],