Skip to content

avatars and Job Artifacts dont get shown after migrated instance

Summary

We installed gitlab in kubernetes and migrated our old instance to the new one, in the new one we cannot see any user avatars or Job logs

Steps to reproduce

install gitlab in Kuberetes and use Netapp S3 storage, upload on own avatar save the profile and refresh the page

Configuration used

(Please provide a sanitized version of the configuration used wrapped in a code block (```yaml))

gitlab:
  appConfig:
    artifacts:
      bucket: gitlab-prod-artifacts
      connection:
        secret: gitlab-rails-storage
        key: rails.s3.yaml
    uploads:
      bucket: gitlab-prod-uploads
      connection:
        secret: gitlab-rails-storage
        key: rails.s3.yaml

Current behavior

somehow this is not at every repo, but i do not know why at some repos and why not at others

we are getting an error 500, when trying to get an object from our S3 Storage

image

image

image

Expected behavior

see the Avatars and job logs

Versions

  • Chart: 6.3.2
  • Platform:
    • Self-hosted: Rancher RKE2
    • Self-hosted: Netapp OnTap S3 Storage
  • Kubernetes: (kubectl version)
    • Client: 1.22.0
    • Server: 1.22.9
  • Helm: (helm version)
    • Client:
    • Server: 3.8.0

Relevant logs

{
  "content_type": "text/html; charset=utf-8",
  "correlation_id": "01GBWD08ZFRWYHESPC7QTT91C4",
  "duration_ms": 528,
  "host": "gitlab.infra.k8s.***.net",
  "level": "info",
  "method": "GET",
  "msg": "access",
  "proto": "HTTP/1.1",
  "referrer": "https://gitlab.infra.k8s.***.net/t632/k8s-deployments/infra/gitlab/-/pipelines/8609",
  "remote_addr": "10.42.3.84:48816",
  "remote_ip": "10.42.3.84",
  "route": "",
  "status": 200,
  "system": "http",
  "time": "2022-09-01T11:29:27Z",
  "ttfb_ms": 527,
  "uri": "/t632/k8s-deployments/infra/gitlab/-/jobs/38569",
  "user_agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/103.0.5060.66 Safari/537.36",
  "written_bytes": 114124
}
{
  "content_type": "text/html; charset=utf-8",
  "correlation_id": "01GBWD09P50N2SG55SX2P7VSY6",
  "duration_ms": 401,
  "host": "gitlab.infra.k8s.***.net",
  "level": "info",
  "method": "GET",
  "msg": "access",
  "proto": "HTTP/1.1",
  "referrer": "https://gitlab.infra.k8s.***.net/t632/k8s-deployments/infra/gitlab/-/jobs/38569",
  "remote_addr": "10.42.5.154:50382",
  "remote_ip": "10.42.5.154",
  "route": "",
  "status": 500,
  "system": "http",
  "time": "2022-09-01T11:29:27Z",
  "ttfb_ms": 400,
  "uri": "/t632/k8s-deployments/infra/gitlab/-/jobs/38569/trace.json?state=",
  "user_agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/103.0.5060.66 Safari/537.36",
  "written_bytes": 3049
}
{"correlation_id":"01GC96JD189RYBFTG2PEC133ZY","duration_s":0.151147771,"error":"open image data stream: stream data from \"https://nas-10-s3-01.***.net/gitlab-prod-uploads/user/avatar/19/avatar.png?AWSAccessKeyId=REDACTED\u0026Signature=g1BCvDD%2B9gSa5lvRZRHtg6Rm6D0%3D\u0026Expires=1662461828\": 403 403 Forbidden","imageresizer.content_type":"image/png","imageresizer.original_filesize":0,"imageresizer.status":"request-failed","imageresizer.target_width":"23","level":"error","method":"GET","msg":"","subsystem":"imageresizer","time":"2022-09-06T10:47:08Z","uri":"/uploads/-/system/user/avatar/19/avatar.png?width=23","written_bytes":0}
{"correlation_id":"01GC96JD17XPJY2037QD00H9Y6","duration_s":0.182394769,"error":"open image data stream: stream data from \"https://nas-10-s3-01.***.net/gitlab-prod-uploads/project/avatar/70/deployment.png?AWSAccessKeyId=REDACTED\u0026Signature=Q20YUiCyO9AFBy9SVdRKtbF%2BvA0%3D\u0026Expires=1662461828\": 403 403 Forbidden","imageresizer.content_type":"image/png","imageresizer.original_filesize":0,"imageresizer.status":"request-failed","imageresizer.target_width":"32","level":"error","method":"GET","msg":"","subsystem":"imageresizer","time":"2022-09-06T10:47:08Z","uri":"/uploads/-/system/project/avatar/70/deployment.png?width=32","written_bytes":0}
{"correlation_id":"01GC96JDPGV8NZBA44TEKJQ5D4","duration_s":0.088097154,"error":"open image data stream: stream data from \"https://nas-10-s3-01.***.net/gitlab-prod-uploads/user/avatar/3/avatar.png?AWSAccessKeyId=REDACTED\u0026Signature=O0aLhSBxGnpj15Whg3cTaTZeUes%3D\u0026Expires=1662461828\": 403 403 Forbidden","imageresizer.content_type":"image/png","imageresizer.original_filesize":0,"imageresizer.status":"request-failed","imageresizer.target_width":"32","level":"error","method":"GET","msg":"","subsystem":"imageresizer","time":"2022-09-06T10:47:08Z","uri":"/uploads/-/system/user/avatar/3/avatar.png?width=32","written_bytes":0}
Edited by Jason Plum