GitLab External Object Storage(AWS S3) Error
Summary
Object not uploaded to external object storage(AWS S3).
Uploading runner cache or avatar image works normally, but uploading ci job artifact and issue editor image does not work.
Steps to reproduce
Deploy Gitlab into AWS on EKS using IRSA for Permissions.
Configuration used
values.yaml (minio.enable false)
gitlab:
appConfig:
object_store:
enabled: true
proxy_download: true
storage_options: {}
# server_side_ encryption:
# server_side_ encryption_kms_key_id
connection:
secret: ${storage_secret}
key: connection
lfs:
enadled: true
proxy_download: true
bucket: ${aws-s3-bucket-name}
artifacts:
enabled: true
proxy_download: true
bucket: ${aws-s3-bucket-name}
uploads:
enabled: true
proxy_download: true
bucket: ${aws-s3-bucket-name}
packages:
enabled: true
proxy_download: true
bucket: ${aws-s3-bucket-name}
kubernetes secret (${storage_secret})
provider: AWS
use_iam_profile: true
region: ap-northeast-2
host: s3.ap-northeast-2.amazonaws.com
endpoint: 'https://s3.ap-northeast-2.amazonaws.com'
values.yaml (Set up service account annotations for toolbox, webservice, and sidekiq pods to use IRSA)
gitlab:
toolbox:
serviceAccount:
annotations:
eks.amazonaws.com/role-arn: ${role-arn}
webservice:
serviceAccount:
annotations:
eks.amazonaws.com/role-arn: ${role-arn}
sidekiq:
serviceAccount:
annotations:
eks.amazonaws.com/role-arn: ${role-arn}
values.yaml (Set up pod environment varialbe)
extraEnv:
AWS_STS_REGIONAL_ENDPOINTS: regional
Current behavior
- The artifacts, the result of CI job, is not uploaded to object storage and an internal server error occurs.
WARNING: Uploading artifacts as "archive" to coordinator... failed id=2733 responseStatus=500 Internal Server Error status=500 token=Wsa36D6M65
WARNING: Retrying...
context=artifacts-uploader error=invalid argument
66WARNING: Uploading artifacts as "archive" to coordinator... failed id=2733 responseStatus=500 Internal Server Error status=500 token=Wsa36D6M
67WARNING: Retrying...
context=artifacts-uploader error=invalid argument
68WARNING: Uploading artifacts as "archive" to coordinator... failed id=2733 responseStatus=500 Internal Server Error status= 500 token=W/sa36D6M
69FATAL: invalid argument
70ERROR: Job failed: command terminated with exit code 1
- When uploading an image to the issue editor, an internal server error occurs.
- In the case of runner cache upload or avatar image upload, it uploads well to aws s3.
- The Pod itself, has S3 permissions from the service account. The aws s3 ls --region commands shows the S3 buckets.
Expected behavior
- Artifact is uploaded to aws s3
Versions
- Chart: v5.7.4
- Platform:
- Cloud: EKS
- Kubernetes: (
kubectl version
) - Helm: (
helm version
)
Relevant logs
- Error log when uploading images to issue editor
{
"correlation id": "01FWK8A5ER6VYG8H0JMWSKQ0JX",
"filename": <REDACT>.jpg,
"level": "info",
"msg": "running exiftool to remove any metadata",
"time": "2022-02-23T12:18:50Z"
}
{
"error": "WebldentityErr: failed to retrieve credentials#n
caused by: SerializationError: failed to unmarshal error message\n\t
status code: 405, request id: \n
caused by: UnmarshalError: failed to unmarshal error message\n\t
00000000 3c 3f 78 6d 6c 20 76 65 72 73 69 6f 6e 3d 22 31 |<?xml version=\"1|\n
00000010 2e 30 22 20 65 6e 63 6f 64 69 6e 67 3d 22 55 54 |.0\" encoding=\"UT|\n
00000020 46 2d 38 22 3f 3e 0a 3c 45 72 72 6f 72 3e 3c 43 |F-8\"?>.<Error> <C|\n
00000030 6f 64 65 3e 4d 65 74 68 6f 64 4e 6f 74 41 6c 6c lode>MethodNotAll|\Hn
00000040 6f 77 65 64 3c 2f 43 6f 64 65 3e 3c 4d 65 73 73 lowed</Code><Mess|\n
00000050 61 67 65 3e 54 68 65 20 73 70 65 63 69 66 69 65 |age> The specifie|\n
00000060 64 20 6d 65 74 68 6f 64 20 69 73 20 6e 6f 74 20 Id method is not |\n
00000070 61 6c 6c 6f 77 65 64 20 61 67 61 69 6e 73 74 20 |allowed against |\n
00000080 74 68 69 73 20 72 65 73 6f 75 72 63 65 2e 3c 2f |this resource. </|\n
00000090 4d 65 73 73 61 67 65 3e 3c 4d 65 74 68 6f 64 3e |Message> <Method>I\n
000000a0 50 4f 53 54 3c 2f 4d 65 74 68 6f 64 3e 3c 52 65 |POST</Method><Re |\n
000000b0 73 6f 75 72 63 65 54 79 70 65 3e 53 45 52 56 49 |sourceType>SERVI|\n
000000c0 43 45 3c 2f 52 65 73 6f 75 72 63 65 54 79 70 65 |CE</ResourceType||\n
000000d0 3e 3c 52 65 71 75 65 73 74 49 64 3e 4a 4d 57 51 |><Requestld>JMWQ|\n
000000e0 54 56 34 45 44 51 58 35 47 35 52 32 3c 2f 52 65 |TV4EDQX5G5R2</Re|\n
000000f0 71 75 65 73 74 49 64 3e 3c 48 6f 73 74 49 64 3e |questld><Hostld>|\n
00000100 66 61 67 39 69 33 67 69 4e 35 38 32 74 6f 46 48 |fag9i3giN582toFH|\n
00000110 53 36 76 43 2b 38 34 45 4c 4a 4a 58 57 7a 68 6a |S6vC+84ELJJXWzhj|\n
00000120 39 6d 6b 79 4f 46 70 2b 69 56 4f 44 34 70 53 71 |9mkyOFp+iVOD4pSq|\n
00000130 42 76 49 66 72 50 65 59 41 36 51 6a 36 74 6b 4e |BvlfrPeYA6Qj6tkN|\n
00000140 33 48 76 76 75 34 6a 65 63 47 59 3d 3c 2f 48 6f |3Hvvu4jecGY=</Ho|\n
00000150 73 74 49 64 3e 3c 2f 45 72 72 6f 72 3e |Istld></Error>|\n\n
caused by: unknown error response tag, {{ Error} []}",
"level": "error",
"msg": "error uploading S3 session",
"time": "2022-02-23T12:18:51Z"
}
{
"correlation_id": "01FWK8A5ER6VYG8H0JMWSKQ0JX",
"error": "handleFileUploads: extract files from multipart: persisting multipart file: unknown error response tag, {{ Error} []}",
"level": "error",
"method": "POST",
"msg": "",
"time": "2022-02-23T12:18:51Z",
"'uri": <REDACT>
}
{
"content_type": "text/plain; charset=utf-8",
"correlation_id": "01FWK8A5ER6VYG8H0JMWSKQ0JX",
"duration_ms": 749,
"host": <REDACT>,
"level": "info",
"method": "POST",
"msg": "access",
'proto": "HTTP/1.1",
"referrer": "https://<REDACT>/-/issues/1",
"remote_addr": "<REDACT>:48526",
"remote ip": "<REDACT>",
"route": "^/([^/]+/){1,}[^/]+/uploads\\z",
"status": 500,
"system": "http",
"time": "2022-02-23T12:18:51Z",
"ttfb_ms": 749,
"uri": "<REDACT>/uploads",
user_agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/98.0.4758.104 Whale/3.13.131.36 Safari/537.36",
"written _bytes": 22
}
- Error log when uploading ci job artifacts
WARNING: Uploading artifacts as "archive" to coordinator... failed id=2733 responseStatus=500 Internal Server Error status=500 token=Wsa36D6M65
WARNING: Retrying...
context=artifacts-uploader error=invalid argument
66WARNING: Uploading artifacts as "archive" to coordinator... failed id=2733 responseStatus=500 Internal Server Error status=500 token=Wsa36D6M
67WARNING: Retrying...
context=artifacts-uploader error=invalid argument
68WARNING: Uploading artifacts as "archive" to coordinator... failed id=2733 responseStatus=500 Internal Server Error status= 500 token=W/sa36D6M
69FATAL: invalid argument
70ERROR: Job failed: command terminated with exit code 1
Edited by 정태균