Container registry is not catching S3 access permission error.
Summary
Container registry tags were not properly deleted. Even though all deletion requests seemed to be successful, the tags remained in place. It turns out to be that the "delete" permission scope was not configured for S3. However,the registry didn't bubble up the S3 access permission error.
This issue was identified during a call with the affected customer. We were able to fix the issue for this particular customer during the call but we may need to do something to avoid further occurrences.
Steps to reproduce
- Create a project and generate some images with different tags in the container registry
- Use S3 as the storage backend.
- Configure S3 permission scopes, leaving "s3:DeleteObject" out.
- Attempt to delete one of the tags from UI, or API, or even the rails console
What is the current bug behavior?
On UI, it shows the tag scheduled for deletion Using API, the call returns 200 response.
What is the expected correct behavior?
It should either delete the tag for real, or return an error message explaining why the tag is not deleted.
Relevant logs and/or screenshots
Example of the delete request in the registry log:
{"content_type":"","correlation_id":"01FQGNSKVMHQEAVE21KYTJN2P4","duration_ms":62,"host":"localhost:5000","level":"info","method":"DELETE","msg":"access","proto":"HTTP/1.1","referrer":"","remote_addr":"127.0.0.1:47498","remote_ip":"127.0.0.1","status":202,"system":"http","time":"2021-12-22T08:58:15Z","ttfb_ms":62,"uri":"/v2/<path to repo>/tags/reference/app-001ac0895480334239fd0d709f47ca2dcb9be2c2-local","user_agent":"GitLab/13.12.12-ee","written_bytes":0}
when we increased the log level of registry to "debug" and of registry storage to "logdebugwithhttpbody", we got:
2022-01-05_09:29:22.86999 2022/01/05 09:29:22 DEBUG: Request s3/DeleteObjects Details:
2022-01-05_09:29:22.86999 ---[ REQUEST POST-SIGN ]-----------------------------
2022-01-05_09:29:22.87000 POST /?delete= HTTP/1.1
2022-01-05_09:29:22.87000 Host: gitlab-gitlabappstack-rvdmag-gitlabregistrybucket-9iw82rpoerp0.s3.eu-west-1.amazonaws.com
2022-01-05_09:29:22.87000 User-Agent: docker-distribution/v3.4.1-gitlab (go1.15.8) aws-sdk-go/1.38.26 (go1.15.8; linux; amd64)
2022-01-05_09:29:22.87000 Content-Length: 535
2022-01-05_09:29:22.87001 Authorization: XXXXX Credential=XXXXXXXXXX/aws4_request, SignedHeaders=content-length;content-md5;host;x-amz-content-sha256;x-amz-date;x-amz-security-token, Signature=XXXXXXXXX
2022-01-05_09:29:22.87002 Content-Md5: XXXXXX
2022-01-05_09:29:22.87002 X-Amz-Content-Sha256: XXXXXXXX
2022-01-05_09:29:22.87003 X-Amz-Date: 20220105T092922Z
2022-01-05_09:29:22.87003 X-Amz-Security-Token: XXXXXXXXX
2022-01-05_09:29:22.87006 Accept-Encoding: gzip
2022-01-05_09:29:22.87006
2022-01-05_09:29:22.87007 <Delete xmlns="http://s3.amazonaws.com/doc/2006-03-01/"><Object><Key>docker/registry/v2/repositories/<path to repo>/_manifests/tags/app-001ac0895480334239fd0d709f47ca2dcb9be2c2-local/current/link</Key></Object><Object><Key>docker/registry/v2/repositories/<path to repo>/_manifests/tags/app-001ac0895480334239fd0d709f47ca2dcb9be2c2-local/index/sha256/aeb09b9cc2ba3592dd84af47a596c4a2270ac4f424999f3de24c9fa2422587b9/link</Key></Object><Quiet>false</Quiet></Delete>
2022-01-05_09:29:22.87008 -----------------------------------------------------
2022-01-05_09:29:22.88512 2022/01/05 09:29:22 DEBUG: Response s3/DeleteObjects Details:
2022-01-05_09:29:22.88514 ---[ RESPONSE ]--------------------------------------
2022-01-05_09:29:22.88516 HTTP/1.1 200 OK
2022-01-05_09:29:22.88516 Connection: close
2022-01-05_09:29:22.88517 Transfer-Encoding: chunked
2022-01-05_09:29:22.88517 Content-Type: application/xml
2022-01-05_09:29:22.88517 Date: Wed, 05 Jan 2022 09:29:23 GMT
2022-01-05_09:29:22.88517 Server: AmazonS3
2022-01-05_09:29:22.88518 X-Amz-Id-2: vzmMXsnD6N2Ztr1jeZC6A76HAm11Fb7mTiYC2K7yaLno8/NlVrgi+K4bAYu420R8tl7nWy+iFyU=
2022-01-05_09:29:22.88518 X-Amz-Request-Id: MX0HA9RM1NVF09WV
2022-01-05_09:29:22.88518
2022-01-05_09:29:22.88519
2022-01-05_09:29:22.88519 -----------------------------------------------------
2022-01-05_09:29:22.88519 2022/01/05 09:29:22 <?xml version="1.0" encoding="UTF-8"?>
2022-01-05_09:29:22.88519 <DeleteResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/"><Error><Key>docker/registry/v2/repositories/<path to repo>/_manifests/tags/app-001ac0895480334239fd0d709f47ca2dcb9be2c2-local/index/sha256/XXXXX/link</Key><Code>AccessDenied</Code><Message>Access Denied</Message></Error><Error><Key>docker/registry/v2/repositories/<path to repo>/_manifests/tags/app-001ac0895480334239fd0d709f47ca2dcb9be2c2-local/current/link</Key><Code>AccessDenied</Code><Message>Access Denied</Message></Error></DeleteResult>
2022-01-05_09:29:22.88521 time="2022-01-05T09:29:22Z" level=debug msg="s3aws.Delete(\"/docker/registry/v2/repositories/<path to repo>/_manifests/tags/app-001ac0895480334239fd0d709f47ca2dcb9be2c2-local\")" auth_user_name=<username> correlation_id=01FRMS4N2FPQD0A55SPZ7GZ19H go_version=go1.15.8 root_repo=<repo name> trace_duration=36.205886ms trace_file=/var/cache/omnibus/src/registry/src/github.com/docker/distribution/registry/storage/driver/base/base.go trace_func="github.com/docker/distribution/registry/storage/driver/base.(*Base).Delete" trace_id=b0afa4fd-8810-4bc5-8f5d-42786963beba trace_line=202 vars_name=<repo path>
2022-01-05_09:29:22.88531 {"content_type":"","correlation_id":"01FRMS4N2FPQD0A55SPZ7GZ19H","duration_ms":117,"host":"gitlab.carnext.io:4567","level":"info","method":"DELETE","msg":"access","proto":"HTTP/1.1","referrer":"","remote_addr":"127.0.0.1:51162","remote_ip":"63.35.68.30","status":202,"system":"http","time":"2022-01-05T09:29:22Z","ttfb_ms":117,"uri":"/v2/<path to repo>/tags/reference/app-001ac0895480334239fd0d709f47ca2dcb9be2c2-local","user_agent":"curl/7.68.0","written_bytes":0}
Results of GitLab environment info
This was reported by a customer operating on GitLab 13.12.12 and container registry 3.4
Possible fixes
- Make sure the registry catches that error and doesn't return a 200 response
- Update our docs to stress on configuring the right permission scopes on S3, and add some steps to the troubleshooting section.