gitlab registry: can't push docker image - 500
Summary
I have gitlab installed in kubernetes in EKS, with storage on S3. I am using my own nginx ingress. Most things are working, but I can't push to the registry. (However, I can
Steps to reproduce
helm chart values:
certmanager:
install: false
gitlab:
migrations:
enabled: false
task-runner:
backups:
cron:
enabled: true
schedule: 17 2 * * *
objectStorage:
backend: s3
config:
key: config
secret: gitlab-backup-storage
webservice:
ingress:
tls:
secretName: gitlab-unicorn-tls
global:
appConfig:
artifacts:
bucket: ff-dev-gitlab-artifacts-k9tz0klr
connection:
key: connection
secret: gitlab-object-storage
backups:
backend: s3
bucket: ff-dev-gitlab-backup-8mea6btq
connection:
key: connection
secret: gitlab-object-storage
tmpBucket: ff-dev-gitlab-backup-tmp-46qitmon
externalDiffs:
bucket: ff-dev-gitlab-external-diffs-yr5iv0ym
connection:
key: connection
secret: gitlab-object-storage
incommingEmail:
address: gitlab@factfiber.com
enabled: true
host: mx1.factfiber.com
password:
secret: gitlab-incoming-email-password
user: gitlab@factfiber.com
lfs:
bucket: ff-dev-gitlab-lfs-2f9gyalz
connection:
key: connection
secret: gitlab-object-storage
packages:
bucket: ff-dev-gitlab-packages-gj6iwy4b
connection:
key: connection
secret: gitlab-object-storage
pseudonymizer:
bucket: ff-dev-gitlab-pseudonymizer-i1kbzk5c
connection:
key: connection
secret: gitlab-object-storage
terraformState:
bucket: ff-dev-gitlab-terraform-state-1czfrdf4
connection:
key: connection
secret: gitlab-object-storage
enabled: false
uploads:
bucket: ff-dev-gitlab-uploads-kz3bvb59
connection:
key: connection
secret: gitlab-object-storage
edition: ce
email:
from: gitlab@factfiber.com
reply_to: noreply@factfiber.com
grafana:
enabled: true
hosts:
domain: factfiber.ai
gitlab:
name: gl.factfiber.ai
registry:
name: reg.factfiber.ai
ingress:
annotations:
cert-manager.io/issuer: letsencrypt-prod-http
kubernetes.io/ingress.class: nginx
nginx.ingress.kubernetes.io/force-ssl-redirect: "true"
nginx.ingress.kubernetes.io/ssl-redirect: "true"
class: nginx
configureCertmanager: false
minio:
enabled: false
registry:
bucket: ff-dev-gitlab-registry-wy5zpugo
shell:
port: 22
smtp:
address: mx1.factfiber.com
authentication: login
enabled: true
openssl_verify_mode: none
password:
secret: gitlab-smtp-password
port: 587
user_name: gitlab@factfiber.com
nginx-ingress:
enabled: false
registry:
ingress:
proxyBodySize: "0"
proxyBuffering: "off"
proxyReadTimeout: 900
tls:
secretName: gitlab-registry-tls
storage:
key: config
secret: gitlab-registry-storageExample Project
(If possible, please create an example project here on GitLab.com that exhibits the problematic behavior, and link to it here in the bug report)
(If you are using an older version of GitLab, this will also determine whether the bug is fixed in a more recent version)
What is the current bug behavior?
The error:
% cat ~/.ssh/gitlab-token| docker login reg.factfiber.ai --username shauncutts --password-stdin
Login Succeeded
% docker push reg.factfiber.ai/factfiber.ai/factfiber.ai/antegraph
The push refers to repository [reg.factfiber.ai/factfiber.ai/factfiber.ai/antegraph]
ab26acccc511: Pushing [==================================================>] 45.06kB
18c7bf8e756f: Pushing [==================================================>] 59.9kB
81866a945421: Pushing [==================================================>] 152.1MB/152.1MB
1895315d8e99: Pushing [==================================================>] 15.87kB
767c7b461585: Pushing [==================================================>] 577kB/577kB
ff0455b905ec: Pushing [==================================================>] 3.072kB
f341a7346dcf: Pushing [> ] 152.1kB/15.14MB
56ff97ac469e: Waiting
126db245dcea: Pushing [> ] 291.8kB/25.94MB
b13e8e0f76f6: Waiting
2d03d4f6507c: Waiting
d28991d81239: Waiting
8891751e0a17: Waiting
2a19bd70fcd4: Waiting
9e53fd489559: Waiting
7789f1a3d4e9: Waiting
received unexpected HTTP status: 500 Internal Server ErrorI thought this was an instance of: https://docs.gitlab.com/ee/administration/packages/container_registry.html#image-push-errors
However, after editing the config-registry map and setting http.relativeurls to true, still same
behavior. See also; #218778 (closed)
What is the expected correct behavior?
Push succeeds
Relevant logs and/or screenshots
Logs from one of the registry pods:
10.2.4.70 - - [22/May/2020:16:50:55 +0000] "POST /v2/factfiber.ai/factfiber.ai/antegraph/blobs/uploads/ HTTP/1.1" 202 0 "" "docker/19.03.8 go/go1.12.17 git-commit/afacb8b kernel/4.19.76-linuxkit os/linux arch/amd64 UpstreamClient(Docker-Client/19.03.8 \\(darwin\\))"
time="2020-05-22T16:50:56.18187193Z" level=error msg="response completed with error" auth.user.name=shauncutts err.code="blob unknown" err.detail=map[data:sha256:4d45dc646fdd7543d33d8fcd2daa4492452aa6e34bc55d3e0607b21440576af3] err.message="blob unknown to registry" go.version=go1.13.9 http.request.host=reg.factfiber.ai http.request.id=e253262e-a5dd-4909-852b-990084746b7a http.request.method=HEAD http.request.remoteaddr=10.2.7.200 http.request.uri="/v2/factfiber.ai/factfiber.ai/antegraph/blobs/sha256:4d45dc646fdd7543d33d8fcd2daa4492452aa6e34bc55d3e0607b21440576af3" http.request.useragent="docker/19.03.8 go/go1.12.17 git-commit/afacb8b kernel/4.19.76-linuxkit os/linux arch/amd64 UpstreamClient(Docker-Client/19.03.8 \(darwin\))" http.response.contenttype="application/json; charset=utf-8" http.response.duration=21.995429ms http.response.status=404 http.response.written=157 vars.digest="sha256:4d45dc646fdd7543d33d8fcd2daa4492452aa6e34bc55d3e0607b21440576af3" vars.name="factfiber.ai/factfiber.ai/antegraph"
10.2.4.70 - - [22/May/2020:16:50:56 +0000] "HEAD /v2/factfiber.ai/factfiber.ai/antegraph/blobs/sha256:4d45dc646fdd7543d33d8fcd2daa4492452aa6e34bc55d3e0607b21440576af3 HTTP/1.1" 404 157 "" "docker/19.03.8 go/go1.12.17 git-commit/afacb8b kernel/4.19.76-linuxkit os/linux arch/amd64 UpstreamClient(Docker-Client/19.03.8 \\(darwin\\))"
time="2020-05-22T16:50:56.182242719Z" level=error msg="response completed with error" auth.user.name=shauncutts err.code="blob unknown" err.detail=map[data:sha256:0686b766d4aab778709350589c60e9af8d408561242981b02842727efba088a8] err.message="blob unknown to registry" go.version=go1.13.9 http.request.host=reg.factfiber.ai http.request.id=434843f5-6b62-4269-9b14-b883c4a94d8f http.request.method=HEAD http.request.remoteaddr=10.2.3.177 http.request.uri="/v2/factfiber.ai/factfiber.ai/antegraph/blobs/sha256:0686b766d4aab778709350589c60e9af8d408561242981b02842727efba088a8" http.request.useragent="docker/19.03.8 go/go1.12.17 git-commit/afacb8b kernel/4.19.76-linuxkit os/linux arch/amd64 UpstreamClient(Docker-Client/19.03.8 \(darwin\))" http.response.contenttype="application/json; charset=utf-8" http.response.duration=22.188191ms http.response.status=404 http.response.written=157 vars.digest="sha256:0686b766d4aab778709350589c60e9af8d408561242981b02842727efba088a8" vars.name="factfiber.ai/factfiber.ai/antegraph"
10.2.4.70 - - [22/May/2020:16:50:56 +0000] "HEAD /v2/factfiber.ai/factfiber.ai/antegraph/blobs/sha256:0686b766d4aab778709350589c60e9af8d408561242981b02842727efba088a8 HTTP/1.1" 404 157 "" "docker/19.03.8 go/go1.12.17 git-commit/afacb8b kernel/4.19.76-linuxkit os/linux arch/amd64 UpstreamClient(Docker-Client/19.03.8 \\(darwin\\))"
10.2.4.70 - - [22/May/2020:16:50:56 +0000] "POST /v2/factfiber.ai/factfiber.ai/antegraph/blobs/uploads/ HTTP/1.1" 202 0 "" "docker/19.03.8 go/go1.12.17 git-commit/afacb8b kernel/4.19.76-linuxkit os/linux arch/amd64 UpstreamClient(Docker-Client/19.03.8 \\(darwin\\))"
10.2.4.70 - - [22/May/2020:16:50:56 +0000] "HEAD /v2/factfiber.ai/factfiber.ai/antegraph/blobs/sha256:fee5db0ff82f7aa5ace63497df4802bbadf8f2779ed3e1858605b791dc449425 HTTP/1.1" 404 157 "" "docker/19.03.8 go/go1.12.17 git-commit/afacb8b kernel/4.19.76-linuxkit os/linux arch/amd64 UpstreamClient(Docker-Client/19.03.8 \\(darwin\\))"
time="2020-05-22T16:50:56.532779217Z" level=error msg="response completed with error" auth.user.name=shauncutts err.code="blob unknown" err.detail=map[data:sha256:fee5db0ff82f7aa5ace63497df4802bbadf8f2779ed3e1858605b791dc449425] err.message="blob unknown to registry" go.version=go1.13.9 http.request.host=reg.factfiber.ai http.request.id=4d736586-be3f-4d5c-9f2d-838f25e1896e http.request.method=HEAD http.request.remoteaddr=10.2.3.177 http.request.uri="/v2/factfiber.ai/factfiber.ai/antegraph/blobs/sha256:fee5db0ff82f7aa5ace63497df4802bbadf8f2779ed3e1858605b791dc449425" http.request.useragent="docker/19.03.8 go/go1.12.17 git-commit/afacb8b kernel/4.19.76-linuxkit os/linux arch/amd64 UpstreamClient(Docker-Client/19.03.8 \(darwin\))" http.response.contenttype="application/json; charset=utf-8" http.response.duration=10.075219ms http.response.status=404 http.response.written=157 vars.digest="sha256:fee5db0ff82f7aa5ace63497df4802bbadf8f2779ed3e1858605b791dc449425" vars.name="factfiber.ai/factfiber.ai/antegraph"
10.2.4.70 - - [22/May/2020:16:50:56 +0000] "POST /v2/factfiber.ai/factfiber.ai/antegraph/blobs/uploads/ HTTP/1.1" 202 0 "" "docker/19.03.8 go/go1.12.17 git-commit/afacb8b kernel/4.19.76-linuxkit os/linux arch/amd64 UpstreamClient(Docker-Client/19.03.8 \\(darwin\\))"
time="2020-05-22T16:50:56.874547991Z" level=error msg="response completed with error" auth.user.name=shauncutts err.code="blob unknown" err.detail=map[data:sha256:d51af753c3d3a984351448ec0f85ddafc580680fd6dfce9f4b09fdb367ee1e3e] err.message="blob unknown to registry" go.version=go1.13.9 http.request.host=reg.factfiber.ai http.request.id=98f555c8-4182-49e5-8c87-024328851ca9 http.request.method=HEAD http.request.remoteaddr=10.2.7.200 http.request.uri="/v2/factfiber.ai/factfiber.ai/antegraph/blobs/sha256:d51af753c3d3a984351448ec0f85ddafc580680fd6dfce9f4b09fdb367ee1e3e" http.request.useragent="docker/19.03.8 go/go1.12.17 git-commit/afacb8b kernel/4.19.76-linuxkit os/linux arch/amd64 UpstreamClient(Docker-Client/19.03.8 \(darwin\))" http.response.contenttype="application/json; charset=utf-8" http.response.duration=13.397805ms http.response.status=404 http.response.written=157 vars.digest="sha256:d51af753c3d3a984351448ec0f85ddafc580680fd6dfce9f4b09fdb367ee1e3e" vars.name="factfiber.ai/factfiber.ai/antegraph"
10.2.4.70 - - [22/May/2020:16:50:56 +0000] "HEAD /v2/factfiber.ai/factfiber.ai/antegraph/blobs/sha256:d51af753c3d3a984351448ec0f85ddafc580680fd6dfce9f4b09fdb367ee1e3e HTTP/1.1" 404 157 "" "docker/19.03.8 go/go1.12.17 git-commit/afacb8b kernel/4.19.76-linuxkit os/linux arch/amd64 UpstreamClient(Docker-Client/19.03.8 \\(darwin\\))"
time="2020-05-22T16:50:56.893967246Z" level=error msg="response completed with error" auth.user.name=shauncutts err.code="blob unknown" err.detail=map[data:sha256:6154df8ff9882934dc5bf265b8b85a3aeadba06387447ffa440f7af7f32b0e1d] err.message="blob unknown to registry" go.version=go1.13.9 http.request.host=reg.factfiber.ai http.request.id=d4ef95d5-9a62-4d53-ae4e-cee61f2b78f6 http.request.method=HEAD http.request.remoteaddr=10.2.7.200 http.request.uri="/v2/factfiber.ai/factfiber.ai/antegraph/blobs/sha256:6154df8ff9882934dc5bf265b8b85a3aeadba06387447ffa440f7af7f32b0e1d" http.request.useragent="docker/19.03.8 go/go1.12.17 git-commit/afacb8b kernel/4.19.76-linuxkit os/linux arch/amd64 UpstreamClient(Docker-Client/19.03.8 \(darwin\))" http.response.contenttype="application/json; charset=utf-8" http.response.duration=334.500618ms http.response.status=404 http.response.written=157 vars.digest="sha256:6154df8ff9882934dc5bf265b8b85a3aeadba06387447ffa440f7af7f32b0e1d" vars.name="factfiber.ai/factfiber.ai/antegraph"
10.2.4.70 - - [22/May/2020:16:50:56 +0000] "HEAD /v2/factfiber.ai/factfiber.ai/antegraph/blobs/sha256:6154df8ff9882934dc5bf265b8b85a3aeadba06387447ffa440f7af7f32b0e1d HTTP/1.1" 404 157 "" "docker/19.03.8 go/go1.12.17 git-commit/afacb8b kernel/4.19.76-linuxkit os/linux arch/amd64 UpstreamClient(Docker-Client/19.03.8 \\(darwin\\))"
time="2020-05-22T16:52:19.73719615Z" level=warning msg="error authorizing context: authorization token required" go.version=go1.13.9 http.request.host=reg.factfiber.ai http.request.id=5a8793cc-acd7-4f79-82b6-7e8e5f46a90e http.request.method=GET http.request.remoteaddr=10.2.7.200 http.request.uri="/v2/" http.request.useragent="docker/19.03.8 go/go1.12.17 git-commit/afacb8b kernel/4.19.76-linuxkit os/linux arch/amd64 UpstreamClient(Docker-Client/19.03.8 \(darwin\))"
10.2.4.70 - - [22/May/2020:16:52:19 +0000] "GET /v2/ HTTP/1.1" 401 87 "" "docker/19.03.8 go/go1.12.17 git-commit/afacb8b kernel/4.19.76-linuxkit os/linux arch/amd64 UpstreamClient(Docker-Client/19.03.8 \\(darwin\\))"
time="2020-05-22T16:52:20.185318629Z" level=error msg="response completed with error" auth.user.name=shauncutts err.code="blob unknown" err.detail=map[data:sha256:ff602a3685897da683446bf7387c10e627df209a40d06338a55dd401b7fde969] err.message="blob unknown to registry" go.version=go1.13.9 http.request.host=reg.factfiber.ai http.request.id=081c46de-a51b-45a9-98d0-47ed32898ce2 http.request.method=HEAD http.request.remoteaddr=10.2.3.177 http.request.uri="/v2/factfiber.ai/factfiber.ai/antegraph/blobs/sha256:ff602a3685897da683446bf7387c10e627df209a40d06338a55dd401b7fde969" http.request.useragent="docker/19.03.8 go/go1.12.17 git-commit/afacb8b kernel/4.19.76-linuxkit os/linux arch/amd64 UpstreamClient(Docker-Client/19.03.8 \(darwin\))" http.response.contenttype="application/json; charset=utf-8" http.response.duration=121.205028ms http.response.status=404 http.response.written=157 vars.digest="sha256:ff602a3685897da683446bf7387c10e627df209a40d06338a55dd401b7fde969" vars.name="factfiber.ai/factfiber.ai/antegraph"
10.2.4.70 - - [22/May/2020:16:52:20 +0000] "HEAD /v2/factfiber.ai/factfiber.ai/antegraph/blobs/sha256:ff602a3685897da683446bf7387c10e627df209a40d06338a55dd401b7fde969 HTTP/1.1" 404 157 "" "docker/19.03.8 go/go1.12.17 git-commit/afacb8b kernel/4.19.76-linuxkit os/linux arch/amd64 UpstreamClient(Docker-Client/19.03.8 \\(darwin\\))"
time="2020-05-22T16:52:20.190111055Z" level=error msg="response completed with error" auth.user.name=shauncutts err.code="blob unknown" err.detail=map[data:sha256:5f885feff4c7d7b2fbafd46490ffe8ed60155e3cebf0d14226c6948de90b3556] err.message="blob unknown to registry" go.version=go1.13.9 http.request.host=reg.factfiber.ai http.request.id=a55ff572-e453-45ba-8a73-5447385f4134 http.request.method=HEAD http.request.remoteaddr=10.2.3.177 http.request.uri="/v2/factfiber.ai/factfiber.ai/antegraph/blobs/sha256:5f885feff4c7d7b2fbafd46490ffe8ed60155e3cebf0d14226c6948de90b3556" http.request.useragent="docker/19.03.8 go/go1.12.17 git-commit/afacb8b kernel/4.19.76-linuxkit os/linux arch/amd64 UpstreamClient(Docker-Client/19.03.8 \(darwin\))" http.response.contenttype="application/json; charset=utf-8" http.response.duration=126.243461ms http.response.status=404 http.response.written=157 vars.digest="sha256:5f885feff4c7d7b2fbafd46490ffe8ed60155e3cebf0d14226c6948de90b3556" vars.name="factfiber.ai/factfiber.ai/antegraph"
10.2.4.70 - - [22/May/2020:16:52:20 +0000] "HEAD /v2/factfiber.ai/factfiber.ai/antegraph/blobs/sha256:5f885feff4c7d7b2fbafd46490ffe8ed60155e3cebf0d14226c6948de90b3556 HTTP/1.1" 404 157 "" "docker/19.03.8 go/go1.12.17 git-commit/afacb8b kernel/4.19.76-linuxkit os/linux arch/amd64 UpstreamClient(Docker-Client/19.03.8 \\(darwin\\))"
10.2.4.70 - - [22/May/2020:16:52:20 +0000] "POST /v2/factfiber.ai/factfiber.ai/antegraph/blobs/uploads/ HTTP/1.1" 202 0 "" "docker/19.03.8 go/go1.12.17 git-commit/afacb8b kernel/4.19.76-linuxkit os/linux arch/amd64 UpstreamClient(Docker-Client/19.03.8 \\(darwin\\))"
10.2.4.70 - - [22/May/2020:16:52:20 +0000] "POST /v2/factfiber.ai/factfiber.ai/antegraph/blobs/uploads/ HTTP/1.1" 202 0 "" "docker/19.03.8 go/go1.12.17 git-commit/afacb8b kernel/4.19.76-linuxkit os/linux arch/amd64 UpstreamClient(Docker-Client/19.03.8 \\(darwin\\))"
time="2020-05-22T16:52:20.444301559Z" level=error msg="error resolving upload: s3aws: AccessDenied: Access Denied
status code: 403, request id: F39CA941F60442C8, host id: aFsrORMfuU0SiYvu1q1GtOv7Dcd4MrBYNl/XynhnS8Lf33oSm1v4g0s98gEI8a+4BcPx7Vv5a2k=" auth.user.name=shauncutts go.version=go1.13.9 http.request.host=reg.factfiber.ai http.request.id=af218416-4b98-448e-ada6-0e8f281ff121 http.request.method=PATCH http.request.remoteaddr=10.2.3.177 http.request.uri="/v2/factfiber.ai/factfiber.ai/antegraph/blobs/uploads/b953f4bd-ec88-4867-823b-593a866934a0?_state=OzEtRjHFFtK_DyK7fJGoviiQ_wQGVl0r0doumuaEGUl7Ik5hbWUiOiJmYWN0ZmliZXIuYWkvZmFjdGZpYmVyLmFpL2FudGVncmFwaCIsIlVVSUQiOiJiOTUzZjRiZC1lYzg4LTQ4NjctODIzYi01OTNhODY2OTM0YTAiLCJPZmZzZXQiOjAsIlN0YXJ0ZWRBdCI6IjIwMjAtMDUtMjJUMTY6NTI6MjAuMjE1Mjk0MzdaIn0%3D" http.request.useragent="docker/19.03.8 go/go1.12.17 git-commit/afacb8b kernel/4.19.76-linuxkit os/linux arch/amd64 UpstreamClient(Docker-Client/19.03.8 \(darwin\))" vars.name="factfiber.ai/factfiber.ai/antegraph" vars.uuid=b953f4bd-ec88-4867-823b-593a866934a0
10.2.4.70 - - [22/May/2020:16:52:20 +0000] "PATCH /v2/factfiber.ai/factfiber.ai/antegraph/blobs/uploads/b953f4bd-ec88-4867-823b-593a866934a0?_state=OzEtRjHFFtK_DyK7fJGoviiQ_wQGVl0r0doumuaEGUl7Ik5hbWUiOiJmYWN0ZmliZXIuYWkvZmFjdGZpYmVyLmFpL2FudGVncmFwaCIsIlVVSUQiOiJiOTUzZjRiZC1lYzg4LTQ4NjctODIzYi01OTNhODY2OTM0YTAiLCJPZmZzZXQiOjAsIlN0YXJ0ZWRBdCI6IjIwMjAtMDUtMjJUMTY6NTI6MjAuMjE1Mjk0MzdaIn0%3D HTTP/1.1" 500 123 "" "docker/19.03.8 go/go1.12.17 git-commit/afacb8b kernel/4.19.76-linuxkit os/linux arch/amd64 UpstreamClient(Docker-Client/19.03.8 \\(darwin\\))"
time="2020-05-22T16:52:20.444392652Z" level=error msg="response completed with error" auth.user.name=shauncutts err.code=unknown err.detail=map[data:s3aws: AccessDenied: Access Denied
status code: 403, request id: F39CA941F60442C8, host id: aFsrORMfuU0SiYvu1q1GtOv7Dcd4MrBYNl/XynhnS8Lf33oSm1v4g0s98gEI8a+4BcPx7Vv5a2k=] err.message="unknown error" go.version=go1.13.9 http.request.host=reg.factfiber.ai http.request.id=af218416-4b98-448e-ada6-0e8f281ff121 http.request.method=PATCH http.request.remoteaddr=10.2.3.177 http.request.uri="/v2/factfiber.ai/factfiber.ai/antegraph/blobs/uploads/b953f4bd-ec88-4867-823b-593a866934a0?_state=OzEtRjHFFtK_DyK7fJGoviiQ_wQGVl0r0doumuaEGUl7Ik5hbWUiOiJmYWN0ZmliZXIuYWkvZmFjdGZpYmVyLmFpL2FudGVncmFwaCIsIlVVSUQiOiJiOTUzZjRiZC1lYzg4LTQ4NjctODIzYi01OTNhODY2OTM0YTAiLCJPZmZzZXQiOjAsIlN0YXJ0ZWRBdCI6IjIwMjAtMDUtMjJUMTY6NTI6MjAuMjE1Mjk0MzdaIn0%3D" http.request.useragent="docker/19.03.8 go/go1.12.17 git-commit/afacb8b kernel/4.19.76-linuxkit os/linux arch/amd64 UpstreamClient(Docker-Client/19.03.8 \(darwin\))" http.response.contenttype="application/json; charset=utf-8" http.response.duration=25.054734ms http.response.status=500 http.response.written=123 vars.name="factfiber.ai/factfiber.ai/antegraph" vars.uuid=b953f4bd-ec88-4867-823b-593a866934a0
10.2.4.70 - - [22/May/2020:16:52:20 +0000] "PATCH /v2/factfiber.ai/factfiber.ai/antegraph/blobs/uploads/42688d36-73e6-43c1-b2f9-1b3f68425d69?_state=7B5slMpQkcrABhMFvI9qmiBOg1ul8GmCiDmmdhjPCwp7Ik5hbWUiOiJmYWN0ZmliZXIuYWkvZmFjdGZpYmVyLmFpL2FudGVncmFwaCIsIlVVSUQiOiI0MjY4OGQzNi03M2U2LTQzYzEtYjJmOS0xYjNmNjg0MjVkNjkiLCJPZmZzZXQiOjAsIlN0YXJ0ZWRBdCI6IjIwMjAtMDUtMjJUMTY6NTI6MjAuMjkzOTYxMjU2WiJ9 HTTP/1.1" 500 123 "" "docker/19.03.8 go/go1.12.17 git-commit/afacb8b kernel/4.19.76-linuxkit os/linux arch/amd64 UpstreamClient(Docker-Client/19.03.8 \\(darwin\\))"
time="2020-05-22T16:52:20.52255195Z" level=error msg="error resolving upload: s3aws: AccessDenied: Access Denied
status code: 403, request id: CE1F488800DA0E19, host id: vU57AyeEqNaQEZK+5q/Qhesz6XVV/1OX9bo4yVeqoO7nP5VqklURba81VL6+WXZyuDcdstwbwRA=" auth.user.name=shauncutts go.version=go1.13.9 http.request.host=reg.factfiber.ai http.request.id=d84f94c8-531a-442b-81cc-9b9bc80165d5 http.request.method=PATCH http.request.remoteaddr=10.2.5.108 http.request.uri="/v2/factfiber.ai/factfiber.ai/antegraph/blobs/uploads/42688d36-73e6-43c1-b2f9-1b3f68425d69?_state=7B5slMpQkcrABhMFvI9qmiBOg1ul8GmCiDmmdhjPCwp7Ik5hbWUiOiJmYWN0ZmliZXIuYWkvZmFjdGZpYmVyLmFpL2FudGVncmFwaCIsIlVVSUQiOiI0MjY4OGQzNi03M2U2LTQzYzEtYjJmOS0xYjNmNjg0MjVkNjkiLCJPZmZzZXQiOjAsIlN0YXJ0ZWRBdCI6IjIwMjAtMDUtMjJUMTY6NTI6MjAuMjkzOTYxMjU2WiJ9" http.request.useragent="docker/19.03.8 go/go1.12.17 git-commit/afacb8b kernel/4.19.76-linuxkit os/linux arch/amd64 UpstreamClient(Docker-Client/19.03.8 \(darwin\))" vars.name="factfiber.ai/factfiber.ai/antegraph" vars.uuid=42688d36-73e6-43c1-b2f9-1b3f68425d69
time="2020-05-22T16:52:20.522644257Z" level=error msg="response completed with error" auth.user.name=shauncutts err.code=unknown err.detail=map[data:s3aws: AccessDenied: Access Denied
status code: 403, request id: CE1F488800DA0E19, host id: vU57AyeEqNaQEZK+5q/Qhesz6XVV/1OX9bo4yVeqoO7nP5VqklURba81VL6+WXZyuDcdstwbwRA=] err.message="unknown error" go.version=go1.13.9 http.request.host=reg.factfiber.ai http.request.id=d84f94c8-531a-442b-81cc-9b9bc80165d5 http.request.method=PATCH http.request.remoteaddr=10.2.5.108 http.request.uri="/v2/factfiber.ai/factfiber.ai/antegraph/blobs/uploads/42688d36-73e6-43c1-b2f9-1b3f68425d69?_state=7B5slMpQkcrABhMFvI9qmiBOg1ul8GmCiDmmdhjPCwp7Ik5hbWUiOiJmYWN0ZmliZXIuYWkvZmFjdGZpYmVyLmFpL2FudGVncmFwaCIsIlVVSUQiOiI0MjY4OGQzNi03M2U2LTQzYzEtYjJmOS0xYjNmNjg0MjVkNjkiLCJPZmZzZXQiOjAsIlN0YXJ0ZWRBdCI6IjIwMjAtMDUtMjJUMTY6NTI6MjAuMjkzOTYxMjU2WiJ9" http.request.useragent="docker/19.03.8 go/go1.12.17 git-commit/afacb8b kernel/4.19.76-linuxkit os/linux arch/amd64 UpstreamClient(Docker-Client/19.03.8 \(darwin\))" http.response.contenttype="application/json; charset=utf-8" http.response.duration=17.042842ms http.response.status=500 http.response.written=123 vars.name="factfiber.ai/factfiber.ai/antegraph" vars.uuid=42688d36-73e6-43c1-b2f9-1b3f68425d69
10.2.4.70 - - [22/May/2020:16:52:25 +0000] "POST /v2/factfiber.ai/factfiber.ai/antegraph/blobs/uploads/ HTTP/1.1" 202 0 "" "docker/19.03.8 go/go1.12.17 git-commit/afacb8b kernel/4.19.76-linuxkit os/linux arch/amd64 UpstreamClient(Docker-Client/19.03.8 \\(darwin\\))"
time="2020-05-22T16:52:25.744397913Z" level=error msg="error resolving upload: s3aws: AccessDenied: Access Denied
status code: 403, request id: 3176760FB5E2BA7F, host id: Ypj9H+RpUEem1xasRMraXFac1gz9oNTyLHJD/mYPAHIz4q7aDoK48blRrRNypgXDCrAnBVLdlz4=" auth.user.name=shauncutts go.version=go1.13.9 http.request.host=reg.factfiber.ai http.request.id=85a7c85d-8deb-4275-8d69-cfb20489d7b4 http.request.method=PATCH http.request.remoteaddr=10.2.5.108 http.request.uri="/v2/factfiber.ai/factfiber.ai/antegraph/blobs/uploads/5bf585e6-894e-4603-aef6-9eefd0c67c54?_state=U0VNHjwO4eRLcc20oG-YkZqXhEPClkyummJnDoCpLAV7Ik5hbWUiOiJmYWN0ZmliZXIuYWkvZmFjdGZpYmVyLmFpL2FudGVncmFwaCIsIlVVSUQiOiI1YmY1ODVlNi04OTRlLTQ2MDMtYWVmNi05ZWVmZDBjNjdjNTQiLCJPZmZzZXQiOjAsIlN0YXJ0ZWRBdCI6IjIwMjAtMDUtMjJUMTY6NTI6MjUuNTQ5MzEzMDI5WiJ9" http.request.useragent="docker/19.03.8 go/go1.12.17 git-commit/afacb8b kernel/4.19.76-linuxkit os/linux arch/amd64 UpstreamClient(Docker-Client/19.03.8 \(darwin\))" vars.name="factfiber.ai/factfiber.ai/antegraph" vars.uuid=5bf585e6-894e-4603-aef6-9eefd0c67c54
10.2.4.70 - - [22/May/2020:16:52:25 +0000] "PATCH /v2/factfiber.ai/factfiber.ai/antegraph/blobs/uploads/5bf585e6-894e-4603-aef6-9eefd0c67c54?_state=U0VNHjwO4eRLcc20oG-YkZqXhEPClkyummJnDoCpLAV7Ik5hbWUiOiJmYWN0ZmliZXIuYWkvZmFjdGZpYmVyLmFpL2FudGVncmFwaCIsIlVVSUQiOiI1YmY1ODVlNi04OTRlLTQ2MDMtYWVmNi05ZWVmZDBjNjdjNTQiLCJPZmZzZXQiOjAsIlN0YXJ0ZWRBdCI6IjIwMjAtMDUtMjJUMTY6NTI6MjUuNTQ5MzEzMDI5WiJ9 HTTP/1.1" 500 123 "" "docker/19.03.8 go/go1.12.17 git-commit/afacb8b kernel/4.19.76-linuxkit os/linux arch/amd64 UpstreamClient(Docker-Client/19.03.8 \\(darwin\\))"
time="2020-05-22T16:52:25.744489753Z" level=error msg="response completed with error" auth.user.name=shauncutts err.code=unknown err.detail=map[data:s3aws: AccessDenied: Access Denied
status code: 403, request id: 3176760FB5E2BA7F, host id: Ypj9H+RpUEem1xasRMraXFac1gz9oNTyLHJD/mYPAHIz4q7aDoK48blRrRNypgXDCrAnBVLdlz4=] err.message="unknown error" go.version=go1.13.9 http.request.host=reg.factfiber.ai http.request.id=85a7c85d-8deb-4275-8d69-cfb20489d7b4 http.request.method=PATCH http.request.remoteaddr=10.2.5.108 http.request.uri="/v2/factfiber.ai/factfiber.ai/antegraph/blobs/uploads/5bf585e6-894e-4603-aef6-9eefd0c67c54?_state=U0VNHjwO4eRLcc20oG-YkZqXhEPClkyummJnDoCpLAV7Ik5hbWUiOiJmYWN0ZmliZXIuYWkvZmFjdGZpYmVyLmFpL2FudGVncmFwaCIsIlVVSUQiOiI1YmY1ODVlNi04OTRlLTQ2MDMtYWVmNi05ZWVmZDBjNjdjNTQiLCJPZmZzZXQiOjAsIlN0YXJ0ZWRBdCI6IjIwMjAtMDUtMjJUMTY6NTI6MjUuNTQ5MzEzMDI5WiJ9" http.request.useragent="docker/19.03.8 go/go1.12.17 git-commit/afacb8b kernel/4.19.76-linuxkit os/linux arch/amd64 UpstreamClient(Docker-Client/19.03.8 \(darwin\))" http.response.contenttype="application/json; charset=utf-8" http.response.duration=23.162376ms http.response.status=500 http.response.written=123 vars.name="factfiber.ai/factfiber.ai/antegraph" vars.uuid=5bf585e6-894e-4603-aef6-9eefd0c67c54
10.2.4.70 - - [22/May/2020:16:52:35 +0000] "POST /v2/factfiber.ai/factfiber.ai/antegraph/blobs/uploads/ HTTP/1.1" 202 0 "" "docker/19.03.8 go/go1.12.17 git-commit/afacb8b kernel/4.19.76-linuxkit os/linux arch/amd64 UpstreamClient(Docker-Client/19.03.8 \\(darwin\\))"
10.2.4.70 - - [22/May/2020:16:52:51 +0000] "POST /v2/factfiber.ai/factfiber.ai/antegraph/blobs/uploads/ HTTP/1.1" 202 0 "" "docker/19.03.8 go/go1.12.17 git-commit/afacb8b kernel/4.19.76-linuxkit os/linux arch/amd64 UpstreamClient(Docker-Client/19.03.8 \\(darwin\\))"
10.2.4.70 - - [22/May/2020:16:52:55 +0000] "POST /v2/factfiber.ai/factfiber.ai/antegraph/blobs/uploads/ HTTP/1.1" 202 0 "" "docker/19.03.8 go/go1.12.17 git-commit/afacb8b kernel/4.19.76-linuxkit os/linux arch/amd64 UpstreamClient(Docker-Client/19.03.8 \\(darwin\\))"
time="2020-05-22T16:52:55.470021401Z" level=error msg="error resolving upload: s3aws: AccessDenied: Access Denied
status code: 403, request id: 1446A9A69B1E2AB9, host id: au76aE0Wrvx8mcYLP9f+KPfrM52O1XUAaMMESjYId0DAjvDDl6dd0Hulk5pdC/f07doJZqxLmGw=" auth.user.name=shauncutts go.version=go1.13.9 http.request.host=reg.factfiber.ai http.request.id=5a0cf161-7a57-4a8b-89c2-ac916589257e http.request.method=PATCH http.request.remoteaddr=10.2.7.200 http.request.uri="/v2/factfiber.ai/factfiber.ai/antegraph/blobs/uploads/cf858225-d51b-46d7-93b4-92df57976f18?_state=RGX9qvsVheRm0CqLq-ocv4gUAMt8kh0ouAggEf6kbqB7Ik5hbWUiOiJmYWN0ZmliZXIuYWkvZmFjdGZpYmVyLmFpL2FudGVncmFwaCIsIlVVSUQiOiJjZjg1ODIyNS1kNTFiLTQ2ZDctOTNiNC05MmRmNTc5NzZmMTgiLCJPZmZzZXQiOjAsIlN0YXJ0ZWRBdCI6IjIwMjAtMDUtMjJUMTY6NTI6NTUuMjUyNDMwMDIyWiJ9" http.request.useragent="docker/19.03.8 go/go1.12.17 git-commit/afacb8b kernel/4.19.76-linuxkit os/linux arch/amd64 UpstreamClient(Docker-Client/19.03.8 \(darwin\))" vars.name="factfiber.ai/factfiber.ai/antegraph" vars.uuid=cf858225-d51b-46d7-93b4-92df57976f18
time="2020-05-22T16:52:55.47011671Z" level=error msg="response completed with error" auth.user.name=shauncutts err.code=unknown err.detail=map[data:s3aws: AccessDenied: Access Denied
status code: 403, request id: 1446A9A69B1E2AB9, host id: au76aE0Wrvx8mcYLP9f+KPfrM52O1XUAaMMESjYId0DAjvDDl6dd0Hulk5pdC/f07doJZqxLmGw=] err.message="unknown error" go.version=go1.13.9 http.request.host=reg.factfiber.ai http.request.id=5a0cf161-7a57-4a8b-89c2-ac916589257e http.request.method=PATCH http.request.remoteaddr=10.2.7.200 http.request.uri="/v2/factfiber.ai/factfiber.ai/antegraph/blobs/uploads/cf858225-d51b-46d7-93b4-92df57976f18?_state=RGX9qvsVheRm0CqLq-ocv4gUAMt8kh0ouAggEf6kbqB7Ik5hbWUiOiJmYWN0ZmliZXIuYWkvZmFjdGZpYmVyLmFpL2FudGVncmFwaCIsIlVVSUQiOiJjZjg1ODIyNS1kNTFiLTQ2ZDctOTNiNC05MmRmNTc5NzZmMTgiLCJPZmZzZXQiOjAsIlN0YXJ0ZWRBdCI6IjIwMjAtMDUtMjJUMTY6NTI6NTUuMjUyNDMwMDIyWiJ9" http.request.useragent="docker/19.03.8 go/go1.12.17 git-commit/afacb8b kernel/4.19.76-linuxkit os/linux arch/amd64 UpstreamClient(Docker-Client/19.03.8 \(darwin\))" http.response.contenttype="application/json; charset=utf-8" http.response.duration=20.568315ms http.response.status=500 http.response.written=123 vars.name="factfiber.ai/factfiber.ai/antegraph" vars.uuid=cf858225-d51b-46d7-93b4-92df57976f18
10.2.4.70 - - [22/May/2020:16:52:55 +0000] "PATCH /v2/factfiber.ai/factfiber.ai/antegraph/blobs/uploads/cf858225-d51b-46d7-93b4-92df57976f18?_state=RGX9qvsVheRm0CqLq-ocv4gUAMt8kh0ouAggEf6kbqB7Ik5hbWUiOiJmYWN0ZmliZXIuYWkvZmFjdGZpYmVyLmFpL2FudGVncmFwaCIsIlVVSUQiOiJjZjg1ODIyNS1kNTFiLTQ2ZDctOTNiNC05MmRmNTc5NzZmMTgiLCJPZmZzZXQiOjAsIlN0YXJ0ZWRBdCI6IjIwMjAtMDUtMjJUMTY6NTI6NTUuMjUyNDMwMDIyWiJ9 HTTP/1.1" 500 123 "" "docker/19.03.8 go/go1.12.17 git-commit/afacb8b kernel/4.19.76-linuxkit os/linux arch/amd64 UpstreamClient(Docker-Client/19.03.8 \\(darwin\\))"
10.2.4.70 - - [22/May/2020:16:53:05 +0000] "POST /v2/factfiber.ai/factfiber.ai/antegraph/blobs/uploads/ HTTP/1.1" 202 0 "" "docker/19.03.8 go/go1.12.17 git-commit/afacb8b kernel/4.19.76-linuxkit os/linux arch/amd64 UpstreamClient(Docker-Client/19.03.8 \\(darwin\\))"
10.2.4.70 - - [22/May/2020:16:53:05 +0000] "POST /v2/factfiber.ai/factfiber.ai/antegraph/blobs/uploads/ HTTP/1.1" 202 0 "" "docker/19.03.8 go/go1.12.17 git-commit/afacb8b kernel/4.19.76-linuxkit os/linux arch/amd64 UpstreamClient(Docker-Client/19.03.8 \\(darwin\\))"
10.2.4.70 - - [22/May/2020:16:53:12 +0000] "POST /v2/factfiber.ai/factfiber.ai/antegraph/blobs/uploads/ HTTP/1.1" 202 0 "" "docker/19.03.8 go/go1.12.17 git-commit/afacb8b kernel/4.19.76-linuxkit os/linux arch/amd64 UpstreamClient(Docker-Client/19.03.8 \\(darwin\\))"
time="2020-05-22T16:53:12.46607912Z" level=error msg="error resolving upload: s3aws: AccessDenied: Access Denied
status code: 403, request id: 18F8685D6FA19619, host id: QLU9AFvL6JsSOvy9Wm9FsRFXy9kwSTR42Xo1F2oeSH6sze1K51hgkSG4anLbMVSgo5DCRBu+Bis=" auth.user.name=shauncutts go.version=go1.13.9 http.request.host=reg.factfiber.ai http.request.id=2ed2b917-5dbc-4b2b-b5dc-026c183a4238 http.request.method=PATCH http.request.remoteaddr=10.2.3.177 http.request.uri="/v2/factfiber.ai/factfiber.ai/antegraph/blobs/uploads/bbf8cc66-1e5b-48e0-b0e9-3cc6c05ac55a?_state=tQ3EogeWnW7n34oVriHKMUUIZHdKgA6qeV_tPYrK9s97Ik5hbWUiOiJmYWN0ZmliZXIuYWkvZmFjdGZpYmVyLmFpL2FudGVncmFwaCIsIlVVSUQiOiJiYmY4Y2M2Ni0xZTViLTQ4ZTAtYjBlOS0zY2M2YzA1YWM1NWEiLCJPZmZzZXQiOjAsIlN0YXJ0ZWRBdCI6IjIwMjAtMDUtMjJUMTY6NTM6MTIuMDA3MjI0ODgzWiJ9" http.request.useragent="docker/19.03.8 go/go1.12.17 git-commit/afacb8b kernel/4.19.76-linuxkit os/linux arch/amd64 UpstreamClient(Docker-Client/19.03.8 \(darwin\))" vars.name="factfiber.ai/factfiber.ai/antegraph" vars.uuid=bbf8cc66-1e5b-48e0-b0e9-3cc6c05ac55a
time="2020-05-22T16:53:12.466179363Z" level=error msg="response completed with error" auth.user.name=shauncutts err.code=unknown err.detail=map[data:s3aws: AccessDenied: Access Denied
status code: 403, request id: 18F8685D6FA19619, host id: QLU9AFvL6JsSOvy9Wm9FsRFXy9kwSTR42Xo1F2oeSH6sze1K51hgkSG4anLbMVSgo5DCRBu+Bis=] err.message="unknown error" go.version=go1.13.9 http.request.host=reg.factfiber.ai http.request.id=2ed2b917-5dbc-4b2b-b5dc-026c183a4238 http.request.method=PATCH http.request.remoteaddr=10.2.3.177 http.request.uri="/v2/factfiber.ai/factfiber.ai/antegraph/blobs/uploads/bbf8cc66-1e5b-48e0-b0e9-3cc6c05ac55a?_state=tQ3EogeWnW7n34oVriHKMUUIZHdKgA6qeV_tPYrK9s97Ik5hbWUiOiJmYWN0ZmliZXIuYWkvZmFjdGZpYmVyLmFpL2FudGVncmFwaCIsIlVVSUQiOiJiYmY4Y2M2Ni0xZTViLTQ4ZTAtYjBlOS0zY2M2YzA1YWM1NWEiLCJPZmZzZXQiOjAsIlN0YXJ0ZWRBdCI6IjIwMjAtMDUtMjJUMTY6NTM6MTIuMDA3MjI0ODgzWiJ9" http.request.useragent="docker/19.03.8 go/go1.12.17 git-commit/afacb8b kernel/4.19.76-linuxkit os/linux arch/amd64 UpstreamClient(Docker-Client/19.03.8 \(darwin\))" http.response.contenttype="application/json; charset=utf-8" http.response.duration=19.006198ms http.response.status=500 http.response.written=123 vars.name="factfiber.ai/factfiber.ai/antegraph" vars.uuid=bbf8cc66-1e5b-48e0-b0e9-3cc6c05ac55a
10.2.4.70 - - [22/May/2020:16:53:12 +0000] "PATCH /v2/factfiber.ai/factfiber.ai/antegraph/blobs/uploads/bbf8cc66-1e5b-48e0-b0e9-3cc6c05ac55a?_state=tQ3EogeWnW7n34oVriHKMUUIZHdKgA6qeV_tPYrK9s97Ik5hbWUiOiJmYWN0ZmliZXIuYWkvZmFjdGZpYmVyLmFpL2FudGVncmFwaCIsIlVVSUQiOiJiYmY4Y2M2Ni0xZTViLTQ4ZTAtYjBlOS0zY2M2YzA1YWM1NWEiLCJPZmZzZXQiOjAsIlN0YXJ0ZWRBdCI6IjIwMjAtMDUtMjJUMTY6NTM6MTIuMDA3MjI0ODgzWiJ9 HTTP/1.1" 500 123 "" "docker/19.03.8 go/go1.12.17 git-commit/afacb8b kernel/4.19.76-linuxkit os/linux arch/amd64 UpstreamClient(Docker-Client/19.03.8 \\(darwin\\))"
10.2.4.70 - - [22/May/2020:16:53:12 +0000] "POST /v2/factfiber.ai/factfiber.ai/antegraph/blobs/uploads/ HTTP/1.1" 202 0 "" "docker/19.03.8 go/go1.12.17 git-commit/afacb8b kernel/4.19.76-linuxkit os/linux arch/amd64 UpstreamClient(Docker-Client/19.03.8 \\(darwin\\))"
10.2.4.70 - - [22/May/2020:16:53:13 +0000] "HEAD /v2/factfiber.ai/factfiber.ai/antegraph/blobs/sha256:4d45dc646fdd7543d33d8fcd2daa4492452aa6e34bc55d3e0607b21440576af3 HTTP/1.1" 404 157 "" "docker/19.03.8 go/go1.12.17 git-commit/afacb8b kernel/4.19.76-linuxkit os/linux arch/amd64 UpstreamClient(Docker-Client/19.03.8 \\(darwin\\))"
time="2020-05-22T16:53:13.068490914Z" level=error msg="response completed with error" auth.user.name=shauncutts err.code="blob unknown" err.detail=map[data:sha256:4d45dc646fdd7543d33d8fcd2daa4492452aa6e34bc55d3e0607b21440576af3] err.message="blob unknown to registry" go.version=go1.13.9 http.request.host=reg.factfiber.ai http.request.id=84c973fa-ec8c-4a88-8fd5-689bdfaee51d http.request.method=HEAD http.request.remoteaddr=10.2.7.200 http.request.uri="/v2/factfiber.ai/factfiber.ai/antegraph/blobs/sha256:4d45dc646fdd7543d33d8fcd2daa4492452aa6e34bc55d3e0607b21440576af3" http.request.useragent="docker/19.03.8 go/go1.12.17 git-commit/afacb8b kernel/4.19.76-linuxkit os/linux arch/amd64 UpstreamClient(Docker-Client/19.03.8 \(darwin\))" http.response.contenttype="application/json; charset=utf-8" http.response.duration=12.544811ms http.response.status=404 http.response.written=157 vars.digest="sha256:4d45dc646fdd7543d33d8fcd2daa4492452aa6e34bc55d3e0607b21440576af3" vars.name="factfiber.ai/factfiber.ai/antegraph"
10.2.4.70 - - [22/May/2020:16:53:12 +0000] "HEAD /v2/factfiber.ai/factfiber.ai/antegraph/blobs/sha256:9afd3a4094a051452b4601d6f5cda7711c3e93c7abad17bd9fab486261e9e55d HTTP/1.1" 404 157 "" "docker/19.03.8 go/go1.12.17 git-commit/afacb8b kernel/4.19.76-linuxkit os/linux arch/amd64 UpstreamClient(Docker-Client/19.03.8 \\(darwin\\))"
time="2020-05-22T16:53:13.178852844Z" level=error msg="response completed with error" auth.user.name=shauncutts err.code="blob unknown" err.detail=map[data:sha256:9afd3a4094a051452b4601d6f5cda7711c3e93c7abad17bd9fab486261e9e55d] err.message="blob unknown to registry" go.version=go1.13.9 http.request.host=reg.factfiber.ai http.request.id=af77a856-bc86-4ef6-b3df-4fe908c0e263 http.request.method=HEAD http.request.remoteaddr=10.2.7.200 http.request.uri="/v2/factfiber.ai/factfiber.ai/antegraph/blobs/sha256:9afd3a4094a051452b4601d6f5cda7711c3e93c7abad17bd9fab486261e9e55d" http.request.useragent="docker/19.03.8 go/go1.12.17 git-commit/afacb8b kernel/4.19.76-linuxkit os/linux arch/amd64 UpstreamClient(Docker-Client/19.03.8 \(darwin\))" http.response.contenttype="application/json; charset=utf-8" http.response.duration=575.178816ms http.response.status=404 http.response.written=157 vars.digest="sha256:9afd3a4094a051452b4601d6f5cda7711c3e93c7abad17bd9fab486261e9e55d" vars.name="factfiber.ai/factfiber.ai/antegraph"
10.2.4.70 - - [22/May/2020:16:53:13 +0000] "POST /v2/factfiber.ai/factfiber.ai/antegraph/blobs/uploads/ HTTP/1.1" 202 0 "" "docker/19.03.8 go/go1.12.17 git-commit/afacb8b kernel/4.19.76-linuxkit os/linux arch/amd64 UpstreamClient(Docker-Client/19.03.8 \\(darwin\\))"
10.2.4.70 - - [22/May/2020:16:53:13 +0000] "POST /v2/factfiber.ai/factfiber.ai/antegraph/blobs/uploads/ HTTP/1.1" 202 0 "" "docker/19.03.8 go/go1.12.17 git-commit/afacb8b kernel/4.19.76-linuxkit os/linux arch/amd64 UpstreamClient(Docker-Client/19.03.8 \\(darwin\\))"
10.2.4.70 - - [22/May/2020:16:53:13 +0000] "HEAD /v2/factfiber.ai/factfiber.ai/antegraph/blobs/sha256:721ada56ab3018810973645913eb705a34ab7aa333d5bbf005b380eb2ac5f8c0 HTTP/1.1" 404 157 "" "docker/19.03.8 go/go1.12.17 git-commit/afacb8b kernel/4.19.76-linuxkit os/linux arch/amd64 UpstreamClient(Docker-Client/19.03.8 \\(darwin\\))"
time="2020-05-22T16:53:13.424933845Z" level=error msg="response completed with error" auth.user.name=shauncutts err.code="blob unknown" err.detail=map[data:sha256:721ada56ab3018810973645913eb705a34ab7aa333d5bbf005b380eb2ac5f8c0] err.message="blob unknown to registry" go.version=go1.13.9 http.request.host=reg.factfiber.ai http.request.id=7ba9bb3e-67fc-45bf-ae19-86a911fbe445 http.request.method=HEAD http.request.remoteaddr=10.2.7.200 http.request.uri="/v2/factfiber.ai/factfiber.ai/antegraph/blobs/sha256:721ada56ab3018810973645913eb705a34ab7aa333d5bbf005b380eb2ac5f8c0" http.request.useragent="docker/19.03.8 go/go1.12.17 git-commit/afacb8b kernel/4.19.76-linuxkit os/linux arch/amd64 UpstreamClient(Docker-Client/19.03.8 \(darwin\))" http.response.contenttype="application/json; charset=utf-8" http.response.duration=21.991425ms http.response.status=404 http.response.written=157 vars.digest="sha256:721ada56ab3018810973645913eb705a34ab7aa333d5bbf005b380eb2ac5f8c0" vars.name="factfiber.ai/factfiber.ai/antegraph"
10.2.4.70 - - [22/May/2020:16:53:13 +0000] "HEAD /v2/factfiber.ai/factfiber.ai/antegraph/blobs/sha256:6154df8ff9882934dc5bf265b8b85a3aeadba06387447ffa440f7af7f32b0e1d HTTP/1.1" 404 157 "" "docker/19.03.8 go/go1.12.17 git-commit/afacb8b kernel/4.19.76-linuxkit os/linux arch/amd64 UpstreamClient(Docker-Client/19.03.8 \\(darwin\\))"
time="2020-05-22T16:53:13.512986124Z" level=error msg="response completed with error" auth.user.name=shauncutts err.code="blob unknown" err.detail=map[data:sha256:6154df8ff9882934dc5bf265b8b85a3aeadba06387447ffa440f7af7f32b0e1d] err.message="blob unknown to registry" go.version=go1.13.9 http.request.host=reg.factfiber.ai http.request.id=8593697d-d614-4ca2-8bb1-0a6fc3ce6dcb http.request.method=HEAD http.request.remoteaddr=10.2.7.200 http.request.uri="/v2/factfiber.ai/factfiber.ai/antegraph/blobs/sha256:6154df8ff9882934dc5bf265b8b85a3aeadba06387447ffa440f7af7f32b0e1d" http.request.useragent="docker/19.03.8 go/go1.12.17 git-commit/afacb8b kernel/4.19.76-linuxkit os/linux arch/amd64 UpstreamClient(Docker-Client/19.03.8 \(darwin\))" http.response.contenttype="application/json; charset=utf-8" http.response.duration=10.164396ms http.response.status=404 http.response.written=157 vars.digest="sha256:6154df8ff9882934dc5bf265b8b85a3aeadba06387447ffa440f7af7f32b0e1d" vars.name="factfiber.ai/factfiber.ai/antegraph"
10.2.4.70 - - [22/May/2020:16:53:13 +0000] "POST /v2/factfiber.ai/factfiber.ai/antegraph/blobs/uploads/ HTTP/1.1" 202 0 "" "docker/19.03.8 go/go1.12.17 git-commit/afacb8b kernel/4.19.76-linuxkit os/linux arch/amd64 UpstreamClient(Docker-Client/19.03.8 \\(darwin\\))"
time="2020-05-22T16:53:13.821162777Z" level=error msg="response completed with error" auth.user.name=shauncutts err.code="blob unknown" err.detail=map[data:sha256:fc878cd0a91c7bece56f668b2c79a19d94dd5471dae41fe5a7e14b4ae65251f6] err.message="blob unknown to registry" go.version=go1.13.9 http.request.host=reg.factfiber.ai http.request.id=33fa5036-d3f3-4942-9328-1150ba9618ce http.request.method=HEAD http.request.remoteaddr=10.2.7.200 http.request.uri="/v2/factfiber.ai/factfiber.ai/antegraph/blobs/sha256:fc878cd0a91c7bece56f668b2c79a19d94dd5471dae41fe5a7e14b4ae65251f6" http.request.useragent="docker/19.03.8 go/go1.12.17 git-commit/afacb8b kernel/4.19.76-linuxkit os/linux arch/amd64 UpstreamClient(Docker-Client/19.03.8 \(darwin\))" http.response.contenttype="application/json; charset=utf-8" http.response.duration=6.546903ms http.response.status=404 http.response.written=157 vars.digest="sha256:fc878cd0a91c7bece56f668b2c79a19d94dd5471dae41fe5a7e14b4ae65251f6" vars.name="factfiber.ai/factfiber.ai/antegraph"
10.2.4.70 - - [22/May/2020:16:53:13 +0000] "HEAD /v2/factfiber.ai/factfiber.ai/antegraph/blobs/sha256:fc878cd0a91c7bece56f668b2c79a19d94dd5471dae41fe5a7e14b4ae65251f6 HTTP/1.1" 404 157 "" "docker/19.03.8 go/go1.12.17 git-commit/afacb8b kernel/4.19.76-linuxkit os/linux arch/amd64 UpstreamClient(Docker-Client/19.03.8 \\(darwin\\))"
10.2.4.70 - - [22/May/2020:16:53:13 +0000] "POST /v2/factfiber.ai/factfiber.ai/antegraph/blobs/uploads/ HTTP/1.1" 202 0 "" "docker/19.03.8 go/go1.12.17 git-commit/afacb8b kernel/4.19.76-linuxkit os/linux arch/amd64 UpstreamClient(Docker-Client/19.03.8 \\(darwin\\))"Output of checks
(If you are reporting a bug on GitLab.com, write: This bug happens on GitLab.com)
Results of GitLab environment info
Expand for output related to GitLab environment info
(For installations with omnibus-gitlab package run and paste the output of: `sudo gitlab-rake gitlab:env:info`) (For installations from source run and paste the output of: `sudo -u git -H bundle exec rake gitlab:env:info RAILS_ENV=production`)
Results of GitLab application Check
Expand for output related to the GitLab application check
(For installations with omnibus-gitlab package run and paste the output of:
sudo gitlab-rake gitlab:check SANITIZE=true)(For installations from source run and paste the output of:
sudo -u git -H bundle exec rake gitlab:check RAILS_ENV=production SANITIZE=true)(we will only investigate if the tests are passing)
Possible fixes
(If you can, link to the line of code that might be responsible for the problem)
Edited by Shaun Cutts