Containerd rewrite does not work with Dependency Proxy
Summary
I use the dependency proxy in conjunction with K3s, which in turn uses Containerd as its container management layer.
I have configured Containerd to use the dependency proxy as a mirror for docker.io
. However, the results are not as I would expect.
The configuration I use worked until a few months ago (we first encountered our issue on November 29 2021, presumably it broke after updating Gitlab to the then most recent stable version.
Steps to reproduce
- Save the following to
/etc/rancher/k3s/registries.yaml
:--- mirrors: docker.io: endpoint: - "open.greenhost.net:443" # This is our GitLab instance rewrite: "^(.*)$": "/stackspin/dependency_proxy/containers/$1" configs: "open.greenhost.net:443": auth: username: <A valid GitLab username> password: <A token for that user with `read_registry` and `write_registry` access> ``
- Install K3s
- Run
crictl -D pull docker.io/hello-world
Example Project
This is quite a large example project, but at least it's the project I'm currently working on and having this issue on: https://open.greenhost.net/stackspin/stackspin/-/merge_requests/757
What is the current bug behavior?
I'll describe this with an image that is currently not present in my Dependency proxy.
# crictl -D pull bitnami/postgresql
DEBU[0000] get image connection
DEBU[0000] connect using endpoint 'unix:///run/k3s/containerd/containerd.sock' with '2s' timeout
DEBU[0000] connected successfully using endpoint: unix:///run/k3s/containerd/containerd.sock
DEBU[0000] PullImageRequest: &PullImageRequest{Image:&ImageSpec{Image:bitnami/postgresql,Annotations:map[string]string{},},Auth:nil,SandboxConfig:nil,}
DEBU[0008] PullImageResponse: nil
FATA[0008] pulling image: rpc error: code = Unknown desc = failed to pull and unpack image "docker.io/bitnami/postgresql:latest": failed to read expected number of bytes: unexpected EOF
Because weirdly enough, for images that are present in the dependency proxy, the result is different (see "expected correct behaviour")
What is the expected correct behavior?
crictl -D pull hello-world
DEBU[0000] get image connection
DEBU[0000] connect using endpoint 'unix:///run/k3s/containerd/containerd.sock' with '2s' timeout
DEBU[0000] connected successfully using endpoint: unix:///run/k3s/containerd/containerd.sock
DEBU[0000] PullImageRequest: &PullImageRequest{Image:&ImageSpec{Image:hello-world,Annotations:map[string]string{},},Auth:nil,SandboxConfig:nil,}
DEBU[0004] PullImageResponse: &PullImageResponse{ImageRef:sha256:feb5d9fea6a5e9606aa995e879d862b825965ba48de054caab5ef356dc6b3412,}
Image is up to date for sha256:feb5d9fea6a5e9606aa995e879d862b825965ba48de054caab5ef356dc6b3412
I can also add new images to the dependency proxy by using the actual dependency proxy address:
# crictl -D pull open.greenhost.net:443/stackspin/dependency_proxy/containers/alpine:3.14
DEBU[0000] get image connection
DEBU[0000] connect using endpoint 'unix:///run/k3s/containerd/containerd.sock' with '2s' timeout
DEBU[0000] connected successfully using endpoint: unix:///run/k3s/containerd/containerd.sock
DEBU[0000] PullImageRequest: &PullImageRequest{Image:&ImageSpec{Image:open.greenhost.net:443/stackspin/dependency_proxy/containers/alpine:3.14,Annotations:map[string]string{},},Auth:nil,SandboxConfig:nil,}
DEBU[0013] PullImageResponse: &PullImageResponse{ImageRef:sha256:0a97eee8041e2b6c0e65abb2700b0705d0da5525ca69060b9e0bde8a3d17afdb,}
Image is up to date for sha256:0a97eee8041e2b6c0e65abb2700b0705d0da5525ca69060b9e0bde8a3d17afdb
So all in all, I can only conclude that my setup works correctly.
Relevant logs and/or screenshots
Logs on our GitLab instance for when I try to pull using the ContainerD mirror (unsuccessfully)
{"method":"HEAD","path":"/v2/stackspin/dependency_proxy/containers/bitnami/mysql/manifests/latest","format":"html","controller":"Groups::DependencyProxyForContainersController","action":"manifest","status":401,"time":"2022-01-17T12:33:57.193Z","params":[{"key":"ns","value":"docker.io"},{"key":"group_id","value":"stackspin"},{"key":"image","value":"bitnami/mysql"},{"key":"tag","value":"latest"}],"remote_ip":"213.108.108.41","ua":"containerd/v1.5.8-k3s2","queue_duration_s":0.00471,"request_urgency":"default","target_duration_s":1,"db_count":0,"db_write_count":0,"db_cached_count":0,"db_replica_count":0,"db_primary_count":0,"db_replica_cached_count":0,"db_primary_cached_count":0,"db_replica_wal_count":0,"db_primary_wal_count":0,"db_replica_wal_cached_count":0,"db_primary_wal_cached_count":0,"db_replica_duration_s":0.0,"db_primary_duration_s":0.0,"cpu_s":0.009591,"mem_objects":3178,"mem_bytes":267056,"mem_mallocs":858,"mem_total_bytes":394176,"pid":4050,"correlation_id":"01FSM0F7XZHC0RYRA53S2HXY7V","db_duration_s":0.0,"view_duration_s":0.00042,"duration_s":0.00236} {"method":"GET","path":"/jwt/auth","format":"html","controller":"JwtController","action":"auth","status":200,"time":"2022-01-17T12:33:57.248Z","params":[{"key":"scope","value":"repository:bitnami/mysql:pull"},{"key":"service","value":"dependency_proxy"}],"correlation_id":"01FSM0F7Z8TR3M76H6ZXDH542H","meta.user":"maarten","meta.caller_id":"JwtController#auth","meta.remote_ip":"213.108.108.41","meta.feature_category":"authentication_and_authorization","meta.client_id":"user/4","remote_ip":"213.108.108.41","user_id":4,"username":"maarten","ua":"containerd/v1.5.8-k3s2","queue_duration_s":0.004747,"request_urgency":"default","target_duration_s":1,"db_count":3,"db_write_count":0,"db_cached_count":0,"db_replica_count":0,"db_primary_count":3,"db_replica_cached_count":0,"db_primary_cached_count":0,"db_replica_wal_count":0,"db_primary_wal_count":0,"db_replica_wal_cached_count":0,"db_primary_wal_cached_count":0,"db_replica_duration_s":0.0,"db_primary_duration_s":0.002,"cpu_s":0.021717,"mem_objects":7872,"mem_bytes":1259496,"mem_mallocs":2252,"mem_total_bytes":1574376,"pid":3740,"db_duration_s":0.00196,"view_duration_s":0.00022,"duration_s":0.01631} {"method":"HEAD","path":"/v2/stackspin/dependency_proxy/containers/bitnami/mysql/manifests/latest","format":"html","controller":"Groups::DependencyProxyForContainersController","action":"manifest","status":200,"time":"2022-01-17T12:33:58.117Z","params":[{"key":"ns","value":"docker.io"},{"key":"group_id","value":"stackspin"},{"key":"image","value":"bitnami/mysql"},{"key":"tag","value":"latest"}],"correlation_id":"01FSM0F804GEPR6M9P3A8PWMR3","meta.user":"maarten","meta.root_namespace":"stackspin","meta.caller_id":"Groups::DependencyProxyForContainersController#manifest","meta.remote_ip":"213.108.108.41","meta.feature_category":"dependency_proxy","meta.client_id":"user/4","remote_ip":"213.108.108.41","user_id":4,"username":"maarten","ua":"containerd/v1.5.8-k3s2","queue_duration_s":0.004843,"request_urgency":"default","target_duration_s":1,"redis_calls":2,"redis_duration_s":0.0029019999999999996,"redis_read_bytes":202,"redis_write_bytes":202,"redis_cache_calls":1,"redis_cache_duration_s":0.002547,"redis_cache_read_bytes":202,"redis_cache_write_bytes":55,"redis_shared_state_calls":1,"redis_shared_state_duration_s":0.000355,"redis_shared_state_write_bytes":147,"db_count":9,"db_write_count":0,"db_cached_count":0,"db_replica_count":0,"db_primary_count":9,"db_replica_cached_count":0,"db_primary_cached_count":0,"db_replica_wal_count":0,"db_primary_wal_count":0,"db_replica_wal_cached_count":0,"db_primary_wal_cached_count":0,"db_replica_duration_s":0.0,"db_primary_duration_s":0.006,"external_http_count":2,"external_http_duration_s":0.24730288796126842,"cpu_s":0.046117,"mem_objects":13567,"mem_bytes":1836264,"mem_mallocs":3400,"mem_total_bytes":2378944,"pid":3902,"db_duration_s":0.00561,"view_duration_s":0.0,"duration_s":0.85625} {"method":"GET","path":"/v2/stackspin/dependency_proxy/containers/bitnami/mysql/manifests/sha256:00bb0a18d63af29610aa2a84f2780345acf667013dc65f5ecf7f1576438d46dd","format":"html","controller":"Groups::DependencyProxyForContainersController","action":"manifest","status":401,"time":"2022-01-17T12:33:58.633Z","params":[{"key":"ns","value":"docker.io"},{"key":"group_id","value":"stackspin"},{"key":"image","value":"bitnami/mysql"},{"key":"tag","value":"sha256:00bb0a18d63af29610aa2a84f2780345acf667013dc65f5ecf7f1576438d46dd"}],"remote_ip":"213.108.108.41","ua":"containerd/v1.5.8-k3s2","queue_duration_s":0.005179,"request_urgency":"default","target_duration_s":1,"db_count":0,"db_write_count":0,"db_cached_count":0,"db_replica_count":0,"db_primary_count":0,"db_replica_cached_count":0,"db_primary_cached_count":0,"db_replica_wal_count":0,"db_primary_wal_count":0,"db_replica_wal_cached_count":0,"db_primary_wal_cached_count":0,"db_replica_duration_s":0.0,"db_primary_duration_s":0.0,"cpu_s":0.010375,"mem_objects":3178,"mem_bytes":270016,"mem_mallocs":858,"mem_total_bytes":397136,"pid":3740,"correlation_id":"01FSM0F9AX4M0WP3GYJ33TCFEA","db_duration_s":0.0,"view_duration_s":0.00045,"duration_s":0.00244} {"method":"GET","path":"/jwt/auth","format":"html","controller":"JwtController","action":"auth","status":200,"time":"2022-01-17T12:33:58.675Z","params":[{"key":"scope","value":"repository:bitnami/mysql:pull"},{"key":"service","value":"dependency_proxy"}],"correlation_id":"01FSM0F9BRVVPNAB8C1CF6RQ31","meta.user":"maarten","meta.caller_id":"JwtController#auth","meta.remote_ip":"213.108.108.41","meta.feature_category":"authentication_and_authorization","meta.client_id":"user/4","remote_ip":"213.108.108.41","user_id":4,"username":"maarten","ua":"containerd/v1.5.8-k3s2","queue_duration_s":0.004101,"request_urgency":"default","target_duration_s":1,"redis_calls":4,"redis_duration_s":0.00095,"redis_read_bytes":811,"redis_write_bytes":229,"redis_cache_calls":4,"redis_cache_duration_s":0.00095,"redis_cache_read_bytes":811,"redis_cache_write_bytes":229,"db_count":3,"db_write_count":0,"db_cached_count":0,"db_replica_count":0,"db_primary_count":3,"db_replica_cached_count":0,"db_primary_cached_count":0,"db_replica_wal_count":0,"db_primary_wal_count":0,"db_replica_wal_cached_count":0,"db_primary_wal_cached_count":0,"db_replica_duration_s":0.0,"db_primary_duration_s":0.002,"cpu_s":0.024162,"mem_objects":8739,"mem_bytes":1464008,"mem_mallocs":2595,"mem_total_bytes":1813568,"pid":3740,"db_duration_s":0.00196,"view_duration_s":0.00026,"duration_s":0.01931} {"method":"GET","path":"/v2/stackspin/dependency_proxy/containers/bitnami/mysql/manifests/sha256:00bb0a18d63af29610aa2a84f2780345acf667013dc65f5ecf7f1576438d46dd","format":"html","controller":"Groups::DependencyProxyForContainersController","action":"manifest","status":200,"time":"2022-01-17T12:33:59.543Z","params":[{"key":"ns","value":"docker.io"},{"key":"group_id","value":"stackspin"},{"key":"image","value":"bitnami/mysql"},{"key":"tag","value":"sha256:00bb0a18d63af29610aa2a84f2780345acf667013dc65f5ecf7f1576438d46dd"}],"correlation_id":"01FSM0F9CQZCMHA99FCR4R3N55","meta.user":"maarten","meta.root_namespace":"stackspin","meta.caller_id":"Groups::DependencyProxyForContainersController#manifest","meta.remote_ip":"213.108.108.41","meta.feature_category":"dependency_proxy","meta.client_id":"user/4","remote_ip":"213.108.108.41","user_id":4,"username":"maarten","ua":"containerd/v1.5.8-k3s2","queue_duration_s":0.005387,"request_urgency":"default","target_duration_s":1,"redis_calls":2,"redis_duration_s":0.000631,"redis_read_bytes":202,"redis_write_bytes":202,"redis_cache_calls":1,"redis_cache_duration_s":0.000299,"redis_cache_read_bytes":202,"redis_cache_write_bytes":55,"redis_shared_state_calls":1,"redis_shared_state_duration_s":0.000332,"redis_shared_state_write_bytes":147,"db_count":9,"db_write_count":0,"db_cached_count":0,"db_replica_count":0,"db_primary_count":9,"db_replica_cached_count":0,"db_primary_cached_count":0,"db_replica_wal_count":0,"db_primary_wal_count":0,"db_replica_wal_cached_count":0,"db_primary_wal_cached_count":0,"db_replica_duration_s":0.0,"db_primary_duration_s":0.006,"external_http_count":2,"external_http_duration_s":0.24156400188803673,"cpu_s":0.049034,"mem_objects":13723,"mem_bytes":1885272,"mem_mallocs":3439,"mem_total_bytes":2434192,"pid":3740,"db_duration_s":0.00527,"view_duration_s":0.0,"duration_s":0.85378} {"method":"GET","path":"/v2/stackspin/dependency_proxy/containers/bitnami/mysql/manifests/sha256:00bb0a18d63af29610aa2a84f2780345acf667013dc65f5ecf7f1576438d46dd","format":"html","controller":"Groups::DependencyProxyForContainersController","action":"manifest","status":200,"time":"2022-01-17T12:34:00.818Z","params":[{"key":"ns","value":"docker.io"},{"key":"group_id","value":"stackspin"},{"key":"image","value":"bitnami/mysql"},{"key":"tag","value":"sha256:00bb0a18d63af29610aa2a84f2780345acf667013dc65f5ecf7f1576438d46dd"}],"correlation_id":"01FSM0FAN6YSMY7WSAZSCJE8M8","meta.user":"maarten","meta.root_namespace":"stackspin","meta.caller_id":"Groups::DependencyProxyForContainersController#manifest","meta.remote_ip":"213.108.108.41","meta.feature_category":"dependency_proxy","meta.client_id":"user/4","remote_ip":"213.108.108.41","user_id":4,"username":"maarten","ua":"containerd/v1.5.8-k3s2","queue_duration_s":0.0045,"request_urgency":"default","target_duration_s":1,"redis_calls":1,"redis_duration_s":0.000336,"redis_write_bytes":147,"redis_shared_state_calls":1,"redis_shared_state_duration_s":0.000336,"redis_shared_state_write_bytes":147,"db_count":9,"db_write_count":0,"db_cached_count":0,"db_replica_count":0,"db_primary_count":9,"db_replica_cached_count":0,"db_primary_cached_count":0,"db_replica_wal_count":0,"db_primary_wal_count":0,"db_replica_wal_cached_count":0,"db_primary_wal_cached_count":0,"db_replica_duration_s":0.0,"db_primary_duration_s":0.005,"external_http_count":2,"external_http_duration_s":0.23032099287956953,"cpu_s":0.044071,"mem_objects":13506,"mem_bytes":1833688,"mem_mallocs":3351,"mem_total_bytes":2373928,"pid":3902,"db_duration_s":0.00468,"view_duration_s":0.0,"duration_s":0.83656} {"method":"GET","path":"/v2/stackspin/dependency_proxy/containers/bitnami/mysql/manifests/sha256:00bb0a18d63af29610aa2a84f2780345acf667013dc65f5ecf7f1576438d46dd","format":"html","controller":"Groups::DependencyProxyForContainersController","action":"manifest","status":200,"time":"2022-01-17T12:34:02.114Z","params":[{"key":"ns","value":"docker.io"},{"key":"group_id","value":"stackspin"},{"key":"image","value":"bitnami/mysql"},{"key":"tag","value":"sha256:00bb0a18d63af29610aa2a84f2780345acf667013dc65f5ecf7f1576438d46dd"}],"correlation_id":"01FSM0FBX6KKZDCB52TZ1AQPBJ","meta.user":"maarten","meta.root_namespace":"stackspin","meta.caller_id":"Groups::DependencyProxyForContainersController#manifest","meta.remote_ip":"213.108.108.41","meta.feature_category":"dependency_proxy","meta.client_id":"user/4","remote_ip":"213.108.108.41","user_id":4,"username":"maarten","ua":"containerd/v1.5.8-k3s2","queue_duration_s":0.00468,"request_urgency":"default","target_duration_s":1,"redis_calls":5,"redis_duration_s":0.001266,"redis_read_bytes":811,"redis_write_bytes":376,"redis_cache_calls":4,"redis_cache_duration_s":0.000945,"redis_cache_read_bytes":811,"redis_cache_write_bytes":229,"redis_shared_state_calls":1,"redis_shared_state_duration_s":0.000321,"redis_shared_state_write_bytes":147,"db_count":9,"db_write_count":0,"db_cached_count":0,"db_replica_count":0,"db_primary_count":9,"db_replica_cached_count":0,"db_primary_cached_count":0,"db_replica_wal_count":0,"db_primary_wal_count":0,"db_replica_wal_cached_count":0,"db_primary_wal_cached_count":0,"db_replica_duration_s":0.0,"db_primary_duration_s":0.005,"external_http_count":2,"external_http_duration_s":0.24000724498182535,"cpu_s":0.045639,"mem_objects":14350,"mem_bytes":2036600,"mem_mallocs":3690,"mem_total_bytes":2610600,"pid":3902,"db_duration_s":0.00492,"view_duration_s":0.0,"duration_s":0.85097} {"method":"GET","path":"/v2/stackspin/dependency_proxy/containers/bitnami/mysql/manifests/sha256:00bb0a18d63af29610aa2a84f2780345acf667013dc65f5ecf7f1576438d46dd","format":"html","controller":"Groups::DependencyProxyForContainersController","action":"manifest","status":200,"time":"2022-01-17T12:34:03.412Z","params":[{"key":"ns","value":"docker.io"},{"key":"group_id","value":"stackspin"},{"key":"image","value":"bitnami/mysql"},{"key":"tag","value":"sha256:00bb0a18d63af29610aa2a84f2780345acf667013dc65f5ecf7f1576438d46dd"}],"correlation_id":"01FSM0FD62TEDWKN6Z3PMJRN8D","meta.user":"maarten","meta.root_namespace":"stackspin","meta.caller_id":"Groups::DependencyProxyForContainersController#manifest","meta.remote_ip":"213.108.108.41","meta.feature_category":"dependency_proxy","meta.client_id":"user/4","remote_ip":"213.108.108.41","user_id":4,"username":"maarten","ua":"containerd/v1.5.8-k3s2","queue_duration_s":0.00461,"request_urgency":"default","target_duration_s":1,"redis_calls":1,"redis_duration_s":0.000355,"redis_write_bytes":147,"redis_shared_state_calls":1,"redis_shared_state_duration_s":0.000355,"redis_shared_state_write_bytes":147,"db_count":9,"db_write_count":0,"db_cached_count":0,"db_replica_count":0,"db_primary_count":9,"db_replica_cached_count":0,"db_primary_cached_count":0,"db_replica_wal_count":0,"db_primary_wal_count":0,"db_replica_wal_cached_count":0,"db_primary_wal_cached_count":0,"db_replica_duration_s":0.0,"db_primary_duration_s":0.005,"external_http_count":2,"external_http_duration_s":0.236992496997118,"cpu_s":0.043399,"mem_objects":13507,"mem_bytes":1850072,"mem_mallocs":3352,"mem_total_bytes":2390352,"pid":4050,"db_duration_s":0.00499,"view_duration_s":0.0,"duration_s":0.84205}
Logs on our GitLab instance when I try to pull using the Dependency Proxy's direct address (successfully):
{"method":"HEAD","path":"/v2/stackspin/dependency_proxy/containers/bitnami/mysql/manifests/latest","format":"html","controller":"Groups::DependencyProxyForContainersController","action":"manifest","status":401,"time":"2022-01-17T12:34:53.477Z","params":[{"key":"group_id","value":"stackspin"},{"key":"image","value":"bitnami/mysql"},{"key":"tag","value":"latest"}],"remote_ip":"213.108.108.41","ua":"containerd/v1.5.8-k3s2","queue_duration_s":0.004955,"request_urgency":"default","target_duration_s":1,"db_count":0,"db_write_count":0,"db_cached_count":0,"db_replica_count":0,"db_primary_count":0,"db_replica_cached_count":0,"db_primary_cached_count":0,"db_replica_wal_count":0,"db_primary_wal_count":0,"db_replica_wal_cached_count":0,"db_primary_wal_cached_count":0,"db_replica_duration_s":0.0,"db_primary_duration_s":0.0,"cpu_s":0.010038,"mem_objects":3053,"mem_bytes":264464,"mem_mallocs":826,"mem_total_bytes":386584,"pid":3740,"correlation_id":"01FSM0GYWS4M2NSC8WHSJWSBQE","db_duration_s":0.0,"view_duration_s":0.00057,"duration_s":0.00257} {"method":"GET","path":"/jwt/auth","format":"html","controller":"JwtController","action":"auth","status":200,"time":"2022-01-17T12:34:53.518Z","params":[{"key":"scope","value":"repository:stackspin/dependency_proxy/containers/bitnami/mysql:pull"},{"key":"service","value":"dependency_proxy"}],"correlation_id":"01FSM0GYXM7HPRYWA2620124Z6","meta.user":"maarten","meta.caller_id":"JwtController#auth","meta.remote_ip":"213.108.108.41","meta.feature_category":"authentication_and_authorization","meta.client_id":"user/4","remote_ip":"213.108.108.41","user_id":4,"username":"maarten","ua":"containerd/v1.5.8-k3s2","queue_duration_s":0.00422,"request_urgency":"default","target_duration_s":1,"db_count":3,"db_write_count":0,"db_cached_count":0,"db_replica_count":0,"db_primary_count":3,"db_replica_cached_count":0,"db_primary_cached_count":0,"db_replica_wal_count":0,"db_primary_wal_count":0,"db_replica_wal_cached_count":0,"db_primary_wal_cached_count":0,"db_replica_duration_s":0.0,"db_primary_duration_s":0.003,"cpu_s":0.0216,"mem_objects":7890,"mem_bytes":1260232,"mem_mallocs":2252,"mem_total_bytes":1575832,"pid":3740,"db_duration_s":0.00334,"view_duration_s":0.00019,"duration_s":0.01715} {"method":"HEAD","path":"/v2/stackspin/dependency_proxy/containers/bitnami/mysql/manifests/latest","format":"html","controller":"Groups::DependencyProxyForContainersController","action":"manifest","status":200,"time":"2022-01-17T12:34:54.413Z","params":[{"key":"group_id","value":"stackspin"},{"key":"image","value":"bitnami/mysql"},{"key":"tag","value":"latest"}],"correlation_id":"01FSM0GYYHWHD15YB73J39HG46","meta.user":"maarten","meta.root_namespace":"stackspin","meta.caller_id":"Groups::DependencyProxyForContainersController#manifest","meta.remote_ip":"213.108.108.41","meta.feature_category":"dependency_proxy","meta.client_id":"user/4","remote_ip":"213.108.108.41","user_id":4,"username":"maarten","ua":"containerd/v1.5.8-k3s2","queue_duration_s":0.0038,"request_urgency":"default","target_duration_s":1,"redis_calls":1,"redis_duration_s":0.000257,"redis_write_bytes":147,"redis_shared_state_calls":1,"redis_shared_state_duration_s":0.000257,"redis_shared_state_write_bytes":147,"db_count":9,"db_write_count":0,"db_cached_count":0,"db_replica_count":0,"db_primary_count":9,"db_replica_cached_count":0,"db_primary_cached_count":0,"db_replica_wal_count":0,"db_primary_wal_count":0,"db_replica_wal_cached_count":0,"db_primary_wal_cached_count":0,"db_replica_duration_s":0.0,"db_primary_duration_s":0.007,"external_http_count":2,"external_http_duration_s":0.2743364665657282,"cpu_s":0.044709,"mem_objects":13249,"mem_bytes":1803336,"mem_mallocs":3285,"mem_total_bytes":2333296,"pid":3740,"db_duration_s":0.00607,"view_duration_s":0.0,"duration_s":0.88444} {"method":"POST","path":"/v2/stackspin/dependency_proxy/containers/bitnami/mysql/manifests/latest/upload/authorize","format":"html","controller":"Groups::DependencyProxyForContainersController","action":"authorize_upload_manifest","status":200,"time":"2022-01-17T12:34:54.879Z","params":[{"key":"group_id","value":"stackspin"},{"key":"image","value":"bitnami/mysql"},{"key":"tag","value":"latest"}],"correlation_id":"01FSM0GYYHWHD15YB73J39HG46","meta.user":"maarten","meta.root_namespace":"stackspin","meta.caller_id":"Groups::DependencyProxyForContainersController#authorize_upload_manifest","meta.remote_ip":"213.108.108.41","meta.feature_category":"dependency_proxy","meta.client_id":"user/4","remote_ip":"213.108.108.41","user_id":4,"username":"maarten","ua":"containerd/v1.5.8-k3s2","request_urgency":"default","target_duration_s":1,"redis_calls":4,"redis_duration_s":0.0008960000000000001,"redis_read_bytes":609,"redis_write_bytes":318,"redis_cache_calls":3,"redis_cache_duration_s":0.000496,"redis_cache_read_bytes":609,"redis_cache_write_bytes":171,"redis_shared_state_calls":1,"redis_shared_state_duration_s":0.0004,"redis_shared_state_write_bytes":147,"db_count":7,"db_write_count":0,"db_cached_count":0,"db_replica_count":0,"db_primary_count":7,"db_replica_cached_count":0,"db_primary_cached_count":0,"db_replica_wal_count":0,"db_primary_wal_count":0,"db_replica_wal_cached_count":0,"db_primary_wal_cached_count":0,"db_replica_duration_s":0.0,"db_primary_duration_s":0.005,"cpu_s":0.03762,"mem_objects":11408,"mem_bytes":1723008,"mem_mallocs":3362,"mem_total_bytes":2179328,"pid":4050,"db_duration_s":0.00456,"view_duration_s":0.00027,"duration_s":0.03348} {"method":"POST","path":"/v2/stackspin/dependency_proxy/containers/bitnami/mysql/manifests/latest/upload","format":"html","controller":"Groups::DependencyProxyForContainersController","action":"upload_manifest","status":200,"time":"2022-01-17T12:34:54.935Z","params":[{"key":"file.gitlab-workhorse-upload","value":"eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJ1cGxvYWQiOnsibWQ1IjoiNTM5OGM3MmFiYzIzNmRhYjkwNjZhMGU1ZDFkNWRmNzMiLCJuYW1lIjoiIiwicGF0aCI6Ii92YXIvb3B0L2dpdGxhYi9naXRsYWItcmFpbHMvc2hhcmVkL2RlcGVuZGVuY3lfcHJveHkvdG1wL3VwbG9hZHMvMDU4MjU2NDAxIiwicmVtb3RlX2lkIjoiIiwicmVtb3RlX3VybCI6IiIsInNoYTEiOiIyOTRhODJlYmY1NGU3MWVmMDUyN2EwZGQzYjE3ZGM1MGEwMTJiMTMyIiwic2hhMjU2IjoiMDBiYjBhMThkNjNhZjI5NjEwYWEyYTg0ZjI3ODAzNDVhY2Y2NjcwMTNkYzY1ZjVlY2Y3ZjE1NzY0MzhkNDZkZCIsInNoYTUxMiI6IjQzNmU5NmUzZDQ3NzQwOTczYzYyMzI4YzQ2MjkxZmEyYThiMDcyNjRiMmFjYTE2ZDU5OTgwMWM3MTA1MTFiODk5ZGUxZGU3MmVjMDNlYjRhZDhmYzQ2MzZlOWI2YmYwYmUxYWYxNTUxZGQyZjk4ZGZiY2JmY2Y5ODY1MWQ5ZjJlIiwic2l6ZSI6IjI2MTkiLCJ1cGxvYWRfZHVyYXRpb24iOiIwLjAwMDM3NzkwMSJ9LCJpc3MiOiJnaXRsYWItd29ya2hvcnNlIn0.anzPVliLdPIPYT33bCVUYUJjcWgEYi3eV6endyLexzs"},{"key":"file.md5","value":"5398c72abc236dab9066a0e5d1d5df73"},{"key":"file.name","value":""},{"key":"file.path","value":"/var/opt/gitlab/gitlab-rails/shared/dependency_proxy/tmp/uploads/058256401"},{"key":"file.remote_id","value":""},{"key":"file.remote_url","value":""},{"key":"file.sha1","value":"294a82ebf54e71ef0527a0dd3b17dc50a012b132"},{"key":"file.sha256","value":"00bb0a18d63af29610aa2a84f2780345acf667013dc65f5ecf7f1576438d46dd"},{"key":"file.sha512","value":"436e96e3d47740973c62328c46291fa2a8b07264b2aca16d599801c710511b899de1de72ec03eb4ad8fc4636e9b6bf0be1af1551dd2f98dfbcbfcf98651d9f2e"},{"key":"file.size","value":"2619"},{"key":"file.upload_duration","value":"0.000377901"},{"key":"file","value":"#"},{"key":"group_id","value":"stackspin"},{"key":"image","value":"bitnami/mysql"},{"key":"tag","value":"latest"}],"correlation_id":"01FSM0GYYHWHD15YB73J39HG46","meta.user":"maarten","meta.root_namespace":"stackspin","meta.caller_id":"Groups::DependencyProxyForContainersController#upload_manifest","meta.remote_ip":"213.108.108.41","meta.feature_category":"dependency_proxy","meta.client_id":"user/4","remote_ip":"213.108.108.41","user_id":4,"username":"maarten","ua":"containerd/v1.5.8-k3s2","queue_duration_s":0.006019,"request_urgency":"default","target_duration_s":1,"redis_calls":2,"redis_duration_s":0.0005809999999999999,"redis_read_bytes":203,"redis_write_bytes":204,"redis_cache_calls":1,"redis_cache_duration_s":0.000296,"redis_cache_read_bytes":203,"redis_cache_write_bytes":57,"redis_shared_state_calls":1,"redis_shared_state_duration_s":0.000285,"redis_shared_state_write_bytes":147,"db_count":13,"db_write_count":4,"db_cached_count":0,"db_replica_count":0,"db_primary_count":13,"db_replica_cached_count":0,"db_primary_cached_count":0,"db_replica_wal_count":0,"db_primary_wal_count":0,"db_replica_wal_cached_count":0,"db_primary_wal_cached_count":0,"db_replica_duration_s":0.0,"db_primary_duration_s":0.01,"cpu_s":0.040094,"mem_objects":13550,"mem_bytes":2219464,"mem_mallocs":3743,"mem_total_bytes":2761464,"pid":3902,"uploaded_file_upload_duration_s":0.000377901,"uploaded_file_size_bytes":2619,"db_duration_s":0.00993,"view_duration_s":0.0,"duration_s":0.04295} {"method":"GET","path":"/v2/stackspin/dependency_proxy/containers/bitnami/mysql/manifests/sha256:00bb0a18d63af29610aa2a84f2780345acf667013dc65f5ecf7f1576438d46dd","format":"html","controller":"Groups::DependencyProxyForContainersController","action":"manifest","status":401,"time":"2022-01-17T12:34:55.136Z","params":[{"key":"group_id","value":"stackspin"},{"key":"image","value":"bitnami/mysql"},{"key":"tag","value":"sha256:00bb0a18d63af29610aa2a84f2780345acf667013dc65f5ecf7f1576438d46dd"}],"remote_ip":"213.108.108.41","ua":"containerd/v1.5.8-k3s2","queue_duration_s":0.00464,"request_urgency":"default","target_duration_s":1,"db_count":0,"db_write_count":0,"db_cached_count":0,"db_replica_count":0,"db_primary_count":0,"db_replica_cached_count":0,"db_primary_cached_count":0,"db_replica_wal_count":0,"db_primary_wal_count":0,"db_replica_wal_cached_count":0,"db_primary_wal_cached_count":0,"db_replica_duration_s":0.0,"db_primary_duration_s":0.0,"cpu_s":0.009321,"mem_objects":3052,"mem_bytes":267168,"mem_mallocs":826,"mem_total_bytes":389248,"pid":3902,"correlation_id":"01FSM0H0GPVAX23X57JD822XSS","db_duration_s":0.0,"view_duration_s":0.00043,"duration_s":0.00237} {"method":"GET","path":"/jwt/auth","format":"html","controller":"JwtController","action":"auth","status":200,"time":"2022-01-17T12:34:55.181Z","params":[{"key":"scope","value":"repository:stackspin/dependency_proxy/containers/bitnami/mysql:pull"},{"key":"service","value":"dependency_proxy"}],"correlation_id":"01FSM0H0HJM80Z9R1PK2BDZHJT","meta.user":"maarten","meta.caller_id":"JwtController#auth","meta.remote_ip":"213.108.108.41","meta.feature_category":"authentication_and_authorization","meta.client_id":"user/4","remote_ip":"213.108.108.41","user_id":4,"username":"maarten","ua":"containerd/v1.5.8-k3s2","queue_duration_s":0.004541,"request_urgency":"default","target_duration_s":1,"redis_calls":1,"redis_duration_s":0.0003,"redis_read_bytes":202,"redis_write_bytes":58,"redis_cache_calls":1,"redis_cache_duration_s":0.0003,"redis_cache_read_bytes":202,"redis_cache_write_bytes":58,"db_count":3,"db_write_count":0,"db_cached_count":0,"db_replica_count":0,"db_primary_count":3,"db_replica_cached_count":0,"db_primary_cached_count":0,"db_replica_wal_count":0,"db_primary_wal_count":0,"db_replica_wal_cached_count":0,"db_primary_wal_cached_count":0,"db_replica_duration_s":0.0,"db_primary_duration_s":0.002,"cpu_s":0.024072,"mem_objects":8106,"mem_bytes":1292168,"mem_mallocs":2339,"mem_total_bytes":1616408,"pid":4050,"db_duration_s":0.00211,"view_duration_s":0.00029,"duration_s":0.01867} {"method":"GET","path":"/v2/stackspin/dependency_proxy/containers/bitnami/mysql/manifests/sha256:00bb0a18d63af29610aa2a84f2780345acf667013dc65f5ecf7f1576438d46dd","format":"html","controller":"Groups::DependencyProxyForContainersController","action":"manifest","status":200,"time":"2022-01-17T12:34:56.048Z","params":[{"key":"group_id","value":"stackspin"},{"key":"image","value":"bitnami/mysql"},{"key":"tag","value":"sha256:00bb0a18d63af29610aa2a84f2780345acf667013dc65f5ecf7f1576438d46dd"}],"correlation_id":"01FSM0H0JGHMZ32MFCMHCG4W1E","meta.user":"maarten","meta.root_namespace":"stackspin","meta.caller_id":"Groups::DependencyProxyForContainersController#manifest","meta.remote_ip":"213.108.108.41","meta.feature_category":"dependency_proxy","meta.client_id":"user/4","remote_ip":"213.108.108.41","user_id":4,"username":"maarten","ua":"containerd/v1.5.8-k3s2","queue_duration_s":0.004032,"request_urgency":"default","target_duration_s":1,"redis_calls":1,"redis_duration_s":0.000274,"redis_write_bytes":147,"redis_shared_state_calls":1,"redis_shared_state_duration_s":0.000274,"redis_shared_state_write_bytes":147,"db_count":13,"db_write_count":3,"db_cached_count":0,"db_replica_count":0,"db_primary_count":13,"db_replica_cached_count":0,"db_primary_cached_count":0,"db_replica_wal_count":0,"db_primary_wal_count":0,"db_replica_wal_cached_count":0,"db_primary_wal_cached_count":0,"db_replica_duration_s":0.0,"db_primary_duration_s":0.012,"external_http_count":2,"external_http_duration_s":0.23251746594905853,"cpu_s":0.05304,"mem_objects":14767,"mem_bytes":2034152,"mem_mallocs":3590,"mem_total_bytes":2624832,"pid":4050,"db_duration_s":0.01142,"view_duration_s":0.0,"duration_s":0.85571} {"method":"GET","path":"/v2/stackspin/dependency_proxy/containers/bitnami/mysql/blobs/sha256:3a5689271ec184d67fca3ff9aedceb472972c1ccd84c5f529fdfdd2b08260fcf","format":"html","controller":"Groups::DependencyProxyForContainersController","action":"blob","status":200,"time":"2022-01-17T12:34:59.653Z","params":[{"key":"group_id","value":"stackspin"},{"key":"image","value":"bitnami/mysql"},{"key":"sha","value":"sha256:3a5689271ec184d67fca3ff9aedceb472972c1ccd84c5f529fdfdd2b08260fcf"}],"correlation_id":"01FSM0H4G1Q98KV7977QBBTRA7","meta.user":"maarten","meta.root_namespace":"stackspin","meta.caller_id":"Groups::DependencyProxyForContainersController#blob","meta.remote_ip":"213.108.108.41","meta.feature_category":"dependency_proxy","meta.client_id":"user/4","remote_ip":"213.108.108.41","user_id":4,"username":"maarten","ua":"containerd/v1.5.8-k3s2","queue_duration_s":0.004629,"request_urgency":"default","target_duration_s":1,"redis_calls":6,"redis_duration_s":0.007313999999999999,"redis_read_bytes":1013,"redis_write_bytes":431,"redis_cache_calls":5,"redis_cache_duration_s":0.001989,"redis_cache_read_bytes":1013,"redis_cache_write_bytes":284,"redis_shared_state_calls":1,"redis_shared_state_duration_s":0.005325,"redis_shared_state_write_bytes":147,"db_count":8,"db_write_count":0,"db_cached_count":0,"db_replica_count":0,"db_primary_count":8,"db_replica_cached_count":0,"db_primary_cached_count":0,"db_replica_wal_count":0,"db_primary_wal_count":0,"db_replica_wal_cached_count":0,"db_primary_wal_cached_count":0,"db_replica_duration_s":0.0,"db_primary_duration_s":0.006,"external_http_count":1,"external_http_duration_s":0.10139547009021044,"cpu_s":0.049759,"mem_objects":13258,"mem_bytes":2045728,"mem_mallocs":3587,"mem_total_bytes":2576048,"pid":3740,"db_duration_s":0.00599,"view_duration_s":0.0,"duration_s":0.44292} {"method":"POST","path":"/v2/stackspin/dependency_proxy/containers/bitnami/mysql/blobs/sha256:3a5689271ec184d67fca3ff9aedceb472972c1ccd84c5f529fdfdd2b08260fcf/upload/authorize","format":"html","controller":"Groups::DependencyProxyForContainersController","action":"authorize_upload_blob","status":200,"time":"2022-01-17T12:35:00.191Z","params":[{"key":"group_id","value":"stackspin"},{"key":"image","value":"bitnami/mysql"},{"key":"sha","value":"sha256:3a5689271ec184d67fca3ff9aedceb472972c1ccd84c5f529fdfdd2b08260fcf"}],"correlation_id":"01FSM0H4G1Q98KV7977QBBTRA7","meta.user":"maarten","meta.root_namespace":"stackspin","meta.caller_id":"Groups::DependencyProxyForContainersController#authorize_upload_blob","meta.remote_ip":"213.108.108.41","meta.feature_category":"dependency_proxy","meta.client_id":"user/4","remote_ip":"213.108.108.41","user_id":4,"username":"maarten","ua":"containerd/v1.5.8-k3s2","request_urgency":"default","target_duration_s":1,"cf_ray":"6cef9ef5bc7e4c26-AMS","redis_calls":1,"redis_duration_s":0.000378,"redis_write_bytes":147,"redis_shared_state_calls":1,"redis_shared_state_duration_s":0.000378,"redis_shared_state_write_bytes":147,"db_count":7,"db_write_count":0,"db_cached_count":0,"db_replica_count":0,"db_primary_count":7,"db_replica_cached_count":0,"db_primary_cached_count":0,"db_replica_wal_count":0,"db_primary_wal_count":0,"db_replica_wal_cached_count":0,"db_primary_wal_cached_count":0,"db_replica_duration_s":0.0,"db_primary_duration_s":0.005,"cpu_s":0.031898,"mem_objects":10416,"mem_bytes":1622192,"mem_mallocs":3011,"mem_total_bytes":2038832,"pid":4050,"db_duration_s":0.00432,"view_duration_s":0.00024,"duration_s":0.02772} {"method":"POST","path":"/v2/stackspin/dependency_proxy/containers/bitnami/mysql/blobs/sha256:3a5689271ec184d67fca3ff9aedceb472972c1ccd84c5f529fdfdd2b08260fcf/upload","format":"html","controller":"Groups::DependencyProxyForContainersController","action":"upload_blob","status":200,"time":"2022-01-17T12:35:00.258Z","params":[{"key":"file.gitlab-workhorse-upload","value":"eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJ1cGxvYWQiOnsibWQ1IjoiNGY5MTYyMTI1NTExN2Y1NTRjN2ZmMGI5M2Y0Y2JlOTQiLCJuYW1lIjoiIiwicGF0aCI6Ii92YXIvb3B0L2dpdGxhYi9naXRsYWItcmFpbHMvc2hhcmVkL2RlcGVuZGVuY3lfcHJveHkvdG1wL3VwbG9hZHMvMTQ3MTcwODc2IiwicmVtb3RlX2lkIjoiIiwicmVtb3RlX3VybCI6IiIsInNoYTEiOiJjMTIyMjkzMGE2ZTA4ZDdjNzg0NGY5MWY2NTRjNDhkMDIyOThhMjk5Iiwic2hhMjU2IjoiM2E1Njg5MjcxZWMxODRkNjdmY2EzZmY5YWVkY2ViNDcyOTcyYzFjY2Q4NGM1ZjUyOWZkZmRkMmIwODI2MGZjZiIsInNoYTUxMiI6IjAzMWEzOTJlZDZmNDM3OThmNDI4YzMwZWU1Y2Y4YTIxZjA4NDU1NzU0MDg3OTAzNjZiYWE5MzRkNDBjMzllNzU2OTY3NzI3Zjc5NmQ1MDQ5OGE4NWE2N2I4N2VmODU2YzcxZDUxNDgyNTAxYzExYmQ5ZjMwZGMwZWQwMjQzNjdlIiwic2l6ZSI6IjU3MDIiLCJ1cGxvYWRfZHVyYXRpb24iOiIwLjAwMDQ5NTM1OCJ9LCJpc3MiOiJnaXRsYWItd29ya2hvcnNlIn0.JOvInVkUYPrM5PxhVHsrIJgszhIuemY8s-yz2D7dkqo"},{"key":"file.md5","value":"4f91621255117f554c7ff0b93f4cbe94"},{"key":"file.name","value":""},{"key":"file.path","value":"/var/opt/gitlab/gitlab-rails/shared/dependency_proxy/tmp/uploads/147170876"},{"key":"file.remote_id","value":""},{"key":"file.remote_url","value":""},{"key":"file.sha1","value":"c1222930a6e08d7c7844f91f654c48d02298a299"},{"key":"file.sha256","value":"3a5689271ec184d67fca3ff9aedceb472972c1ccd84c5f529fdfdd2b08260fcf"},{"key":"file.sha512","value":"031a392ed6f43798f428c30ee5cf8a21f0845575408790366baa934d40c39e756967727f796d50498a85a67b87ef856c71d51482501c11bd9f30dc0ed024367e"},{"key":"file.size","value":"5702"},{"key":"file.upload_duration","value":"0.000495358"},{"key":"file","value":"#"},{"key":"group_id","value":"stackspin"},{"key":"image","value":"bitnami/mysql"},{"key":"sha","value":"sha256:3a5689271ec184d67fca3ff9aedceb472972c1ccd84c5f529fdfdd2b08260fcf"}],"correlation_id":"01FSM0H4G1Q98KV7977QBBTRA7","meta.user":"maarten","meta.root_namespace":"stackspin","meta.caller_id":"Groups::DependencyProxyForContainersController#upload_blob","meta.remote_ip":"213.108.108.41","meta.feature_category":"dependency_proxy","meta.client_id":"user/4","remote_ip":"213.108.108.41","user_id":4,"username":"maarten","ua":"containerd/v1.5.8-k3s2","queue_duration_s":0.005962,"request_urgency":"default","target_duration_s":1,"cf_ray":"6cef9ef5bc7e4c26-AMS","redis_calls":1,"redis_duration_s":0.000438,"redis_write_bytes":147,"redis_shared_state_calls":1,"redis_shared_state_duration_s":0.000438,"redis_shared_state_write_bytes":147,"db_count":12,"db_write_count":4,"db_cached_count":0,"db_replica_count":0,"db_primary_count":12,"db_replica_cached_count":0,"db_primary_cached_count":0,"db_replica_wal_count":0,"db_primary_wal_count":0,"db_replica_wal_cached_count":0,"db_primary_wal_cached_count":0,"db_replica_duration_s":0.0,"db_primary_duration_s":0.019,"cpu_s":0.044872,"mem_objects":12914,"mem_bytes":2162464,"mem_mallocs":3579,"mem_total_bytes":2679024,"pid":4050,"uploaded_file_upload_duration_s":0.000495358,"uploaded_file_size_bytes":5702,"db_duration_s":0.01854,"view_duration_s":0.0,"duration_s":0.05426} {"method":"GET","path":"/v2/stackspin/dependency_proxy/containers/bitnami/mysql/blobs/sha256:f1cabe50ef568332f8933f4077111f0be1344572d24f97d24c49d15d507d880b","format":"html","controller":"Groups::DependencyProxyForContainersController","action":"blob","status":200,"time":"2022-01-17T12:35:00.768Z","params":[{"key":"group_id","value":"stackspin"},{"key":"image","value":"bitnami/mysql"},{"key":"sha","value":"sha256:f1cabe50ef568332f8933f4077111f0be1344572d24f97d24c49d15d507d880b"}],"correlation_id":"01FSM0H5KCM3ZDZXPA3YYX261Q","meta.user":"maarten","meta.root_namespace":"stackspin","meta.caller_id":"Groups::DependencyProxyForContainersController#blob","meta.remote_ip":"213.108.108.41","meta.feature_category":"dependency_proxy","meta.client_id":"user/4","remote_ip":"213.108.108.41","user_id":4,"username":"maarten","ua":"containerd/v1.5.8-k3s2","queue_duration_s":0.005106,"request_urgency":"default","target_duration_s":1,"redis_calls":1,"redis_duration_s":0.000265,"redis_write_bytes":147,"redis_shared_state_calls":1,"redis_shared_state_duration_s":0.000265,"redis_shared_state_write_bytes":147,"db_count":8,"db_write_count":0,"db_cached_count":0,"db_replica_count":0,"db_primary_count":8,"db_replica_cached_count":0,"db_primary_cached_count":0,"db_replica_wal_count":0,"db_primary_wal_count":0,"db_replica_wal_cached_count":0,"db_primary_wal_cached_count":0,"db_replica_duration_s":0.0,"db_primary_duration_s":0.005,"external_http_count":1,"external_http_duration_s":0.10172282345592976,"cpu_s":0.040359,"mem_objects":11801,"mem_bytes":1720912,"mem_mallocs":3056,"mem_total_bytes":2192952,"pid":3902,"db_duration_s":0.00448,"view_duration_s":0.0,"duration_s":0.42581} {"method":"GET","path":"/v2/stackspin/dependency_proxy/containers/bitnami/mysql/blobs/sha256:41b479190c3ad34105037f24369ce93173b7c33bd1e588ccc37bf7ab87bcf8f2","format":"html","controller":"Groups::DependencyProxyForContainersController","action":"blob","status":200,"time":"2022-01-17T12:35:00.806Z","params":[{"key":"group_id","value":"stackspin"},{"key":"image","value":"bitnami/mysql"},{"key":"sha","value":"sha256:41b479190c3ad34105037f24369ce93173b7c33bd1e588ccc37bf7ab87bcf8f2"}],"correlation_id":"01FSM0H5M1G5XEZQS68WAZ8FWJ","meta.user":"maarten","meta.root_namespace":"stackspin","meta.caller_id":"Groups::DependencyProxyForContainersController#blob","meta.remote_ip":"213.108.108.41","meta.feature_category":"dependency_proxy","meta.client_id":"user/4","remote_ip":"213.108.108.41","user_id":4,"username":"maarten","ua":"containerd/v1.5.8-k3s2","queue_duration_s":0.004907,"request_urgency":"default","target_duration_s":1,"redis_calls":1,"redis_duration_s":0.001101,"redis_write_bytes":147,"redis_shared_state_calls":1,"redis_shared_state_duration_s":0.001101,"redis_shared_state_write_bytes":147,"db_count":8,"db_write_count":0,"db_cached_count":0,"db_replica_count":0,"db_primary_count":8,"db_replica_cached_count":0,"db_primary_cached_count":0,"db_replica_wal_count":0,"db_primary_wal_count":0,"db_replica_wal_cached_count":0,"db_primary_wal_cached_count":0,"db_replica_duration_s":0.0,"db_primary_duration_s":0.006,"external_http_count":1,"external_http_duration_s":0.10141550097614527,"cpu_s":0.047818,"mem_objects":11802,"mem_bytes":1741392,"mem_mallocs":3057,"mem_total_bytes":2213472,"pid":4050,"db_duration_s":0.00522,"view_duration_s":0.0,"duration_s":0.44412} {"method":"GET","path":"/v2/stackspin/dependency_proxy/containers/bitnami/mysql/blobs/sha256:4de6e9ef77a36fce6ada7437c84d656ad57346d62917604176fca5b25fa4056f","format":"html","controller":"Groups::DependencyProxyForContainersController","action":"blob","status":200,"time":"2022-01-17T12:35:00.820Z","params":[{"key":"group_id","value":"stackspin"},{"key":"image","value":"bitnami/mysql"},{"key":"sha","value":"sha256:4de6e9ef77a36fce6ada7437c84d656ad57346d62917604176fca5b25fa4056f"}],"correlation_id":"01FSM0H5N1N9MR04ETJ9ZT74SH","meta.user":"maarten","meta.root_namespace":"stackspin","meta.caller_id":"Groups::DependencyProxyForContainersController#blob","meta.remote_ip":"213.108.108.41","meta.feature_category":"dependency_proxy","meta.client_id":"user/4","remote_ip":"213.108.108.41","user_id":4,"username":"maarten","ua":"containerd/v1.5.8-k3s2","queue_duration_s":0.005843,"request_urgency":"default","target_duration_s":1,"redis_calls":1,"redis_duration_s":0.000406,"redis_write_bytes":147,"redis_shared_state_calls":1,"redis_shared_state_duration_s":0.000406,"redis_shared_state_write_bytes":147,"db_count":8,"db_write_count":0,"db_cached_count":0,"db_replica_count":0,"db_primary_count":8,"db_replica_cached_count":0,"db_primary_cached_count":0,"db_replica_wal_count":0,"db_primary_wal_count":0,"db_replica_wal_cached_count":0,"db_primary_wal_cached_count":0,"db_replica_duration_s":0.0,"db_primary_duration_s":0.006,"external_http_count":1,"external_http_duration_s":0.10140489134937525,"cpu_s":0.041424,"mem_objects":11802,"mem_bytes":1757776,"mem_mallocs":3058,"mem_total_bytes":2229856,"pid":3902,"db_duration_s":0.00525,"view_duration_s":0.0,"duration_s":0.42479} {"method":"GET","path":"/v2/stackspin/dependency_proxy/containers/bitnami/mysql/blobs/sha256:73729094e3568d281b4e0027ba04600149fb86082ad1e0c8353c26452e025419","format":"html","controller":"Groups::DependencyProxyForContainersController","action":"blob","status":200,"time":"2022-01-17T12:35:00.845Z","params":[{"key":"group_id","value":"stackspin"},{"key":"image","value":"bitnami/mysql"},{"key":"sha","value":"sha256:73729094e3568d281b4e0027ba04600149fb86082ad1e0c8353c26452e025419"}],"correlation_id":"01FSM0H5MJGQX2DSDXB1TSEHJD","meta.user":"maarten","meta.root_namespace":"stackspin","meta.caller_id":"Groups::DependencyProxyForContainersController#blob","meta.remote_ip":"213.108.108.41","meta.feature_category":"dependency_proxy","meta.client_id":"user/4","remote_ip":"213.108.108.41","user_id":4,"username":"maarten","ua":"containerd/v1.5.8-k3s2","queue_duration_s":0.006827,"request_urgency":"default","target_duration_s":1,"redis_calls":1,"redis_duration_s":0.00025,"redis_write_bytes":147,"redis_shared_state_calls":1,"redis_shared_state_duration_s":0.00025,"redis_shared_state_write_bytes":147,"db_count":8,"db_write_count":0,"db_cached_count":0,"db_replica_count":0,"db_primary_count":8,"db_replica_cached_count":0,"db_primary_cached_count":0,"db_replica_wal_count":0,"db_primary_wal_count":0,"db_replica_wal_cached_count":0,"db_primary_wal_cached_count":0,"db_replica_duration_s":0.0,"db_primary_duration_s":0.009,"external_http_count":1,"external_http_duration_s":0.10261026863008738,"cpu_s":0.041362,"mem_objects":11801,"mem_bytes":1720912,"mem_mallocs":3056,"mem_total_bytes":2192952,"pid":3740,"db_duration_s":0.00878,"view_duration_s":0.0,"duration_s":0.46462} {"method":"GET","path":"/v2/stackspin/dependency_proxy/containers/bitnami/mysql/blobs/sha256:19d690b633e33fab8dcdb616ddc662bb126a241b36d8875054c08a84ad737fc7","format":"html","controller":"Groups::DependencyProxyForContainersController","action":"blob","status":200,"time":"2022-01-17T12:35:00.850Z","params":[{"key":"group_id","value":"stackspin"},{"key":"image","value":"bitnami/mysql"},{"key":"sha","value":"sha256:19d690b633e33fab8dcdb616ddc662bb126a241b36d8875054c08a84ad737fc7"}],"correlation_id":"01FSM0H5NZCAMCMQ6DF4MC94DM","meta.user":"maarten","meta.root_namespace":"stackspin","meta.caller_id":"Groups::DependencyProxyForContainersController#blob","meta.remote_ip":"213.108.108.41","meta.feature_category":"dependency_proxy","meta.client_id":"user/4","remote_ip":"213.108.108.41","user_id":4,"username":"maarten","ua":"containerd/v1.5.8-k3s2","queue_duration_s":0.007052,"request_urgency":"default","target_duration_s":1,"redis_calls":1,"redis_duration_s":0.000694,"redis_write_bytes":147,"redis_shared_state_calls":1,"redis_shared_state_duration_s":0.000694,"redis_shared_state_write_bytes":147,"db_count":8,"db_write_count":0,"db_cached_count":0,"db_replica_count":0,"db_primary_count":8,"db_replica_cached_count":0,"db_primary_cached_count":0,"db_replica_wal_count":0,"db_primary_wal_count":0,"db_replica_wal_cached_count":0,"db_primary_wal_cached_count":0,"db_replica_duration_s":0.0,"db_primary_duration_s":0.005,"external_http_count":1,"external_http_duration_s":0.1012633042410016,"cpu_s":0.039561,"mem_objects":11804,"mem_bytes":1757776,"mem_mallocs":3058,"mem_total_bytes":2229936,"pid":4050,"db_duration_s":0.00454,"view_duration_s":0.0,"duration_s":0.42361} {"method":"GET","path":"/v2/stackspin/dependency_proxy/containers/bitnami/mysql/blobs/sha256:81bf975e185f02351023f2bf780826a91a183164aa1c574c51fc541cc97e3972","format":"html","controller":"Groups::DependencyProxyForContainersController","action":"blob","status":200,"time":"2022-01-17T12:35:00.855Z","params":[{"key":"group_id","value":"stackspin"},{"key":"image","value":"bitnami/mysql"},{"key":"sha","value":"sha256:81bf975e185f02351023f2bf780826a91a183164aa1c574c51fc541cc97e3972"}],"correlation_id":"01FSM0H5NGKYZTXMXW3MM4J5CW","meta.user":"maarten","meta.root_namespace":"stackspin","meta.caller_id":"Groups::DependencyProxyForContainersController#blob","meta.remote_ip":"213.108.108.41","meta.feature_category":"dependency_proxy","meta.client_id":"user/4","remote_ip":"213.108.108.41","user_id":4,"username":"maarten","ua":"containerd/v1.5.8-k3s2","queue_duration_s":0.012926,"request_urgency":"default","target_duration_s":1,"redis_calls":1,"redis_duration_s":0.00128,"redis_write_bytes":147,"redis_shared_state_calls":1,"redis_shared_state_duration_s":0.00128,"redis_shared_state_write_bytes":147,"db_count":8,"db_write_count":0,"db_cached_count":0,"db_replica_count":0,"db_primary_count":8,"db_replica_cached_count":0,"db_primary_cached_count":0,"db_replica_wal_count":0,"db_primary_wal_count":0,"db_replica_wal_cached_count":0,"db_primary_wal_cached_count":0,"db_replica_duration_s":0.0,"db_primary_duration_s":0.017,"external_http_count":1,"external_http_duration_s":0.10075845941901207,"cpu_s":0.037763,"mem_objects":11801,"mem_bytes":1737296,"mem_mallocs":3057,"mem_total_bytes":2209336,"pid":3740,"db_duration_s":0.01661,"view_duration_s":0.0,"duration_s":0.43678} {"method":"GET","path":"/v2/stackspin/dependency_proxy/containers/bitnami/mysql/blobs/sha256:28a756701188d2ddee0911638724dff7664ca8099160da8414e16b79c203f746","format":"html","controller":"Groups::DependencyProxyForContainersController","action":"blob","status":200,"time":"2022-01-17T12:35:00.897Z","params":[{"key":"group_id","value":"stackspin"},{"key":"image","value":"bitnami/mysql"},{"key":"sha","value":"sha256:28a756701188d2ddee0911638724dff7664ca8099160da8414e16b79c203f746"}],"correlation_id":"01FSM0H5Q44HN17BC5R254287Q","meta.user":"maarten","meta.root_namespace":"stackspin","meta.caller_id":"Groups::DependencyProxyForContainersController#blob","meta.remote_ip":"213.108.108.41","meta.feature_category":"dependency_proxy","meta.client_id":"user/4","remote_ip":"213.108.108.41","user_id":4,"username":"maarten","ua":"containerd/v1.5.8-k3s2","queue_duration_s":0.009668,"request_urgency":"default","target_duration_s":1,"redis_calls":1,"redis_duration_s":0.000432,"redis_write_bytes":147,"redis_shared_state_calls":1,"redis_shared_state_duration_s":0.000432,"redis_shared_state_write_bytes":147,"db_count":8,"db_write_count":0,"db_cached_count":0,"db_replica_count":0,"db_primary_count":8,"db_replica_cached_count":0,"db_primary_cached_count":0,"db_replica_wal_count":0,"db_primary_wal_count":0,"db_replica_wal_cached_count":0,"db_primary_wal_cached_count":0,"db_replica_duration_s":0.0,"db_primary_duration_s":0.005,"external_http_count":1,"external_http_duration_s":0.09891590755432844,"cpu_s":0.043101,"mem_objects":11802,"mem_bytes":1741392,"mem_mallocs":3057,"mem_total_bytes":2213472,"pid":3902,"db_duration_s":0.0046,"view_duration_s":0.0,"duration_s":0.42515} {"method":"GET","path":"/v2/stackspin/dependency_proxy/containers/bitnami/mysql/blobs/sha256:1acccb53a4242bb9d155c13eb9225e3b05f26e381a1c8e3269d055a2787b8393","format":"html","controller":"Groups::DependencyProxyForContainersController","action":"blob","status":200,"time":"2022-01-17T12:35:00.925Z","params":[{"key":"group_id","value":"stackspin"},{"key":"image","value":"bitnami/mysql"},{"key":"sha","value":"sha256:1acccb53a4242bb9d155c13eb9225e3b05f26e381a1c8e3269d055a2787b8393"}],"correlation_id":"01FSM0H5PEZKHYP6FB9HART6TJ","meta.user":"maarten","meta.root_namespace":"stackspin","meta.caller_id":"Groups::DependencyProxyForContainersController#blob","meta.remote_ip":"213.108.108.41","meta.feature_category":"dependency_proxy","meta.client_id":"user/4","remote_ip":"213.108.108.41","user_id":4,"username":"maarten","ua":"containerd/v1.5.8-k3s2","queue_duration_s":0.008566,"request_urgency":"default","target_duration_s":1,"redis_calls":1,"redis_duration_s":0.000976,"redis_write_bytes":147,"redis_shared_state_calls":1,"redis_shared_state_duration_s":0.000976,"redis_shared_state_write_bytes":147,"db_count":8,"db_write_count":0,"db_cached_count":0,"db_replica_count":0,"db_primary_count":8,"db_replica_cached_count":0,"db_primary_cached_count":0,"db_replica_wal_count":0,"db_primary_wal_count":0,"db_replica_wal_cached_count":0,"db_primary_wal_cached_count":0,"db_replica_duration_s":0.0,"db_primary_duration_s":0.022,"external_http_count":1,"external_http_duration_s":0.1018092455342412,"cpu_s":0.056702,"mem_objects":11802,"mem_bytes":1741392,"mem_mallocs":3057,"mem_total_bytes":2213472,"pid":4293,"db_duration_s":0.02095,"view_duration_s":0.0,"duration_s":0.48014} {"method":"GET","path":"/v2/stackspin/dependency_proxy/containers/bitnami/mysql/blobs/sha256:23c438466695cca70bffc703fc418f81411a251877d001132bd52dc8fe829484","format":"html","controller":"Groups::DependencyProxyForContainersController","action":"blob","status":200,"time":"2022-01-17T12:35:00.927Z","params":[{"key":"group_id","value":"stackspin"},{"key":"image","value":"bitnami/mysql"},{"key":"sha","value":"sha256:23c438466695cca70bffc703fc418f81411a251877d001132bd52dc8fe829484"}],"correlation_id":"01FSM0H5R9NGRGARGKTDQWD8RK","meta.user":"maarten","meta.root_namespace":"stackspin","meta.caller_id":"Groups::DependencyProxyForContainersController#blob","meta.remote_ip":"213.108.108.41","meta.feature_category":"dependency_proxy","meta.client_id":"user/4","remote_ip":"213.108.108.41","user_id":4,"username":"maarten","ua":"containerd/v1.5.8-k3s2","queue_duration_s":0.00593,"request_urgency":"default","target_duration_s":1,"redis_calls":1,"redis_duration_s":0.000801,"redis_write_bytes":147,"redis_shared_state_calls":1,"redis_shared_state_duration_s":0.000801,"redis_shared_state_write_bytes":147,"db_count":8,"db_write_count":0,"db_cached_count":0,"db_replica_count":0,"db_primary_count":8,"db_replica_cached_count":0,"db_primary_cached_count":0,"db_replica_wal_count":0,"db_primary_wal_count":0,"db_replica_wal_cached_count":0,"db_primary_wal_cached_count":0,"db_replica_duration_s":0.0,"db_primary_duration_s":0.005,"external_http_count":1,"external_http_duration_s":0.10154136177152395,"cpu_s":0.040159,"mem_objects":11803,"mem_bytes":1741392,"mem_mallocs":3057,"mem_total_bytes":2213512,"pid":4050,"db_duration_s":0.00509,"view_duration_s":0.0,"duration_s":0.4278} {"method":"GET","path":"/v2/stackspin/dependency_proxy/containers/bitnami/mysql/blobs/sha256:dc5d2b1813bb5d45582444bd56d7a27a5d6a9bb7bc439ebb5be91f1040c3a782","format":"html","controller":"Groups::DependencyProxyForContainersController","action":"blob","status":200,"time":"2022-01-17T12:35:00.945Z","params":[{"key":"group_id","value":"stackspin"},{"key":"image","value":"bitnami/mysql"},{"key":"sha","value":"sha256:dc5d2b1813bb5d45582444bd56d7a27a5d6a9bb7bc439ebb5be91f1040c3a782"}],"correlation_id":"01FSM0H5QQ2K24M9BMKJBH5NDH","meta.user":"maarten","meta.root_namespace":"stackspin","meta.caller_id":"Groups::DependencyProxyForContainersController#blob","meta.remote_ip":"213.108.108.41","meta.feature_category":"dependency_proxy","meta.client_id":"user/4","remote_ip":"213.108.108.41","user_id":4,"username":"maarten","ua":"containerd/v1.5.8-k3s2","queue_duration_s":0.008182,"request_urgency":"default","target_duration_s":1,"redis_calls":1,"redis_duration_s":0.00447,"redis_write_bytes":147,"redis_shared_state_calls":1,"redis_shared_state_duration_s":0.00447,"redis_shared_state_write_bytes":147,"db_count":8,"db_write_count":0,"db_cached_count":0,"db_replica_count":0,"db_primary_count":8,"db_replica_cached_count":0,"db_primary_cached_count":0,"db_replica_wal_count":0,"db_primary_wal_count":0,"db_replica_wal_cached_count":0,"db_primary_wal_cached_count":0,"db_replica_duration_s":0.0,"db_primary_duration_s":0.015,"external_http_count":1,"external_http_duration_s":0.10204044077545404,"cpu_s":0.045358,"mem_objects":11802,"mem_bytes":1757776,"mem_mallocs":3058,"mem_total_bytes":2229856,"pid":4293,"db_duration_s":0.01444,"view_duration_s":0.0,"duration_s":0.45459} {"method":"POST","path":"/v2/stackspin/dependency_proxy/containers/bitnami/mysql/blobs/sha256:f1cabe50ef568332f8933f4077111f0be1344572d24f97d24c49d15d507d880b/upload/authorize","format":"html","controller":"Groups::DependencyProxyForContainersController","action":"authorize_upload_blob","status":200,"time":"2022-01-17T12:35:01.291Z","params":[{"key":"group_id","value":"stackspin"},{"key":"image","value":"bitnami/mysql"},{"key":"sha","value":"sha256:f1cabe50ef568332f8933f4077111f0be1344572d24f97d24c49d15d507d880b"}],"correlation_id":"01FSM0H5KCM3ZDZXPA3YYX261Q","meta.user":"maarten","meta.root_namespace":"stackspin","meta.caller_id":"Groups::DependencyProxyForContainersController#authorize_upload_blob","meta.remote_ip":"213.108.108.41","meta.feature_category":"dependency_proxy","meta.client_id":"user/4","remote_ip":"213.108.108.41","user_id":4,"username":"maarten","ua":"containerd/v1.5.8-k3s2","request_urgency":"default","target_duration_s":1,"cf_ray":"6cef9efc491f4c26-AMS","redis_calls":1,"redis_duration_s":0.003349,"redis_write_bytes":147,"redis_shared_state_calls":1,"redis_shared_state_duration_s":0.003349,"redis_shared_state_write_bytes":147,"db_count":7,"db_write_count":0,"db_cached_count":0,"db_replica_count":0,"db_primary_count":7,"db_replica_cached_count":0,"db_primary_cached_count":0,"db_replica_wal_count":0,"db_primary_wal_count":0,"db_replica_wal_cached_count":0,"db_primary_wal_cached_count":0,"db_replica_duration_s":0.0,"db_primary_duration_s":0.006,"cpu_s":0.042668,"mem_objects":10472,"mem_bytes":1622192,"mem_mallocs":3011,"mem_total_bytes":2041072,"pid":3740,"db_duration_s":0.00586,"view_duration_s":0.00021,"duration_s":0.05125} {"method":"POST","path":"/v2/stackspin/dependency_proxy/containers/bitnami/mysql/blobs/sha256:41b479190c3ad34105037f24369ce93173b7c33bd1e588ccc37bf7ab87bcf8f2/upload/authorize","format":"html","controller":"Groups::DependencyProxyForContainersController","action":"authorize_upload_blob","status":200,"time":"2022-01-17T12:35:01.347Z","params":[{"key":"group_id","value":"stackspin"},{"key":"image","value":"bitnami/mysql"},{"key":"sha","value":"sha256:41b479190c3ad34105037f24369ce93173b7c33bd1e588ccc37bf7ab87bcf8f2"}],"correlation_id":"01FSM0H5M1G5XEZQS68WAZ8FWJ","meta.user":"maarten","meta.root_namespace":"stackspin","meta.caller_id":"Groups::DependencyProxyForContainersController#authorize_upload_blob","meta.remote_ip":"213.108.108.41","meta.feature_category":"dependency_proxy","meta.client_id":"user/4","remote_ip":"213.108.108.41","user_id":4,"username":"maarten","ua":"containerd/v1.5.8-k3s2","request_urgency":"default","target_duration_s":1,"cf_ray":"6cef9efc89ff4c26-AMS","redis_calls":1,"redis_duration_s":0.000295,"redis_write_bytes":147,"redis_shared_state_calls":1,"redis_shared_state_duration_s":0.000295,"redis_shared_state_write_bytes":147,"db_count":7,"db_write_count":0,"db_cached_count":0,"db_replica_count":0,"db_primary_count":7,"db_replica_cached_count":0,"db_primary_cached_count":0,"db_replica_wal_count":0,"db_primary_wal_count":0,"db_replica_wal_cached_count":0,"db_primary_wal_cached_count":0,"db_replica_duration_s":0.0,"db_primary_duration_s":0.022,"cpu_s":0.042637,"mem_objects":10430,"mem_bytes":1622640,"mem_mallocs":3013,"mem_total_bytes":2039840,"pid":4293,"db_duration_s":0.02168,"view_duration_s":0.00032,"duration_s":0.07461} {"method":"POST","path":"/v2/stackspin/dependency_proxy/containers/bitnami/mysql/blobs/sha256:4de6e9ef77a36fce6ada7437c84d656ad57346d62917604176fca5b25fa4056f/upload/authorize","format":"html","controller":"Groups::DependencyProxyForContainersController","action":"authorize_upload_blob","status":200,"time":"2022-01-17T12:35:01.382Z","params":[{"key":"group_id","value":"stackspin"},{"key":"image","value":"bitnami/mysql"},{"key":"sha","value":"sha256:4de6e9ef77a36fce6ada7437c84d656ad57346d62917604176fca5b25fa4056f"}],"correlation_id":"01FSM0H5N1N9MR04ETJ9ZT74SH","meta.user":"maarten","meta.root_namespace":"stackspin","meta.caller_id":"Groups::DependencyProxyForContainersController#authorize_upload_blob","meta.remote_ip":"213.108.108.41","meta.feature_category":"dependency_proxy","meta.client_id":"user/4","remote_ip":"213.108.108.41","user_id":4,"username":"maarten","ua":"containerd/v1.5.8-k3s2","request_urgency":"default","target_duration_s":1,"cf_ray":"6cef9efc9a444c26-AMS","redis_calls":1,"redis_duration_s":0.000347,"redis_write_bytes":147,"redis_shared_state_calls":1,"redis_shared_state_duration_s":0.000347,"redis_shared_state_write_bytes":147,"db_count":7,"db_write_count":0,"db_cached_count":0,"db_replica_count":0,"db_primary_count":7,"db_replica_cached_count":0,"db_primary_cached_count":0,"db_replica_wal_count":0,"db_primary_wal_count":0,"db_replica_wal_cached_count":0,"db_primary_wal_cached_count":0,"db_replica_duration_s":0.0,"db_primary_duration_s":0.017,"cpu_s":0.054542,"mem_objects":10415,"mem_bytes":1601728,"mem_mallocs":3010,"mem_total_bytes":2018328,"pid":4050,"db_duration_s":0.01621,"view_duration_s":0.0005,"duration_s":0.102} {"method":"POST","path":"/v2/stackspin/dependency_proxy/containers/bitnami/mysql/blobs/sha256:19d690b633e33fab8dcdb616ddc662bb126a241b36d8875054c08a84ad737fc7/upload/authorize","format":"html","controller":"Groups::DependencyProxyForContainersController","action":"authorize_upload_blob","status":200,"time":"2022-01-17T12:35:01.387Z","params":[{"key":"group_id","value":"stackspin"},{"key":"image","value":"bitnami/mysql"},{"key":"sha","value":"sha256:19d690b633e33fab8dcdb616ddc662bb126a241b36d8875054c08a84ad737fc7"}],"correlation_id":"01FSM0H5NZCAMCMQ6DF4MC94DM","meta.user":"maarten","meta.root_namespace":"stackspin","meta.caller_id":"Groups::DependencyProxyForContainersController#authorize_upload_blob","meta.remote_ip":"213.108.108.41","meta.feature_category":"dependency_proxy","meta.client_id":"user/4","remote_ip":"213.108.108.41","user_id":4,"username":"maarten","ua":"containerd/v1.5.8-k3s2","request_urgency":"default","target_duration_s":1,"cf_ray":"6cef9efcead54c26-AMS","redis_calls":1,"redis_duration_s":0.000455,"redis_write_bytes":147,"redis_shared_state_calls":1,"redis_shared_state_duration_s":0.000455,"redis_shared_state_write_bytes":147,"db_count":7,"db_write_count":0,"db_cached_count":0,"db_replica_count":0,"db_primary_count":7,"db_replica_cached_count":0,"db_primary_cached_count":0,"db_replica_wal_count":0,"db_primary_wal_count":0,"db_replica_wal_cached_count":0,"db_primary_wal_cached_count":0,"db_replica_duration_s":0.0,"db_primary_duration_s":0.026,"cpu_s":0.0377,"mem_objects":10416,"mem_bytes":1622208,"mem_mallocs":3011,"mem_total_bytes":2038848,"pid":3740,"db_duration_s":0.0254,"view_duration_s":0.00039,"duration_s":0.07735} {"method":"POST","path":"/v2/stackspin/dependency_proxy/containers/bitnami/mysql/blobs/sha256:81bf975e185f02351023f2bf780826a91a183164aa1c574c51fc541cc97e3972/upload/authorize","format":"html","controller":"Groups::DependencyProxyForContainersController","action":"authorize_upload_blob","status":200,"time":"2022-01-17T12:35:01.466Z","params":[{"key":"group_id","value":"stackspin"},{"key":"image","value":"bitnami/mysql"},{"key":"sha","value":"sha256:81bf975e185f02351023f2bf780826a91a183164aa1c574c51fc541cc97e3972"}],"correlation_id":"01FSM0H5NGKYZTXMXW3MM4J5CW","meta.user":"maarten","meta.root_namespace":"stackspin","meta.caller_id":"Groups::DependencyProxyForContainersController#authorize_upload_blob","meta.remote_ip":"213.108.108.41","meta.feature_category":"dependency_proxy","meta.client_id":"user/4","remote_ip":"213.108.108.41","user_id":4,"username":"maarten","ua":"containerd/v1.5.8-k3s2","request_urgency":"default","target_duration_s":1,"cf_ray":"6cef9efceada4c26-AMS","redis_calls":1,"redis_duration_s":0.01426,"redis_write_bytes":147,"redis_shared_state_calls":1,"redis_shared_state_duration_s":0.01426,"redis_shared_state_write_bytes":147,"db_count":7,"db_write_count":0,"db_cached_count":0,"db_replica_count":0,"db_primary_count":7,"db_replica_cached_count":0,"db_primary_cached_count":0,"db_replica_wal_count":0,"db_primary_wal_count":0,"db_replica_wal_cached_count":0,"db_primary_wal_cached_count":0,"db_replica_duration_s":0.0,"db_primary_duration_s":0.049,"cpu_s":0.045648,"mem_objects":10416,"mem_bytes":1622208,"mem_mallocs":3011,"mem_total_bytes":2038848,"pid":3740,"db_duration_s":0.04918,"view_duration_s":0.00019,"duration_s":0.12554} {"method":"POST","path":"/v2/stackspin/dependency_proxy/containers/bitnami/mysql/blobs/sha256:73729094e3568d281b4e0027ba04600149fb86082ad1e0c8353c26452e025419/upload/authorize","format":"html","controller":"Groups::DependencyProxyForContainersController","action":"authorize_upload_blob","status":200,"time":"2022-01-17T12:35:01.465Z","params":[{"key":"group_id","value":"stackspin"},{"key":"image","value":"bitnami/mysql"},{"key":"sha","value":"sha256:73729094e3568d281b4e0027ba04600149fb86082ad1e0c8353c26452e025419"}],"correlation_id":"01FSM0H5MJGQX2DSDXB1TSEHJD","meta.user":"maarten","meta.root_namespace":"stackspin","meta.caller_id":"Groups::DependencyProxyForContainersController#authorize_upload_blob","meta.remote_ip":"213.108.108.41","meta.feature_category":"dependency_proxy","meta.client_id":"user/4","remote_ip":"213.108.108.41","user_id":4,"username":"maarten","ua":"containerd/v1.5.8-k3s2","request_urgency":"default","target_duration_s":1,"cf_ray":"6cef9efcca8b4c26-AMS","redis_calls":4,"redis_duration_s":0.0084,"redis_read_bytes":609,"redis_write_bytes":318,"redis_cache_calls":3,"redis_cache_duration_s":0.007005,"redis_cache_read_bytes":609,"redis_cache_write_bytes":171,"redis_shared_state_calls":1,"redis_shared_state_duration_s":0.001395,"redis_shared_state_write_bytes":147,"db_count":7,"db_write_count":0,"db_cached_count":0,"db_replica_count":0,"db_primary_count":7,"db_replica_cached_count":0,"db_primary_cached_count":0,"db_replica_wal_count":0,"db_primary_wal_count":0,"db_replica_wal_cached_count":0,"db_primary_wal_cached_count":0,"db_replica_duration_s":0.0,"db_primary_duration_s":0.075,"cpu_s":0.055242,"mem_objects":11064,"mem_bytes":1713760,"mem_mallocs":3266,"mem_total_bytes":2156320,"pid":3902,"db_duration_s":0.07001,"view_duration_s":0.00031,"duration_s":0.158} {"method":"POST","path":"/v2/stackspin/dependency_proxy/containers/bitnami/mysql/blobs/sha256:28a756701188d2ddee0911638724dff7664ca8099160da8414e16b79c203f746/upload/authorize","format":"html","controller":"Groups::DependencyProxyForContainersController","action":"authorize_upload_blob","status":200,"time":"2022-01-17T12:35:01.527Z","params":[{"key":"group_id","value":"stackspin"},{"key":"image","value":"bitnami/mysql"},{"key":"sha","value":"sha256:28a756701188d2ddee0911638724dff7664ca8099160da8414e16b79c203f746"}],"correlation_id":"01FSM0H5Q44HN17BC5R254287Q","meta.user":"maarten","meta.root_namespace":"stackspin","meta.caller_id":"Groups::DependencyProxyForContainersController#authorize_upload_blob","meta.remote_ip":"213.108.108.41","meta.feature_category":"dependency_proxy","meta.client_id":"user/4","remote_ip":"213.108.108.41","user_id":4,"username":"maarten","ua":"containerd/v1.5.8-k3s2","request_urgency":"default","target_duration_s":1,"cf_ray":"6cef9efcfb044c26-AMS","redis_calls":1,"redis_duration_s":0.000353,"redis_write_bytes":147,"redis_shared_state_calls":1,"redis_shared_state_duration_s":0.000353,"redis_shared_state_write_bytes":147,"db_count":7,"db_write_count":0,"db_cached_count":0,"db_replica_count":0,"db_primary_count":7,"db_replica_cached_count":0,"db_primary_cached_count":0,"db_replica_wal_count":0,"db_primary_wal_count":0,"db_replica_wal_cached_count":0,"db_primary_wal_cached_count":0,"db_replica_duration_s":0.0,"db_primary_duration_s":0.094,"cpu_s":0.045173,"mem_objects":10430,"mem_bytes":1622640,"mem_mallocs":3013,"mem_total_bytes":2039840,"pid":4050,"db_duration_s":0.09348,"view_duration_s":0.00031,"duration_s":0.19074} {"method":"POST","path":"/v2/stackspin/dependency_proxy/containers/bitnami/mysql/blobs/sha256:1acccb53a4242bb9d155c13eb9225e3b05f26e381a1c8e3269d055a2787b8393/upload/authorize","format":"html","controller":"Groups::DependencyProxyForContainersController","action":"authorize_upload_blob","status":200,"time":"2022-01-17T12:35:01.623Z","params":[{"key":"group_id","value":"stackspin"},{"key":"image","value":"bitnami/mysql"},{"key":"sha","value":"sha256:1acccb53a4242bb9d155c13eb9225e3b05f26e381a1c8e3269d055a2787b8393"}],"correlation_id":"01FSM0H5PEZKHYP6FB9HART6TJ","meta.user":"maarten","meta.root_namespace":"stackspin","meta.caller_id":"Groups::DependencyProxyForContainersController#authorize_upload_blob","meta.remote_ip":"213.108.108.41","meta.feature_category":"dependency_proxy","meta.client_id":"user/4","remote_ip":"213.108.108.41","user_id":4,"username":"maarten","ua":"containerd/v1.5.8-k3s2","request_urgency":"default","target_duration_s":1,"cf_ray":"6cef9efd2b4e4c26-AMS","redis_calls":1,"redis_duration_s":0.000816,"redis_write_bytes":147,"redis_shared_state_calls":1,"redis_shared_state_duration_s":0.000816,"redis_shared_state_write_bytes":147,"db_count":7,"db_write_count":0,"db_cached_count":0,"db_replica_count":0,"db_primary_count":7,"db_replica_cached_count":0,"db_primary_cached_count":0,"db_replica_wal_count":0,"db_primary_wal_count":0,"db_replica_wal_cached_count":0,"db_primary_wal_cached_count":0,"db_replica_duration_s":0.0,"db_primary_duration_s":0.123,"cpu_s":0.049841,"mem_objects":10417,"mem_bytes":1622208,"mem_mallocs":3011,"mem_total_bytes":2038888,"pid":4293,"db_duration_s":0.12373,"view_duration_s":0.00026,"duration_s":0.25381} {"method":"POST","path":"/v2/stackspin/dependency_proxy/containers/bitnami/mysql/blobs/sha256:dc5d2b1813bb5d45582444bd56d7a27a5d6a9bb7bc439ebb5be91f1040c3a782/upload/authorize","format":"html","controller":"Groups::DependencyProxyForContainersController","action":"authorize_upload_blob","status":200,"time":"2022-01-17T12:35:01.653Z","params":[{"key":"group_id","value":"stackspin"},{"key":"image","value":"bitnami/mysql"},{"key":"sha","value":"sha256:dc5d2b1813bb5d45582444bd56d7a27a5d6a9bb7bc439ebb5be91f1040c3a782"}],"correlation_id":"01FSM0H5QQ2K24M9BMKJBH5NDH","meta.user":"maarten","meta.root_namespace":"stackspin","meta.caller_id":"Groups::DependencyProxyForContainersController#authorize_upload_blob","meta.remote_ip":"213.108.108.41","meta.feature_category":"dependency_proxy","meta.client_id":"user/4","remote_ip":"213.108.108.41","user_id":4,"username":"maarten","ua":"containerd/v1.5.8-k3s2","request_urgency":"default","target_duration_s":1,"cf_ray":"6cef9efd3b8b4c26-AMS","redis_calls":1,"redis_duration_s":0.007014,"redis_write_bytes":147,"redis_shared_state_calls":1,"redis_shared_state_duration_s":0.007014,"redis_shared_state_write_bytes":147,"db_count":7,"db_write_count":0,"db_cached_count":0,"db_replica_count":0,"db_primary_count":7,"db_replica_cached_count":0,"db_primary_cached_count":0,"db_replica_wal_count":0,"db_primary_wal_count":0,"db_replica_wal_cached_count":0,"db_primary_wal_cached_count":0,"db_replica_duration_s":0.0,"db_primary_duration_s":0.109,"cpu_s":0.052842,"mem_objects":10416,"mem_bytes":1622208,"mem_mallocs":3011,"mem_total_bytes":2038848,"pid":3902,"db_duration_s":0.10748,"view_duration_s":0.00025,"duration_s":0.21024} {"method":"POST","path":"/v2/stackspin/dependency_proxy/containers/bitnami/mysql/blobs/sha256:19d690b633e33fab8dcdb616ddc662bb126a241b36d8875054c08a84ad737fc7/upload","format":"html","controller":"Groups::DependencyProxyForContainersController","action":"upload_blob","status":200,"time":"2022-01-17T12:35:01.693Z","params":[{"key":"file.gitlab-workhorse-upload","value":"eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJ1cGxvYWQiOnsibWQ1IjoiNmZhYjk4NjU4MzJmNWU3MzEyZjM1MDZlYzIzMzYzYTIiLCJuYW1lIjoiIiwicGF0aCI6Ii92YXIvb3B0L2dpdGxhYi9naXRsYWItcmFpbHMvc2hhcmVkL2RlcGVuZGVuY3lfcHJveHkvdG1wL3VwbG9hZHMvMDgzMzUzMzkyIiwicmVtb3RlX2lkIjoiIiwicmVtb3RlX3VybCI6IiIsInNoYTEiOiI3ZTFlNjRjM2NlOGRiNjYwZDQ4M2M2YmJjYjJkNzNjN2E0ODExYjZmIiwic2hhMjU2IjoiMTlkNjkwYjYzM2UzM2ZhYjhkY2RiNjE2ZGRjNjYyYmIxMjZhMjQxYjM2ZDg4NzUwNTRjMDhhODRhZDczN2ZjNyIsInNoYTUxMiI6ImU5ZWM3ZWFkZjYxZjE2MTE1NzJjZWY2ZTI3NTdjOWRjOWE1N2IxMjllMDYyNGE2ZDgwMmNlMDU1ZDI1MGY5NzZhNTgzYmMzYzAzZWE5ZTdmNWIyYzg5MzQ2ZjZmMzczNzVlY2I0YjQ1YmM4OWU0YjQyYzNlODljZDIwOWIzNjRiIiwic2l6ZSI6IjEyMCIsInVwbG9hZF9kdXJhdGlvbiI6IjAuMDAwMjcxNjEifSwiaXNzIjoiZ2l0bGFiLXdvcmtob3JzZSJ9.QYuDap9s3XLyiAeHuSAzwi_CAy_i_EWZJ2fTiIxLS_g"},{"key":"file.md5","value":"6fab9865832f5e7312f3506ec23363a2"},{"key":"file.name","value":""},{"key":"file.path","value":"/var/opt/gitlab/gitlab-rails/shared/dependency_proxy/tmp/uploads/083353392"},{"key":"file.remote_id","value":""},{"key":"file.remote_url","value":""},{"key":"file.sha1","value":"7e1e64c3ce8db660d483c6bbcb2d73c7a4811b6f"},{"key":"file.sha256","value":"19d690b633e33fab8dcdb616ddc662bb126a241b36d8875054c08a84ad737fc7"},{"key":"file.sha512","value":"e9ec7eadf61f1611572cef6e2757c9dc9a57b129e0624a6d802ce055d250f976a583bc3c03ea9e7f5b2c89346f6f37375ecb4b45bc89e4b42c3e89cd209b364b"},{"key":"file.size","value":"120"},{"key":"file.upload_duration","value":"0.00027161"},{"key":"file","value":"#"},{"key":"group_id","value":"stackspin"},{"key":"image","value":"bitnami/mysql"},{"key":"sha","value":"sha256:19d690b633e33fab8dcdb616ddc662bb126a241b36d8875054c08a84ad737fc7"}],"correlation_id":"01FSM0H5NZCAMCMQ6DF4MC94DM","meta.user":"maarten","meta.root_namespace":"stackspin","meta.caller_id":"Groups::DependencyProxyForContainersController#upload_blob","meta.remote_ip":"213.108.108.41","meta.feature_category":"dependency_proxy","meta.client_id":"user/4","remote_ip":"213.108.108.41","user_id":4,"username":"maarten","ua":"containerd/v1.5.8-k3s2","queue_duration_s":0.03725,"request_urgency":"default","target_duration_s":1,"cf_ray":"6cef9efcead54c26-AMS","redis_calls":2,"redis_duration_s":0.012921,"redis_read_bytes":203,"redis_write_bytes":204,"redis_cache_calls":1,"redis_cache_duration_s":0.009959,"redis_cache_read_bytes":203,"redis_cache_write_bytes":57,"redis_shared_state_calls":1,"redis_shared_state_duration_s":0.002962,"redis_shared_state_write_bytes":147,"db_count":12,"db_write_count":4,"db_cached_count":0,"db_replica_count":0,"db_primary_count":12,"db_replica_cached_count":0,"db_primary_cached_count":0,"db_replica_wal_count":0,"db_primary_wal_count":0,"db_replica_wal_cached_count":0,"db_primary_wal_cached_count":0,"db_replica_duration_s":0.0,"db_primary_duration_s":0.092,"cpu_s":0.062752,"mem_objects":13131,"mem_bytes":2214240,"mem_mallocs":3668,"mem_total_bytes":2739480,"pid":3740,"uploaded_file_upload_duration_s":0.00027161,"uploaded_file_size_bytes":120,"db_duration_s":0.09079,"view_duration_s":0.0,"duration_s":0.22874} {"method":"POST","path":"/v2/stackspin/dependency_proxy/containers/bitnami/mysql/blobs/sha256:23c438466695cca70bffc703fc418f81411a251877d001132bd52dc8fe829484/upload/authorize","format":"html","controller":"Groups::DependencyProxyForContainersController","action":"authorize_upload_blob","status":200,"time":"2022-01-17T12:35:01.700Z","params":[{"key":"group_id","value":"stackspin"},{"key":"image","value":"bitnami/mysql"},{"key":"sha","value":"sha256:23c438466695cca70bffc703fc418f81411a251877d001132bd52dc8fe829484"}],"correlation_id":"01FSM0H5R9NGRGARGKTDQWD8RK","meta.user":"maarten","meta.root_namespace":"stackspin","meta.caller_id":"Groups::DependencyProxyForContainersController#authorize_upload_blob","meta.remote_ip":"213.108.108.41","meta.feature_category":"dependency_proxy","meta.client_id":"user/4","remote_ip":"213.108.108.41","user_id":4,"username":"maarten","ua":"containerd/v1.5.8-k3s2","request_urgency":"default","target_duration_s":1,"cf_ray":"6cef9efd3b7c4c26-AMS","redis_calls":1,"redis_duration_s":0.009246,"redis_write_bytes":147,"redis_shared_state_calls":1,"redis_shared_state_duration_s":0.009246,"redis_shared_state_write_bytes":147,"db_count":7,"db_write_count":0,"db_cached_count":0,"db_replica_count":0,"db_primary_count":7,"db_replica_cached_count":0,"db_primary_cached_count":0,"db_replica_wal_count":0,"db_primary_wal_count":0,"db_replica_wal_cached_count":0,"db_primary_wal_cached_count":0,"db_replica_duration_s":0.0,"db_primary_duration_s":0.087,"cpu_s":0.053715,"mem_objects":10471,"mem_bytes":1622208,"mem_mallocs":3011,"mem_total_bytes":2041048,"pid":4293,"db_duration_s":0.08645,"view_duration_s":0.00198,"duration_s":0.25885} {"method":"POST","path":"/v2/stackspin/dependency_proxy/containers/bitnami/mysql/blobs/sha256:f1cabe50ef568332f8933f4077111f0be1344572d24f97d24c49d15d507d880b/upload","format":"html","controller":"Groups::DependencyProxyForContainersController","action":"upload_blob","status":200,"time":"2022-01-17T12:35:01.716Z","params":[{"key":"file.gitlab-workhorse-upload","value":"eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJ1cGxvYWQiOnsibWQ1IjoiMTUyYmMwZTNiZWQwMDc3ODNkYjQxMWE1YzZjNWU5ZGMiLCJuYW1lIjoiIiwicGF0aCI6Ii92YXIvb3B0L2dpdGxhYi9naXRsYWItcmFpbHMvc2hhcmVkL2RlcGVuZGVuY3lfcHJveHkvdG1wL3VwbG9hZHMvMjM1NjE3Mzg3IiwicmVtb3RlX2lkIjoiIiwicmVtb3RlX3VybCI6IiIsInNoYTEiOiIxODY1MjVkMGQ1M2I0ZjUwZDVlMDQ1ODcyMzViZjk1YzA5YjllYmQwIiwic2hhMjU2IjoiZjFjYWJlNTBlZjU2ODMzMmY4OTMzZjQwNzcxMTFmMGJlMTM0NDU3MmQyNGY5N2QyNGM0OWQxNWQ1MDdkODgwYiIsInNoYTUxMiI6IjBkMzE1OWNhN2FiZjMwMTc4MWUxMzMyMDU3MTA0NGIwZjIyOTk3ZDNjNmMwNTNiYTA5ODVmMDAxMmI3MTg2MDVkZjI4NGZjYzk3MTU0YTFlZDlmNGNmMjZlNWRmMWY0OWExOGVhZWE0MjRmMjk2YzdhYTNiODBjY2MyMTYzZTY0Iiwic2l6ZSI6IjYwMiIsInVwbG9hZF9kdXJhdGlvbiI6IjAuMDAxOTEzNjYifSwiaXNzIjoiZ2l0bGFiLXdvcmtob3JzZSJ9.nDbALWVhCrPG9B3lWhjnKVCZlILicmUJZUGShcrDZ9U"},{"key":"file.md5","value":"152bc0e3bed007783db411a5c6c5e9dc"},{"key":"file.name","value":""},{"key":"file.path","value":"/var/opt/gitlab/gitlab-rails/shared/dependency_proxy/tmp/uploads/235617387"},{"key":"file.remote_id","value":""},{"key":"file.remote_url","value":""},{"key":"file.sha1","value":"186525d0d53b4f50d5e04587235bf95c09b9ebd0"},{"key":"file.sha256","value":"f1cabe50ef568332f8933f4077111f0be1344572d24f97d24c49d15d507d880b"},{"key":"file.sha512","value":"0d3159ca7abf301781e13320571044b0f22997d3c6c053ba0985f0012b718605df284fcc97154a1ed9f4cf26e5df1f49a18eaea424f296c7aa3b80ccc2163e64"},{"key":"file.size","value":"602"},{"key":"file.upload_duration","value":"0.00191366"},{"key":"file","value":"#"},{"key":"group_id","value":"stackspin"},{"key":"image","value":"bitnami/mysql"},{"key":"sha","value":"sha256:f1cabe50ef568332f8933f4077111f0be1344572d24f97d24c49d15d507d880b"}],"correlation_id":"01FSM0H5KCM3ZDZXPA3YYX261Q","meta.user":"maarten","meta.root_namespace":"stackspin","meta.caller_id":"Groups::DependencyProxyForContainersController#upload_blob","meta.remote_ip":"213.108.108.41","meta.feature_category":"dependency_proxy","meta.client_id":"user/4","remote_ip":"213.108.108.41","user_id":4,"username":"maarten","ua":"containerd/v1.5.8-k3s2","queue_duration_s":0.017997,"request_urgency":"default","target_duration_s":1,"cf_ray":"6cef9efc491f4c26-AMS","redis_calls":2,"redis_duration_s":0.000995,"redis_read_bytes":203,"redis_write_bytes":204,"redis_cache_calls":1,"redis_cache_duration_s":0.000483,"redis_cache_read_bytes":203,"redis_cache_write_bytes":57,"redis_shared_state_calls":1,"redis_shared_state_duration_s":0.000512,"redis_shared_state_write_bytes":147,"db_count":12,"db_write_count":4,"db_cached_count":0,"db_replica_count":0,"db_primary_count":12,"db_replica_cached_count":0,"db_primary_cached_count":0,"db_replica_wal_count":0,"db_primary_wal_count":0,"db_replica_wal_cached_count":0,"db_primary_wal_cached_count":0,"db_replica_duration_s":0.0,"db_primary_duration_s":0.171,"cpu_s":0.09921,"mem_objects":13157,"mem_bytes":2173136,"mem_mallocs":3666,"mem_total_bytes":2699416,"pid":4293,"uploaded_file_upload_duration_s":0.00191366,"uploaded_file_size_bytes":602,"db_duration_s":0.16918,"view_duration_s":0.0,"duration_s":0.37518} {"method":"POST","path":"/v2/stackspin/dependency_proxy/containers/bitnami/mysql/blobs/sha256:28a756701188d2ddee0911638724dff7664ca8099160da8414e16b79c203f746/upload","format":"html","controller":"Groups::DependencyProxyForContainersController","action":"upload_blob","status":200,"time":"2022-01-17T12:35:01.787Z","params":[{"key":"file.gitlab-workhorse-upload","value":"eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJ1cGxvYWQiOnsibWQ1IjoiMmEwMDQ4YzJiOGM5YmVkNGRiMDExN2I3NGY4ZDdiNTUiLCJuYW1lIjoiIiwicGF0aCI6Ii92YXIvb3B0L2dpdGxhYi9naXRsYWItcmFpbHMvc2hhcmVkL2RlcGVuZGVuY3lfcHJveHkvdG1wL3VwbG9hZHMvMjcyMzIyMDA5IiwicmVtb3RlX2lkIjoiIiwicmVtb3RlX3VybCI6IiIsInNoYTEiOiJhNzIwYzg1OThiMDk5NDQxZTdkNDYyYTgwYWQ3ZGY3Yjc0YzdkMjc2Iiwic2hhMjU2IjoiMjhhNzU2NzAxMTg4ZDJkZGVlMDkxMTYzODcyNGRmZjc2NjRjYTgwOTkxNjBkYTg0MTRlMTZiNzljMjAzZjc0NiIsInNoYTUxMiI6IjBjNzAxZDQ4M2ZjMjMzNTkwNjIzYTgyNzNmYTZkNzJhMTk2N2Q1ODI3ZWFiMWEyNDc2OWM0ZGMzYzFhYTY2ZDk0MGRkNjZjM2U5MmMzNzg5MTkyNTczNWY5MDA1ZjUzNmQ0NjI2NGE1ODRmNDgzODk2Mzk4NjQwMjllNjY5Nzk3Iiwic2l6ZSI6IjI2NSIsInVwbG9hZF9kdXJhdGlvbiI6IjAuMDAwNTE0NDk3In0sImlzcyI6ImdpdGxhYi13b3JraG9yc2UifQ.pLDNkgRnG1549TcT4SpPMlOHusD5WKkS-8-_9C6NiyE"},{"key":"file.md5","value":"2a0048c2b8c9bed4db0117b74f8d7b55"},{"key":"file.name","value":""},{"key":"file.path","value":"/var/opt/gitlab/gitlab-rails/shared/dependency_proxy/tmp/uploads/272322009"},{"key":"file.remote_id","value":""},{"key":"file.remote_url","value":""},{"key":"file.sha1","value":"a720c8598b099441e7d462a80ad7df7b74c7d276"},{"key":"file.sha256","value":"28a756701188d2ddee0911638724dff7664ca8099160da8414e16b79c203f746"},{"key":"file.sha512","value":"0c701d483fc233590623a8273fa6d72a1967d5827eab1a24769c4dc3c1aa66d940dd66c3e92c37891925735f9005f536d46264a584f48389639864029e669797"},{"key":"file.size","value":"265"},{"key":"file.upload_duration","value":"0.000514497"},{"key":"file","value":"#"},{"key":"group_id","value":"stackspin"},{"key":"image","value":"bitnami/mysql"},{"key":"sha","value":"sha256:28a756701188d2ddee0911638724dff7664ca8099160da8414e16b79c203f746"}],"correlation_id":"01FSM0H5Q44HN17BC5R254287Q","meta.user":"maarten","meta.root_namespace":"stackspin","meta.caller_id":"Groups::DependencyProxyForContainersController#upload_blob","meta.remote_ip":"213.108.108.41","meta.feature_category":"dependency_proxy","meta.client_id":"user/4","remote_ip":"213.108.108.41","user_id":4,"username":"maarten","ua":"containerd/v1.5.8-k3s2","queue_duration_s":0.03477,"request_urgency":"default","target_duration_s":1,"cf_ray":"6cef9efcfb044c26-AMS","redis_calls":1,"redis_duration_s":0.007215,"redis_write_bytes":147,"redis_shared_state_calls":1,"redis_shared_state_duration_s":0.007215,"redis_shared_state_write_bytes":147,"db_count":12,"db_write_count":4,"db_cached_count":0,"db_replica_count":0,"db_primary_count":12,"db_replica_cached_count":0,"db_primary_cached_count":0,"db_replica_wal_count":0,"db_primary_wal_count":0,"db_replica_wal_cached_count":0,"db_primary_wal_cached_count":0,"db_replica_duration_s":0.0,"db_primary_duration_s":0.031,"cpu_s":0.083056,"mem_objects":12913,"mem_bytes":2142624,"mem_mallocs":3579,"mem_total_bytes":2659144,"pid":3902,"uploaded_file_upload_duration_s":0.000514497,"uploaded_file_size_bytes":265,"db_duration_s":0.03009,"view_duration_s":0.0,"duration_s":0.1373} {"method":"POST","path":"/v2/stackspin/dependency_proxy/containers/bitnami/mysql/blobs/sha256:23c438466695cca70bffc703fc418f81411a251877d001132bd52dc8fe829484/upload","format":"html","controller":"Groups::DependencyProxyForContainersController","action":"upload_blob","status":200,"time":"2022-01-17T12:35:01.826Z","params":[{"key":"file.gitlab-workhorse-upload","value":"eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJ1cGxvYWQiOnsibWQ1IjoiZjlkM2U2MjkwYzU1YWRhNmUyNmU3M2Q4ZjNjZDFkNzciLCJuYW1lIjoiIiwicGF0aCI6Ii92YXIvb3B0L2dpdGxhYi9naXRsYWItcmFpbHMvc2hhcmVkL2RlcGVuZGVuY3lfcHJveHkvdG1wL3VwbG9hZHMvNDQyNzMwMDM4IiwicmVtb3RlX2lkIjoiIiwicmVtb3RlX3VybCI6IiIsInNoYTEiOiIzNjBmMDM4MmRjNGU5YTc4MDY5MGI2NmU3YTQ4OGU5MjE5OWVlNzAyIiwic2hhMjU2IjoiMjNjNDM4NDY2Njk1Y2NhNzBiZmZjNzAzZmM0MThmODE0MTFhMjUxODc3ZDAwMTEzMmJkNTJkYzhmZTgyOTQ4NCIsInNoYTUxMiI6ImEyNTgyNWMxY2M1MzA1NzlhMTYyZTg2N2Q5YmMyOGI1ZDhjMTI1ODEyMmIxOTMwNTUyZDM5OTdkZTY1NGQ3ODk5OTNmNmE3MGZlNzJkYTlmZWM1ZWI0OWNkOGJhNTdlMzRlMGIyNGNiYTllMzliY2VlOTlkN2YzOWM1N2VjN2JiIiwic2l6ZSI6IjExNSIsInVwbG9hZF9kdXJhdGlvbiI6IjAuMDAwMzU1MzMxIn0sImlzcyI6ImdpdGxhYi13b3JraG9yc2UifQ.RqQ0ctRxvV2GisvaKipJtULfjmT2aO6cNrzcXIUrKRQ"},{"key":"file.md5","value":"f9d3e6290c55ada6e26e73d8f3cd1d77"},{"key":"file.name","value":""},{"key":"file.path","value":"/var/opt/gitlab/gitlab-rails/shared/dependency_proxy/tmp/uploads/442730038"},{"key":"file.remote_id","value":""},{"key":"file.remote_url","value":""},{"key":"file.sha1","value":"360f0382dc4e9a780690b66e7a488e92199ee702"},{"key":"file.sha256","value":"23c438466695cca70bffc703fc418f81411a251877d001132bd52dc8fe829484"},{"key":"file.sha512","value":"a25825c1cc530579a162e867d9bc28b5d8c1258122b1930552d3997de654d789993f6a70fe72da9fec5eb49cd8ba57e34e0b24cba9e39bcee99d7f39c57ec7bb"},{"key":"file.size","value":"115"},{"key":"file.upload_duration","value":"0.000355331"},{"key":"file","value":"#"},{"key":"group_id","value":"stackspin"},{"key":"image","value":"bitnami/mysql"},{"key":"sha","value":"sha256:23c438466695cca70bffc703fc418f81411a251877d001132bd52dc8fe829484"}],"correlation_id":"01FSM0H5R9NGRGARGKTDQWD8RK","meta.user":"maarten","meta.root_namespace":"stackspin","meta.caller_id":"Groups::DependencyProxyForContainersController#upload_blob","meta.remote_ip":"213.108.108.41","meta.feature_category":"dependency_proxy","meta.client_id":"user/4","remote_ip":"213.108.108.41","user_id":4,"username":"maarten","ua":"containerd/v1.5.8-k3s2","queue_duration_s":0.011471,"request_urgency":"default","target_duration_s":1,"cf_ray":"6cef9efd3b7c4c26-AMS","redis_calls":1,"redis_duration_s":0.000291,"redis_write_bytes":147,"redis_shared_state_calls":1,"redis_shared_state_duration_s":0.000291,"redis_shared_state_write_bytes":147,"db_count":12,"db_write_count":4,"db_cached_count":0,"db_replica_count":0,"db_primary_count":12,"db_replica_cached_count":0,"db_primary_cached_count":0,"db_replica_wal_count":0,"db_primary_wal_count":0,"db_replica_wal_cached_count":0,"db_primary_wal_cached_count":0,"db_replica_duration_s":0.0,"db_primary_duration_s":0.013,"cpu_s":0.066659,"mem_objects":12927,"mem_bytes":2142608,"mem_mallocs":3581,"mem_total_bytes":2659688,"pid":3740,"uploaded_file_upload_duration_s":0.000355331,"uploaded_file_size_bytes":115,"db_duration_s":0.01279,"view_duration_s":0.0,"duration_s":0.07164}
What stands out in these logs is that the JWT call differs! I don't know how that's generated, but that's the only real difference I could find in the first log lines, and it seems relevant to me (not knowing how the dependency proxy works exactly).
There are errors in the nginx log too:
tail /var/log/gitlab/nginx/gitlab_error.log 2022/01/17 15:37:29 [error] 20787#0: *943577 upstream prematurely closed connection while reading upstream, client: 213.108.108.41, server: open.greenhost.net, request: "GET /v2/stackspin/dependency_proxy/containers/rancher/mirrored-pause/manifests/sha256:59eec8837a4d942cc19a52b8c09ea75121acc38114a2c68b98983ce9356b8610?ns=docker.io HTTP/1.1", upstream: "http://unix:/var/opt/gitlab/gitlab-workhorse/sockets/socket:/v2/stackspin/dependency_proxy/containers/rancher/mirrored-pause/manifests/sha256:59eec8837a4d942cc19a52b8c09ea75121acc38114a2c68b98983ce9356b8610?ns=docker.io", host: "open.greenhost.net:443" 2022/01/17 15:37:31 [error] 20787#0: *943579 upstream prematurely closed connection while reading upstream, client: 213.108.108.41, server: open.greenhost.net, request: "GET /v2/stackspin/dependency_proxy/containers/rancher/mirrored-pause/manifests/sha256:59eec8837a4d942cc19a52b8c09ea75121acc38114a2c68b98983ce9356b8610?ns=docker.io HTTP/1.1", upstream: "http://unix:/var/opt/gitlab/gitlab-workhorse/sockets/socket:/v2/stackspin/dependency_proxy/containers/rancher/mirrored-pause/manifests/sha256:59eec8837a4d942cc19a52b8c09ea75121acc38114a2c68b98983ce9356b8610?ns=docker.io", host: "open.greenhost.net:443" 2022/01/17 15:37:32 [error] 20787#0: *943583 upstream prematurely closed connection while reading upstream, client: 213.108.108.41, server: open.greenhost.net, request: "GET /v2/stackspin/dependency_proxy/containers/rancher/mirrored-pause/manifests/sha256:59eec8837a4d942cc19a52b8c09ea75121acc38114a2c68b98983ce9356b8610?ns=docker.io HTTP/1.1", upstream: "http://unix:/var/opt/gitlab/gitlab-workhorse/sockets/socket:/v2/stackspin/dependency_proxy/containers/rancher/mirrored-pause/manifests/sha256:59eec8837a4d942cc19a52b8c09ea75121acc38114a2c68b98983ce9356b8610?ns=docker.io", host: "open.greenhost.net:443" 2022/01/17 15:37:34 [error] 20787#0: *943588 upstream prematurely closed connection while reading upstream, client: 213.108.108.41, server: open.greenhost.net, request: "GET /v2/stackspin/dependency_proxy/containers/rancher/mirrored-pause/manifests/sha256:59eec8837a4d942cc19a52b8c09ea75121acc38114a2c68b98983ce9356b8610?ns=docker.io HTTP/1.1", upstream: "http://unix:/var/opt/gitlab/gitlab-workhorse/sockets/socket:/v2/stackspin/dependency_proxy/containers/rancher/mirrored-pause/manifests/sha256:59eec8837a4d942cc19a52b8c09ea75121acc38114a2c68b98983ce9356b8610?ns=docker.io", host: "open.greenhost.net:443" 2022/01/17 15:37:35 [error] 20787#0: *943594 upstream prematurely closed connection while reading upstream, client: 213.108.108.41, server: open.greenhost.net, request: "GET /v2/stackspin/dependency_proxy/containers/rancher/mirrored-pause/manifests/sha256:59eec8837a4d942cc19a52b8c09ea75121acc38114a2c68b98983ce9356b8610?ns=docker.io HTTP/1.1", upstream: "http://unix:/var/opt/gitlab/gitlab-workhorse/sockets/socket:/v2/stackspin/dependency_proxy/containers/rancher/mirrored-pause/manifests/sha256:59eec8837a4d942cc19a52b8c09ea75121acc38114a2c68b98983ce9356b8610?ns=docker.io", host: "open.greenhost.net:443" 2022/01/17 15:37:36 [error] 20787#0: *943601 upstream prematurely closed connection while reading upstream, client: 213.108.108.41, server: open.greenhost.net, request: "GET /v2/stackspin/dependency_proxy/containers/rancher/mirrored-pause/manifests/sha256:59eec8837a4d942cc19a52b8c09ea75121acc38114a2c68b98983ce9356b8610?ns=docker.io HTTP/1.1", upstream: "http://unix:/var/opt/gitlab/gitlab-workhorse/sockets/socket:/v2/stackspin/dependency_proxy/containers/rancher/mirrored-pause/manifests/sha256:59eec8837a4d942cc19a52b8c09ea75121acc38114a2c68b98983ce9356b8610?ns=docker.io", host: "open.greenhost.net:443" 2022/01/17 15:37:38 [error] 20787#0: *943605 upstream prematurely closed connection while reading upstream, client: 213.108.108.41, server: open.greenhost.net, request: "GET /v2/stackspin/dependency_proxy/containers/rancher/mirrored-pause/manifests/sha256:59eec8837a4d942cc19a52b8c09ea75121acc38114a2c68b98983ce9356b8610?ns=docker.io HTTP/1.1", upstream: "http://unix:/var/opt/gitlab/gitlab-workhorse/sockets/socket:/v2/stackspin/dependency_proxy/containers/rancher/mirrored-pause/manifests/sha256:59eec8837a4d942cc19a52b8c09ea75121acc38114a2c68b98983ce9356b8610?ns=docker.io", host: "open.greenhost.net:443" 2022/01/17 15:37:39 [error] 20787#0: *943612 upstream prematurely closed connection while reading upstream, client: 213.108.108.41, server: open.greenhost.net, request: "GET /v2/stackspin/dependency_proxy/containers/rancher/mirrored-pause/manifests/sha256:59eec8837a4d942cc19a52b8c09ea75121acc38114a2c68b98983ce9356b8610?ns=docker.io HTTP/1.1", upstream: "http://unix:/var/opt/gitlab/gitlab-workhorse/sockets/socket:/v2/stackspin/dependency_proxy/containers/rancher/mirrored-pause/manifests/sha256:59eec8837a4d942cc19a52b8c09ea75121acc38114a2c68b98983ce9356b8610?ns=docker.io", host: "open.greenhost.net:443" 2022/01/17 15:37:40 [error] 20787#0: *943615 upstream prematurely closed connection while reading upstream, client: 213.108.108.41, server: open.greenhost.net, request: "GET /v2/stackspin/dependency_proxy/containers/rancher/mirrored-pause/manifests/sha256:59eec8837a4d942cc19a52b8c09ea75121acc38114a2c68b98983ce9356b8610?ns=docker.io HTTP/1.1", upstream: "http://unix:/var/opt/gitlab/gitlab-workhorse/sockets/socket:/v2/stackspin/dependency_proxy/containers/rancher/mirrored-pause/manifests/sha256:59eec8837a4d942cc19a52b8c09ea75121acc38114a2c68b98983ce9356b8610?ns=docker.io", host: "open.greenhost.net:443" 2022/01/17 15:37:42 [error] 20787#0: *943618 upstream prematurely closed connection while reading upstream, client: 213.108.108.41, server: open.greenhost.net, request: "GET /v2/stackspin/dependency_proxy/containers/rancher/mirrored-pause/manifests/sha256:59eec8837a4d942cc19a52b8c09ea75121acc38114a2c68b98983ce9356b8610?ns=docker.io HTTP/1.1", upstream: "http://unix:/var/opt/gitlab/gitlab-workhorse/sockets/socket:/v2/stackspin/dependency_proxy/containers/rancher/mirrored-pause/manifests/sha256:59eec8837a4d942cc19a52b8c09ea75121acc38114a2c68b98983ce9356b8610?ns=docker.io", host: "open.greenhost.net:443"
Output of checks
Results of GitLab environment info
Expand for output related to GitLab environment info
System information System: Debian 9.13 Current User: git Using RVM: no Ruby Version: 2.7.5p203 Gem Version: 3.1.4 Bundler Version:2.1.4 Rake Version: 13.0.6 Redis Version: 6.0.16 Git Version: 2.33.1. Sidekiq Version:6.3.1 Go Version: unknown GitLab information Version: 14.6.2 Revision: 0a901d60f8a Directory: /opt/gitlab/embedded/service/gitlab-rails DB Adapter: PostgreSQL DB Version: 12.7 URL: https://open.greenhost.net HTTP Clone URL: https://open.greenhost.net/some-group/some-project.git SSH Clone URL: git@open.greenhost.net:some-group/some-project.git Using LDAP: no Using Omniauth: yes Omniauth Providers: github, openid_connect GitLab Shell Version: 13.22.1 Repository storage paths: - default: /var/opt/gitlab/git-data/repositories GitLab Shell path: /opt/gitlab/embedded/service/gitlab-shell Git: /opt/gitlab/embedded/bin/git
Results of GitLab application Check
Expand for output related to the GitLab application check
Checking GitLab subtasks ...Checking GitLab Shell ...
GitLab Shell: ... GitLab Shell version >= 13.22.1 ? ... OK (13.22.1) Running /opt/gitlab/embedded/service/gitlab-shell/bin/check Internal API available: OK Redis available via internal API: OK gitlab-shell self-check successful
Checking GitLab Shell ... Finished
Checking Gitaly ...
Gitaly: ... default ... OK
Checking Gitaly ... Finished
Checking Sidekiq ...
Sidekiq: ... Running? ... yes Number of Sidekiq processes (cluster/worker) ... 1/1
Checking Sidekiq ... Finished
Checking Incoming Email ...
Incoming Email: ... Checking Reply by email ...
IMAP server credentials are correct? ... Checking open@greenhost.net yes Mailroom enabled? ... skipped MailRoom running? ... skipped
Checking Reply by email ... Finished
Checking Incoming Email ... Finished
Checking LDAP ...
LDAP: ... LDAP is disabled in config/gitlab.yml
Checking LDAP ... Finished
Checking GitLab App ...
Git configured correctly? ... yes Database config exists? ... yes All migrations up? ... yes Database contains orphaned GroupMembers? ... no GitLab config exists? ... yes GitLab config up to date? ... yes Log directory writable? ... yes Tmp directory writable? ... yes Uploads directory exists? ... yes Uploads directory has correct permissions? ... yes Uploads directory tmp has correct permissions? ... yes Systemd unit files or init script exist? ... skipped (omnibus-gitlab has neither init script nor systemd units) Systemd unit files or init script up-to-date? ... skipped (omnibus-gitlab has neither init script nor systemd units) Projects have namespace: ... 8/1 ... yes 8/2 ... yes 8/3 ... yes 8/5 ... yes 8/6 ... yes 12/7 ... yes 8/8 ... yes 8/9 ... yes 8/10 ... yes 16/11 ... yes 12/13 ... yes 8/14 ... yes 8/16 ... yes 18/17 ... yes 8/18 ... yes 8/23 ... yes 33/25 ... yes 16/26 ... yes 8/27 ... yes 8/29 ... yes 8/30 ... yes 33/31 ... yes 16/33 ... yes 16/34 ... yes 8/35 ... yes 5/36 ... yes 5/38 ... yes 4/39 ... yes 8/40 ... yes 8/41 ... yes 8/42 ... yes 8/43 ... yes 8/44 ... yes 16/45 ... yes 16/46 ... yes 8/48 ... yes 8/50 ... yes Redis version >= 5.0.0? ... yes Ruby version >= 2.7.2 ? ... yes (2.7.5) Git version >= 2.33.0 ? ... yes (2.33.1) Git user has default SSH configuration? ... yes Active users: ... 1724 Is authorized keys file accessible? ... yes GitLab configured to store new projects in hashed storage? ... yes All projects are in hashed storage? ... yes
Checking GitLab App ... Finished
Checking GitLab subtasks ... Finished
Possible fixes
What stands out in these logs is that the JWT call differs! I don't know how that's generated, but that's the only real difference I could find in the first log lines, and it seems relevant to me (not knowing how the dependency proxy works).