Kaniko cannot push images to GitLab Container Registry

We are trying to change our current shell executor to k8s executor and we have to use Kaniko to build rails images in k8s cluster.

Kaniko cannot push builded image to registry. There is no error at build stage. But kaniko pushed image with basic/simple Dockerfile. I think that there should not be special requirements for Dockerfile to build images in Kaniko.

https://docs.gitlab.com/ee/ci/docker/using_kaniko.html

Build stage in .gitlab-ci.yml
build:
  <<: *default_tags
  stage: build
  interruptible: true
  image:
    name: gcr.io/kaniko-project/executor:debug
    entrypoint: [""]
  script:
    - echo "{\"auths\":{\"$CI_REGISTRY\":{\"username\":\"$CI_REGISTRY_USER\",\"password\":\"$CI_REGISTRY_PASSWORD\"}}}" > /kaniko/.docker/config.json
    - /kaniko/executor --context $CI_PROJECT_DIR --dockerfile $CI_PROJECT_DIR/Dockerfile --destination $CI_REGISTRY_IMAGE:$CI_COMMIT_REF_SLUG.$CI_PIPELINE_ID
Kaniko access logs on GitLab Container Registry
2020-04-09_15:39:12.20956 time="2020-04-09T18:39:12.209505769+03:00" level=warning msg="error authorizing context: authorization token required" go.version=go1.12.13 http.request.host=registry.example.ru http.request.id=af238654-4ee6-48d7-9ea9-c41811431e13 http.request.method=GET http.request.remoteaddr=185.91.54.74 http.request.uri="/v2/" http.request.useragent="Go-http-client/2.0"                       
2020-04-09_15:39:12.20960 127.0.0.1 - - [09/Apr/2020:18:39:12 +0300] "GET /v2/ HTTP/1.1" 401 87 "" "Go-http-client/2.0"                                                                                       
2020-04-09_15:39:12.37783 time="2020-04-09T18:39:12.377649134+03:00" level=info msg="authorized request" go.version=go1.12.13 http.request.contenttype="application/json" http.request.host=registry.example.ru http.request.id=ad24b426-cb64-46ee-a2f3-6ee273e512fe http.request.method=POST http.request.remoteaddr=185.91.54.74 http.request.uri="/v2/a.amirkhonov/app/blobs/uploads/" http.request.useragent=go-containerregistry vars.name="a.amirkhonov/app"                                                                                                                                                                
2020-04-09_15:39:12.39724 time="2020-04-09T18:39:12.397193984+03:00" level=info msg="response completed" go.version=go1.12.13 http.request.contenttype="application/json" http.request.host=registry.example.ru http.request.id=ad24b426-cb64-46ee-a2f3-6ee273e512fe http.request.method=POST http.request.remoteaddr=185.91.54.74 http.request.uri="/v2/a.amirkhonov/app/blobs/uploads/" http.request.useragent=go-containerregistry http.response.duration=20.867145ms http.response.status=202 http.response.written=0                                                                                                           
2020-04-09_15:39:12.39726 127.0.0.1 - - [09/Apr/2020:18:39:12 +0300] "POST /v2/a.amirkhonov/app/blobs/uploads/ HTTP/1.1" 202 0 "" "go-containerregistry"                                                    
2020-04-09_15:39:12.43686 time="2020-04-09T18:39:12.436834385+03:00" level=warning msg="error authorizing context: insufficient scope" go.version=go1.12.13 http.request.host=registry.example.ru http.request.id=e7cb97c0-4ce9-49d7-978e-4aa49d2a9acd http.request.method=DELETE http.request.remoteaddr=185.91.54.74 http.request.uri="/v2/a.amirkhonov/app/blobs/uploads/4857efcc-5c10-48f1-816d-eca13e2367a2?_state=H45TtcrPRjsXG1gMGtH7Q7sVyYVMe-M2atfLzPS25yl7Ik5hbWUiOiJhLmFtaXJraG9ub3Yvc2x1cm0iLCJVVUlEIjoiNDg1N2VmY2MtNWMxMC00OGYxLTgxNmQtZWNhMTNlMjM2N2EyIiwiT2Zmc2V0IjowLCJTdGFydGVkQXQiOiIyMDIwLTA0LTA5VDE1OjM5OjEyLjM3Nzc0Njg0N1oifQ%3D%3D" http.request.useragent=go-containerregistry vars.name="a.amirkhonov/app" vars.uuid=4857efcc-5c10-48f1-816d-eca13e2367a2                                                              
2020-04-09_15:39:12.43689 127.0.0.1 - - [09/Apr/2020:18:39:12 +0300] "DELETE /v2/a.amirkhonov/app/blobs/uploads/4857efcc-5c10-48f1-816d-eca13e2367a2?_state=H45TtcrPRjsXG1gMGtH7Q7sVyYVMe-M2atfLzPS25yl7Ik5hbWUiOiJhLmFtaXJraG9ub3Yvc2x1cm0iLCJVVUlEIjoiNDg1N2VmY2MtNWMxMC00OGYxLTgxNmQtZWNhMTNlMjM2N2EyIiwiT2Zmc2V0IjowLCJTdGFydGVkQXQiOiIyMDIwLTA0LTA5VDE1OjM5OjEyLjM3Nzc0Njg0N1oifQ%3D%3D HTTP/1.1" 401 163 "" "go-containerregistry"
2020-04-09_15:39:12.53998 time="2020-04-09T18:39:12.539952247+03:00" level=warning msg="error authorizing context: insufficient scope" go.version=go1.12.13 http.request.host=registry.example.ru http.request.id=e70435b2-53f2-47b1-9d03-89e2033365d3 http.request.method=DELETE http.request.remoteaddr=185.91.54.74 http.request.uri="/v2/a.amirkhonov/app/blobs/uploads/4857efcc-5c10-48f1-816d-eca13e2367a2?_state=H45TtcrPRjsXG1gMGtH7Q7sVyYVMe-M2atfLzPS25yl7Ik5hbWUiOiJhLmFtaXJraG9ub3Yvc2x1cm0iLCJVVUlEIjoiNDg1N2VmY2MtNWMxMC00OGYxLTgxNmQtZWNhMTNlMjM2N2EyIiwiT2Zmc2V0IjowLCJTdGFydGVkQXQiOiIyMDIwLTA0LTA5VDE1OjM5OjEyLjM3Nzc0Njg0N1oifQ%3D%3D" http.request.useragent=go-containerregistry vars.name="a.amirkhonov/app" vars.uuid=4857efcc-5c10-48f1-816d-eca13e2367a2                                                              
2020-04-09_15:39:12.54003 127.0.0.1 - - [09/Apr/2020:18:39:12 +0300] "DELETE /v2/a.amirkhonov/app/blobs/uploads/4857efcc-5c10-48f1-816d-eca13e2367a2?_state=H45TtcrPRjsXG1gMGtH7Q7sVyYVMe-M2atfLzPS25yl7Ik5hbWUiOiJhLmFtaXJraG9ub3Yvc2x1cm0iLCJVVUlEIjoiNDg1N2VmY2MtNWMxMC00OGYxLTgxNmQtZWNhMTNlMjM2N2EyIiwiT2Zmc2V0IjowLCJTdGFydGVkQXQiOiIyMDIwLTA0LTA5VDE1OjM5OjEyLjM3Nzc0Njg0N1oifQ%3D%3D HTTP/1.1" 401 163 "" "go-containerregistry"
2020-04-09_15:40:31.41284 time="2020-04-09T18:40:31.412821379+03:00" level=info msg="authorized request" go.version=go1.12.13 http.request.host="localhost:5000" http.request.id=7e3fd0b2-f755-46c3-a301-5d4dc5189048 http.request.method=GET http.request.remoteaddr="127.0.0.1:47808" http.request.uri="/v2/a.amirkhonov/app/tags/list" http.request.useragent="Faraday v0.15.4" vars.name="a.amirkhonov/app"        
2020-04-09_15:40:31.41296 time="2020-04-09T18:40:31.412946948+03:00" level=info msg="response completed" go.version=go1.12.13 http.request.host="localhost:5000" http.request.id=7e3fd0b2-f755-46c3-a301-5d4dc5189048 http.request.method=GET http.request.remoteaddr="127.0.0.1:47808" http.request.uri="/v2/a.amirkhonov/app/tags/list" http.request.useragent="Faraday v0.15.4" http.response.contenttype="application/json; charset=utf-8" http.response.duration=1.083974ms http.response.status=200 http.response.written=47                                                                                                     
2020-04-09_15:40:31.41297 127.0.0.1 - - [09/Apr/2020:18:40:31 +0300] "GET /v2/a.amirkhonov/app/tags/list HTTP/1.1" 200 47 "" "Faraday v0.15.4"                                                             
2020-04-09_15:40:31.41582 time="2020-04-09T18:40:31.415798544+03:00" level=info msg="authorized request" go.version=go1.12.13 http.request.host="localhost:5000" http.request.id=6b8e9a76-5044-4143-b21a-2139bef1038f http.request.method=HEAD http.request.remoteaddr="127.0.0.1:47810" http.request.uri="/v2/a.amirkhonov/app/manifests/stand" http.request.useragent="Faraday v0.15.4" vars.name="a.amirkhonov/app" vars.reference=stand
2020-04-09_15:40:31.41614 time="2020-04-09T18:40:31.416122733+03:00" level=info msg="response completed" go.version=go1.12.13 http.request.host="localhost:5000" http.request.id=6b8e9a76-5044-4143-b21a-2139bef1038f http.request.method=HEAD http.request.remoteaddr="127.0.0.1:47810" http.request.uri="/v2/a.amirkhonov/app/manifests/stand" http.request.useragent="Faraday v0.15.4" http.response.contenttype="application/vnd.docker.distribution.manifest.v2+json" http.response.duration=1.856918ms http.response.status=200 http.response.written=1233                                                                       
2020-04-09_15:40:31.41617 127.0.0.1 - - [09/Apr/2020:18:40:31 +0300] "HEAD /v2/a.amirkhonov/app/manifests/stand HTTP/1.1" 200 1233 "" "Faraday v0.15.4"                                                    
2020-04-09_15:40:31.41793 time="2020-04-09T18:40:31.417913664+03:00" level=info msg="authorized request" go.version=go1.12.13 http.request.host="localhost:5000" http.request.id=d51537b1-7d88-4ee9-8d87-75b08e0dad91 http.request.method=GET http.request.remoteaddr="127.0.0.1:47812" http.request.uri="/v2/a.amirkhonov/app/manifests/stand" http.request.useragent="Faraday v0.15.4" vars.name="a.amirkhonov/app" vars.reference=stand
2020-04-09_15:40:31.41823 time="2020-04-09T18:40:31.41821183+03:00" level=info msg="response completed" go.version=go1.12.13 http.request.host="localhost:5000" http.request.id=d51537b1-7d88-4ee9-8d87-75b08e0dad91 http.request.method=GET http.request.remoteaddr="127.0.0.1:47812" http.request.uri="/v2/a.amirkhonov/app/manifests/stand" http.request.useragent="Faraday v0.15.4" http.response.contenttype="application/vnd.docker.distribution.manifest.v2+json" http.response.duration=1.393542ms http.response.status=200 http.response.written=1233
2020-04-09_15:40:31.41826 127.0.0.1 - - [09/Apr/2020:18:40:31 +0300] "GET /v2/a.amirkhonov/app/manifests/stand HTTP/1.1" 200 1233 "" "Faraday v0.15.4"                                                     
2020-04-09_15:40:31.42097 time="2020-04-09T18:40:31.420946647+03:00" level=info msg="authorized request" go.version=go1.12.13 http.request.host="localhost:5000" http.request.id=b4f32863-3aa1-4651-a038-42c6ee494e7e http.request.method=GET http.request.remoteaddr="127.0.0.1:47814" http.request.uri="/v2/a.amirkhonov/app/blobs/sha256:71ac87711d327b37873f2912af2d66ec05b80ae304e6fae8ecdfe86e262ae27c" http.request.useragent="Faraday v0.15.4" vars.digest="sha256:71ac87711d327b37873f2912af2d66ec05b80ae304e6fae8ecdfe86e262ae27c" vars.name="a.amirkhonov/app"                                                            
2020-04-09_15:40:31.42162 time="2020-04-09T18:40:31.421581194+03:00" level=info msg="response completed" go.version=go1.12.13 http.request.host="localhost:5000" http.request.id=b4f32863-3aa1-4651-a038-42c6ee494e7e http.request.method=GET http.request.remoteaddr="127.0.0.1:47814" http.request.uri="/v2/a.amirkhonov/app/blobs/sha256:71ac87711d327b37873f2912af2d66ec05b80ae304e6fae8ecdfe86e262ae27c" http.request.useragent="Faraday v0.15.4" http.response.contenttype="application/octet-stream" http.response.duration=2.427101ms http.response.status=200 http.response.written=2617                                      
2020-04-09_15:40:31.42164 127.0.0.1 - - [09/Apr/2020:18:40:31 +0300] "GET /v2/a.amirkhonov/app/blobs/sha256:71ac87711d327b37873f2912af2d66ec05b80ae304e6fae8ecdfe86e262ae27c HTTP/1.1" 200 2617 "" "Faradayv0.15.4"
2020-04-09_15:41:06.01230 time="2020-04-09T18:41:06.012272772+03:00" level=warning msg="error authorizing context: authorization token required" go.version=go1.12.13 http.request.host=registry.example.ru http.request.id=8d74aed4-cb20-4eb6-8132-66c1fc82b341 http.request.method=GET http.request.remoteaddr=185.91.54.74 http.request.uri="/v2/" http.request.useragent="Go-http-client/1.1"                      
2020-04-09_15:41:06.01234 127.0.0.1 - - [09/Apr/2020:18:41:06 +0300] "GET /v2/ HTTP/1.1" 401 87 "" "Go-http-client/1.1"                                                                                      
2020-04-09_15:41:06.26161 time="2020-04-09T18:41:06.261588305+03:00" level=info msg="authorized request" go.version=go1.12.13 http.request.host=registry.example.ru http.request.id=796c58d7-a19c-4367-9635-673c2bf134b7 http.request.method=GET http.request.remoteaddr=185.91.54.74 http.request.uri="/v2/a.amirkhonov/app/manifests/develop.69523" http.request.useragent="Go-http-client/1.1" vars.name="a.amirkhonov/app" vars.reference=develop.69523
2020-04-09_15:41:06.26173 time="2020-04-09T18:41:06.261706119+03:00" level=error msg="response completed with error" auth.user.name=a.amirkhonov err.code="manifest unknown" err.detail="unknown tag=develop.69523" err.message="manifest unknown" go.version=go1.12.13 http.request.host=registry.example.ru http.request.id=796c58d7-a19c-4367-9635-673c2bf134b7 http.request.method=GET http.request.remoteaddr=185.91.54.74 http.request.uri="/v2/a.amirkhonov/app/manifests/develop.69523" http.request.useragent="Go-http-client/1.1" http.response.contenttype="application/json; charset=utf-8" http.response.duration=1.141988ms http.response.status=404 http.response.written=103 vars.name="a.amirkhonov/app" vars.reference=develop.69523                                                                                         
2020-04-09_15:41:06.26175 127.0.0.1 - - [09/Apr/2020:18:41:06 +0300] "GET /v2/a.amirkhonov/app/manifests/develop.69523 HTTP/1.1" 404 103 "" "Go-http-client/1.1"
Edited by Akbarkhon Amirkhonov