Access denied to registry (use CI_JOB_TOKEN)

Hello all and sorry for my English

Briefly: 20% builds completed with an error: unauthorized: HTTP Basic: Access denied At runtime: docker push <IMAGE_NAME>

Detail:

Gitlab-ce 11.1.1 in a container

Internal registry

Gitlab-runner 11.1.0

.gitlab-ci.yaml:

image: docker:latest

# Pipeline stages
stages:
- prepare
- build

# Build image environment prepare
before_script:
  - docker login -u $CI_REGISTRY_USER -p $CI_JOB_TOKEN $CI_REGISTRY

  # Docker image names
  BASE_NAME: "base"
  API_NAME: "api"
  CRON_NAME: "cron"
  RETAIL_NAME: "retail"
  WEB_NAME: "web"

  # Docker container names
  API_CONTAINER_NAME: "api-application"
  CRON_CONTAINER_NAME: "cron-application"
  RETAIL_CONTAINER_NAME: "retail-application"
  WEB_CONTAINER_NAME: "web-application"

  # Image names
  BASE_IMAGE: "$CI_REGISTRY_IMAGE/$BASE_NAME"
  API_IMAGE: "$CI_REGISTRY_IMAGE/$API_NAME"
  CRON_IMAGE: "$CI_REGISTRY_IMAGE/$CRON_NAME"
  RETAIL_IMAGE: "$CI_REGISTRY_IMAGE/$RETAIL_NAME"
  WEB_IMAGE: "$CI_REGISTRY_IMAGE/$WEB_NAME"

  # Build image names
  BASE_BUILD_IMAGE: "$CI_REGISTRY_IMAGE/$BASE_NAME:$CI_COMMIT_REF_NAME"
  API_BUILD_IMAGE: "$CI_REGISTRY_IMAGE/$API_NAME:$CI_COMMIT_REF_NAME"
  CRON_BUILD_IMAGE: "$CI_REGISTRY_IMAGE/$CRON_NAME:$CI_COMMIT_REF_NAME"
  RETAIL_BUILD_IMAGE: "$CI_REGISTRY_IMAGE/$RETAIL_NAME:$CI_COMMIT_REF_NAME"
  WEB_BUILD_IMAGE: "$CI_REGISTRY_IMAGE/$WEB_NAME:$CI_COMMIT_REF_NAME"

  # Release image names
  BASE_RELEASE_IMAGE: "$CI_REGISTRY_IMAGE/$BASE_NAME:latest"
  API_RELEASE_IMAGE: "$CI_REGISTRY_IMAGE/$API_NAME:latest"
  CRON_RELEASE_IMAGE: "$CI_REGISTRY_IMAGE/$CRON_NAME:latest"
  RETAIL_RELEASE_IMAGE: "$CI_REGISTRY_IMAGE/$RETAIL_NAME:latest"
  WEB_RELEASE_IMAGE: "$CI_REGISTRY_IMAGE/$WEB_NAME:latest"

# Templates
.prepare_template: &prepare_definition
  stage: prepare
  when: manual
  only:
    - master
    - /^release-.*$/
  tags:
    - build

.build_template: &build_definition
  stage: build
  only:
    - master
    - /^release-.*$/
  tags:
    - build

### BUILD ###

# Build base OS image
build-base:
  <<: *prepare_definition
  script:
    - docker build --no-cache -f Dockerfile -t $BASE_RELEASE_IMAGE .
    - docker push $BASE_RELEASE_IMAGE
    - docker rmi $BASE_RELEASE_IMAGE

# Build API application image
build-api:
  <<: *build_definition
  script:
    - docker build $BUILD_REPO_ARGS --build-arg environment=uat --no-cache -f api/Dockerfile -t $API_BUILD_IMAGE .
    - docker push $API_BUILD_IMAGE
    - docker rmi $API_BUILD_IMAGE

# Build cron application image
build-cron:
  <<: *build_definition
  script:
    - docker build $BUILD_REPO_ARGS --build-arg environment=uat --no-cache -f cron/Dockerfile -t $CRON_BUILD_IMAGE .
    - docker push $CRON_BUILD_IMAGE
    - docker rmi $CRON_BUILD_IMAGE

# Build retail application image
build-retail:
  <<: *build_definition
  script:
    - docker build $BUILD_REPO_ARGS --build-arg environment=uat --no-cache -f retail/Dockerfile -t $RETAIL_BUILD_IMAGE .
    - docker push $RETAIL_BUILD_IMAGE
    - docker rmi $RETAIL_BUILD_IMAGE

# Build web application image
build-web:
  <<: *build_definition
  script:
    - docker build $BUILD_REPO_ARGS --build-arg environment=uat --no-cache -f web/Dockerfile -t $WEB_BUILD_IMAGE .
    - docker push $WEB_BUILD_IMAGE
    - docker rmi $WEB_BUILD_IMAGE

All builds used gitlan-runner-tag 'build'. I register gitlab-runner. /etc/gitlab-runner/config.toml:

concurrent = 10
check_interval = 3

[[runners]]
  name = "BUILD"
  limit = 4
  url = "<REGISTRY_URL"
  token = "<TOKEN>"
  executor = "shell"
  [runners.cache]

With this settings, we can run all 4 builds (excluding base-image) at the same time. And this work, but ~20% builds finish with error:

$ docker push <IMAGE>
The push refers to repository [<URL>]
b8912353ab19: Preparing
f26785188f66: Preparing
409148046411: Preparing
...
...
...
0a999339f04f: Waiting
bcc97fbfc9e1: Waiting
unauthorized: HTTP Basic: Access denied
ERROR: Job failed: exit status 1

Before this docker pull <IMAGE> finish correctly. If I retry build - likely it finished normally.

Everything looks as if the job-token expires. However, builds can work a maximum of 30 minutes.

Authorization token duration (minutes) set 1440.

I try:

  1. Used 1 runner for 1 process. /etc/gitlab-runner/config.toml:
concurrent = 10
check_interval = 3

[[runners]]
  name = "BUILD [1]"
  limit = 1
  url = "<GIT_URL>"
  token = "<TOKEN>"
  executor = "shell"
  [runners.cache]

[[runners]]
  name = "BUILD [2]"
  limit = 1
  url = "<GIT_URL>"
  token = "<TOKEN>"
  executor = "shell"
  [runners.cache]

[[runners]]
  name = "BUILD [3]"
  limit = 1
  url = "<GIT_URL>"
  token = "<TOKEN>"
  executor = "shell"
  [runners.cache]

[[runners]]
  name = "BUILD [4]"
  limit = 1
  url = "<GIT_URL>"
  token = "<TOKEN>"
  executor = "shell"
  [runners.cache]

However, the error remained

  1. I register GitLab user, with the necessary rights, add him to CI\CD variables, and change .gitlab-ci.yaml:
# Build image environment prepare
before_script:
#  - docker login -u $CI_REGISTRY_USER -p $CI_JOB_TOKEN $CI_REGISTRY
  - docker login -u $BUILD_USER -p $BUILD_PASSWORD $CI_REGISTRY

It's work, but I want to use best way.

Please told me, what I do wrong!

Edited by 🤖 GitLab Bot 🤖