Skip to content
Snippets Groups Projects

Can't pull image: invalid character '<' looking for beginning of value

    • Closed Issue created by TriplEight @TriplEight

    Summary

    Klar scanner refuses to read image manifest and fails the job.

    Steps to reproduce

    My PR to update the scanner to the changes https://github.com/paritytech/scripts/pull/247 and the current config of the scanner.

    CI config is described below, TLDR:

    I had to switch to buildah to build images without dind. Buildah seems to use some different format of manifest. And Klar fails to read it.

    Example Project

    # build and push the image using buildah
dockerize:
  stage:                           build
  image:                           quay.io/buildah/stable
  tags:
    - kubernetes-build
  script:
    - buildah bud
      --squash
      --format=docker
      --tag "docker.io/paritytech/tools:latest"
      --file "dockerfiles/tools/Dockerfile" dockerfiles
  - buildah login --username "$DOCKER_USER" --password "$DOCKER_PASSWORD" docker.io
  - buildah push --format=v2s2 "docker.io/paritytech/tools:latest"
  - buildah logout docker.io

# scan image using Klar
container_scanning:
  # Template does not work, had to override its config from
  # https://gitlab.com/gitlab-org/gitlab/blob/master/lib/gitlab/ci/templates/Security/Container-Scanning.gitlab-ci.yml
  # these settings are from the template 
  stage:                           test
  image:                           $SECURE_ANALYZERS_PREFIX/klar:$CS_MAJOR_VERSION
  services:
    - name:                        $CLAIR_DB_IMAGE
      alias:                       clair-vulnerabilities-db
  allow_failure:                   true
  script:
    - /analyzer run
  artifacts:
    reports:
      container_scanning:          gl-container-scanning-report.json
  variables:
    SECURE_ANALYZERS_PREFIX:       "registry.gitlab.com/gitlab-org/security-products/analyzers"
  # these are our overrides
    CS_MAJOR_VERSION:              3
    CLAIR_DB_IMAGE:                arminc/clair-db:latest
    CLAIR_TRACE:                   "true"
    GIT_STRATEGY:                  fetch
    CI_APPLICATION_REPOSITORY:     docker.io/paritytech/tools:latest
    CI_APPLICATION_TAG:            latest
    DOCKERFILE_PATH:               dockerfiles/tools/Dockerfile
  tags:
    - kubernetes-build

    What is the current bug behavior?

    Scan job fails with an error: https://gitlab.parity.io/parity/infrastructure/scripts/-/jobs/751611

    Error log 
    [INFO] [klar] [2020-12-17T20:08:55Z] ▶ Clair API started successfully.
[INFO] [klar] [2020-12-17T20:08:55Z] ▶ Scanning container from registry 'docker.io/paritytech/tools:latest' for vulnerabilities with severity level 'Unknown' or higher with klar '' and clair '2.1.4'
[WARN] [klar] [2020-12-17T20:08:55Z] ▶ Clair log contents: 
  {"Event":"notifier service is disabled","Level":"info","Location":"notifier.go:77","Time":"2020-12-17 20:08:53.872460"}
  {"Event":"starting health API","Level":"info","Location":"api.go:85","Time":"2020-12-17 20:08:53.872574","port":6061}
  {"Event":"updater service is disabled.","Level":"info","Location":"updater.go:78","Time":"2020-12-17 20:08:53.872767"}
  
[INFO] [klar] [2020-12-17T20:08:56Z] ▶ Shutting down Clair server with PID: 19
[INFO] [klar] [2020-12-17T20:08:56Z] ▶ Clair server shut down successfully
[ERRO] [klar] [2020-12-17T20:08:56Z] ▶ Error encountered while scanning container 'docker.io/paritytech/tools:latest': ----> HTTP REQUEST:
GET /v2/paritytech/tools/manifests/latest HTTP/1.1
Host: docker.io
Accept: application/vnd.docker.distribution.manifest.v2+json, application/vnd.docker.distribution.manifest.v1+prettyjws
Authorization: [REDACTED]
<---- HTTP RESPONSE:
HTTP/1.1 404 Not Found
Transfer-Encoding: chunked
Cache-Control: max-age=21600, public
Connection: keep-alive
Content-Language: en
Content-Type: text/html; charset=UTF-8
Date: Thu, 17 Dec 2020 20:08:56 GMT
Etag: W/"1608235736"
Expires: Sun, 19 Nov 1978 05:00:00 GMT
Last-Modified: Thu, 17 Dec 2020 20:08:56 GMT
Link: <https://www.docker.com/node/18056>; rel="canonical"
Link: <https://www.docker.com/node/18056>; rel="revision"
Link: <https://www.docker.com/node/18056>; rel="canonical"
Link: <https://www.docker.com/node/18056>; rel="revision"
Server: nginx
Strict-Transport-Security: max-age=31536000
Vary: Accept-Encoding
Via: 1.1 42b60ee17f75a.cloudfront.net (CloudFront)
X-Amz-Cf-Id: xxx
X-Amz-Cf-Pop: FRA2-C1
X-Cache: Error from cloudfront
X-Content-Type-Options: nosniff
X-Drupal-Cache: MISS
X-Drupal-Dynamic-Cache: UNCACHEABLE
X-Frame-Options: SAMEORIGIN
X-Generator: Drupal 8 (https://www.drupal.org)
X-Powered-By: PHP/7.3.14
X-Ua-Compatible: IE=edge
8000
  
<!DOCTYPE html>
<html lang="en" dir="ltr" prefix="og: https://ogp.me/ns#">

# I'll truncate the entire 404 HTML from docker.com

</html>
0
ImageV1 decode error
Can't pull image: invalid character '<' looking for beginning of value
exit status 2
Uploading artifacts for failed job
00:00
Uploading artifacts...
WARNING: gl-container-scanning-report.json: no matching files 
ERROR: No files to upload                          
Cleaning up file based variables
00:00
ERROR: Job failed: command terminated with exit code 1

    What is the expected correct behavior?

    Previously, when I used dind to build images, the scanner job was always successful, tried kaniko and it worked as well.

    Results of GitLab environment info

    GitLab Enterprise Edition 13.6.3-ee

    Designs

      Child items ...

      2. Activity

      Please register or sign in to reply