Skip to content

Helm repo search not returning published charts

Context

GitLab recently added support for a Helm chart registry to support publishing and sharing using your GitLab project. However, several bugs were discovered and opened. This issue deals with one particular problem and links to the related issues.

Problem to solve

In the original bug report, several items were reported:

  1. The last uploaded artifact (for a given name/version) is hidden behind the first uploaded one. This is a bug, which is fixed in the MR !68968 (merged), which is included in milestone 14.3.
  2. All uploaded artifacts from a given name/version are in the index. The problem here is twofold. First, 20 is a too low limit and can be easily be hit. Secondly, duplicated files will be returned and so, the limit is not applied at the right level. This issue is resolved in the MR !69223 (merged), also included in milestone 14.3.
  3. The upload endpoint is not failing when re-uploading the same name/version. This is by design. The Helm chart registry allows duplicates to be uploaded. this is by design, like other package types. However, it would be good to make it configurable. #340767 proposes adding a setting to control how duplicate uploads are handled.

Proposal

  • Introduce a new helm finder for packages.
    • This finder look for packages given a project and channel.
    • It will return the 300 most recent packages. This is a hard limit that we impose here.
      • This limit is similar to what we do with the nuget endpoint APIs.
      • We will iterate on this value, meaning that we will revisit this if necessary in the future.
  • Update the index presenter to work on a set of packages (instead a set of package files).
    • The presenter here needs to get the most recent package file. This is a common need so we created a new scope in the Packages::PackageFile.
  • Created a new scope in Packages::PackageFile to return the most recent package files given a set of packages.
    • This will create a query based on a lateral join with package files to select only the most recent package file for each package.
    • This scope will only be useful for package types where a push operation is a single file upload. That's the case for helm but not all package types work like this (for example, with maven, the push operation uploads at least 2 files)
  • Added/Updated the related specs
    • Making sure that if the presenter or API endpoint hits packages with duplicated package files, only the most recent package file.

Original bug report

Hi I'm using a Self Hosted version of Gitlab with:

GitLab: 14.1.0
GitLab Shell: 13.19.0
GitLab Workhorsev: 14.1.0
GitLab APIv4
Ruby: 2.7.2p137
Rails: 6.1.3.2
PostgreSQL: 13.3
Redis: 4.0.14

Everytime I push the same helm package with same version in my project, a new file is added to the artifac:

$ curl --user gitlab-ci-token:$CI_JOB_TOKEN      https://mygitlab.com/api/v4/projects/424/packages/helm/stable/index.yaml
---
apiVersion: v1
entries: {}
generated: '2021-08-02T15:30:47.089418218Z'
serverInfo:
  contextPath: "/api/v4/projects/424/packages/helm"
  
$  helm push Downloads/datadog-2.19.9.tgz repo-1
Pushing datadog-2.19.9.tgz to repo-1...
Done.

$  helm push Downloads/datadog-2.19.9.tgz repo-1
Pushing datadog-2.19.9.tgz to repo-1...
Done.

$  curl --user gitlab-ci-token:$CI_JOB_TOKEN  https://mygitlab.com/api/v4/projects/424/packages/helm/stable/index.yaml
---
apiVersion: v1
entries:
  datadog:
  - home: https://www.datadoghq.com
    icon: https://datadog-live.imgix.net/img/dd_logo_70x75.png
    name: datadog
    sources:
    - https://app.datadoghq.com/account/settings#agent/kubernetes
    - https://github.com/DataDog/datadog-agent
    version: 2.19.9
    keywords:
    - monitoring
    - alerting
    - metric
    apiVersion: v1
    appVersion: '7'
    description: Datadog Agent
    maintainers:
    - name: Datadog
      email: support@datadoghq.com
    created: '2021-08-02T15:31:07.037026000Z'
    digest:
    urls:
    - charts/datadog-2.19.9.tgz
  - home: https://www.datadoghq.com
    icon: https://datadog-live.imgix.net/img/dd_logo_70x75.png
    name: datadog
    sources:
    - https://app.datadoghq.com/account/settings#agent/kubernetes
    - https://github.com/DataDog/datadog-agent
    version: 2.19.9
    keywords:
    - monitoring
    - alerting
    - metric
    apiVersion: v1
    appVersion: '7'
    description: Datadog Agent
    maintainers:
    - name: Datadog
      email: support@datadoghq.com
    created: '2021-08-02T15:31:12.625876000Z'
    digest:
    urls:
    - charts/datadog-2.19.9.tgz
generated: '2021-08-02T15:31:16.465531121Z'
serverInfo:
  contextPath: "/api/v4/projects/424/packages/helm"

image

The expected behaviour would be to fail with an Already Exists HTTP error (like for the Terraform module Registry).

Thanks

Edited by Tim Rizzi