Skip to content

Release Environments - Check and remediate impact of Bitnami chart/docker image changes

Important

This is a clone of production-engineering#27125 (closed), but specific for Release Environments.

Overview

Bitnami have announced that their large repository of open source charts/docker images are now being put behind a paywall effective August 28th:

The current Debian-based images will be moved from the current public catalog docker.io/bitnami to a legacy catalog docker.io/bitnamilegacy, and will no longer receive any updates.

A limited number of new hardened images are available under a new catalog docker.io/bitnamisecure.

We need to evaluate the impact of this change in our image builds, Helm charts and Kubernetes workloads, and remediate it before August 28th.

See also:

Risks

Without any action from our part, from August 28th:

  • Image builds pulling images from docker.io/bitnami will fail, breaking CI pipelines in multiple projects and affecting our ability to keep our tools and workloads up-to-date
  • Kubernetes workloads using images from docker.io/bitnami will fail to start as they won't be able to pull those images anymore

This is the second time in 8 months that major Bitnami catalog changes are threatening our infrastructure, after the introduction of aggressive rate limits on their Helm charts registry to promote Bitnami Premium.

Summary

  • bitnami/postgresql image is used by gitlab-prafect-db-automation job
  • bitnami/external-dns image is used by external-dns
  • All other images come from GitLab Helm chart.

Solutions

General evaluation

If we make an assumption that Distribution team will fix the Helm chart and backport it to the 3 maintained versions, then only gitlab-prafect-db-automation left to be fixed. However, even in that case, it means Release Environments won't work anymore with GitLab version < 18.0.

Plan of actions

Edited by Dat Tang