CNG: Use smaller base images
Using Distroless (instead of Debian) and UBI Micro is a challenge. This issues discusses highlights some of the them and discusses how to address them.
Common challenges
Shell scripts
Current CNG images rely on shell and in some cases Bash to implement the entrypoint and healthchek of Docker containers.
It could be very difficult and perhaps undesirable to remove the shell. Replacing the scripts seems almost impossible. On the other hand, having a shell helps debugging the images in their runtime environment. Not to mention that the toolbox image requires a shell to operate.
The first step is to make these scripts portable so that they can run on a POSIX shell. I could only find one instance in gitlab-base
where shopt
which is Bash-specific is used but there could be other instances, for example arrays or /dev/tcp
built-in.
This will simplify migrating to other base images where a different POSIX shell is available, for example Ash.
NOTE: The best practices need to be clearly described in developer documentation.
Base commands and packages
Different shell scripts that are packaged with the CNG images use a limited number of commands to operate, here is a partial list of these commands:
adduser, awk, basename, cat, chmod, chown, cp, curl, find, grep, ls, mkdir, netstat, pgrep, ps, rm, sort, stat, tail, tar, touch, xtail
With the exception of curl
and xtail
the rest of the commands can be sourced from BusyBox.
xtail
can be built from source into a single binary and it only depends on libc
(another asset image).
But the case for curl
is more complicated. Although it has an official image (see curlimages/curl
) it is based on Alpine and it has a wide range of dependencies. Sourcing it from the official image or building it from source can not be justified, considering the degree that it is being used.
It is not an application dependency for GitLab (we can treat Toolbox as an exception). In a few runtime images it is used for health checking scripts. This can be reasonably replaced with nc
and a smart shell tweak to extract the HTTP status code and even headers.
Another small piece is the ca-certificates
package. We can safely use curl
CA certificate extracts and upgrade the CA certificates without using a package.
A number of images use pgrep
and ps
to monitor the health of the container by checking if the main process is running. Since we changed the PID1 to the main process of the container this health check seems moot.
Using Distroless and UBI Micro
Using the previous steps we can use Distroless as a base image. UBI Micro has a shell and part of the required commands are included. But the same solution as Distroless can be applied to UBI Micro to provide the core software modules.
How to source dependencies
- Core utilities, including shell, are sourced from well-known and well-maintained official images, including busybox.
- Anything that we build in Omnibus GitLab is built from source as is maintained as an asset image.
- Core libraries that are included in a specific distribution are sourced from official package repositories but are installed in the build stage, possibly with a package manager, and the binaries are injected in the final image. This will remove the dependency of the final image to a package manager and relaxes the distribution-specific dependencies.
Why not Alpine?
At this stage we are not prepared to build GitLab software components with Musl. Even Omnibus GitLab, which very often is the source of our build expertise, does not support it.
The case of Rails image
Ruby Gems, and in particular those with native extensions, can complicate things for the Rails image and others that are based on it.
The following table shows the list of dependencies of the existing Gems with native extensions that are installed in the current gitLab-rails
image:
ld-linux-x86-64.so.2, libanl.so.1, libc.so.6, libcom_err.so.2, libcrypt.so.1, libcrypto.so.1.1, libdl.so.2, libffi.so.7, libgcc_s.so.1, libicudata.so.67, libicui18n.so.67, libicuio.so.67, libicuuc.so.67, libjemalloc.so.2, libk5crypto.so.3, libkeyutils.so.1, libkrb5.so.3, libkrb5support.so.0, libm.so.6, libpq.so.5, libpthread.so.0, libre2.so.9, libresolv.so.2, librt.so.1, libruby.so.2.7, libssl.so.1.1, libstdc++.so.6, libutil.so.1, libyaml-0.so.2, libz.so.1
This is not an exhaustive list of dependencies. For example it does not include the transitive dependencies. But it is enough to argue in favor of the above framework for sourcing software dependencies.
The following table shows how the dependencies are sourced currently and what is the alternative for Distroless and UBI Micro.
Dependency |
Current Source |
Proposed Source |
||
Debian |
UBI |
Distroless |
UBI Micro |
|
ld-linux-x86-64.so.2 |
bullseye-slim |
ubi8/ubi |
distroless/base |
ubi8/ubi-micro |
libanl.so.1 |
bullseye-slim |
ubi8/ubi |
distroless/base |
ubi8/ubi-micro |
libc.so.6 |
bullseye-slim |
ubi8/ubi |
distroless/base |
ubi8/ubi-micro |
libcom_err.so.2 |
bullseye-slim |
ubi8/ubi |
deb (libcom-err2) |
rpm (libcom_err) |
libcrypt.so.1 |
bullseye-slim |
ubi8/ubi |
deb (libcrypt1) |
rpm (libxcrypt) |
libcrypto.so.1.1 |
bullseye-slim |
ubi8/ubi |
distroless/base |
OMNIBUS (openssl) |
libdl.so.2 |
bullseye-slim |
ubi8/ubi |
distroless/base |
ubi8/ubi-micro |
libffi.so.7 |
bullseye-slim |
ubi8/ubi |
OMNIBUS (libffi) |
OMNIBUS (libffi) |
libgcc_s.so.1 |
bullseye-slim |
ubi8/ubi |
distroless/cc |
ubi8/ubi-micro |
libicudata.so.67 |
apt (libicu) |
dnf (libicu) |
OMNIBUS (libicu) |
OMNIBUS (libicu) |
libicui18n.so.67 |
apt (libicu) |
dnf (libicu) |
OMNIBUS (libicu) |
OMNIBUS (libicu) |
libicuio.so.67 |
apt (libicu) |
dnf (libicu) |
OMNIBUS (libicu) |
OMNIBUS (libicu) |
libicuuc.so.67 |
apt (libicu) |
dnf (libicu) |
OMNIBUS (libicu) |
OMNIBUS (libicu) |
libjemalloc.so.2 |
SOURCE (gitlab-ruby) |
SOURCE (gitlab-ruby) |
OMNIBUS (jemalloc) |
OMNIBUS (jemalloc) |
libk5crypto.so.3 |
bullseye-slim |
ubi8/ubi |
OMNIBUS (krb5) |
OMNIBUS (krb5) |
libkeyutils.so.1 |
bullseye-slim |
ubi8/ubi |
deb (libkeyutils1) |
rpm (keyutils-libs |
libkrb5.so.3 |
bullseye-slim |
ubi8/ubi |
OMNIBUS (krb5) |
OMNIBUS (krb5) |
libkrb5support.so.0 |
bullseye-slim |
ubi8/ubi |
OMNIBUS (krb5) |
OMNIBUS (krb5) |
libm.so.6 |
bullseye-slim |
ubi8/ubi |
distroless/base |
ubi8/ubi-micro |
libpq.so.5 |
SOURCE (postgresql) |
SOURCE (postgresql) |
OMNIBUS (postgresql) |
OMNIBUS (postgresql) |
libpthread.so.0 |
bullseye-slim |
ubi8/ubi |
distroless/base |
ubi8/ubi-micro |
libre2.so.9 |
apt (libre2-9) |
SOURCE (gitlab-ruby) |
OMNIBUS (libre2) |
OMNIBUS (libre2) |
libresolv.so.2 |
bullseye-slim |
ubi8/ubi |
distroless/base |
ubi8/ubi-micro |
librt.so.1 |
bullseye-slim |
ubi8/ubi |
distroless/base |
ubi8/ubi-micro |
libruby.so.2.7 |
SOURCE (gitlab-ruby) |
SOURCE (gitlab-ruby) |
OMNIBUS (ruby) |
OMNIBUS (ruby) |
libssl.so.1.1 |
bullseye-slim |
ubi8/ubi |
distroless/base |
OMNIBUS (openssl) |
libstdc++.so.6 |
bullseye-slim |
ubi8/ubi |
distroless/cc |
rpm (libstdc++) |
libutil.so.1 |
bullseye-slim |
ubi8/ubi |
distroless/base |
ubi8/ubi-micro |
libyaml-0.so.2 |
apt (libyaml) |
ubi8/ubi |
OMNIBUS (libyaml) |
OMNIBUS (libyaml) |
libz.so.1 |
bullseye-slim |
ubi8/ubi |
OMNIBUS (zlib) |
OMNIBUS (zlib) |