Supportable customizable base images

Goal: Good support for custom base images

Today we use rpm-ostree to generate base images. It has been working that way for years, and is a known quantity. There's also a few other tools out there that do builds. There's a whole lot of intersecting/overlapping things going on here.

Requirements

Requirement: "Container native"

The build process MUST be executable as part of a Dockerfile/Containerfile with podman. It MAY also work with other tools, e.g. docker.

An important component of this is that it MUST work to customize the image with Containerfile/Dockerfile fragments.

Goal: Avoid forking

At a base level, one can of course just fork the source git repository for our base image - but this implies a lot of overhead/maintenance that we already pay for the image-based variants in Fedora. Our source layout isn't designed really for forking.

Goal: Base image can ship non-RPM content

We have non-RPM content in the base image today. Hence the "obvious" implementation of custom base images that accepts only RPMs as input would be problematic.

Goal: Align with other base image builds

We should also align with how other base images are built; ideally with the Fedora/CentOS Stream "app" images (i.e. quay.io/centos/centos:stream9 etc.).

Goal: Avoid creating new things

Related to the above; there's a whole lot of prior art out there. Let's avoid creating a new file format for example.

Goal: Declarative

There's many advantages to having a declarative format; i.e. not just RUN dnf -y install. We need this in order to handle "lockfiles" sanely for example. Following from the above: We should pick existing declarative formats, not create new ones.

Goal: Reduce role of ostree

Let's aim to shrink over time the role of ostree in the base image builds.

Implementation: base image contains tools

Our default base image is already large. Let's just ensure that this image has the tools to customized new versions of itself.

This continues to "anchor" the user experience around container flow and helps avoid forking and decoupled build systems.

Skeleton no-op build

This shows the raw structure that could be used to generate a new custom base image:

FROM quay.io/fedora/fedora-bootc as builder
RUN --mount=type=cache,target=/var/cache self-image-builder init
RUN --mount=type=bind,rw=true,src=.,dst=/buildcontext,bind-propagation=shared self-image-builder finish /buildcontext/image.ociarchive

# This magical bit extends the oci-archive (tar) file generated in the build stage.
FROM oci-archive:./image.ociarchive
# Need to reference builder here to force ordering. But since we have to run
# something anyway, we might as well cleanup after ourselves.
RUN --mount=type=bind,from=builder,src=.,target=/var/tmp --mount=type=bind,rw=true,src=.,dst=/buildcontext,bind-propagation=shared rm /buildcontext/image.ociarchive

This process would re-fetch all input RPMs from rpm-md repositories - default version locking them to the versions in the base image. The non-RPM content in the base image would be applied afterwards.

Starting from a small subset

RUN --mount=type=cache,target=/var/cache self-image-builder init --install=bootc,kernel

Here bootc,kernel is a strawman for the most minimal image we would support. But I'd also like to support e.g.

RUN --mount=type=cache,target=/var/cache self-image-builder init --install=@core

The @ is a historical signifier for comps groups but here we're expanding this to something like a "content set" (think treefiles/kickstarts) that could include also other non-RPM content too. (I wanted to use the term "module" but it's taken)

Customizing the rootfs

In this example we're copying configuration from a container into the target /rootfs, and then also ensuring that the initramfs is regenerated before finalization.

FROM quay.io/examplecorp/bootc-configs:latest as config
RUN --mount=type=cache,target=/var/cache self-image-builder init
COPY --from=config /usr/ /rootfs/usr/
RUN self-image-build run dracut
RUN --mount=type=bind,rw=true,src=.,dst=/buildcontext,bind-propagation=shared self-image-builder finish /buildcontext/image.ociarchive