D

defs2bst

Scripts to convert Baserock definitions to BuildStream projects

Name Last Update
defs2bst Loading commit data...
README.rst Loading commit data...
defs2bst.py Loading commit data...

Migration path from YBD projects

This document outlines how to move from a YBD definitions project such as Baserock, to use BuildStream.

System Requirements

Before using the accompanied scripts, you will need:

  • An installation of YBD
  • An installation of BuildStream
  • A checkout of this repository

Migration Steps

Preparing project configuration

Any of the aliases you are currently using with YBD for your projects must be declared in a project.conf.

If you are converting a baserock system, which includes a bootstrap of the runtime, this part has already been done for you and a working copy exists here:

https://gitlab.com/BuildStream/buildstream-tests/blob/gnu-toolchain/project.conf

Preparing the new base

Because of BuildStream's strict no host tool policy, it is impossible to convert the base 'bootstrap mode' components in an existing YBD project.

For the baserock project, a manual conversion of the build-essential stratum was required and is available at:

https://gitlab.com/BuildStream/buildstream-tests/tree/gnu-toolchain

How was this part migrated ?

For reference, the work which was needed to manually convert the toolchain bootstrap build-essential are as follows:

  • project.conf defines the arch specific stuff which was previously hardcoded into YBD, so the stage1 target and target etc are all defined by the gnu-runtime project.conf
  • Direct and easy changes for git source representation, these are very straight forward as BuildStream's git source supports everything YBD did for its repo, ref and submodules attributes.
  • Added stage1.bst & stage2.bst stacks, everything built in stage2 build-depends on stage1.bst, and the final build-essential products build-depend on stage2.bst (note build-depend means to depend _only_ for building, not propagated forward).
  • Instead of using host tools we build on the GNOME flatpak sdk/platform bundles, this choice was rather arbitrary and it can be replaced instead with an output of gnu-toolchain proper (so the build becomes circular and independent of third parties).
  • Some build-essential morph files use $(dirname $(pwd)) for a sysroot, which is weird, it means the morph files rely on building at one directory below the slash sysroot - in buildstream we build in /buildstream/build which is two - had to replace these with $(dirname $(dirname $(pwd))) instead
  • Remove the devices sections from the fhs-dirs elements, not allowed to create static device nodes in buildstream.
  • Elements in stage2 and stage3 need some customized environment variables to place /tools in the path. YBD previously did this by asserting that any chunk which sets an explicit prefix will augment the path of every chunk which indirectly depends on that custom prefix chunk, BuildStream does not make this assumption so the environments need to be specified either project wide, or on each element which needs to be built with a custom PATH.

What if I'm migrating something else ?

If you have a YBD project that does have a bootstrap process and diverges significantly from Baserock's build-essential stratum, you will need to follow the steps above to perform the same task and end up with a base runtime.

Otherwise, it's fairly common to have a YBD project that uses a tarball SDK to build things on top of, in that case the above procedure is much simpler. One should start by creating a project.conf and creating an import element using a git source for importing the same base SDK you were previously using.

Converting YBD definitions

At this point you should know:

  • What is the name of the bst element which provides the base you want to build on
  • What is the definitions target that you want to build against that base

The conversion scripts will dump bst files into an output directory based on the <target>.yml output of ybd, which it will invoke by itself. Note that the conversion script may be run many times in a row on different targets, in this way it is possible to convert a definitions repository with multiple targets.

Valid targets inlcude stratum and system morph files; cluster conversions are not supported.

To invoke the defs2bst converter, the arguments you need are:

./defs2bst/defs2bst.py --ybd <ybd.py> \
                       --definitions <definitions repo> \
                       --output <buildstream elements directory> \
                       --rebase <definitions base dependency> <buildstream base dependency> \
                       <ybd target morph> <target architecture>
  • --ybd: The path to the ybd.py script in a ybd checkout
  • --definitions: The path to the definitions checkout you want to convert from
  • --output: The elements directory in the target buildstream project
  • --rebase: The old base dependency you are replacing with a new manually created base dependency
  • The remainder are the <target> <arch> arguments which defs2bst will use to invoke ybd

Example Conversion

Assuming you are using the gnu-toolchain branch from buildstream-tests repository as your base, and that you want to build a GNOME system as defined by baserock definitions, you can basically follow these steps:

mkdir /path/to/workdir
cd /path/to/workdir

# Clone everything you need
git clone https://gitlab.com/baserock/definitions.git
git clone https://gitlab.com/baserock/ybd.git
git clone https://gitlab.com/BuildStream/defs2bst.git
git clone https://gitlab.com/BuildStream/buildstream-tests.git

# Get the gnu-toolchain branch
cd buildstream-test
git checkout gnu-toolchain
cd ..

# Here you may want to edit ybd.conf and that sorta thing, note
# that since we need to run ybd, ybd will want to download all of the
# gits into it's mirror so you may want to point your ybd.conf to
# a path where you have already checked everything out.

# Run the conversion
./defs2bst/defs2bst.py --ybd ybd/ybd.py \
                       --definitions definitions \
                       --output buildstream-tests/elements \
                       --rebase strata/build-essential.morph gnu-toolchain.bst \
                       gnome/systems/gnome-system-x86_64.morph x86_64

Cleaning up after the conversion

At this point, you really should build the result and ensure that the conversion worked. While the conversion works 99% of the time, there are some cases it will fail. This is because we prefer a conversion that does not export DESTDIR and PREFIX into the environment.

At this stage there are only a few cases I am aware of where the resulting build can fail.

docutils/xml-catalog.bst

This file will be converted with a failing post-install-commands due to lack of DESTDIR in the environment.

To fix it, simply change post-install-commands like so:

post-install-commands:
- DESTDIR="%{install-root}" ./post-install.sh

gnome/gnome/empathy.bst

This is not really a failed conversion but a random case of builds failing with parallelism enabled, this might fail, but best to fix this with the following addition to the bst file:

variables:
  notparallel: true

Creating the initramfs

Before we can deploy, we'll want an initramfs to include in the boot partition.

You can convert the initramfs system from baserock but the result is ridiculously simple, it's only going to be the gnu-toolchain along with the initramfs scripts.

So I recommend instead to just create this manually, here are the steps involved:

Import the initramfs scripts into your project

You really dont want a separate git repo hold these two important scripts which are just more convenient to hold on to in your repository.

To do this:

git clone git://git.baserock.org/baserock/baserock/initramfs-scripts.git
mkdir ${buildstream}/files/initramfs
cp initramfs-scripts/init ${buildstream}/files/initramfs
cp initramfs-scripts/shutdown ${buildstream}/files/initramfs
git add ${buildstream}/files/initramfs
git commit -m "Adding the initramfs scripts" ${buildstream}/files/initramfs

Besides this, you will need to fix the init script for two reasons:

  • The baserock init script has bugs in it's parsing of kernel command line args
  • We dont create static dev nodes, so you need the script to create /dev/console directly before calling switch_root

Best to just take my copy from the buildstream-tests repository.

Defining the initramfs elements

Here I'll just go over each of the elements involved in creating the initramfs.gz you will need for the boot partition

  • elements/initramfs/initramfs-scripts.bst
kind: import
sources:
- kind: local
  path: files/initramfs
  • elements/initramfs/initramfs.bst
kind: compose
description: Initramfs composition
depends:
- filename: gnu-toolchain.bst
  type: build
- filename: initramfs/initramfs-scripts.bst
  type: build

config:
  include:
  - runtime
  • elements/initramfs/initramfs-gz.bst
kind: script
description: The compressed initramfs
depends:
- filename: initramfs/initramfs.bst
  type: build
- filename: foundation.bst
  type: build

config:
  base: foundation.bst
  input: initramfs/initramfs.bst

  commands:
  - mkdir -p %{install-root}/boot
  - (find . -print0 | cpio -0 -H newc -o) |
    gzip -c > %{install-root}/boot/initramfs.gz

Creating an image deployment

At this stage you should be able to build everything but not deploy.

BuildStream itself does not yet have a convenience element with an understanding of creating bootable images, but we do have a sample deployment using the script element which will be a starting point for creating a more convenient and reusable image deployment element.

This deployment will require a new and separate stack element to provide the tooling you will need to use for the deployment, further we will require that you perform a manual conversion for any install-files extensions you may have been using previously.

We also recommend that you maintain the initramfs boot scripts in your buildstream repository proper as this is usually more practical than having them in a separate git repository.

Element structure

Assuming that you have run defs2bst.py on gnome-system-x86_64.morph as an example, it will have generated the element gnome/gnome-system-x86_64-content.bst

We suggest you first rename this to gnome/gnome-system-content.bst, since you will want to be handling architecture specifics using buildstream's arch conditionals instead of maintaining a separate set of everything.

You will then want to create the following new elements:

  • deploy-tools.bst: A stack element depending on all the tools you need in the deployment
  • gnome/gnome-system-files.bst: A conversion of the install-files, described below
  • gnome/gnome-system.bst: A compose element which will depend on gnome-system-content.bst and gnome/gnome-system-files.bst
  • gnome/gnome-system-image.bst: A script element which will perform the deployment

Below we will discuss all of these elements and how to create them.

Converting install files

Converting the install-files extensions is quite straight forward.

Here is an example of what needs to be done, again using the GNOME system as an example:

# The ${definitions} variable is the old definitions repository
# and the ${buildstream} variable is the new converted project

# Create a local directory to store the files
mkdir -p ${buildstream}/files/gnome-files

# Copy in the install files from the old definitions repo
cp -a ${definitions}/install-files/gnome/* ${buildstream}/files/gnome-files

# Remove the manifest, we dont install that
rm ${buildstream}/files/gnome-files/manifest

# At this point, you should read the manifest and ensure that
# you have all the permissions you want for these files, but
# they are mostly config data which is already 0644 so you
# dont have much to do.
#
# Git will happily record the permissions you set.

Now that you have added the files into your converted repository, create a gnome-system-files.bst element in the gnome subdirectory, the element should look like this:

kind: import
description: Extra files to add to the system

# We depend on this, because we want these files to
# replace anything which was already there.
depends:
- gnome/gnome-system-content.bst

# Use a local source pointing to the files
sources:
- kind: local
  path: files/gnome-files

Creating the deploy-tools stack

This stack can be used as the base for deploying any system using the technique we're going to go over in the final step.

It needs to depend on the following:

  • Bash
  • GNU coreutils, for some shell tools, we will use the truncate and dd commands
  • A recent enough version of e2fsprogs. We need the mkfs programs to support the newer mkfs.ext4 -d option for populating the filesystem image while creating it.
  • syslinux to install the boot loader on a vfat partition
  • mtools is required for copying files into a vfat filesystem
  • util-linux is needed for mkswap
  • parted is required for partitioning

Assuming you are converting a baserock or baserock derived system, then you already have everything except for mtools at your disposal.

You will note that baserock installs syslinux tools itself as a part of the intel based bsp stacks. This is wrong, you dont need syslinux itself installed on the target, so lets just put syslinux and nasm itself into the deploy-tools stack.

Ultimately you will want a deploy-tools.bst in the elements directory that looks like this:

kind: stack
description: Deployment tooling
depends:
- foundation.bst
- tools.bst
- deploy-tools/mtools.bst
- deploy-tools/nasm.bst
- deploy-tools/syslinux.bst

The nasm and syslinux elements are the ones you've moved over from the the automatically converted bsp.

The mtools should look like this:

kind: autotools
depends:
- foundation.bst
sources:
- kind: git
  url: upstream:mtools
  track: mtools-4.0.18
  ref: af0c3edb9706e470b45a9c8dd6debcc9e2d543c2

And that's it, now you have a deploy-tools stack which can be used to deploy images.

The gnome-system composition element

Here we compose a single artifact including everything we want to have on the target system.

Here is what the element looks like:

kind: compose
description: GNOME system
depends:
- filename: gnome/gnome-system-x86_64-content.bst
  type: build
- filename: gnome/gnome-system-files.bst
  type: build
- filename: initramfs/initramfs-gz.bst
  type: build

config:
  # Include only the runtime and locale domains in this composition
  include:
  - runtime
  - locale

The gnome-system-image script element

Finally, the last part is to create a system image creation script which composes a bootable image from our built system entirely in user space.

For this part, we'll want to enhance things by creating new scriptable elements which can be reused with some parameters, but for now the script element is entirely sufficient.

Here is the sauce you need to create a bootable image:

kind: script
description: Create a deployment of the GNOME system
depends:
- filename: gnome/gnome-system.bst
  type: build
- filename: deploy-base.bst
  type: build

variables:
  # Size of the disk to create
  #
  # Should be able to calculate this based on the space
  # used, however it must be a multiple of (63 * 512) bytes
  # as mtools wants a size that is devisable by sectors (512 bytes)
  # per track (63).
  boot-size: 252000K

  rootfs-size: 4G
  swap-size: 1G
  sector-size: 512

config:
  base: deploy-base.bst
  input: gnome/gnome-system.bst

  commands:

  - |
    # Split up the boot directory and the other
    cd /buildstream
    mkdir -p /buildstream/sda1
    mkdir -p /buildstream/sda2

    mv %{build-root}/boot/* /buildstream/sda1
    mv %{build-root}/* /buildstream/sda2

  - |
    # Generate an fstab
    cat > /buildstream/sda2/etc/fstab << EOF
    /dev/sda2   /       ext4   defaults,rw,noatime   0 1
    /dev/sda1   /boot   vfat   defaults              0 2
    /dev/sda3   none    swap   defaults              0 0
    EOF

  - |
    # Create the syslinux config
    mkdir -p /buildstream/sda1/syslinux
    cat > /buildstream/sda1/syslinux/syslinux.cfg << EOF
    PROMPT 0
    TIMEOUT 5

    ALLOWOPTIONS 1
    SERIAL 0 115200

    DEFAULT boot
    LABEL boot

    KERNEL /vmlinuz
    INITRD /initramfs.gz

    APPEND root=/dev/sda2 rootfstype=ext4 rootdelay=20 init=/usr/lib/systemd/systemd
    EOF

  - |
    # Create the vfat image
    truncate -s %{boot-size} /buildstream/sda1.img
    mkdosfs /buildstream/sda1.img

  - |
    # Copy all that stuff into the image
    mcopy -D s -i /buildstream/sda1.img -s /buildstream/sda1/* ::/

  - |
    # Install the bootloader on the image, it should get the config file
    # from inside the vfat image, I think
    syslinux --directory /syslinux/ /buildstream/sda1.img

  - |
    # Now create the root filesys on sda2
    truncate -s %{rootfs-size} /buildstream/sda2.img
    mkfs.ext4 -F -i 8192 /buildstream/sda2.img -L root -d /buildstream/sda2

  - |
    # Create swap
    truncate -s %{swap-size} /buildstream/sda3.img
    mkswap -L swap /buildstream/sda3.img

  - |

    ########################################
    #         Partition the disk           #
    ########################################

    # First get the size in bytes
    sda1size=$(stat --printf="%s" /buildstream/sda1.img)
    sda2size=$(stat --printf="%s" /buildstream/sda2.img)
    sda3size=$(stat --printf="%s" /buildstream/sda3.img)

    # Now convert to sectors
    sda1sec=$(( ${sda1size} / %{sector-size} ))
    sda2sec=$(( ${sda2size} / %{sector-size} ))
    sda3sec=$(( ${sda3size} / %{sector-size} ))

    # Now get the offsets in sectors, first sector is MBR
    sda1offset=1
    sda2offset=$(( ${sda1offset} + ${sda1sec} ))
    sda3offset=$(( ${sda2offset} + ${sda2sec} ))

    # Get total disk size in sectors and bytes
    sdasectors=$(( ${sda3offset} + ${sda3sec} ))
    sdabytes=$(( ${sdasectors} * %{sector-size} ))

    # Create the main disk and do the partitioning
    truncate -s ${sdabytes} /buildstream/sda.img
    parted -s /buildstream/sda.img mklabel msdos
    parted -s /buildstream/sda.img unit s mkpart primary fat32 ${sda1offset} $(( ${sda1offset} + ${sda1sec} - 1 ))
    parted -s /buildstream/sda.img unit s mkpart primary ext2 ${sda2offset} $(( ${sda2offset} + ${sda2sec} - 1 ))
    parted -s /buildstream/sda.img unit s mkpart primary linux-swap ${sda3offset} $(( ${sda3offset} + ${sda3sec} - 1 ))

    # Make partition 1 the boot partition
    parted -s /buildstream/sda.img set 1 boot on

    # Now splice the existing filesystems directly into the image
    dd if=/buildstream/sda1.img of=/buildstream/sda.img \
        ibs=%{sector-size} obs=%{sector-size} conv=notrunc \
        count=${sda1sec} seek=${sda1offset}

    dd if=/buildstream/sda2.img of=/buildstream/sda.img \
        ibs=%{sector-size} obs=%{sector-size} conv=notrunc \
        count=${sda2sec} seek=${sda2offset}

    dd if=/buildstream/sda3.img of=/buildstream/sda.img \
        ibs=%{sector-size} obs=%{sector-size} conv=notrunc \
        count=${sda3sec} seek=${sda3offset}

  - |
    # Move the image where it will be collected
    mv /buildstream/sda.img %{install-root}
    chmod 0644 %{install-root}/sda.img