If a dockerfile was requested for a package that does not request any CrossPackage,
lcitool would incorrectly generate a line like "eatmydata apt-get install --no-install-recommends -y []".
Fix this by omitting the line whenever the cross_pkgs are empty in the varmap.

Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>

Right now, the same key is used for both a native and a cross x86_64 target.
The two however can have different packages when the build is done on a non-x86_64
host.

To fix this, change the cache key for native targets to not include the "-x86_64"
suffix.  The set of packages for native targets, as far as lcitool is concerned,
is the same for all host architectures, so there is no reason to mention x86_64.

Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>

This patch adds an abstract class for containers.
This class will provide methods such as "build"
and "run" which would be used for building container
images and running workloads in a container.

Signed-off-by: Abdulwasiu Apalowo <abdulwasiuapalowo@gmail.com>

This patch defines the separate Docker and Podman
classes by subclassing the Container class.

The engine classes are imported in "__init__.py"
in order to import them directly from
"lcitool.containers"

Signed-off-by: Abdulwasiu Apalowo <abdulwasiuapalowo@gmail.com>

This patch introduces a couple of classes to
provide various error message for different
container action. "ContainerError" class is
subclassed to create these classes.

{Podman,Docker}RunError for error during
"{podman,docker} run".

{Podman,Docker}BuildError for error during
{podman,docker} build".

Exception instance attribute is also added to
Container, Podman and Docker.

Signed-off-by: Abdulwasiu Apalowo <abdulwasiuapalowo@gmail.com>

This patch introduces a static method called
"_exec" into the Container class. This method
will run engine commands through the
subprocess.run method.

Signed-off-by: Abdulwasiu Apalowo <abdulwasiuapalowo@gmail.com>

The helper method checks whether a container engine
(docker, podman) is available to use. It does it by
running `/path/to/engine version`

Signed-off-by: Abdulwasiu Apalowo <abdulwasiuapalowo@gmail.com>

This is a public wrapper around the "_check()" method.

Signed-off-by: Abdulwasiu Apalowo <abdulwasiuapalowo@gmail.com>

This is a convenience wrapper around the `pwd.getpwnam`
and `pwd.getpwuid` library methods, returning the standard
passwd 7-tuple which will by used in future patches to set
the correct container engine options.

Signed-off-by: Abdulwasiu Apalowo <abdulwasiuapalowo@gmail.com>

Generate some of the options needed to run workloads
in a container. It replaces some of the logic in
`libvirt.git/ci/Makefile` and adds a simpler interface
with the use of "datadir".

If `datadir` is supplied, it is mounted to the
"datadir/" directory in the user's home directory inside
the container.

if `script` is supplied, it is copied into a temp. directory,
and also mounted to the "script" file inside the
container. It is required to kick-start the operation.

NB: The working directory is changed to the home directory if
--workload-dir or --script is supplied to the CLI options

/etc/{passwd,group} files are copied to a temporary dir
(temp_dir). The respective files are then mounted to
/etc/{passwd,group} in the container.

Signed-off-by: Abdulwasiu Apalowo <abdulwasiuapalowo@gmail.com>

Adds a private method called "_extra_args" that is
used to perform podman user namespace manipulation.

The workdone here is taken from `libvirt.git/ci/Makefile`
(libvirt's current container handling Makefile). It converts
the podman namespace logic into Python code for use with
lcitool's container support.

Signed-off-by: Abdulwasiu Apalowo <abdulwasiuapalowo@gmail.com>

This patch overrides "_build_args" in Podman class
to return a list containing engine options and the
podman-specific id mapping as options passed to the
podman engine.

Signed-off-by: Abdulwasiu Apalowo <abdulwasiuapalowo@gmail.com>

This is used to remove an image from the engine's
local repository. This is especially useful during
image build to ensure that images don't contain stale contents.

Signed-off-by: Abdulwasiu Apalowo <abdulwasiuapalowo@gmail.com>

This returns a string in JSON format which contains details
of all the images contained in an engine's local repository
excluding all dangling image(s).
This is achieved by running
    <engine> images --format "{{json .}}" --filter dangling=true

Signed-off-by: Abdulwasiu Apalowo <abdulwasiuapalowo@gmail.com>

Deserialize string in JSON format to Python list.

Signed-off-by: Abdulwasiu Apalowo <abdulwasiuapalowo@gmail.com>

Deserialize string in JSON format to Python list.

Signed-off-by: Abdulwasiu Apalowo <abdulwasiuapalowo@gmail.com>

This introduces a method that checks if an image exists
in podman's local repository.

Signed-off-by: Abdulwasiu Apalowo <abdulwasiuapalowo@gmail.com>

This introduces a method that checks if an image exists
in the docker's local repository.

Signed-off-by: Abdulwasiu Apalowo <abdulwasiuapalowo@gmail.com>

The method ultimately translates to
"<container_engine> run {options} {image}"

Signed-off-by: Abdulwasiu Apalowo <abdulwasiuapalowo@gmail.com>

Adds a public "build" method to Container class.
This method will be used for building container
images from a Dockerfile.

Signed-off-by: Abdulwasiu Apalowo <abdulwasiuapalowo@gmail.com>

Add helper method to get the appropriate engine class.

Signed-off-by: Abdulwasiu Apalowo <abdulwasiuapalowo@gmail.com>

This patch adds an optional argument called
"engine". The default value is "podman".

It will be used by the "run" and "shell"
sub-commands to determine the engine to use(from the
list of available engines) for image build and/or
running workloads on containers.

e.g --engine podman OR --engine docker

Signed-off-by: Abdulwasiu Apalowo <abdulwasiuapalowo@gmail.com>

This is the absolute path to the directory which is
mounted as the "datadir/" folder in the user's
home directory inside the container.

The path is expected to contain all the files and
folders required to perform any workload in the
container. In order to build a project, the path
passed to --workload-dir would contain the source code
of the project and other necessary files and folder.

The logic behind providing a --workload-dir argument is to
provide a simpler interface to a directory which would
contain all the data required to perform an operation inside
the container rather than having to pass different arguments
such as the home directory, working directory, source-code
directory et.c to be mounted inside the container.

e.g --workload-dir /tmp/container_dir

Signed-off-by: Abdulwasiu Apalowo <abdulwasiuapalowo@gmail.com>

The script argument is the path of the executable
file which is used to run the workload contained in
--workload-dir CLI option. It can either be a
relative or absolute path.

It is the orchestrator of the content of WORKLOAD_DIR
(directory which contains all the files and folders
required to perform any workload on the container).

NB: The file is mounted into the container and can be
run in the container as a standalone workload (e.g an
"echo hello world" script)

e.g --script script.sh

Signed-off-by: Abdulwasiu Apalowo <abdulwasiuapalowo@gmail.com>

This adds "--env" optional argument which is an
option to "run" and "shell" subcommand.

"--env" denotes the environmental variables which is
set in the container. It can take multiple values e.g
--env FOO=bar --env BAR=baz

Signed-off-by: Abdulwasiu Apalowo <abdulwasiuapalowo@gmail.com>

This adds the "--user" optional argument which is
an option to "run" and "shell" subcommand.

"$USER" is the current user running in the container.
It can either be an integer (which is the UID) or
a string (which is the username). e.g
    --user 0 OR --user root

Signed-off-by: Abdulwasiu Apalowo <abdulwasiuapalowo@gmail.com>

This patch introduces the image argument for
representing the image to use. It could be
names, image IDs, full registry paths and tags.

e.g registry.gitlab.com/ci-fedora-36:latest

Signed-off-by: Abdulwasiu Apalowo <abdulwasiuapalowo@gmail.com>

This is to prepare for the introduction of `container_projectopt`
(for container's "--projects" option) and to differentiate
where each one is used

Signed-off-by: Abdulwasiu Apalowo <abdulwasiuapalowo@gmail.com>

This patch introduces the --projects argument
which accepts a glob of lcitool supported projects.

It is used with --target argument to generate a
Dockerfile and build image from scratch.

e.g --projects libvirt*

Signed-off-by: Abdulwasiu Apalowo <abdulwasiuapalowo@gmail.com>

This patch adds a "_validate" function which validates
the CLI argument by parsing the command line to
ensure that the required arguments are passed to
lcitool

Signed-off-by: Abdulwasiu Apalowo <abdulwasiuapalowo@gmail.com>

This adds another top-level command "container"
which will introduce other subcommands in later patches.

The "container" command allow the ability to plug
another subcommand, which nests lcitool two levels deep.
i.e
    lcitool container {command} -h

Signed-off-by: Abdulwasiu Apalowo <abdulwasiuapalowo@gmail.com>

This adds the "engines" container subcommand
which is used to list all the available and
usable engines.

It allows users to decide on the engine to
select when spinning up containers and/or
building images.

Signed-off-by: Abdulwasiu Apalowo <abdulwasiuapalowo@gmail.com>

This adds the "build" subcommand which is used to
build container image from args.projects and
args.target.

The Dockerfile string generated from using the
`DockerfileFormatter` class is copied to a temporary file
which is passed to the build command.

Tags are generated by prefixing the target name with
"lcitool.", e.g "lcitool.fedora-36" to prevent trashing
the output of "podman images" with incomprehensible tags.

If an image is present, it is removed before starting another
build to ensure the image is built from scratch.

Ensure that (--target & --projects) arguments are passed with
"build" subcommand.

Signed-off-by: Abdulwasiu Apalowo <abdulwasiuapalowo@gmail.com>

This adds the "run" subcommand to lcitool
which is used to run workload in a container
It is run with:
    lcitool container run --help

Validate the command line arguments for "run"
to ensure that --script is required, and it
works with and without --workload-dir argument

It also stops running if image doesn't exist locally.

Signed-off-by: Abdulwasiu Apalowo <abdulwasiuapalowo@gmail.com>

Adds "shell" subcommand to the container command.
It is run with:
    lcitool container shell --help

The subcommand works standalone, but also accepts --script
and --workload-dir options.

Signed-off-by: Abdulwasiu Apalowo <abdulwasiuapalowo@gmail.com>

Check that "lcitool container engines" returns
a status code of zero (0).

Signed-off-by: Abdulwasiu Apalowo <abdulwasiuapalowo@gmail.com>

Test case for Podman()._extra_args(). This also tests
Container()._passwd method using mocked `pwd` module.

Signed-off-by: Abdulwasiu Apalowo <abdulwasiuapalowo@gmail.com>

Test that {Podman,Docker}._build_args method
returns the correct list.

Signed-off-by: Abdulwasiu Apalowo <abdulwasiuapalowo@gmail.com>

Extensive documentation on the container functionality of
lcitool.

Signed-off-by: Abdulwasiu Apalowo <abdulwasiuapalowo@gmail.com>

'warn' is no longer an accepted keyword argument for the builtin
command module and without it newer Ansible (ansible-core 2.12+) fails
with:

"Unsupported parameters for (ansible.legacy.command) module: warn.
Supported parameters include: stdin_add_newline, removes, creates,
stdin, chdir, executable, _uses_shell, strip_empty_ends, argv,
_raw_params."

Signed-off-by: Erik Skultety <eskultet@redhat.com>

Fedora has supported this mingw package for many years.

Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>

Paolo Bonzini authored Nov 08, 2022 and Erik Skultety committed Jan 17, 2023

Now that the Projects object is propagated down from Application, make
the extra data directory part of the object instead.

Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>

Paolo Bonzini authored Nov 08, 2022 and Erik Skultety committed Jan 17, 2023

This provides more freedom to use Pathlib when loading the facts
for the projects and inventory.  Without this kind of early
loading, the monkeypatch can potentially cause the load to fail.

Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>

Paolo Bonzini authored Nov 08, 2022 and Erik Skultety committed Jan 17, 2023

Instead of passing down just a str|None, create a class that
looks for files both with pkg_resources and in the user
data directory.

Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>

Paolo Bonzini authored Dec 08, 2022 and Erik Skultety committed Jan 17, 2023

This simplifies the code, removing the usage of pkg_resources.  There is
no command line interface change yet; the data directory is *not* used
to allow local mappings, as the file is always marked as internal.

Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>

Paolo Bonzini authored Dec 08, 2022 and Erik Skultety committed Jan 17, 2023

This simplifies the code, removing the usage of pkg_resources.
There is no command line interface change yet; the data directory
is *not* used to allow local mappings.

Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>

The 'black' tool provides opinionated formatting of python code.
This adds a minimal container for running 'black' as a CI job.

Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>

This allows the manifest to request use of 'black' for python code
formatting checks.

Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>

The 'flake8' tool provides opinionated code style checks of
python code. This adds a minimal container for running 'flake8'
as a CI job.

Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>

This allows the manifest to request use of 'flake8' for python code
formatting checks.

Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>

Erik Skultety authored Jan 17, 2023 and Daniel P. Berrangé committed Jan 23, 2023

Some of the modules (mostly new) were missing the license identifiers.

Signed-off-by: Erik Skultety <eskultet@redhat.com>

Fix module comments in "lcitool/containers/docker.py" and
"lcitool/containers/podman.py"

Signed-off-by: Abdulwasiu Apalowo <abdulwasiuapalowo@gmail.com>

Add copyright information to some modules in 'lcitool/containers'.

Signed-off-by: Abdulwasiu Apalowo <abdulwasiuapalowo@gmail.com>

EPEL9 retired capstone as soon as it was added to c9s for
the future RHEL-9.2 release. This dooms anyone still using
RHEL < 9.2 who still needs capstone from EPEL, so we
workaround by manually picking the most recent version

This relies on the fact that DNF accepts URLs to package
downloads, as an alternative to plain package names. This
is good enough for generating containers, but could break
if we get other code in lcitool that expects it to always
be a literal package name.

https://bugzilla.redhat.com/show_bug.cgi?id=2124181


Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>

In Alpine 3.16, there are php81 packages in parallel with php8

In Alpine Edge, there are php82 packages in parallel with php81,
and no php8.

We can't use the php82 packages yet, however, since the pecl-imagick
package isn't available.

So this switches to the php81 stack for 3.16 and Edge. Edge will
need updating again to php82 when it is fully available.

Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>

This package was recently discontinued:

  * Sat Dec 17 2022 Dirk Müller <dmueller@suse.com>
  - update to 5.2.10:
  - drop unused xz-devel-static which is no longer supported when using
    - -with-pic (which is needed for shared libs)

Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>

Alpine ships many versions of llvm in parallel. There is no reason why
we need to fixate on version 11 though, especially since it is dropped
in Alpine Edge. Switching to the unversioned llvm alias lets apk pick
the newest version per releae stream.

Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>

This is available since 3.14 at least.

Further, alpine edge now refuses to install any packages using
'pip' unless a virtualenv is utilized, so the pip fallback is
unusable.

Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>

Marc-André Lureau authored Jan 09, 2023 and Paolo Bonzini committed Feb 08, 2023

Signed-off-by: Marc-André Lureau <marcandre.lureau@redhat.com>

This reverts commit 50d7df99.
New versions of Sphinx (6.x) actually *require* docutils>=0.18,
and the readthedocs theme has therefore been updated as well.

Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>

Paolo Bonzini authored Dec 08, 2022 and Erik Skultety committed Feb 10, 2023

Allow specifying individual projects to override the version they require
of a required version of a PyPI or CPAN package.  Projects that want
to test with an old version of Meson could therefore disable the distro
version and install the old version directly from PyPI.

Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>

Paolo Bonzini authored Nov 08, 2022 and Erik Skultety committed Feb 10, 2023

Allow overriding target facts in a project.  This is useful to specify
the paths to a specific version of Python.

Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>