use buildbot as guts of build system

buildbot is a large and actively developed framework for running automated builds and tests. It seems a natural framework to use for fdroidserver. It is written in Python, handles all sorts of source repositories, manages the whole build process,

It might not have the best support for Android, but that we already have, so that could be integrated into buildbot.

Buildbot is GPLv2 only, while fdroidserver is AGPLv3+. AFAIU those licenses are incompatible. So, no way.

wow, that sucks... it might still make sense to use buildbot, but just call its executables rather than use it as a library. That would also mean we could use the version in Debian, which doesn't support Python 3

Another thing to consider is whether we can use the version of buildbot that is in Debian/jessie, so that it can be deployed on the existing buildserver setup without a major overhaul.

@uniqx

But is licensing an issue here? Can't we just use Buildbot's automatic build feature and upload it somewhere at the end of the build job?

Can't we just use Buildbot's automatic build feature

We can, and IMO it's a good idea design-wise to keep buildbot code separated from fdroidserver code.

assigned to @uniqx

unassigned @uniqx

mentioned in issue #523 (closed)

mentioned in issue admin#164 (closed)

I just looked again at buildbot, Mozilla, Chromium, WebKit and others use it as their core release build infrastructure. I think its the right choice. I think CI systems like Drone or GitLab-CI are unlikely to work as well, since CI usually means you don't have to care too much about security. Also, buildbot is a Python framework, so we can fix it ourselves. GitLab-CI is Ruby/Go and Drone is Go.

We will need to architect it so that the buildbot stuff is run as an executable, since GPLv2-only and AGPLv3-or-later are not compatible, as @relan said. But that seems worthwhile anyway. I think buildbot will only be needed in build and checkupdates anyway.

I think we should not try to force the existing fdroid build process flow onto buildbot, but instead build our buildbot system based on buildbot patterns, then adapt our commands as needed. We will be able to throw away so much buggy, custom code that it will really be worth it.

added 1 deleted label

The essential we need in the buildbot workers is that each build starts in a clean, reset snapshot. It seems that buildbot operates under the assumption that workers are running when they are supposed to be running builds. fdroid build only starts the buildserver VM when it is about to be used. So we'll need to change that assumption. I think we can make buildbot just reset and reboot each worker after it completes a build run. Then once the worker has rebooted, it'll register itself with the buildbot master, which will issue it a new build.

Moving to buildbot will change a lot of the build workflow, but it is clear to me that it will more than pay off.

Here are some projects using buildbot for builds that we can draw from:

• https://github.com/MythTV/ansible
• https://github.com/hicknhack-software/ansible-buildbot

So if we implement a standalone buildserver management tool built on buildbot that is licensed GPLv2-or-later, then we don't have any license conflicts. I think this would make sense anyway, so that we have a toolkit for managing the buildserver that separate from fdroid build.

Part of this consideration is the structure of the buildbot workers. I think they should be libvirt QEMU+KVM instances that use read-only NFS shares for a shared cache of downloaded Android SDK and Gradle binaries. Using NFS means this shared disk is shared over the network, so that buildserver instances can be on multiple bare metal boxes while sharing a single binary cache.

I think a better solution is to use #418 for all caching

@uniqx and I were discussing yesterday, this could be broken up into nice solvable chunks like this:

• Make an fdroid run-this-build subcommand which only runs the build process for a single release from a single app in the current machine it is running in (e.g. not fdroid build --server). This could happen now and would be useful refactoring in any case.
• Make fdroid build just take the command line args and generate a list of builds to run.
• Make a toolkit for creating/running/updating/resetting buildserver instances.
• Configure buildbot to run N buildserver instances, accepting jobs from the list that fdroid build generates, and running builds one at a time using fdroid run-this-build
• Make buildbot reset the VM state after each run, so that if a VM is running, it is either ready to accept a new build, or it is actively running a build.
• Figure out better naming scheme than fdroid build, fdroid run-this-build, fdroid build --server, etc.

mentioned in merge request !814 (merged)

In the process of updating the Repomaker flatpak package, I've been using their builder. It is built with buildbot and it quite nice. It is something we could draw from: https://flathub.org/builds/#/

changed the description

added help-wanted label

marked this issue as related to #872

marked this issue as related to #776

changed title from consider using buildbot as guts of build system to use buildbot as guts of build system

mentioned in issue #872

more fodder https://web.mit.edu/asedeno/www/automating-build-test-and-release-with-buildbot.pdf

mentioned in merge request !879 (merged)

I started writing up this on the wiki https://gitlab.com/fdroid/wiki/-/wikis/Internal/Buildbot/

marked this issue as related to #901

mentioned in merge request !935 (merged)

mentioned in issue #906

changed milestone to %buildbot

added Calyx Institute label

mentioned in merge request !1030

mentioned in merge request !1254 (merged)

mentioned in merge request !1311 (merged)

mentioned in merge request !1283

marked this issue as blocked by admin#383

removed time estimate

assigned to @uniqx and @eighthave

unassigned @uniqx

added NGI0 label

unassigned @eighthave