Technical analysis of Bamboo Jenkins importer
Bamboo has created a Jenkins importer (https://confluence.atlassian.com/bamboo/importing-data-from-jenkins-317949257.html, video of it in action at https://www.youtube.com/watch?v=9SGWRWcJTyA. We should do a technical analysis to see how feasible building something with feature parity would be, if so how much work is needed, or if that would even work at all for GitLab.
Result of the research
Overview - translating the differences
- Jenkins home has a
config.xml
which contains Jenkins server information. We "may" not be interested at all in this file. - Inside
$JENKINS_HOME/jobs
we have the definition of a Jenkins job. This does not match to GitLab jobs but it's rather a self contained pipeline where each steps of the job are run sequentially, pretty much like a 1 step per stage. - Translation issues: not all Jenkins build steps and post-build actions directly translate to GitLab jobs. We need to do our best effort to tackle the most important ones first. For those items where the translation is not complete we would need to log it properly for users to understand where the gaps are and how to fix it manually.
- Each step of the Jenkins build has it's own set of parameters and settings. The structure is not as standardized as GitLab jobs. To effectively translate each build we need to create adapters for each of their types. We could start with the basic steps + those added by the most common plugins but we won't be able to cover them all. Having these adapters open-source it means that users can contribute to supporting more of them and improve the importer.
- As it's likely impossible to support all plugins, there will always be something not properly imported. We need to clearly define whether we want the importer to fail or to import as much as it can, skipping steps that cannot be imported. Logging and "import coverage" is essential for user to understand the gaps. This aspect would mark the difference between a broken tool and one that is useful anyway.
- Multiple Jenkins projects can actually build on the same repository. This, in principle, is equivalent to having child pipelines running in parallel. When importing a Jenkins project we should look at what repository is being used and "merge" the resulting pipeline structure with any existing pipeline structures into the
.gitlab-ci.yml
of the repository. For example: ifproject-1-a
andproject-1-b
both build forproject-1
repository. When importingproject-1-a
we build the first version of.gitlab-ci.yml
. Then, when importingproject-1-b
we need to overlap the resulting pipeline structure with the existing one, in a way that they can run in parallel. This could be done leveraging child parent pipelines. - It's important to understand that we cannot achieve 100% translation as some concepts may not even exist on GitLab and vice-versa. We will be looking at translating the max common denominator between Jenkins and GitLab. Implementation of the remaining configurations may vary depending on the user's preferences so the will need to be done manually.
project-1-a:
trigger:
include: project-1-a.yml
project-1-b:
trigger:
include: project-1-b.yml
Below are more details of the main types of pipelines that Jenkins supports by default. There may be more depending on the plugins used though.
Each of these pipelines could be chained together (as we do with multi-project pipelines) or project subscriptions.
Free-style project
The jobs/<job-name>/config.xml
uses a top level <project>
node.
For example. Having the Build
section of a Jenkins project
job1
- shell-script-1
- shell-script-2
- send-slack-notification
should translate to a child pipeline job1.yml
stages:
- step-1
- step-2
- step-3
shell-script-1:
stage: step-1
script: shell-script-1
shell-script-2:
stage: step-2
script: shell-script-2
send-slack-notification:
stage: step-3
script: send-slack-notification
Post-build Actions in Jenkins can map to normal jobs that just occur at the end of the pipeline.
Pipeline
The jobs/<job-name>/config.xml
changes by using a top level <flow-definition plugin="workflow-job@2.36">
to define a different structure.
The General
and Build Triggers
section is the same as the free-style project.
The Pipeline
section is where the pipeline script, as in a Jenkinsfile
is defined. This can use:
- declarative syntax
- scripted syntax
Parsing pipeline scripts is very hard. Can we evaluate this script using a Jenkins sandbox?. This could be possible by using https://lackastack.net/posts/gitlabci-jenkinsfile from #208277 (closed) and build a script that generates a Jenkins sandbox container based on the customer's version of the Jenkins ecosystem.
Multi-configuration project (Matrix Project Plugin)
The jobs/<job-name>/config.xml
changes by using a top level <matrix-project plugin="matrix-project@1.14">
to define a different structure.
This a typical matrix-style project to allow multiple combinations of configurations.
The General
, SCM
, Build Triggers
, Build
and Post-build Actions
is the same as the free-style project.
The main difference is the definition of Matrix-style configurations where user defines axis and values and produces configurations that run in parallel where for each configuration, each step runs in sequence (like a free-style project).
How does this level of import relate to running a Jenkinsfile?
Translating information from config.xml
applies to a higher level translation than either #208276 or #208275. The declarative and scripted Jenkinsfile
translator would be responsible to turn part of the Pipeline Jenkins item to a .gitlab-ci.yml
snippet. This issue is more about translating a higher level of details to a .gitlab-ci.yml
.
example: translate a Jenkins Pipeline project to GitLab
Like every Jenkins projects a Pipeline will have a General
section which contains some metadata. We might not be able to translate some of these configurations but some may be translatable like importing SLACKTOKEN
as pipeline variable.
Build triggers
: e.g. is this a pipeline to be triggered on push
event? Or should we register a scheduled pipeline with the same cronjob? Is this pipeline triggered when another pipeline triggers? (e.g. multi-project pipeline)
and finally the Pipeline
section which is where a user can input the content of a Jenkinsfile
or a path to a Jenkinsfile
inside the workspace.
The issues #208276 and #208275 are only focusing on translating the content of a Jenkinsfile
, which is the last section above.
All the configurations set via the UI are persisted in a config.xml
file. An importer like Bamboo looks at the config.xml
file and tries to replicate a pipeline or sequence of builds. I'm not sure whether Bamboo is able to import and translate Jenkinsfile
s. I couldn't find any documentation about it. Bamboo also requires the Jenkins Build Pipeline plugin to be installed in the Jenkins instance prior to reading the config.xml
. What this plugin does is allowing to connect a sequence of Jenkins projects/builds in a form of a pipeline. These connections between builds (e.g. run project-B after project-A) seem to be saved in the config.xml
.
In short, a Bamboo-like importer would import static project configurations but not a translated form of a Jenkinsfile
which is a pipeline as a code. This importer would work well for Free-style project or Multi-configuration project. In addition, for Pipeline projects, this importer could carry some extra information such as whether a pipeline is triggered on push
or via a crontab shedule.