### Problem to solve Converting Jenkins installations into GitLab can be complex and error prone. What if instead, we just made it easy to put Jenkins in a little box and run it inside of GitLab while providing it the right inputs and gathering its outputs? It might be the best of both worlds to support an ongoing migration. Then, we can follow up to take the longer effort of writing various translators which will have more complexity. ### Intended users <!-- Who will use this feature? If known, include any of the following: types of users (e.g. Developer), personas, or specific company roles (e.g. Release Manager). It's okay to write "Unknown" and fill this field in later. * [Rachel (Release Manager)](https://about.gitlab.com/handbook/marketing/product-marketing/roles-personas/#rachel-release-manager) * [Parker (Product Manager)](https://about.gitlab.com/handbook/marketing/product-marketing/roles-personas/#parker-product-manager) * [Delaney (Development Team Lead)](https://about.gitlab.com/handbook/marketing/product-marketing/roles-personas/#delaney-development-team-lead) * [Sasha (Software Developer)](https://about.gitlab.com/handbook/marketing/product-marketing/roles-personas/#sasha-software-developer) * [Presley (Product Designer)](https://about.gitlab.com/handbook/marketing/product-marketing/roles-personas/#presley-product-designer) * [Devon (DevOps Engineer)](https://about.gitlab.com/handbook/marketing/product-marketing/roles-personas/#devon-devops-engineer) * [Sidney (Systems Administrator)](https://about.gitlab.com/handbook/marketing/product-marketing/roles-personas/#sidney-systems-administrator) * [Sam (Security Analyst)](https://about.gitlab.com/handbook/marketing/product-marketing/roles-personas/#sam-security-analyst) * [Dana (Data Analyst)](https://about.gitlab.com/handbook/marketing/product-marketing/roles-personas/#dana-data-analyst) Personas are described at https://about.gitlab.com/handbook/marketing/product-marketing/roles-personas/ --> ### Further details Previously discussed in the gitlab~4107678 epic https://gitlab.com/groups/gitlab-org/-/epics/2735#note_295172334 ### Proposal Our initial MVC will be oriented around the simultaneous work to document @ctimberlake1's blog post (https://lackastack.net/posts/gitlabci-jenkinsfile/) that is happening in https://gitlab.com/gitlab-org/gitlab/issues/37032. This currently is an entirely manual process, but offers some opportunities for automation to make it easier to do. As such, our plan is to start with a CLI in https://gitlab.com/gitlab-org/gitlab/issues/208848 to achieve the first part of the configuration needed in the blog post. This tool would: * Run on the customer's current Jenkins server * Be pointed at the Jenkins home directory and the needed jenkinsfile-runner version * Once triggered, would create and push a container to our registry that can be used as the runtime environment used in the remainder of the blog post. Official documentation being delivered in https://gitlab.com/gitlab-org/gitlab/issues/37032 would then need to be updated to ensure it has the latest details on: * How to create a runner capable of running the container (there are sizing concerns, and a shared runner would likely not be sufficient) * How to use set up your pipeline correctly to run your Jenkinsfile in the container * How to make use of some GitLab features, e.g. uploading artifacts created by the Jenkinsfile jobs, capturing logs, or running only in the presence of a Jenkinsfile for example. There is already a proof of concept set up following the blog post documented at https://gitlab.com/gitlab-org/gitlab/issues/37032#note_298623934 which can be referred to here. ### Where does this MVC end Adding features to this wrapper could go on and on until we've fully embedded Jenkins inside of GitLab, so it's important to call out what our target is for now: - Implement the wrapper MVC per https://gitlab.com/gitlab-org/gitlab/-/issues/208848, and on to potentially everything documented in https://gitlab.com/gitlab-org/gitlab/issues/37032 - Provide documentation on how to use the tool (https://gitlab.com/gitlab-org/gitlab/issues/37032) - Show a summary of what succeeded or failed so a decision can be made on how to proceed Once this is working, feedback on additional features, including translators, will be evaluated as needed. Our target is to be able to import a meaningful amount (20%) of the existing Jenkins jobs, and would require some manual troubleshooting. We are not aiming for 100% automated due to the complexity that grows in supporting more and more use cases. In the end we want the use of the wrapper to be a good, but not permanent, experience. It gets you through the migration but not lets you run things forever there. ### Research notes Additional notes from @ctimberlake1: > One of the issues with remaking the pipelines is that some of our customers have massive pipelines. One of our major and strategic partners - Who has requested this - I just learned has over 200,000 Jenkins Pipelines that are older. There's no possible way to move all of those over by hand, and even if we narrowed it down to active pipelines and critical pipelines, I suspect that number will still be out of range. > > What if we look at the spirit of this request as opposed to the letter of the request? We merely want to run Jenkins pipelines in GitLab Seemlessly. > > I've done a lot of thinking on this; It's something important to me, and I'd like to help however I can. Hear me out - We're thus far approaching this from the perspective of converting the pipelines. We're also lumping the pipelines and process together, or atleast I have been up until now. > > *Speaking on the basis of Jenkins Groovy/Imperative Pipelines only:* > > Consider the downsides of an interpreted conversion. We'll never be able to 100% interpret and convert an Jenkins Imperative pipelines, as it's basically an application unto itself. That means there will be hand holding no matter what. This shifts the burden of maintaining, and conversion onto the individual pipeline. So if something fails or is incompatible, It's with that individual pipeline. We'll be tracing these issues forever. Customers will be upset and we'll encounter "Well, all X of my pipelines work, but this one doesn't.", and Teams with "Well, we tried it but 2% of our pipelines didn't work, so we're staying with Jenkins." > > **New/Old Proposal?** Linked above is an article I wrote about integrating Jenkins and running it inside of GitLab as an image. Jenkins+Jenkins Worker as one component that can be invoked by GitLab-Runner and ran. I mention numerous pitfalls and issues with it. But what if we could use that process as a jumping off point to build a compatibility layer inside of GitLab and the GitLab Runner, where we could run any/all Jenkins Pipelines inside it. > > This would of course require GitLab-Runner to have enough ram and resources to spin it up. It would also require some configuration of plugins from the user. But this is something we could potentially automate. The upside to this is we would have 100% Compatibility now and into the future. We would also have a single point of failure to maintain and manage. > > We ship a GitLab Docker Image w/ Jenkins installed in it. Then we pull any plugins/config necessary for Jenkins from a repo's individual `.jenkins` directory. The GitLab-Runner spins up the image; The image then does a git pull, moves the `.jenkins` directory into jenkins_home, spins up Jenkins, runs the CI, and dies. > > For companies with more established versions of Jenkins or support. We include instructions on how to build their own Jenkins Docker image based off ours that doesn't require a `.jenkins` directory, they can just plugin all their plugins and configuration in that image and call it good. > > Then, when a Repo has a .Jenkinsfile in it, we can offer the option to "Import Jenkins", which will drop a preconfigured `.gitlab-ci.yml` file in the repo that is setup to used Jenkins. We can even leverage the Jenkins Integrations and improve it to pass data to/from GitLab. ### Permissions and Security <!-- What permissions are required to perform the described actions? Are they consistent with the existing permissions as documented for users, groups, and projects as appropriate? Is the proposed behavior consistent between the UI, API, and other access methods (e.g. email replies)?--> ### Documentation <!-- See the Feature Change Documentation Workflow https://docs.gitlab.com/ee/development/documentation/feature-change-workflow.html Add all known Documentation Requirements here, per https://docs.gitlab.com/ee/development/documentation/feature-change-workflow.html#documentation-requirements If this feature requires changing permissions, this document https://docs.gitlab.com/ee/user/permissions.html must be updated accordingly. --> ### Availability & Testing <!-- This section needs to be retained and filled in during the workflow planning breakdown phase of this feature proposal, if not earlier. What risks does this change pose to our availability? How might it affect the quality of the product? What additional test coverage or changes to tests will be needed? Will it require cross-browser testing? Please list the test areas (unit, integration and end-to-end) that needs to be added or updated to ensure that this feature will work as intended. Please use the list below as guidance. * Unit test changes * Integration test changes * End-to-end test change See the test engineering planning process and reach out to your counterpart Software Engineer in Test for assistance: https://about.gitlab.com/handbook/engineering/quality/test-engineering/#test-planning --> ### What does success look like, and how can we measure that? <!-- Define both the success metrics and acceptance criteria. Note that success metrics indicate the desired business outcomes, while acceptance criteria indicate when the solution is working correctly. If there is no way to measure success, link to an issue that will implement a way to measure this. --> ### What is the type of buyer? <!-- Which leads to: in which enterprise tier should this feature go? See https://about.gitlab.com/handbook/product/pricing/#four-tiers --> ### Links / references