Kubernetes CI/CD without Auto DevOps Documentation

I don't want to use Auto DevOps. But the only documentation I can find about using Kubernetes with GitLab CI/CD (IE: Actually deploying my project to a k8s cluster) involves Auto DevOps. This is disappointing.

mentioned in issue #58949 (closed)

@tstivers Thanks for the feedback

/cc @marcia

added Category:Kubernetes Management deploy devopsrelease [DEPRECATED] documentation ~2975006 + 1 deleted label

mentioned in issue #59089 (closed)

changed milestone to %12.1

added [deprecated] Accepting merge requests label

@mikelewis This is exactly the issue I was referring to in one of our recent meetings. Auto DevOps is not so straightforward; having to know GKE + k8s is kinda overwhelming for a feature that should work out-of-the-box.

Besides improving the docs we should really look into improving the feature. ¯\＿(ツ)＿/¯

@eread are the Auto DevOps docs in your plans?

Auto DevOps and k8s is ~Configure, so this would go to you @eread. https://about.gitlab.com/handbook/product/categories/#auto-devops--kubernetes-group

added devopsconfigure [DEPRECATED] label and removed devopsrelease [DEPRECATED] + 1 deleted label

added Category:Auto DevOps label

CC @danielgruesso

@marcia We definitely need to take a look at them as part of: &784 (closed).

Let's add this issue to that Epic.

added to epic &784 (closed)

mentioned in epic &784 (closed)

@tstivers what's missing from https://docs.gitlab.com/ee/user/project/clusters/index.html or https://docs.gitlab.com/ee/user/group/clusters/index.html that you need?

@eread, I can't speak for @tstivers, but here are some of our notes:

• The docs for actually making the connection to EKS (I'll focus on EKS as that's where we are) were great. Super easy to follow and reproduce.
• The EKS docs are out of sync with the main k8s docs you linked as they main k8s docs are now relevant for RBAC, but the EKS docs don't mention this
• The docs want us to install tiller, an ingress controller, cert manager, and prometheus. Any k8s cluster that wasn't created yesterday will almost certainly already have these components installed and even your own docs mention the concerns and confusion that can arise by duplicating them

Frankly, in terms of our own actual following of the provided docs, that's where we stopped. There was not enough here to assure us that continuing wasn't going to completely bork our cluster. We have no idea what ingress class name is being used by the gitlab installed ingress controller so it could very well conflict with our installed one. What happens with two tiller's installed? Cert-manager, same question?

Maybe if there were some passage in the docs providing some assurance that there wouldn't be conflicts, we'd have given it a try.

Despite not actually continuing beyond this point, we did examine the rest of the docs for viability, but there really wasn't anything about how to actually USE (i.e. deploy our apps in CI) the attached k8s cluster except auto-devops, which is, I think, the point @tstivers was making. If we're not going to use auto-devops, is there any way to leverage an attached kubernetes cluster with the provided benefits (monitoring and deploy boards being the big ones)? I.e. how do I deploy my application to a configured k8s cluster if I don't use auto-devops? Or is that completely missing the point? Are the attached k8s cluster there ONLY for the purpose of auto-devops?

Finally, we were (are?) seriously considering setting up a cluster JUST for Gitlab to connect to and using only the Gitlab installed control and monitoring applications, which is fine, but that doesn't really solve the problem long term because eventually, we'd need to deploy to production and we're back at square one of having to figure out how to do that within the context of Gitlab's attached k8s clusters or just giving in and stepping outside the context of Gitlab's known k8s clusters.

Thanks @MWatter. Let me loop in @DylanGriffith and @tkuah to take a look through these notes.

@MWatter thanks for this awesome write up. It's really great feedback and as someone so close to the code it's really easy for me to miss the fact that some of this stuff isn't clearly outlined in our features or in our docs so we appreciate this kind of feedback a lot.

Let me provide some answers and some links but it's definitely clear from this we should update docs even if I can answer your questions/concerns now.

First of all the detail that may help the most to know is that you don't need to install any of the Gitlab managed apps via our UI (ie. Tiller, Ingress, Cert Manager,...). You can just add a cluster and be done with it. At the very least this will mean that all of your CI jobs are getting passed a $KUBECONFIG CI variable that they can use for running kubectl commands. Similarly doing this does not require using Auto DevOps at all. Auto DevOps is just a pre-built GitLab CI template and you can write your own .gitlab-ci.yml and this can allow you to do deployments however suits you best.

We do at least have one guide we published a while ago about deploying java apps to K8s using GitLab CI https://about.gitlab.com/2016/12/14/continuous-delivery-of-a-spring-boot-application-with-gitlab-ci-and-kubernetes/. While this may not be exactly what you want it should be illustrative of how to interact with Kubernetes in .gitlab-ci.yml. A quick Google did find a lot of other published blog posts and things too. I don't want to downplay your criticism of our docs here and I do think you're right that we should improve them but this may help unblock you for the moment.

Then if you find you want to use the Kubernetes integration in this way it won't be giving you a lot of benefit apart from creating CI variables for you but you can get more value by labelling your pods so they can be detected by our Deploy Boards. Similarly this labelling should enable web terminals to work and also pod logs.

We have no idea what ingress class name is being used by the gitlab installed ingress controller so it could very well conflict with our installed one

We are using this chart https://github.com/helm/charts/tree/master/stable/nginx-ingress at version 1.1.2 for now and if I'm reading helm inspect --version 1.1.2 stable/nginx-ingress correctly then I think the class name is nginx since we are not overriding this setting.

What happens with two tiller's installed?

This should be fine as long as your Tiller is not installed in the gitlab-managed-apps namespace where we install our Tiller.

Cert-manager, same question?

I actually don't know about this one. Presumably this will cause problems unless cert-manager has some way of deploying them independently (like the class name thing) because otherwise multiple things will be trying to create certificates.

Hopefully that answered all the questions you posed but let me know if you've got any more and we can use this as a starting point for improving our docs.

At the very least this will mean that all of your CI jobs are getting passed a $KUBECONFIG CI variable that they can use for running kubectl commands.

That alone is golden and I didn't even realize it. Thank you!

Then if you find you want to use the Kubernetes integration in this way it won't be giving you a lot of benefit apart from creating CI variables for you but you can get more value by labelling your pods so they can be detected by our Deploy Boards. Similarly this labelling should enable web terminals to work and also pod logs.

Nice!

Thank you for the huge amount of information. We will definitely be putting it to good use.

mentioned in issue gitlab-design#387 (closed)

assigned to @eread

Alright, let's see if we can resolve this issue.

I'm wondering if:

• We need to be clearer here: https://docs.gitlab.com/ee/user/project/clusters/index.html#what-you-can-get-with-the-kubernetes-integration that you get CI/CD with Kubernetes clusters, not just AutoDevOps. So we could perhaps repurpose https://docs.gitlab.com/ee/user/project/clusters/index.html#auto-devops to say "CI/CD using AutoDevOps or your own configuration".
• We need to say here: https://docs.gitlab.com/ee/user/project/clusters/index.html#installing-applications that you can still install applications manually as an alternative.

What are some other must-haves, @DylanGriffith @MWatter @tstivers

@eread My biggest complaint is there's just no indication that Kubernetes integration can be used at all without Auto DevOps. So I'd mostly like to see documentation just as comprehensive as the Auto DevOps documentation, that outlines how to do things like deploying to a Kubernetes cluster without using Auto DevOps.

Thanks.

@eread I believe simply adding a section to the docs to the effect of Deploying your application to kubernetes which walks the user through how to setup up gitlab-ci.yml with a simple build, test, deploy workflow is a great start. Perhaps adding a note regarding Auto DevOps to that section as well. We can use one of our existing applications such as the minimal ruby app.

Thanks for chiming in! That's really helpful.

@ccasella it would also be great and a huge help if we could write a blog post about how you deployed your personal site to kubernetes without Auto DevOps. We could get marketing help here as well.

I'll work on it and let you know asap

That would be cool, and could inform what the docs need to cover.

Thanks!

@ccasella Have you had a chance to progress this at all?

In the absence of a full blog, @ccasella, could you sketch out some steps for me that I could turn into documentation? You know much more about this than me!

Thanks.

Working on it, I'll let you have something as soon as possible (I hope before the end of this week)

@eread

https://gitlab.com/ccasella/manual-devops

This is a real simple project, please let me know if there is any part that you want to improve, expand or if you have any feedback.

Thanks so much, @ccasella.

Would you say your project is a cut-down version of what's here: https://docs.gitlab.com/ee/user/project/clusters/index.html?

In my instance I’m totally not using the GitLab integration. So it’s not following that doc

I wonder if documenting Kubernetes without our integration is outside the scope of our docs?

Any any rate, I've tried to improve the visibility of what we have https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/31081 (http://review-docs-docs-zbwtzs.178.62.207.141.xip.io/ce/user/project/clusters/index.html#deploying-to-a-kubernetes-cluster) with a view to adding more.

removed [deprecated] Accepting merge requests label

changed milestone to %12.2

mentioned in merge request !31081 (merged)

mentioned in merge request !31225 (merged)

mentioned in merge request !31318 (merged)

closed via merge request !31318 (merged)

mentioned in commit 8702dbcd

@eread let's re-open this one and close when we have an example. @ccasella is already preparing a good example at https://gitlab.com/ccasella/manual-integrated-devops/ so I think we're pretty close.

reopened

mentioned in issue #65110 (closed)

Ok, I completed the doc, you can find it in the project README https://gitlab.com/ccasella/manual-integrated-devops. Let me know if you need support or any further information

Oh thanks!

Do we have anywhere more permanent and official such a project could live? @axil seems to be quite across where sample things live.

Now, if I had to come up with some steps for what you did:

1. Enable GitLab Kubernetes integration.
2. Created Kubernetes manifest files, to configure the cluster.
3. Configure .gitlab-ci.yml, with:
   • Building an image using Kaniko, because simple and secure, and pushing it to a Registry.
   • Using ${KUBE_INGRESS_BASE_DOMAIN} to talk to the Cluster, and using kubectl to do the deploy using the configuration.
   • The difference between deploying to Staging and deploying to Production. Is the variables used in the files that kubectl consumes.

Questions:

1. Why are you creating an image in the Build step? Later, you use https://gitlab.com/gitlab-org/cluster-integration/helm-install-image for the deployment.
2. What else would I have to do if I didn't enable the GitLab Kubernetes integration?

@eread that's mostly correct with some clarifications/answers

Using ${KUBE_INGRESS_BASE_DOMAIN} to talk to the Cluster, and using kubectl to do the deploy using the configuration

Technically KUBE_INGRESS_BASE_DOMAIN is just a configuration option that the scripts use to figure out what URL the deployed app will have. It's the resulting base domain of the app you are deploying and not the URL of the cluster. The cluster URL (and other details) are actually in a variable called KUBECONFIG which is implicitly detected by kubectl so doesn't need to be explicitly referenced.

1. Why are you creating an image in the Build step? Later, you use https://gitlab.com/gitlab-org/cluster-integration/helm-install-image for the deployment.

The helm-install-image is used as a base image to run the deployment script. The image being built in the Build step is your app's docker image. So there are 2 images. Your app that will be deployed to the cluster (created in Build). An image that is really just a utility used during deploy stage to get access to the kubectl binary used to communicate with the cluster.

What else would I have to do if I didn't enable the GitLab Kubernetes integration?

You would need to add specific CI variables used by kubectl (in particular KUBECONFIG). You would also need to manually install your own Ingress to route traffic to your application (and possibly provision your own SSL certs if you wanted HTTPS). Even if you did this then you wouldn't get any of the following handled automatically:

1. Separate namespaces and service accounts created automatically for your CI jobs which means projects and environments (about to be merged in https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/30711) can be isolated from each other by default. You could implement this yourself but it would mean manually creating all the namespaces ahead of time and configuring a separate KUBECONFIG CI variable for every single environment.
2. Pod logs, web terminals, monitoring

We have https://gitlab.com/gitlab-examples

we can migrate this project there after the completion!

added groupconfigure [DEPRECATED] label

mentioned in issue #66572 (closed)

mentioned in issue #66594 (closed)

changed milestone to %12.3

added missed:12.2 label

mentioned in issue #66618 (closed)

Hey!

GitLab is moving all development for both GitLab Community Edition and Enterprise Edition into a single codebase. The current gitlab-ce repository will become a read-only mirror, without any proprietary code. All development is moved to the current gitlab-ee repository, which we will rename to just gitlab in the coming weeks. As part of this migration, issues will be moved to the current gitlab-ee project.

If you have any questions about all of this, please ask them in our dedicated FAQ issue.

https://gitlab.com/gitlab-org/gitlab-ee/issues/13855

You can also find more information here:

• https://gitlab.com/gitlab-org/gitlab-ee/issues/13304
• https://about.gitlab.com/2019/08/23/a-single-codebase-for-gitlab-community-and-enterprise-edition/

Frequently Asked Questions

For an up to date list of questions and answers, please take a look at https://gitlab.com/gitlab-org/gitlab-ee/issues/13855

What will we do with the repository names?

https://gitlab.com/gitlab-org/gitlab-ee/ will become https://gitlab.com/gitlab-org/gitlab, and https://gitlab.com/gitlab-org/gitlab-ce will become https://gitlab.com/gitlab-org/gitlab-foss.

Why rename gitlab-ce to gitlab-foss?

Using "gitlab" and "gitlab-ce" would be confusing, so we decided to rename gitlab-ce to gitlab-foss to make the purpose of this FOSS repository more clear

I created a merge requests for CE, and this got closed. What do I need to do?

You will need to create a merge request at https://gitlab.com/gitlab-org/gitlab-ee/. This link will eventually redirect to https://gitlab.com/gitlab-org/gitlab.

How does the licensing work in this new setup?

Everything in the ee/ directory is proprietary. Everything else is free and open source software. If your merge request does not change anything in the ee/ directory, the process of contributing changes is the same as when using the gitlab-ce repository.

Will you accept merge requests on the gitlab-ce/gitlab-foss project after it has been renamed?

No. Merge requests submitted to this project will be closed automatically.

Will I still be able to view old issues and merge requests in gitlab-ce/gitlab-foss?

Yes.

How will this affect users of GitLab CE using Omnibus?

No changes will be necessary, as the packages built remain the same.

How will this affect users of GitLab CE that build from source?

Once the project has been renamed, you will need to change your Git remotes to use this new URL. GitLab will take care of redirecting Git operations so there is no hard deadline, but we recommend doing this as soon as the projects have been renamed.

Where can I see a timeline of the remaining steps?

https://gitlab.com/gitlab-org/gitlab-ee/issues/13304

Will community contributions submitted to the new "gitlab" repository be available in the new gitlab-foss repository?

Yes, these changes will be synced to gitlab-foss automatically, several times per day.

locked this issue