For application development in containers, you want an environment that you can freely mess around in, has development dependencies and possibly even credentials to push directly to a repository.
GitLab can provide default development container images that allow you to quickly work on a project. In the projects repository you then work on building a production container
Proposal
A way to specify a development container, such as the following in .gitlab-ci.yml:
development:-ruby-on-rails
Which will spin up a container for you with this image, that you can open a terminal for to do development in.
The container has ssh credentials, so you can push to GitLab.
In the future, a Web IDE or local IDE could connect to this container to provide your entire dev environment.
Details
You can also specify your own container for development:
The repository that contains the code is used for development and to build a production container, the production container image is added to the container repository. So we can install git credentials to the development container.
I name this practice 'split container'. It means you have two containers:
The development one, this has all development dependencies, similar to registry.gitlab.com/gitlab-examples/openshift-deploy
The production one, this is the container that is added to the registry under the project name
GitLab will provide development containers for all common platforms. You can also provide your own development container by specifying it in .gitlab-ci.yml. But the repo and images of your custom container are not related to your application code. This because you can reuse the development container across different projects.
When you create a project, select the auto deploy template, and press the terminal button it should be possible to get the development container. Maybe we can define a 'development' stage in our template to accommodate this.
Designs
Child items
...
Show closed items
Linked items
0
Link issues together to show that they're related or that one is blocking others.
Learn more.
says about adding extra packages in order to push back. However, in most cases it will basically not work, because you do run deployment with compiled application (ex. Go) and you may not really have an access to source codes. Secondly, credentials. You don’t really have credentials to push back (no SSH keys), so you would be forced to use HTTPS which if you have 2FA enabled will not work either, because you would have to use Personal Access Token.
The more I think about it, the more I feel this is the wrong approach. It makes a great demo, but it's bad best practices. You don't need git on your production servers, so why include it in your base docker image?
@ayufan@markpundsack I agree that you can't use our terminal to develop applications if there are build dependencies. And making git access work is hard. But it is a step.
I think the future is that production and development environments will look very much the same. And that development will look much more like production (multiple containers in the cloud). I'm not sure how to incorporate development dependencies, maybe with something like using docker container layers.
All the cloud ide's started with special development environments and failed (Nitrous shut down today https://news.ycombinator.com/item?id=12841489 ). We'll start from the production environment and will be successful because this is more representative of the end state.
TLDR; it is easier to iterate from production to a dev environment than is the other way.
@sytses So I can see the argument for having production and development environments look the same. But I still feel like this is missing something. Let me see if I can explain.
When developing locally, you have a local box with some development tools such as git, docker, and docker-compose. You then run docker-compose up and start N docker services. Those docker services may very well be identical to production. But you still need the local box to develop those services.
What you're asking for with this issue is to ssh into one of those running production services, and develop inside that container. I'm not sure that's ever going to be a best practice.
In our case, the cloud version of that local box is closer to our GitLab runners. The runners grab the latest copy of the code, run whatever is needed to build (and test and deploy) the app.
@sytses From a Slack conversation, I now understand that you see a future where development skips docker compose, and uses kubernetes, just like production.
Sidecar containers that load the git repository and any needed binaries might be part of the answer. The production image would be "pure".
@markpundsack can you make a proposal? right now we're deploying a very simple static website instead of application and I'm afraid people won't be able to make the translation.
I'm still having trouble thinking what the MVP here is. We can certainly add RUN apk install git to the Dockerfile, but then there's a question of credentials. We don't need to preload SSH keys or anything; we could rely on https login/password, but then doesn't that get stored somewhere like ~/.gitconfig? And then those creds would be on a public-facing machine. That alone would be a security risk I wouldn't recommend. It would also mean if someone else opened up the terminal on that same pod, they'd have access to someone else's creds. Using the app's image for development is just scary and fundamentally wrong at a certain level. @ayufan's proposal of creating a specific dev image, and declaring it in your .gitlab-ci.yml is the most promising proposal I've seen. Sidecar loading of a dev addon with Kubernetes is also intriguing. But neither of these options is easy.
Edit: I guess git has a cache mode where it will keep creds in memory for 15 minutes. Maybe that's enough? It'll be fine for a demo, at least. Let's give it a shot and see how it feels. At the very least, it'll give us a sense of what the future will be like, even if it's not practical right now.
TLDR: Personally I liked the demo in Amsterdam done by @sytses until he clicked the "magic" terminal button and started to update Hello World directly in the "environment". I already heard enterprise customers screaming "why do we have processes, environments and QA teams when any team superhero can jump directly into environment and "tweak" things that are maybe already tested and verified!"
I do think production and development will and should get closer together but I would nuke "terminal" button on environments page and restrict all access to staging and production environments other than through pipeline defined steps that allow proper compilation, minification, test and validation steps before application is deployed to the environment. CI/CD pipeline is great value - lets keep the value.
However, I do agree that developers need their special "developer" environment that allows them to go in and tweak and try things in any step of the process. This environment can also be created and shown by default for every team member under "Environments" in Gitlab, it is backed by full .gitlab-ci.yml definition to ensure that their environment definition is as similar as possible (as this is the holy goal right?) like all other environments. But it should be something just for that specific logged-in user. People can not see each-others developer environments.
Could it be possible to have developers to use their own "private" runners, that would allow them to jump into a private runner (that gets code anyway during CI) and start tweaking things there, re-running runner (or entire pipeline) when they want to see how does the change they just created affects the system. Here, and only here, on special "developer environment" page you would have terminal button or other ways listed, how to connect with your favorite IDE to the private runner so that you can edit things. These runners, as they would be "private" (maybe even run on dev machines only) could have env variables defined during runner installation that would allow authentication and authorization of the git and other operations from those runners to perform actions as a certain developer that "owns" them.
@perica-zivkovic Thanks for commenting. I agree that many users do not want to give all of their developers access to production. I'm sure that we can evolve a solution that accounts for that.
Thanks @sytses, I have been looking a bit for similar solutions and found a nice web based editor engine that Microsoft is using for Visual Studio online - Monaco Editor.
Based on this some people created GHEdit (MIT licensed source code available) that claims it can do in place source code browsing and editing on GitHub project. I did not had time to try it out but if someone could find the time to fork it as GLEdit it might be possible to integrate it with GitLab
The repository that contains the code is used for development and to build a production container, the production container image is added to the container repository. So we can install git credentials to the development container.
I name this practise 'split container'. It means you have two containers:
The development one, this has all development dependencies, similar to registry.gitlab.com/gitlab-examples/openshift-deploy
The production one, this is the container that is added to the registry under the project name
GitLab will provide development containers for all common platforms. You can also provide your own development container by specifying it in .gitlab-ci.yml. But the repo and images of your custom container are not related to your application code. This because you can reuse the development container accross different projects.
When you create a project, select the auto deploy template, and press the terminal button it should be possible to get the development container. Maybe we can define a 'development' stage in our template to accommodate this.
I really like the idea of formalising the concept of two containers. It makes sense to give it a unique name. I'm not sure whether "split containers" is the best name for it. How about "two phase containers"? Because one of them is for building and the other for running so these are two phases.
Job van der Voortchanged title from Split containers to Split containers: separate dev / prod containers
changed title from Split containers to Split containers: separate dev / prod containers
whose git credentials would we load on the container?
We can generate credentials that will be one-time credentials, injected into your dev session, and valid for your user. Maybe a similar way to how ssh-agent works, by providing a git-authentication-credentials.
I assume that we would execute everything through GitLab, so we could make thing like this to happen.
I'm not sure I understand the current proposal. Are the dev and prod containers completely separate? Meaning, I run a generic dev container, fetch the current code, compile it there, and run the webserver locally in the dev container? And the only interface to the prod container is that I ultimately do a Docker build and push up the production image?
That's fine, but kind of kills the dev/prod parity that everyone is craving.
Or is the dev container supposed to work in concert with a running "production" image? Meaning, I run a generic dev container, but also have another container spun up to run the application, and I connect to that container via the dev container? If so, how do I run edited code? If I have to rebuild the docker image every time, it's going to be a horrible experience. For scripted languages, you could mount the filesystem between the development and production containers. For compiled languages, you'd have to share build artifacts as well. Either way, the production would have to be aware of the changes to pick them up. Default ruby production web servers don't reload source code, for example.
It seems like we have two potential proposals to implement split containers:
a) Use runners as development environments. Pros: They already know how to start from an image, including additional services, then checkout codebase at specific SHA, and run additional commands. They're on-demand and disappear when unused. With Kubernetes executor, these are pods in the cluster. Easy to spin up multiple dev containers at the same time, for the same or different people. Credentials are already handled by build token. Cons: They don't know how to expose ports. Not set up for SSHing in for web terminal.
b) Define development environment. Pros: Already know how to create/update and SSH into environments. Cons: Environments are generally persistent which is wasteful and may cost money. Could mitigate with manual start/stop actions. It's not easy to spin up different environments for different developers.
@sytses Yeah, my words might have been unclear. In all examples above, the dev container is not application-specific (although might be framework-specific or company-specific). I'm trying to understand how the development code is built and served.
If it's built and served within the dev container, then it blows dev/prod parity. If it's build in the dev container, but served from a prod container, then it's a hard workflow.
If I liken it to a local laptop-based development flow. A common pattern is:
Edit files locally
Build app, downloading dependencies to local machine
Serve app using dev server
Commit to VCS and push to origin
CI/CD tests and builds production image
And a common Docker pattern is:
Edit files locally
Build Docker image with shared filesystem, downloading dependencies into Docker image
Serve app via dev server in Docker image (using docker or docker-compose)
Commit to VCS and push to origin
CI/CD tests and builds production image
Neither pattern is truly dev/prod equivalent if you're building with development dependencies and running a dev server, but at least under Docker you've removed discrepancies between system resources (like macOS vs Linux).
But what people appear to want is:
Edit files locally
Build production Docker image
Somehow develop using that production image
Push production image
What I don't have a clear vision for, is how to make development using the production image not suck. To take a specific example, if you load a rails app in production mode, it's going to ignore changes to source code made after bootup. Even if you bundle install any new gems and rake assets:precompile any assets, you're going to have to kill the webserver to reload your code. If you're lucky, your webserver will listen to HUP and reload, but many won't. And then of course there's debug-level information that Rails outputs to logs and renders on 500s, but only when in development mode. If you actually need to debug a running process, it's even worse as you likely won't have pry or your chosen debugger in the production image at all, and at any rate there's no way to kill the webserver to then reload it under a debugger, without triggering the Docker supervisor to respawn the container.
Maybe for a web terminal, having it suck, but be possible, is better than nothing; so it's a net-positive. Or maybe we stick with the common Docker pattern of having a different docker-compose configuration that is close, but not exactly production. The talk of "production container" in this issue implies we're trying to solve the last flow, but it sounds like we're only supporting the first flow.
There are two additional flows I've heard that are interesting:
Dev Wrapper
Edit files locally
Build production image, downloading production dependencies into Docker image
Build dev image with production image as base image, and adds development dependencies and dev server
Serve app via dev server in dev image (using docker or docker-compose), sharing local filesystem
...
Sidecar container
Edit files locally
Build production image, downloading production dependencies into Docker image
Build dev image as a sidecar to production image, and adds development dependencies and dev server
Serve app via dev server in dev image (using docker or docker-compose), sharing local filesystem
...
Now what gets really interesting with the sidecar container is when we consider it in the context of this issue, which is really about developing in the cloud rather than on your local laptop. For all the other flows, you're just wrapping the whole flow in another container. e.g. spin up a generic "dev container", then edit files, create dev and/or production images, commit to VCS. But imagine we combine the dev container with the dev sidecar, and apply it to a production image. The flow now looks like:
Sidecar container in the cloud
Build production image
Attach generic dev container with git, rvm, npm, etc.
Fetch repo (because production image is missing .git directory)
[Optional] Build dev dependencies - This part is fuzzy
Edit files
[Optional] Serve app via dev server - This too is fuzzy
Commit to VCS and push to origin
CI/CD tests and re-builds production image
At its heart, a sidecar container is functionally the same as sharing the filesystem between your dev environment and the production container, so it might have the same limitations. And it's also very similar to the Dev Wrapper, but with an advantage of only needing to be built once rather than every time the prod image changes so it's faster to spin up. It's also got the possibility of attaching to an existing review app.
Some of the items above are fuzzy because I don't actually know how sidecar containers work with Kubernetes. My thinking is that you build dependencies into the ephemeral filesystem of the sidecar, and with most frameworks this will be additive and even for subtractive changes, it'll just ignore what's not needed. And then for running the dev server, imagine we attach to a production container, but don't actually run it's CMD. Or just ignore that running CMD. Either way, spin up a dev server, mapped to an appropriate exposed port, and route to that port so you can test your changes. I got stuck for a while thinking that we somehow had to take over a review app's running server, but if we think of it as a separate process with its own routing, it's a little easier to wrap my head around. Maybe the dev container isn't a sidecar to a production container, it's the other way around. e.g. Spin up a dev app/container and attach the last-known good production image to it.
In this scenario, you'll still rebuild the production image when you're done, so it's not 100% dev/prod equal, but it's as close as you can get. You'll be running based on a production image, but with a reasonable dev flow.
@markpundsack I think that we should let go of dev prod parity. Both should use the same OS and database version, but they should not be the same.
A development container is used throughout the company and for multiple applications and sometimes programming languages. A production container is specific to that application and is never used for development.
The development container can be seen as a Heroku buildpack for development. The goal is not to have it be exactly the same as production but to have it the same for the whole team that is developing.
@sytses Sounds good. And simplifies things greatly. We can always tackle dev/prod parity in a later iteration. :)
So, focusing on a common dev container with various languages, as well as docker and docker-compose on it, we're back to how to implement that, and we've got two proposals.
@ayufan I'd like to flesh out these options and see what a first iteration could look like.
@markpundsack Runner seems simpler, just need to export port 80. Already have git clone and build token to push back. Sid's OK with this since we'll have a lot of non Kubernetes usage too.
Both proposals have merit, but I'm leaning towards runners-as-dev-env. It feels more elegant and with our current Kubernetes-based architecture, runners are spun up as pods in the cluster, just like environments. The only major thing missing is the routing, but that feels pretty easy to tackle.
Feels easy to prototype:
Create base image.
Create manual action in .gitlab-ci.yml to spin up dev container, using base image; add a sleep 30m to leave runner around.
Manually add routing to runner, port X000
Manually SSH to runner (via Kubernetes dashboard if necessary)
For product, we probably need to:
Create base image(s).
Figure out how we want to trigger this - via manual action or dedicated UI.
Figure out how to keep the runner alive as long as you're SSH'd in, destroy when done.
Automatically add routing. e.g. dev-7A8BCD.$KUBE_DOMAIN
Automatically open web terminal on runner
Extend CI_BUILD_TOKEN to allow writes to main repo, sub repos, and Git LFS. (Already has write access to container registry, if needed.)
Questions:
Should we let someone join someone else's dev container?
Should dev containers persist? (If so, perhaps an environment approach is better)
Should this run in a pipeline, which then blocks pipeline status and MR merge-ability, or should this be some external thing, that happens to re-use runners?
Do we need any helpers to make git commits easier?
@sytses No, that doesn't solve this. That solves for efficiently building a production container, but does nothing for letting a person develop in one, with development dependencies, git push credentials, etc.