Cloud development with a web IDE
Cloud Development is needed to complete the original idea to production vision. We expected Koding to fulfill this part, but it hasn't worked out as we had hoped.
The initial thinking was to combine a "dev environment" + web terminal + git push back, where the dev environment is similar to a review app in that is ephemeral and only available when needed. They have to be different in that they'll have development dependencies and tools, and the ability to commit changes and push them back to origin.
The Web IDE provides new possibilities allowing editing to be done in an editor much more like a local IDE like VS Code instead of needing editing to be done through a web terminal using vim/emacs/nano etc (many developers are not comfortable making complex multi-file edits in these terminal based editors).
The current thinking is to use CI runners to power the "dev environment" rather than something closer to review apps – meaning we would combine Web IDE and CI runners, and expose a Web Terminal to the runner and Live Previews. This is possible because of a change to CI runners that is being made to allow a Web Terminal to be attached.
A challenge of using the Web IDE is that the editing of files is happening outside of a "dev environment" which means that the code changes need to be applied to the environment somehow. This is not a problem when the editing is done directly through the terminal because the editing is happening directly on the disk of the dev environment. We will need to build a solution to sync files between the IDE and the dev environment.
Using CI runners we will be able to spin up ephemeral environments by starting a new job that is not part of a pipeline like we do for ChatOps. This will allow many developers to work on the same project or branch at the same time without clashing with each other. We will need to work out how to persist work for longer editing sessions and between sessions, like the persistent disk in personal computers that allows changes to be saved but not committed.
For implementation, we might declare an explicit environment in .gitlab-ci.yml
, just like review apps, but I'm leaning towards creating temporary environments that run directly on the runner with ports exposed and routing ingress. Since runners are already k8s friendly, this is very convenient and would more easily let us spin them up when needed with less overhead. With might need a way to automatically garbage collect idle environments and persist disk between instantiations.
Either way, we'd boot up the container and install dev dependencies according to the image defined in .gitlab-ci.yml
explicitly for development. We can provide a base image that works for lots of languages/frameworks.
Timeline
Shipped
- Web IDE editor gitlab-org/gitlab-ce#37808
- Review staged changes as diffs gitlab-org/gitlab-ee#4568
- Stage and commit individual files gitlab-org/gitlab-ee#4541
- Fuzzy file finder gitlab-org/gitlab-ce#44841
- View CI job statuses in the Web IDE gitlab-org/gitlab-ce#44604
- View CI job logs in the Web IDE gitlab-org/gitlab-ce#46245
- Workspace switching - switch between assigned and authored merge requests without leaving the Web IDE gitlab-org/gitlab-ce#45184
-
View Merge Request description in the Web IDE gitlab-org/gitlab-ce#45187
A merge request description should describe the reason a change was needed and what it is trying to do. This information is important when reviewing changes in the Web IDE as it provides context for what the changes are trying to do.
-
(Backstage) complete Rails and Workhorse changes for community contribution to https://gitlab.com/gitlab-org/gitlab-ce/issues/25990 https://gitlab.com/gitlab-org/gitlab-ce/issues/47116
This will provide a web socket that can be used for a web terminal and transferring files to/from the CI runner.
-
(Proof of concept) open web terminal from CI runner in Web IDE https://gitlab.com/gitlab-org/gitlab-ee/issues/5281
Click a button in the Web IDE to launch a Web IDE job without a pipeline (like ChatOps), attach to the runner, and open a web terminal to run commands in the CI runner. This will inform how we structure work in the following release.
-
Run tests in real time before committing in the Web IDE https://gitlab.com/gitlab-org/gitlab-ee/issues/5426
In combination with viewing job traces, a user with a failing merge request will now be able to fix the code side-by-side to the job trace. When the code is fixed, they will be able to re-run the failing test in real time by typing the appropriate command in the web terminal (this could be automated and integrated in the editor in a future iteration) to verify their fix works. This means that they fix, test and commit like they would locally right in the Web IDE.
This iteration provides immediate utility to users and uses our super-power of building an integrated product by integrating Create and Verify steps. Syncing the files to the dev environment from the editor also tackles a significant part of achieving live preview of changes.
-
Sync files to CI runner from Web IDE https://gitlab.com/gitlab-org/gitlab-ee/issues/5276 so that when you run tests in the terminal they are run against the code changes being made in the Web IDE. We'll need to sync changes to/from the compute environment since we need to warm up the container and install dependencies in advance so the user doesn't wait for minutes at a time to preview a change.
-
Live preview &170
A user will be able to make changes to their application in the Web IDE editor and preview them live. This will be possible because the changes will be synced from the editor to the dev environment, and a dev server will that is watching for changes will restart and serve the changes. This commands to start the dev server will be defined in the
.gitlab-ci.yml
file. The primary technical challenge will be opening a port from the runner and making it accessible to the user.This iteration achieves a one-way flow of editing, testing and previewing.
Future
-
Sync changes from dev environment to Web IDE &197 (closed)
Adding a dependency is commonly done with a command line tool that will be available in the dev environment. This will update the dependency file (e.g.
Gemfiles
,package.json
) and also update lockfiles too which are managed automatically by tools. Tools for scaffolding that create files, or code transpilers need the files they create to be committed. We need to provide a way to sync these back to the Web IDE. -
Merge request discussions in the Web IDE &72
During the review process comments are left on the code changes. This makes it easy to comment on a specific piece of code that needs to be changed. If there is more than one piece of feedback it's unlikely I'll be able to remember it all, and will need to refer back to the comments to address them with my fixup commit. We should show the merge request discussions inline in the Web IDE, allow comments to be added, discussions started, and discussions resolved.
-
Web IDE with persistent disk gitlab-org/gitlab-ee#4012
-
Live linting &70 (closed)
-
Live shared Web IDE for pair programming &198 (closed)
Links
- Part of GitLab DevOps - Big I2P areas: https://gitlab.com/gitlab-org/gitlab-ce/issues/32639
- Split containers: separate dev / prod containers: https://gitlab.com/gitlab-org/gitlab-ce/issues/23966
Older issues:
- MVP of container code editor: https://gitlab.com/gitlab-org/gitlab-ce/issues/22863
- IDE / editor with a local editor via mirroring / sync: https://gitlab.com/gitlab-org/gitlab-ce/issues/22876