GitLab.org / GitLab Community Edition

I believe Koding is using FUSE to mount your local directory to the remote machine: https://github.com/koding/koding/blob/master/go/src/koding/kd/Readme.markdown#kd

Does FUSE have an open unresolved security bug? I just happened to google it and that came up.

Instead of putting the online editor first #22863 (closed) we can alterntively have the local editor as the primary use case.

Great. We should. I hope it is easier. This is all very complicated stuff.

Yes. After some research, make an online IDE is quite complicated. Unless we remove collaboration. The main problem is the file syncing, especially when collaborating. For collaboration you have operational transformation or prosemirror using another (centralized) method. But prosemirror does not seem to support being used with codemirror (for writing code). ATM I think it is unsupported.

We could use prosemirror for our live editing of issues descriptions though.

The terminal, on the other hand, is much more straight forward. It isn't collaborative and works through an API. Very nice.

As a side note: I would use the web editor if:

1. It was really really good. As good as/better than Sublime, which I use on a daily basis. Which so far none of them are or come close. It would also by that definition have to be fast. There couldn't be any lag at all. In the terminal or the editor.
2. It allowed me to review code and make quick changes. Instead of checking out a branch.
3. It was super quick to open. To open Sublime, I press cmd+space and type subl and hit enter, then I start coding. It would have to be that quick.

I think these are some of the things that stop people, I am sure there are other reasons. We should try and figure out why people don't use them and fix those things in our own iteration. I am sure that I would like to use it if it were perfect.

@sytses I am wondering one other thing. Instead of copying this method that no one wants to use. Could we do our own simpler form of collaboration for the IDE(without OT, or anything complicated) which would work (in theory) using only git as it's main tool of syncing, something like this:

Users who code don't need to be able to overwrite each other like in google docs. In fact that's probably more unrealistic for anything other than a code interview. For true collaborative coding I think it could be nicer to each have your own version where you can see each others versions at the same time. But you also need to know what the difference is between what you are writing and what they are writing.

This would work in two ways.

1. You see a side by side of yours and their code. Could also be above and below. The simplest way.
2. In another alternative view you could see a side by side of your version on one side (while you type) and a diff on the right, with the difference between yours and theirs. Because if you are truly coding together you shouldn't disagree on many things so if there is a disagreement you'll be able to resolve them.

3. OR that view could also be a diff against master. This would be a view where what you are typing is on the left, and on the right is a diff view with a combination of your version and and their version combined and merged against master (sort of like a practice run merge). You would then see the conflicts between you and your collaborative coder. True git coding.

4. Each user can have 1 instance of the IDE open. For example if jschatz1 is signed in then he can only have 1 instance/tab of the IDE open.

5. We dedicate a directory to that user or a docker container to that user, for each user who wants to collaborate.

6. If another user is signed in, they also get a docker container or dedicated directory.

7. We sync between the 2 (or more) user's to one master directory (or container). That becomes the collaboration. This syncing is done manually with the press of a button. While you view the two different directories no operational transformations/ or syncing has to take place. It is literally just two different versions in two different directories or containers.

8. We use git to sync between the two directories to the master directory. Or use git with separate containers.

9. Once ready, this master can then be synced with the main repo. So we would use git for the whole process. Which is our bread and butter.

@jschatz1 Great thinking!

1. I agree that collaboration in the code editor is out of scope.
2. I agree that making an online IDE is quite complicated.
3. I think that for the sync we can reuse the kite protocol from Koding https://github.com/koding/koding/blob/master/go/src/koding/kd/Readme.markdown#kd
4. I think it is unlikely that the editor will be fast to start or work with any time soon, so for now I'm happy if we just have a terminal.
5. Your one other thing idea is interesting but I think collaboration is out of scope for now.

Mentioned in issue #22863 (closed)

Made the issue confidential

I've made this confidential until we resolve discussions with Koding.

Made the issue visible

Alternatively we can reuse an existing sync protocol:

1. Add a special unison container and volume https://github.com/leighmcculloch/docker-unison#docker-compose
2. Need to install unison and fswatch locally https://github.com/leighmcculloch/docker-unison#installing-fswatch-locally
3. This doesn't solve the remote aspect, you still need ssh/workhorse/tunneling for that.
4. Unison is GPLv3 but the container and local client are separate from GitLab
5. There is https://hub.docker.com/r/immense/unison/ that is an extension of https://github.com/leighmcculloch/docker-unison to support mounting into a docker host.
6. There is https://github.com/onnimonni/docker-unison that seems similar and has "Thanks for leighmcculloch for showing me how to use unison with docker."
7. There are is https://github.com/EugenMayer/docker-sync/wiki that includes fswatch, rsync, and unison out of the box.

Installing it as a separate container is nice but might be strange to use in production.

@sytses could we use the same idea that you've just written to make the online ide? Each user would connect to a docker container and we could use the unison method to keep them in sync? The only thing we'd have to do is make sure that each user is only connected to one docker container. Right? Basically using the same method you're talking about for local editing as for remote editing. Then you wouldn't have to deal with any operational transformations or anything like that. Because the hard part of the collaboration is the operational transformations, which it sounds like you've solved.

@jschatz1 we could. But I don't understand your proposal. So the user would connect to the container using unison. I think that means that the files from the container are synced to the filesystem of the user. At that point the user should use a local ide or not?

My idea of how to make the IDE sync work:

1. You click on an icon in GitLab, it is next to where we now show the url for the terminal.
2. The icon opens a url, with a schema specific for our solution (although we should open source the protocol so others can reuse it). In the url is the following information: server and project location, pod and container to connect to, a token to identify the user.
3. Beforehand the user has installed a local client that is registered to handle this type of schema.
4. The filesystem of the container is synched to the users machine by the local client (I propose to a default directory system specific to the local client).
5. The local client also opens up the favorite editor of the user (make this configurable with smart default by seeing what editors are installed via which).
6. The local editor talks to an endpoint of GitLab, similar to how our terminal works.
7. GitLab uses kubernetes/docker exec to run unison and fswatch on the container (this is a different approach that the volume approach suggested in https://github.com/leighmcculloch/docker-unison#docker ).
8. GitLab makes a reference implementation of this for MacOS, using unison and fswatch. The community will hopefully make Linux and Windows clients.

From the user the experience is 'click editor icon' => 'have the project in your favorite local editor'. The solution can deal with short of long periods of losing inactivity. In the short term you don't need API calls to do autosuggest, in the long term you don't have to be afraid your changes disappear.

The token in the url is secure because people use https. GitLab is in between the local client and the container. Checking the local client for security vulnerabilities is essential.

Added koding label

I have tested a few remote collaboration plugins for various editors which exhibit the behaviour I would like from a container syncing service; working on a remote project (possible with other people) using a local editor like Sublime or Atom.

• https://packagecontrol.io/packages/RemoteCollab
• https://atom.io/packages/atom-pair
• https://floobits.com/help/plugins/atom

The Project Lead for Che responded to a comment by Sid on a thread about Google Clouds Browser Editor with how they achieve syncing remote files locally:

With Eclipse Che, we provide a simple CLI that lets you live sync your workspace to your desktop IDE. You just "che mount" and we use a fuse-based file system with unison sync to keep the hosted workspaces synchronized with the desktop. You can then use your local IDE and still have your workspaces entirely in the cloud.

Thanks @SeanPackham. Those plugins are cool but it would be hard for us to support all editors. I'm not sure if there is a standard we can use.

Using che mount requires "an agent container which is volume mounted to the projects in the workspace containers". https://eclipse-che.readme.io/docs/desktop-ide-mounting describes it as a Docker container that combines sshfs with unison. I like that approach.

I think our solution will need to following attributes that che mount doesn't have (yet):

1. It doesn't require you to modify your kubernetes pod files. This to have parity between development and production. For the terminal we used kubectl and docker exec to do this.
2. It should launch from a url, not require che mount c:\users\tyler\che\mount <ws-ssh-port> and ssh usernames/passwords.
3. It should launch your favorite editor, not only Eclipse. And preferably it should run without Eclipse.

If we got the local sync thing working we can add a cloud editor by:

1. Launching a workspace container with the Eclipse Workspace agent
2. File sync the workspace container to the container we want to edit
3. Launch Eclipse Che in the browser and connect it to the workspace container

This is an extra sync step compared to installing the Eclipse Workspace agent on the container we want to edit directly. This extra sync steps gains us the following benefits:

1. We use the same process for local and cloud editors.
2. The cloud editor can be run without modifying the Kubernetes pod file of our application. Reducing the differences between development and production.
3. You can use any online editor you want, different people can use their favorite online editor.

@SeanPackham and Tyler, what do you think?

Mentioned in issue gitlab-com/www-gitlab-com#871 (closed)

I think it makes sense for us to separate the "sync & mount" part of our container apart from the Che-specific stuff. And then we could collaborate on a general tool that uses Docker containers to connect to any SSH file system and then synchronize a target directory with a locally hosted directory. One nice thing about Unison inside is that it has a very flexible language for ignoring and choosing what to synchronize, so you can optimize performance. The requirements would only be that the remote target is SSHable. We make Che workspaces with an SSH login (today it's only user / pass, but we will have automatic key generation in a week or so). But that part isn't really essential to what you need to achieve.

So, to keep the separation of concerns clear, and recognize that you can manage each differently:

1. A general purpose mount and sync utility, implemented over Docker, which works against any local OS and any remote SSH server. We would separate our existing container into a general purpose one that we all can collaborate on.

2. We have Che extensions to this that we use to know how to connect to a Che workspace specifically, and if you are using our CLI, you don't even hvae to type the ws name if you want, so we do things such as scanning all of the workspaces for a single Che server to detect which ones have SSH active, which means they are mountable. I am not sure this stuff is needed for you guys.

3. If you want to auto-launch a Che workspace / Che container, we have a syntax for that. It runs anywhere that Docker is at, so it can be scheduled with your scheduler. But that is a different objective, than I think what you are targeting with this thread. But the Che thing is interesting for Gitlab if we want to bake in the ability for anyone to create a hosted workspace that is running your git repo files in a Docker runtime, that is controlled by Gitlab's scheduler, much the way you manage CI jobs.

@tjewell Thanks for the reply, I'm excited to see how we can collaborate and reuse your pioneering work.

1. This would be awesome, we would reuse this instead of building our own.
2. I'm also not sure if we need this.
3. Something like this, but I don't understand your proposal, probably because I'm tired after a transatlantic flight.

Sid - I have been in an offsite hackathon with our top engineers for 10 days in central america. Responding to these ideas is at the top of my list, but need a couple more days to get organized. We had a product manager do some extensive testing on che-mount and came to the table with additional ideas to contribute, and a potential roadmap.

Tyler Jewell | CEO | tyler@​codenvy.​com | 9​78​.8​84​.53​55

Thank you Tyler for giving some information on how this would be implemented. From your last reply Sid it seems you like Tyler's suggestions, I think it is worth prototyping this?

@tjewell cool! thanks for the update, enjoy the time together.

@SeanPackham I certainly like "A general purpose mount and sync utility, implemented over Docker, which works against any local OS and any remote SSH server." but I suggest we wait for @tjewell 's team to come with a proposal. For now we can focus on the TODO's in https://about.gitlab.com/handbook/sales/vision-demo/

Discussion on Hacker News https://news.ycombinator.com/item?id=12841928

From the updated description, it seems like the remote env is most valuable for large, complex apps, with multiple services such that running them on your local machine would be too painful.

It's unfortunate that it means that the long tail of small apps that run fine on a local laptop won't benefit (much) from it.

Quickly switch between projects (no need to boot them, they are running already)

@sytses This implies the containers are always running, but that can still get cost-prohibitive. Or are you assuming that the VMs are running, but the containers are not, and that the containers spin up quickly?

@markpundsack exactly, the VM's are there and have a mixed workload, a container scheduler can quickly spin up the containers.

@sytses From #22876 (comment 17198025):

This is an extra sync step compared to...

If you load the workspace container as a kubernetes sidecar, you can skip the extra sync step, right? The workspace and the main container would just share (part of) the filesystem directly. If this all works, this sounds ideal.

With this workflow, most of your work would be done locally, with your own editor. And I'm assuming you're going to run git locally to commit your changes after testing so you don't need the workspace container or the production container to contain git binaries, for example. What is left to require a "terminal" session to the production container? Restarting web servers, re-running bundle install, debugging running processes, etc.? Is there any way to do those via local docker commands to the remote, production container?

@sytses Or would you still want a terminal that spawned a container that shared the files with the production container, like the workspace container, but without needing to sync locally, and with git installed and creds set up so you could push changes back? Or is the online terminal just not necessary?

@markpundsack if you load the workspace as a sidecar you can skip the extra sync step. But as I mentioned this extra sync steps gains us the following benefits:

1. We use the same process for local and cloud editors (always sync).
2. The cloud editor can be run without modifying the Kubernetes pod file of our application. Reducing the differences between development and production.
3. You can use any online editor you want, different people can use their favorite online editor. (otherwise people have to specify it in their pod file)

You are talking about a production container. I don't understand this since it would be an environment launched especially for development. One goal is to make the kubernetes.pod file of development resemble the production kubernetes pod file as much as possible, ideally being the same. I think that in development there should be a git binary on the machine and a git directory structure. I'm not sure how this can be done without creating differences between development and production.

I was going to ask the same questions as Sid.

There are a number of challenges between having an identical set of containers for development and production. But the goals that you are seeking are identical to the goals of Che. The Che workspaces are very close to having identical container sets. So for Gitlab, you can give users multiple options.

Option 1: Deploy a set of identical containers with some sort of file sync to local IDE. The containers would not be in production & identical, but you would not have developer "services". The synchronization is likely to be slow unless you make a large number of improvements to the synchronization mechanism ... or ... you get comfortable with the idea that you will need to modify the production containers with some additional stuff like specialized unison clients, different SSH keys, and the so forth. Additionally, many production containers do not have the same source trees that the development containers have, so this model works fantastic if you have dynamic languages where the source tree is also running in production, but less so in compiled enviornments where libraries are in the production images but not the source.

Option 2: You allow for some basic modifications to the production containers to allow for SSH, unison clients, and maybe even a specialized web-terminal. In Che, we do something similar to this and we package up these improvements as dynamically deployable agents. Some agents have to be deployed inside the very same containers that you want to develop against (such as SSH or terminal since you need root access) and other agents can be deployed as their own independent containers, such as language servers (for intellisense + debugging) and unison sync. Any situation where agents only require access to the source code, but not the root of the container, those services can be deployed as separate agents and then they can share a volume mount to the project source code.

Option 3: You can deploy through your scheduler a Che workspace, which itself is a set of containers that will match production. We then do the heavy lifting around getting all of the agents into the right location and ensuring that the new workspace is setup for an optimized source code sync. We also do the heavy lifting of getting the source code into the container (it lives outside usually), setting up a language server + debugger. The only differences in the underlying base images between a che workspace and the production containers is that we require a non-terminating command. We then inject everything else it needs like terminal, SSH, unison sync, and language servers as agents after the containers have started.

#1 (closed) is the quick hit for you and once we finish our improvements to the unison sync over fuse file system could be a good enough low hanging fruit. #2 (closed) is probably the hardest path, and puts you on the path to building out your own cloud IDE system #3 (closed) is probably mostly ready to go, and could be a simple flag that lets users switch between #1 (closed) and #3 (closed).

We now run che entierly as a container and the startup sequence is simple and straight forward.

Tyler Jewell | CEO | tyler@​codenvy.​com | 9​78​.8​84​.53​55

We can consider two types of terminal accesses:

1. terminal access for development purposes: provisioning an environment with all tools and credentials allowing you to conenct to production cluster,
2. terminal access for debugging purposes: connecting to existing deployments.

Currently the terminal implemented as part of I2P is about allowing the debugging access, as we are working on top of existing deployments that can't really be easily extended with additional tools nor git credentials.

The problem with terminal support for debugging purposes working on top of existing deployments is that we don't really know what we should connect to, because usually these deployments will consist of multiple replicas and multiple services.

Having a terminal access for development purposes allows us to manage this deployments from outside, making it really easy to tinker with them in order to update or scale existing running deployments.

The development terminal can be long or short living, allowing us to also have a terminal access to existing deployments.

The deployment terminals "kind of" makes sense, but they usability is very limited as they are ephemeral, by design short living, suspectible to Kubernetes auto-scaling policies. Secondly, if we do want to tinker with deployments it can also be tricky to edit them, as they maybe a lot of them being currently live.

Development terminal makes a lot of sense if configured properly, because it can act as a portal to existing system. The development terminal actually opens us possibilities to expose existing filesystem to the user, making it possible for him to: - mount filesystem in the browser (with help of Webdav), - mount filesystem on local computer (with help of Fuse over SSH), - use editors like a atom. Also, it allows us to do very good job in a way how long this environment do live, thus control the persistence of data stored in terminal.

However, I do see a benefit of adding a support for better deployments that we would be aware of. We can effectively built a Kubernetes API straight into GitLab allowing us to be very smart in managing the deployments: doing a real stop, access terminal, monitoring the health of the application, reading logs. It's a little like a replicating a OpenShift functionality, but straight from GitLab interface.

The development terminal could be done in multiple ways, but it would still require the same thing to start with: first-class Kubernetes integration in GitLab.

What does mean Kubernetes integration in GitLab?

1. You would store credentials in GitLab,
2. You would be able to run development terminal for your user and your project,
3. You would have the same access level as your CI,
4. You would have to use smart deployments to deploy your application.

Maybe we could effectively skip the CI part of the story and provide the way to interact with environments, but not to run that in context of deployment.

TL;DR

Maybe we should basically provide a terminal that would provision a stable development environment for working with this "smart deployment".

So basically instead of entering into existing container, we would provision a new Pod, with a new Container, with Credentials and Git sources, running on top of your choosen docker image, allowing you to effectively work on your toolchain, with everything that you would require if working locally.

Later we could extend that concept with additional containers, sharing the filesystem and allowing you to live-edit the files from your computer.

This is completely out-of-CI thing, very simple, requiring only single docker image, but giving the stable development environment with full access to production-like running environment.

@ayufan and me discussed yesterday and we'll do the simple terminal access in #22864 (closed) (not smart deployments nor what was proposed here for now)

Removed koding label

@tjewell we would likely go for option one, we're into quick hits (iterating). Instead of installing dependencies we might use docker exec (kubectl exec) to install, configure, and run unison in one go. We realize this is only usable for dynamic languages. The containers will probably have the working directory and dependencies, the local synced one will probably have the working directory and .git.

Regarding option 3 "which itself is a set of containers that will match production". You mean that the kubernetes pod file matches production with the addition of a non-terminating command?

We just committed and merged our updated unison sync mechanism. It's two ways:

1. A desktop-based docker container for mac / windows / linux which runs sshfs + unison with a host volume mount. It connects to a remote SSH environment, synchronizes the content into the host container, which then moves it onto the host volume. We have also added in capabilities for unison profiles which give users fine-grained control over includes / excludes, so that you can optimize exactly what is synchronized between the local and the remote system. It's really essential as when you are dealing with developers that have large maven, npm, or other library repositories, you do not want to be messing with those over the wire.

2. We now have a unison agent that is deployable into a workspace container. This unison agent does a local scan, which when triggered by the host unison agent, increases the performance of the sync by 10 x. A 200 minute initial sync for a 500MB project can get down to 2 minutes. We dynamically deploy this agent into a container that has project code mounted into it as long as SSH is present. This sync agent is coupled to the SSH agent, so now there is a more fluid way to add / remove SSH + sync.

Between the exclude / include + dual unison agents, the performance is really nice. I just spun up a system where I was developing against the Che repository (>1M lines of code + 1GB maven repo) and I couldn't tell that there was a live sync, and I was using a desktop IDE. And it's cross platform thanks to Docker. No more crazy putty or other complicated windows stuff.

On the option #3 (closed) - I was just pointing out that we work very hard with Che to make the workspaces represent any off the shelf Dockerfile or Composefile. For a long time we had requirements for additional software to be inside those containers. Now, all of that stuff is abstracted out into agents that are selectively deployed into the containers based upon either user definition, team leader definition, or in some cases based upon what a user does, ie "Oh! You just opened a .cs file, time for us to deploy the csharp language server agent into your container so that you can get intellisense + debugging".

Our only requirements now are that the base agents need to have bash inside. This is generally universally true for most of the common base images. And then the non-terminating command. But even with that we have a way to override embedded commands with non-terminating ones. So essentially, we can layer on the Che-ness to any set of containers that are out there.

So whether you wanted to take a set of production containers and just add terminal + sync to them --- or you wanted to give users the option to convert those containers into a che workspace with the embedded IDE + all of the agents that are deployed, you could go either direction.

Tyler Jewell | CEO | tyler@​codenvy.​com | 9​78​.8​84​.53​55

Mentioned in issue #25190 (closed)

@markglenfletcher for this unlabelled issue I suggest Discussion feature proposal Edge

added idea-to-production label

added direction label

Thanks for the extensive comment @tjewell, interesting stuff.

changed title from MVP of container editor with a local editor as the primary use case to IDE / editor with a local editor via mirroring / sync

mentioned in issue #32637

added moonshots label

We're talking about Theia https://github.com/theia-ide/theia

We need auto save via NFS, stash, or some way to make spinning down the containers and the make client parallel CI possible

Theia browser editor connects to node backend process per user, the backend process starts a language dependent language server for every type of file (based on the language) you have open at the time. (similar to Che, kinda assumed one container per user)

From my perspective I would see it something like that:

1. I browse the repository and view files - exists, can be improved with file list, tabs etc.
2. I edit files and can commit those - exists, can be improved with a bulk edit, tabs etc.
3. If I trigger action that requires environment (for example open terminal tab) - a docker container started with all necessary (database, web server etc; docker-compose.yml? ) and modified code synced between browser and container during a user session.

Similar to what happens on my machine locally: start vim to do anything, then start gdk when I need it and keep it running until I am done (push my change).

@dzaporozhets cool idea!

@jschatz1 is going to try to ship Theia in 9.4 for repository browsing

I am working off this: #31890

So @dzaporozhets and I talked about this at length. There are a couple of things that need figuring out. But as a POC. BE will render the README. FE Will render rest.