CI runners checkout code from Git, but when being used with the Web IDE local changes should be mirrored to the runner so that builds and tests run against the state in the Web IDE. It is confusing to have a terminal next to an editor interacting with different versions of the code.
Target audience
As a software engineer, when I am viewing or editing a repository in the Web IDE I want to be able to make changes and test them using the Web Terminal before committing them, so that I know that I am committing code that works.
Further details
This also significantly improves the ability of people to contribute to projects they don't contribute to regularly, by allowing them to view, edit, and test without install local dependencies.
Proposal
Implement a method for mirroring changes from the Web IDE to the runner. The editor will establish a websocket connection to the mirroring service running in the CI build, using an event-based messaging protocol. In this communication, the editor will send diffs from the current changes to the service.
Editor contents will be saved to the development environment when the editor loses focus or is closed. The Save keyboard shortcut (CTRL+s, Command+s) can be used to save your changes immediately.
What does success look like, and how can we measure that?
This feature improves the utility of the Web Terminal in the Web IDE, and should increase the use of the Web IDE for Ultimate users. This can be monitored through the existing Web IDE usage ping.
In order to implement these services and the other ones related to services in CI build we need to implement some foundations which we'll reuse in each of them. This way we won't need to reinvent the wheel every time and manage them in a homogenized way.
There are three places where we have to make modifications:
Rails: we need to update the Web IDE config yaml to accept a new ports section. We also need add a new endpoint and a way to handle the communication from/to Workhorse
webide-file-sync#1 (closed) Binary files changes don't generate diffs. Therefore, we have to upload the whole file and this is inefficient. We should explore again other options like using the rsync protocol.
We should also keep in mind the future need to mirror file changes from the runner to the Web IDE so that it is convenient to inspect generated files (e.g. viewing Javascript output of Coffeescript). The first iteration should not need to implement this.
@jramsay If I recall correctly, the repository was mounted as a volume in the runtime container. The POC is there: https://gitlab.com/mbergeron/gitlab-web-ide, so any modification done would be available there.
@jramsay so the scope is to have the local copy available to the runner before it is pushed remotely? Or even before it has been staged/committed locally?
@bikebilly yes, the intention is to be able use the runner for tests, linters and formatters before making a commit. Because of the incremental changes I've made in the Web IDE would need to be copied to the runner. Essentially, just like hit Ctrl-S in Atom, switching to the terminal and running rspec.
Deferring to %11.3 once further progress on the Web Terminal has been made. Although not all use cases are covered, there is incremental benefit in shipping the terminal without the file changes being synced. For example, making a change and testing it can be done by making a change, committing the change, the web terminal will restart the relevant tests run. This will be faster than waiting for all tests to run, but of course the end goal is to be able to preview and test changes before committing them.
We'll need close involvement of a Runner domain expert to help us determine the best solution, and to either implement the necessary changes themselves, or guide one of the Golang devs on the ~Create team to do so.
Until we settle on an approach, it's hard for me to predict how much work will be required, and whether this will be something any Golang dev could do, or whether we actually need a domain expert.
@ayufan What do you think? Do we need to borrow someone from ~Verify for 11.6? :)
That is something that is discussed in the doc. We may use websocket channels using the same websocket connection. The problem with that approach is how to get everything synced both ways. For this specific case, we discussed that we'd use patches instead of every update on the file but that lead us to the need of additional requirements like, for example, having file watchers in the job (to be aware of every CRUD in that part of the connection) and translating those changes into something the web ide could interpret and apply.
@jlenny I think @DouweM was wanting to flag that we will need review and input from someone from the ~Verify team in order to accomplish this issue, and to make sure that this was considered in scheduling/capacity.
@jramsay @jlenny We first need to hear from @ayufan how much work this is going to require on the runner side of things, and if we are actually going to need someone from ~Verify to help us out with that, or if we'll be able to do this ourselves. If not, it's something for you, @jlenny, to consider for scheduling :)
Sure thing. @grzesiek is the tech lead on the ~Verify team, and @erushton is the EM. Once they know how much time is needed from someone I can help schedule it. Thanks everyone!
FYI I've started a test plan for this but it's fairly sparse atm. I'll update it once there's some more detail about the implementation and expected update/sync behavior.
@fjsanpedro@ayufan@grzesiek wanted to pick up the conversation here on this next iteration. From the various discussions previously around the web terminal, I recall we were leaning towards extending gitlab-webide.yml to declare various services like rsync for mirroring files from the web ide to the runner, and http for exposing a preview for example. Is this roughly correct?
It would help me if you could add some detail to the proposal to explain the technical approach that you're intending to take, and the different elements that need to implemented on the frontend and backend. Thanks!
@jlenny would it be possible for you set aside some capacity from ~Verify so that @fjsanpedro has enough support to keep this moving forward in %11.8? /cc @erushton@DouweM
@jramsay it would have to be very minimal, we're overwhelmed at the moment with confidential issues with SLA that are taking priority. @erushton would better be able to speak to engineering team capacity.
@jramsay the approach wasn't clear and now I'm more concerned because of the problems we already have with the web ide terminal and kubernetes. I'd like to find a solution that actually would work at least with GCP. @ayufan@grzesiek and also @SteveAzz (because he already knows the problems with the web ide terminal) I'll schedule a meeting this week about to figure out the better way to go.
What it was certain, at least for me, is that we would need to use more WebSockets, and there was a comment about a concern from the production team with this technology. @jarv how do you feel using more websockets in our codebase from the production perspective?
What it was certain, at least for me, is that we would need to use more WebSockets, and there was a comment about a concern from the production team with this technology. @jarv how do you feel using more websockets in our codebase from the production perspective?
I think that the idea is that we redirect WebSocket connection to Runner, right? And Runner handles that.
And GitLab via Runner is allowed to connect to gitlab/file-manipulator service similar as we do with terminal today, this can be a bi-directional web-socket connection, it is up service to implement the correct protocol.
What it was certain, at least for me, is that we would need to use more WebSockets, and there was a comment about a concern from the production team with this technology. @jarv how do you feel using more websockets in our codebase from the production perspective?
Thanks @jarv. We have kind of thought of a similar way but using runner and jobs labels for webide for example, where we could redirect the connection to specific runners (like a webide fleet). But I think your approach would be more straightforward and harder to bypass, because if you don't set any label in the job it would go to the default fleet.
Provide a flexible way to interact with runner jobs. The example required by WebIDE is to allow direct file manipulation. Additional goals, allow to easily preview changes and use compute of runner machines.
Proposal
Implement extension to .gitlab-ci.yml and gitlab-webide.yml (not really in configs, but between Coordinator and Rails):
Extend Runner to provide: /session/uuid/proxy/<build|or|service-name>/<port>/<requested-uri> that proxies HTTP call to the build or service container on this port providing bi-directional communication. Actually, we maybe don't need at all ports:, but some executors might need to explicitly expose these ports in order to make them reachable (to verify).
Extend Workhorse to provide endpoint that will proxy to /proxy similar as we do for /exec,
Implement endpoint on Rails that generate metadata required for connecting to runner.
Convention
I would say that: runner/workhorse/rails do not understand what type of data are being passed through, they are just responsible for bi-directional passthrough of HTTP connection that can later be updated by implementator (service running on that port) to websocket connection. It is up to callee (frontend) to decide what type of request is that: is that websocket? use wss://, is that regular GET/POST, use xhr.
I would say that for every build image we inject ports: 8080 to make runner to expose this port and allow user to run webserver in theirs webterminal session.
@jramsay it doesn't exist, it's just an example to illustrate @ayufan point of view. It means that after we implement the proxy stuff, we will be able to invocate any service and make bidirectional connections to it (including for example a web server).
@jramsay it seems a good tool, but we have the same problem as with rsync. We'll have to rewrite it to be able to send the data in a way JS understand. Also, we have to build a client in JS that can speak to this tool. Furthermore, I'm not sure if we could use this tool directly when using the kubernetes executor.