## Implementation Plan <details> <summary>Click to expand</summary> 1. https://gitlab.com/gitlab-org/gitlab/-/issues/512722+s 1. https://gitlab.com/gitlab-org/gitlab/-/issues/511503+s 1. Preliminary refactor MR 1: https://gitlab.com/gitlab-org/gitlab/-/merge_requests/174589+s (branch: `caw-ws-refactor-component-logic-1`) 1. Preliminary refactor MR 2: https://gitlab.com/gitlab-org/gitlab/-/merge_requests/174792+s (branch: `caw-ws-refactor-component-logic-2`) 1. Preliminary refactor MR 3: https://gitlab.com/gitlab-org/gitlab/-/merge_requests/176947+s (branch: `caw-ws-refactor-component-logic-3`) 1. Preliminary refactor MR 4: https://gitlab.com/gitlab-org/gitlab/-/merge_requests/177547+s (branch: `caw-ws-refactor-component-logic-4`) 1. Preliminary refactor MR 5: https://gitlab.com/gitlab-org/gitlab/-/merge_requests/180191+s (branch: `caw-ws-refactor-component-logic-5`) 1. Preliminary refactor MR 6: https://gitlab.com/gitlab-org/gitlab/-/merge_requests/180824+s (branch: `caw-ws-refactor-component-logic-6`) 1. https://gitlab.com/gitlab-org/gitlab/-/issues/520866+s 1. https://gitlab.com/gitlab-org/gitlab/-/merge_requests/182365+s (branch: `caw-ws-add-actual-state-workspace-var`) 1. https://gitlab.com/gitlab-org/gitlab/-/merge_requests/183632+s (branch: `caw-ws-add-actual-state-workspace-var-1-5`) 1. https://gitlab.com/gitlab-org/gitlab/-/merge_requests/184104+s (branch: `caw-ws-add-actual-state-workspace-var-1-6`) 1. https://gitlab.com/gitlab-org/gitlab/-/merge_requests/184120+s (branch: `caw-ws-add-actual-state-workspace-var-1-7`) 1. https://gitlab.com/gitlab-org/gitlab/-/merge_requests/184452+s (branch: `caw-ws-add-actual-state-workspace-var-1-8`) 1. https://gitlab.com/gitlab-org/gitlab/-/merge_requests/185529+s (branch: `caw-ws-fix-desired-config-generator-annotations`) 1. https://gitlab.com/gitlab-org/gitlab/-/merge_requests/182392+s (branch: `caw-ws-add-actual-state-workspace-var-2`) 1. https://gitlab.com/gitlab-org/gitlab/-/merge_requests/185150+s (branch: `caw-ws-add-actual-state-workspace-var-3`) 1. https://gitlab.com/gitlab-org/gitlab/-/merge_requests/185207+s (branch: `caw-ws-add-actual-state-workspace-var-4`) 1. https://gitlab.com/gitlab-org/gitlab/-/issues/524205+s 1. https://gitlab.com/gitlab-org/gitlab/-/merge_requests/185562+s (branch `caw-ws-add-agent-config-string-vs-symbol-test-coverage`) 3. https://gitlab.com/gitlab-org/gitlab/-/issues/520870+s (branch: `TODO`) 4. https://gitlab.com/gitlab-org/gitlab/-/issues/520882+s 1. https://gitlab.com/gitlab-org/gitlab/-/merge_requests/185023+s (branch: `da-inject-workspace-scripts-in-fixtures`) 6. https://gitlab.com/gitlab-org/gitlab/-/issues/520884+s (branch: `TODO`) - NOTE: This will need to be coordinated with https://gitlab.com/gitlab-org/gitlab-build-images/-/merge_requests/905+s, to ensure that we get the latest version 1. https://gitlab.com/gitlab-org/gitlab/-/issues/520867+s 1. https://gitlab.com/gitlab-org/gitlab/-/merge_requests/182354+s (branch: `caw-ws-refactor-component-logic-7`) 3. ----- At this point, https://gitlab.com/groups/gitlab-org/-/epics/15769+s can be started and built on top of existing postStart hook support ----- 4. https://gitlab.com/gitlab-org/gitlab/-/issues/525262+s (branch: `TODO`) 5. https://gitlab.com/gitlab-org/gitlab/-/issues/505988+s (branch: `TODO`) 6. https://gitlab.com/gitlab-org/gitlab/-/issues/392853+s (branch: `TODO`) 7. https://gitlab.com/gitlab-org/gitlab/-/issues/520868+s (branch: `TODO`) 8. https://gitlab.com/gitlab-org/gitlab/-/issues/520869+s (branch: `TODO`) 9. https://gitlab.com/gitlab-org/gitlab/-/issues/524911+s (do after %"17.10" ) (branch: `TODO`) TODO: Add follow-up epic to support `preStop` hooks as well. </details> ## Problem Running automated commands on launch of a workspace is one of the most basic and foundational concepts in Remote Development, and every single competitor offers this. However, the way we have it implemented it up to today is **confusing, and at minimum requires a lot of extra manual work, or doesn't even offer the functionality that users expect**. #### Conflict with official .devfile documentation If you follow the .devfile documentation, you will see that they instruct you to use the [events](https://devfile.io/docs/2.1.0/adding-event-bindings) and [command](https://devfile.io/docs/2.1.0/adding-commands) keywords to define different actions that can be run at specific moments, e.g. `preStart` or `postStart`. However, all the events that would be valid for the major use cases (e.g. automatically installing dependencies) are intentionally being overridden, as [they are being used to start the VS Code server](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/lib/remote_development/workspaces/create/tools_component_injector.rb?ref_type=heads#L68). The only point where we make this deviation away from the `.devfile` expectation visible is [in a small note in our documentation](https://docs.gitlab.com/ee/user/workspace/index.html#validation-rules), which makes this extremely tough to discover for users, and there have already been moments during dogfooding where this has led to people (including myself and actual members of the team) to either [having to ask internally how this should be done](https://gitlab.slack.com/archives/C03KE0L9NC9/p1719847757766659), or just trying to use these commands, and then [having to reach out](https://gitlab.com/gitlab-org/gitlab-ui/-/merge_requests/4249#note_1930507071 "chore: Avoid unsupported commands") to the ~"group::ide" team to learn more about why it isn't working as expected, and figuring out what alternatives there are. ## Lack of valid alternatives #### VS Code tasks.json When this problem is being brought up, [using VS Code tasks.json is being recommended as workaround](https://gitlab.slack.com/archives/C03KE0L9NC9/p1719297212091629?thread_ts=1719263751.961559&cid=C03KE0L9NC9). However, there are multiple problems with this. One of them is that it also affects users if they use VS Code locally, which often times creates unwanted consequences. That led to it having to be reverted in `gitlab-org/gitlab-ui` due to engineering feedback: https://gitlab.com/gitlab-org/gitlab-ui/-/merge_requests/4461 **Blocking the high-priority initiatives to enable other IDEs** As this is a VS Code native feature, it would not be available anymore once we [bring in other IDEs](https://gitlab.com/groups/gitlab-org/-/epics/10635 "Create capability for additional editors that can be injected into the workspace"). **Refresh triggering the same script to run again** The only possible options to run these defined tasks `manually` or on `folder open`. However, that means that the same script will run again when you refresh the page. **Having to write manual scripts to block re-runs of the same script** To prevent the problem mention in the aspect above, teams [have to write their own custom workarounds](https://gitlab.com/gitlab-org/gitlab-development-kit/-/merge_requests/3571 "Automate GDK bootstrapping using VSCode's task") just to avoid that the workspace runs the same script again, which is quite a lot of work. **This recommendation is not documented right now** While this is the recommended workaround from the team, it's not mentioned anywhere in our documentation, so it's not even accessible for customers and other users. #### Devfile `preStart` Users could use the `preStart` event command, but **major use cases like installing dependencies are not possible here**, and due to a 10 minute timeout for launching workspaces, it is **also not a viable solution for most custom scripts**. ## How our competitors solve this #### Eclipse Che Che is the most similar to Workspaces, because it uses devfiles. GitHub Codespaces and GitPod use devcontainers. Che also uses Kubernetes via devfile-operator, whereas Codespaces/GitPod rely on VM-based (non-kubernetes) approaches. Here is a writeup of how Che handles this: https://gitlab.com/gitlab-org/gitlab/-/issues/512722#note_2313352319 #### GitHub Using the `devcontainer.json` [specification](https://containers.dev/implementors/spec/), which next to many other moments, also allows users to [define commands to run](https://docs.github.com/en/codespaces/setting-up-your-project-for-codespaces/adding-a-dev-container-configuration/setting-up-your-nodejs-project-for-codespaces#devcontainerjson) after launch via `postCreateCommand`. #### Gitpod A custom `gitpod.yml` file, which uses the `tasks` keyword that has [many options to run scripts and commands at specific moments](https://www.gitpod.io/docs/references/gitpod-yml#tasks), the equivalent for "after launch of workspace" being `init`. ## Solution ### Overview of solution Introduce an abstraction in the Workspaces domain logic to support the `devfile` standard `postStart` event (see docs for [postStart object](https://devfile.io/docs/2.3.0/adding-event-bindings#post-start-object) and [events](https://devfile.io/docs/2.3.0/devfile-schema#events)). This abstraction will support modifying the devfile-generated kubernetes resources to leverage the standard Kubernetes `postStart` lifecycle hook. ### Justifications for this solution 1. Will solve the immediate requirements of this epic with relatively minimal effort - i.e., all the changes should be isolated to the specific steps in the ROP business logic on the Rails side. 2. Still leverages all our existing devfile-based architecture all of the existing solution and docs are built around. 3. Implements the existing devfile spec for a `postStart` hook, which will always run after the container is successfully started and ready (see devfile docs for [postStart object](https://devfile.io/docs/2.3.0/adding-event-bindings#post-start-object) and [events](https://devfile.io/docs/2.3.0/devfile-schema#events))... * ..._BUT_ implements it in a way which makes sense for our particular use case usage of the devfile standard, which _is different_ than any of the existing implementations of devfile standard (odo, theia/che, or the devfile CLI reference implementation). We can then use our implementation to open a discussion with the devfile team, and help drive the discussion around the effort to include one (or more) of the various approaches directly into the standard `devfile` CLI executable. 4. Still follows the "spirit" of the devcontainers specification - because devcontainers also supports a `postStart` concept, which also is defined as being run when a container is successfully started and ready (see devcontainer docs for [lifecycle scripts](https://containers.dev/implementors/json_reference/#lifecycle-scripts) and [lifecycle hooks](https://containers.dev/implementors/features/#lifecycle-hooks)). Also note that the plumbing and abstraction we introduce as part of this effort - the ability to run multiple specific commands in multiple specific containers - will also be leveraged as the basis of multiple other epics, including https://gitlab.com/groups/gitlab-org/-/epics/15769+, https://gitlab.com/groups/gitlab-org/-/epics/13984+, and others. This initial implementation does _NOT_ imply that we do not want to eventually support the devcontainer standard for GitLab Workspaces (in addition or instead of devfile standard) - but that is a _much_ larger scope of work with significant implications for the future product direction, and thus needs to be a separate discussion with proper attention to investigation and planning. ### Details of handling `postStart` functionality based on investigation into Che functionality See https://gitlab.com/gitlab-org/gitlab/-/issues/512722#note_2313352319 for details of the Che investigation. Here's a summary of the plan outlined there: ### High level overview of implementation plan 1. We will no longer overwrite the container entrypoint/cmd. We will leave them untouched, unless there is no entrypoint/cmd, in which case we will add one for `tail -f /dev/null`, so the container will not exit after starting. 2. All commands we want to run in containers, both "our" commands (i.e. vscode/sshd startup) and "user-defined" devfile postStart hooks, will be invoked by invoking them via kubernetes container-level postStart hook. This is what Che does. 3. We will group all poststart commands to be invoked via a single script, which is invoked as a background process via a single kubernetes poststart hook. The hook will aggregate all STDOUT/STDERR from the commands and pipe it to `/tmp/poststart-stdout.txt` and `/tmp/poststart-stderr.txt`. This is what Che does. 4. To address the two concerns raised above about container readiness, we will leverage the existing rails-side logic (in `desired_config_generator.rb`) to check the workspace actual_state received via the reconciliation loop, and use it to control a boolean kubernetes file-based secret. The state of this file-based variable will then be used to to "delay" the running of user-defined devfile postStart events. ### Plan for addressing potential issues with pod/container startup race conditions - Kubernetes postStart lifecycle hooks do _NOT_ block on the container being ready. - And we are implementing the devfile postStart script support via a Kuberetes poststart hook - This is why we are introducing the concept of "waiting" for the pod to be ready before running user-provided poststart hooks. This will be done via observing the reconciled actual state on a file-mounted secret (https://gitlab.com/gitlab-org/gitlab/-/issues/520866+) - However, we will _STILL_ run the "internal" scripts to start VS Code and sshd BEFORE the pod is marked ready. - This introduces the potential for race conditions where the pod is ready but the internal scripts are not complete. For example, the UI shows a link to the VS Code URL and port because the pod is Running, but VS Code is not actually running yet. Or SSH is not running yet. - Since these things run quickly, this is unlikely to be a problem, at least for the initial release. - However, if we do encounter problems like this, we can fix it by introducing a kubernetes HealthCheck resource, which will check the VS Code (and maybe optionally sshd) ports, and not mark the pod ready until they are open. sshd might be not be easy though, because we have no way to know if sshd is actually configured in the container. We would have to drop a file or something to indicate if the sshd executable was actually found in the container, and only check the port if it was and was able to be executed successfully. ### Out-of-scope requirements for this solution UPDATE: With the new plan described [here](https://gitlab.com/gitlab-org/gitlab/-/issues/512722#note_2313352319), based on Che's implementation we _CAN_ preserve the container's ENTRYPOINT/CMD. ~~We are explicitly _excluding_ the following use case from this epic:~~ ~~_As a user, any ENTRYPOINT/CMD specified in my devfile's container(s) should be preserved and executed when the container is started_~~ ~~Reasoning:~~ 1. ~~Fixing this limitation is not necessary in order to fulfill the implementation plan outlined above to provide support for the `postStart` event in a devfile.~~ 1. ~~This limitation exists with the current implementation. In [`ee/lib/remote_development/workspace_operations/create/tools_component_injector.rb`](https://gitlab.com/gitlab-org/gitlab/blob/2e81909e48e0a95683962e5cb080bcb3e0d56edc/ee/lib/remote_development/workspace_operations/create/tools_component_injector.rb#L67-68), we currently override the `['container']['command']` and `['container']['args']`~~ 1. ~~Fixing this limitation would add significant complexity. The current plan only involves isolated additions/refactoring on the Rails side to the ROP chain classes around `ee/lib/remote_development/workspace_operations/create/tools_component_injector.rb`. In contrast, in order to preserve container `ENTRYPOINT/COMMAND` would involve adding new logic agent-side/kubernetes-side to retrieve these values after the container definition has been retrieved and parsed by Kubernetes. We _cannot_ do this on the rails side only during devfile processing time - we only have access to the registry path of the image, and even if we were to attempt to download and process it, the rails web processes might not even have access to it anyway (e.g. in the case of internal private registries which are only allowed access from the kubernetes infrastructure).~~ ## Testing notes - On cloud sandbox, we have a cluster set up and a VM running the GDK. This can be used to test in a real production-like environment. Notes: - SSH (ask Vishal or Chad for a key): `ssh -i ~/.ssh/dev-create-remote-us-east-2-key-pair.pem ubuntu@gdk-sandbox.remote-development-test.com` - GDK is installed under `~/gdk/gdk` - Use `kubens`/`kubectl` from within the VM - See docs for connecting to workspace container to check status/logs/etc: https://gitlab.com/gitlab-org/workspaces/gitlab-workspaces-docs/-/blob/main/doc/local-development-debugging.md#one-liners-to-invoke-commands-in-the-workspace-main-container