BUG: Errors are not properly handled when project cloning fails
MR: Add error handling for workspace project cloning (!163280 - merged)
Description
SUMMARY: Large projects like www-gitlab-com are not cloned properly in a workspace
Context - #464114 (comment 1988482042)
When creating workspace from a large project like www-gitlab-com, the cloning fails but the workspace still gets created. Ideally if the cloning fails, the workspace should have reported a Failed state. Furthermore, we need to address the reasons for cloning failure.
Error -
ERROR 2024-07-08T20:30:57.235633578Z [resource.labels.containerName: gl-cloner-injector-gl-cloner-injector-command-1] Cloning into '/projects/www-gitlab-com'...
ERROR 2024-07-08T20:32:17.886462747Z [resource.labels.containerName: gl-cloner-injector-gl-cloner-injector-command-1] error: RPC failed; curl 92 HTTP/2 stream 0 was not closed cleanly: INTERNAL_ERROR (err 2)
ERROR 2024-07-08T20:32:17.886524742Z [resource.labels.containerName: gl-cloner-injector-gl-cloner-injector-command-1] error: 52504 bytes of body are still expected
ERROR 2024-07-08T20:32:18.133259912Z [resource.labels.containerName: gl-cloner-injector-gl-cloner-injector-command-1] fetch-pack: unexpected disconnect while reading sideband packet
ERROR 2024-07-08T20:32:18.159193083Z [resource.labels.containerName: gl-cloner-injector-gl-cloner-injector-command-1] fatal: early EOF
ERROR 2024-07-08T20:32:18.197845878Z [resource.labels.containerName: gl-cloner-injector-gl-cloner-injector-command-1] fatal: fetch-pack: invalid index-pack output
ERROR 2024-07-08T20:32:19.806111232Z [resource.labels.containerName: gl-cloner-injector-gl-cloner-injector-command-1] Cloning into '/projects/www-gitlab-com'...The reason why the workspace ends up in a ready state is a flaw in our cloning logic. The logic is
- If folder(with project name) exists, skip the cloning. The assumption here is that the existence of the folder is indicative of the fact that a successful cloning has occurred. This is an incorrect assumption as has been proved by this case.
- If the folder does not exists, clone the project.
So currently when user tries to create a workspace from www-gitlab-com, the following steps happen
- Project cloning container starts.
- Folder(with project name) does not exist. Thus, start the cloning process. This will create a folder.
- The cloning fails because of some reason.
- Because of Kubernetes logic, the project cloning container restarts.
- The folder exists and thus the cloning is skipped.
- The workspace gets marked ready.
So the actionable items here is
- Fix the flawed cloning logic
We also need to provide more control over cloning options, that is captured in separate issues.
Related issues
- Allow configuration of project cloning approach (#414011 - closed), which is an issue to provide more extensive control over the cloning mechanisms.
- Expose settings to control the size of volume w... (&12807), which is about exposing settings for the the memory and volume size of the project cloner init container
- Increase hardcoded defaults for project cloner ... (#477834 - closed), which is a stopgap solution to workspace failures, by increasing the hardcoded project cloner memory and volume size
Acceptance Criteria
TODO: Fill out (required)
-
[Describe what must be achieved to complete this issue.] -
[Describe another requirement needed to complete this issue.] -
[Add additional acceptance criteria as needed.]
Technical Requirements
TODO: Fill out or delete (optional) [If applicable, please list out any technical requirements for this feature/enhancement.]
Design Requirements
TODO: Fill out or delete (optional) [If applicable, please provide a link to the design specifications for this feature/enhancement.]
Impact Assessment
TODO: Fill out or delete (optional) [Please describe the impact this feature/enhancement will have on the user experience and/or the product as a whole.]
User Story
TODO: Fill out or delete (optional) [Provide a user story to illustrate the use case for this feature/enhancement. Include examples to help communicate the intended functionality.]