Skip to content

[WebIDE][POC] Web IDE Fast File system based on Origin Private File System

Enrique Alcántararequested to merge
499-private-origin-file-system-poc into main

Issue: [WebIDE] [POC] Explore using origin private fil... (#499 - closed)

What does this MR do and why?

This Merge Request implements a proof of concept of an alternative file system and source control client implementations for the Web IDE based on the following technologies:

  • Origin private file system (OPFS) A web browser API that allows creating a private file system for a Web Application that can't accessed directly by the user and is not available on a specific origin.
  • ZenFS A library that simulates Node.js File System API. It's an abstraction layer on top of OPFS.
  • Isomorphic git A pure JavaScript implementation of Git that can run in Web Browsers.

Goals

Scope

This proof of concept doesn't introduce new functionality to the Web IDE. It aims to provide a drop-in replacement to the current file system implementation in the Web IDE. The file system and git client implemented in this proof of concept offers the following features:

  1. Loading a repository.
  2. Reading files.
  3. Writing files.
  4. Source control status.
  5. File diffs.
  6. Committing and pushing files.

How it works?

  • This implementation sandboxes the Web IDE in-memory source control (currently used in production) and the OPFS-based file system in a Web Worker to avoid blocking the Extensions Host thread.
  • For large repositories, in-memory file source control loads faster than the OPFS-based source control. Imagine a repository which several gigabytes of files. It's much faster to load the file-tree using the GitLab REST API than downloading these files using the git clone operation.
  • The Web Worker contains a SourceControlSystemsOrchestrator class that loads the initial File Tree view using the REST API and concurrently starts the git clone operation using isomorphic-git. When the clone repository operation finishes successfully, the orchestrator replaces the in-memory file system with the OPFS-based alternative.
  • Files are transferred between the Extension Host web worker and the File System web worker. Transferring objects avoids cloning and just moves references to memory spaces between threads which makes the file system reads/writes very fast regardless of the file size.

image.png

Risks

This enhancement requires adding CORS headers to the git HTTP protocol endpoints in the GitLab Monolith. This is a highly requested feature in the community that hasn't been implemented yet Adding CORS headers for git clone and push (gitlab#20590). We have to collaborate with the application security team to identify the risks of adding CORS headers to these endpoints.

There is a 2nd proof of concept that implements the changes required in the GitLab Monolith gitlab!197123 (closed)

Web browser support

94.01% of all web browsers tracked by caniuse https://caniuse.com/mdn-api_storagemanager_getdirectory.

MR acceptance checklist

Please evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.

Screenshots or screen recordings

Screenshots are required for UI changes, and strongly recommended for all other merge requests.

This is a quick demo of the file system where I also explain some of the motivations and goals of this change 😄 .

https://youtu.be/75Zf9Fk0MMk?si=mt4ukioy9M2VhBKh

How to set up and validate locally

Numbered steps to set up and validate the change are strongly suggested.

  1. Follow the GitLab Monolith MR's instructions to add CORS headers to the git HTTP protocol endpoints [Rails][POC] Web IDE Fast File system based on ... (gitlab!197123 - closed).
  2. In the Web IDE, run yarn start:example.
  3. All the functionality in the Web IDE should continue working as usual.
Edited by Enrique Alcántara

Merge request reports