Skip to content

WIP Use virtual filesystem interface when serving files

Jacob Vosmaer requested to merge jv-pages-vfs into master

This is a proof of concept for running GitLab Pages from object storage, using a "one blob per file" approach like you would with S3 static site hosting.

The main goal of this approach was to have minimal changes to working code. This is possible because the existing code uses only 3 Go standard library functions to talk to the filesystem when serving files (os.Lstat, os.Readlink and os.OpenFile), and we can reimplement those on object storage. Because we use a library that abstracts across cloud providers, we get support for S3, Azure and Google in one go.

Existing pages deployments, e.g. backed by NFS, would keep working exactly the way they do now, without any requirement to migrate.

The approach in this MR requires Rails changes as well.

  1. Pages deployment in Rails should use unique directories for each deploy (compare with "hashed storage" used for repositories etc.), rather than renaming directories. This change can be applied without migrations to existing installations. This would be a non-trivial change but it is something we have done before with other resources (repositories, uploads etc.).
  2. We would have to build a optional migration system in Rails to be able to migrate existing NFS-backed deployments to object storage. Such migrations would be optional. The idea would be to configure Rails to deploy every Pages site twice, both to NFS and object storage. At first Pages keeps serving from NFS like before. We would run a script that looks for old Pages sites that are not on object storage yet, and that copies their data from NFS to object storage. Once all existing Pages sites have been copied from NFS to object storage we would reconfigure pages to serve the data from object storage and decommission NFS.

Related to #437 (closed)

Edited by Jaime Martinez

Merge request reports