Transition GitLab pages to object storage instead of NFS

First part of the refactoring: &1316 (closed)

• Add support for object storage as a backend in gitlab-pages
• Add ability to create a .zip file during the pages deploy process
• Add ability to deploy to object storage during the pages deploy process
• Create a migration strategy to move sites from file storage to object storage

Note: Don't forget to use project name independent URLs for object storage(see hashed storage). discussion

Possible ways to implement

Serve directly from zip artifacts

See gitlab-pages!136 (closed) for and idea

PoC serving from object storage using pre-signed URLs gitlab-pages!299 (closed)

Download zip artifacts by pages and use current disc serving

Build CDN between object storage and pages

Open questions

will that allow as to get rid of jailing?

mentioned in epic &1316 (closed)

added to epic &1316 (closed)

marked this issue as related to #25411 (closed)

added Category:Pages devopsrelease [DEPRECATED] + 2 deleted labels

marked this issue as related to #28784 (closed)

changed the description

I extracted this issue from &1316 (closed) mostly to link blocked issues here.

cc @krasio @jaime @jhampton @jmeshell

this is super helpful @vshushlin - should we have it on the cicdactive board?

This is really great, thanks @vshushlin!

@jmeshell, good call, that might be nice.

Is cicdactive meant to indicate something we start to work on and finish with a milestone or 2?

This issue is more of an epic and we'll break it down to a few smaller actionable once.

marked this issue as related to gitlab-pages#137 (closed)

changed milestone to %Backlog

added [deprecated] Accepting merge requests + 1 deleted label

mentioned in issue gitlab-pages#355 (closed)

marked this issue as related to gitlab-pages#355 (closed)

marked this issue as related to gitlab-pages#196 (closed)

gitlab-pages!136 (diffs) has an attempt to serve from zip artifacts.

cc @jaime it might help you with your PoC

@grzesiek can you post your idea about using CDN between object storage and Pages?

changed the description

@ayufan do we have any reason for jailing except for preventing pages from serving random files from the file system?

One of the approaches we're discussing is to download zip artifacts and cache them on the pages side. We very much want to get rid of jailing, but in that case probably need to replace it with something.

do we have any reason for jailing except for preventing pages from serving random files from the file system?

This is the only reason. I hope that we get rid of chrooting magic, once we switch to .zip.

@ayufan @nolith we want to start discussions on implementing object storage.

I invited you to our next Thu call and we have some time to discuss it async before.

Please post your thoughts on that. I found some old links, but if you remember something else, please add them

cc @nick.thomas

@vshushlin

I had PoC that was able to serve directly from .zip (regardless if it is local or remote stored). It required some caching approaches to cache metadata info, but maybe this is the best first step to understand how performant it could be?

We know that this is not the most performant, but maybe this is a good starting point. Maybe if pages would be in front of some CDN this would be simply enough (as for performance goes?)

gitlab-pages!136 (closed)

The idea behind using .zip is that we can then effectively implement GitLab Pages: Review Apps basically for every MR, as we serve directly from archive, and we do not need any storage to unpack.