Needed features to sufficiently enhance GitLab Pages for the docs site

Status

In light of the GitLab Pages feature enhancements since this issue was created, we are planning to stay with GitLab Pages indefinitely unless new site requirements arise that prompt the consideration of a different approach.

Obstacles to continued use of GitLab Pages

Limitation	Description	Issue	Milestone	Required for /help deprecation
Lack of storage (Completed)	The Pages artifacts is now capped to 1GB. While we could in theory change that for all GitLab.com, there's not a way to change it per project.	gitlab#16610 (closed)	%12.6	Yes
No support for splat redirects	Right now we have a bunch of meta refresh tags (~358 in EE, tested with ag "redirect_to" doc | wc -l) Also we need this in order to drop "/ee" from our help URLs.	gitlab-pages#24 (closed) (Alternative we're likely to implement first #96 (closed))	%13.4	The first Pages redirect feature was shipped with GitLab 12.4, but it only supports single file redirects. If we wanted to drop /ce /ee paths at the same time as we deprecate /help (or before), this would need to come first. However, this is not our recommendation as /ee in URLs is mainly an aesthetic issue and could cause some confusion about product name, whereas /help poses major UX issues. Also: if and when we want to archive very old versions, this type of redirect will also become necessary.
No incremental update support	We have many previous versions that don't need to be uploaded as artifacts. With the current deployment process, we upload master and the 3 previous versions.	gitlab#29265	%Backlog	Not yet, because we've set our deployments to be less frequent, but the deployment time may catch up to that, so the sooner we have this feature implemented and configured, the better.
No multiple Pages support	The current way of deploying versions is a bit hacky and it involves many moving parts. Ideally, this should be a GitLab feature.	gitlab#16208 (closed)	%Backlog	No, but this could change in the future depending on how our larger deployments scale.
Issues with large pages sites	The bigger the artifacts, the longer the extraction in the Pages server is. Sometimes it might also time out and the artifacts never get uploaded pages:deploy fails.	https://gitlab.com/gitlab-org/gitlab-ce/issues/37085	Closed	N/A. GL can how handle larger Pages sizes (Axil confirming this).

Alternative

We want to be able to use and dogfood GitLab Pages, but having our own server means:

1. We can expand the storage size as we see fit. (Update: This is now also possible with GitLab Pages.)
2. We can rsync the files much easier without the need to zip and unzip the whole site.
3. We can rsync only the docs versions that change. Old ones don't need to be moved.
4. We can use Nginx to have proper 301 redirects and not rely on meta refresh tags anymore.

Current process

1. Every month we create the new stable (single) version that lives inside a Docker image (the HTML files of all products) and is uploaded to the Registry.
2. There's a schedule that runs every once an hour on gitlab-docs master that pulls the products' master branches. It uses registry.gitlab.com/gitlab-com/gitlab-docs:latest which includes the previous 3 stable versions that are fetched from the 3 single version Docker images + master from the previous hourly build. They are then all uploaded to Pages.
3. The final Docker image that is used for production, includes the previous 3 stable versions that are fetched from the 3 single version Docker images + master. This is happening by building Dockerfile.master and ultimately uploaded as registry.gitlab.com/gitlab-com/gitlab-docs:latest. It's built every four hours from gitlab-docs master.