GitLab Pages without DNS wildcard - MVC
### Description
Currently, creating a GitLab pages project would require a domain of the form `name.example.io` or `name.pages.example.io`. This setup forces admins to setup wildcard DNS records and SSL/TLS certificates (i.e. `*.pages.example.io`). Some services typically used by small structures or individuals (~~such as Let's Encrypt~~) disallow this.
Removing the requirement for wildcard certificates would allow using services ~~like Let's Encrypt~~ on the main GitLab pages domain, _without the need to setup a custom domain_. It would also allow for easy configuration of custom HTTP(S) proxies like Apache. Finally, IMHO having an exponential amount of off-the-radar subdomains is really bad when you try to keep your DNS space clean and tidy.
~~"feature proposal" gitlab-ce~~4107696
### Proposal
In order to remove the need for wildcard dns, the namespace can be included in the URL path instead of the domain.
Thus, `https://name.pages.example.io/index.html` would become `https://pages.example.io/name/index.html`.
This could be done by setting a specific variable in `/etc/gitlab/gitlab.rb`, e.g.
```ruby
gitlab_pages["namespace_in_path"] = true;
```
This variable would then change the URL parsing rule used by GitLab pages so that it will look for the namespace in the URL path instead of the subdomain.
In order to ensure compatibility with existing setups, this feature would obviously be turned off by default.
As part of [pages rearchitecture](https://gitlab.com/groups/gitlab-org/-/epics/1316 "GitLab Pages Architecture Updates") we're moving host resolving to the rails API(https://gitlab.com/gitlab-org/gitlab/blob/a5d7d416e8f4c101120a051549f4f1c0f9db8be9/lib/api/internal/pages.rb#L27).
Also, there are a few configuration possibilities:
1. parameter in omnibus, we generally try to avoid this, as this is more complicated then tweaking the UI, but we'll need this if we add some additional parameter for `gitlab-pages` daemon
2. UI checkbox `use path instead of subdomains` in admin panel(if we don't need new parameters for pages
3. `pages-daemon` param itself
We need to choose between:
* [implementation inside pages daemon](https://gitlab.com/gitlab-org/gitlab/-/issues/29841#note_305782840 "Tech Eval: GitLab Pages without DNS wildcard (golang version)")
* [implementation using nginx proxy](https://gitlab.com/gitlab-org/gitlab/-/issues/211677 "Tech Eval: GitLab Pages without DNS wildcard (NGINX-config version)")
### If we choose any one of the below two implementation:
I would declare this feature `experimental` and start merging some code, maybe some users can try this new mode in their environment. This serves a few purposes:
1. It's hard to be 100% sure that we won't find some major problems with this approach in the future. And if in that case it will be nice to have options:
1. redesign implementation(including configuration parameters)
2. remove the feature completely
2. it will allow us to get some early feedback from users. Since we won't enable this mode on .com, some production cases will be nice to see.
### Pages daemon implementation
So we can divide development into 3 main phases:
#### Developing `experimental` version(we can call it MVC):
1. Finish the PoC https://gitlab.com/gitlab-org/gitlab-pages/-/merge_requests/256 and merge it
2. Test the authentication workflow and it will probably require some additional fixes
3. Omnibus MR to configure both rails and pages parameters
4. gitlab-pages MR to stop requiring wildcard certificates
5. one of:
1. omnibus MR to create pages auth application with base url `pages.example.com/projects/auth` instead of `projects.pages.example.com/auth`
2. or gitlab-pages MR to allow serving `projects.pages.example.com` in the new mode(may require additional SSL certificate)
6. very basic MR for documentation
it will have `/weight 6`
<details>
<summary>Everything else is outside of this MVC</summary>
#### Collecting some feedback from users and fixing bugs
Ask some users upvoted issue to try this feature.
#### Moving feature to `stable` status
If there will no major problems discovered by us and by first adopters, we can declare this feature stableWe'll need to:
1. Write better documentation for it, including migration from the current setup(e.g. recreating auth application to have new redirect url)
2. write more extensive test suite for this mode including:
1. tests for auth workflow
2. tests for artifacts proxying
3. remove all the mentions and warnings about `experimental` status
</details>
### Nginx implementation:
So we can divide development into 3 main phases:
#### Developing `experimental` version (we can call it MVC):
- [x] Implement NGINX config as mentioned [here](https://gitlab.com/gitlab-org/gitlab/-/issues/211677#note_1646965167 "Tech Eval: GitLab Pages without DNS wildcard (NGINX-config version)") in Omnibus: https://gitlab.com/gitlab-org/omnibus-gitlab/-/merge_requests/7250
- [x] Implement path-based cookie: https://gitlab.com/gitlab-org/gitlab-pages/-/merge_requests/929
- [x] Test the authentication workflow. (Note: We have already tested this in the research issue.)
- [x] GitLab MR to update/add experimental Pages URL in Deploy \> Pages: https://gitlab.com/gitlab-org/gitlab/-/merge_requests/137615
- [x] Basic MR for documentation: https://gitlab.com/gitlab-org/gitlab/-/merge_requests/137289
Next iteration:
- [ ] After initial feedback will add support in Helm charts: https://gitlab.com/gitlab-org/charts/gitlab/-/merge_requests/3491
It will have `/weight 6`.
<details>
<summary>Everything else is outside of this MVC</summary>
#### Collecting some feedback from users and fixing bugs
Ask some users upvoted issue to try this feature.
#### Moving feature to `stable` status
If there will no major problems discovered by us and by first adopters, we can declare this feature stableWe'll need to:
1. Write better documentation for it, including migration from the current setup(e.g. recreating SSL certificate)
2. Write more extensive test suite for this mode including. I am not sure, how much is possible as there is only one QA test for GitLab-pages create.
3. Remove all the mentions and warnings about `experimental` status: https://gitlab.com/gitlab-org/gitlab/-/issues/434370+
</details>
### Feedback issue:
Please add your feedback or report any issues you encounter after trying out the experimental solution: https://gitlab.com/gitlab-org/gitlab/-/issues/434372+
### Links / references
[Let's Encrypt FAQ](https://letsencrypt.org/docs/faq/)
### Reserves
This setup might have security or technical issues I do not know of. I have not been able to find the reasons behind the choice to include namespaces as subdomains, and I would be interested to know about them.
_This page may contain information related to upcoming products, features and functionality. It is important to note that the information presented is for informational purposes only, so please do not rely on the information for purchasing or planning purposes. Just like with all projects, the items mentioned on the page are subject to change or delay, and the development, release, and timing of any products, features, or functionality remain at the sole discretion of GitLab Inc._
_This page may contain information related to upcoming products, features and functionality. It is important to note that the information presented is for informational purposes only, so please do not rely on the information for purchasing or planning purposes. Just like with all projects, the items mentioned on the page are subject to change or delay, and the development, release, and timing of any products, features, or functionality remain at the sole discretion of GitLab Inc._
<!-- triage-serverless v3 PLEASE DO NOT REMOVE THIS SECTION -->
*This page may contain information related to upcoming products, features and functionality.
It is important to note that the information presented is for informational purposes only, so please do not rely on the information for purchasing or planning purposes.
Just like with all projects, the items mentioned on the page are subject to change or delay, and the development, release, and timing of any products, features, or functionality remain at the sole discretion of GitLab Inc.*
<!-- triage-serverless v3 PLEASE DO NOT REMOVE THIS SECTION -->
issue