Compatible and conflicting features for namespace sharding
I would like to summarize the strategy we are currently discussing around topics like "database sharding", "service extraction" and "application sharding" in a rather cohesive doc in the handbook.
I'll use this issue as a scratch pad.
"If we shard by root namespace, people only care about people in their organization."
I believe the common ground for "how to scale GitLab" (the application) is the idea that everything belongs to a namespace. This is a great summary: https://www.youtube.com/watch?v=TGulb4sGJ9g&feature=youtu.be&t=1324
This idea has motivated the thinking around root-namespace sharding because so many GitLab features and their data reside in a namespace. A problem is that GitLab (the application) doesn't have a notion of an "organization". Your user can be active in many (root) namespaces and this is what makes it difficult to shard like this at the moment: How do you check which issues your user is assigned to (as an example)? The scope of this is global - the access-pattern is "by user", which doesn't benefit from namespace-based sharding (you end up checking all namespaces). Similar issue with /explore , user dashboards, TODOs. CI scheduling is site-wide/global AFAIK.
For now, I'd like to compile an overview of features that conflict with namespace-based sharding. This should be informative for conversations about feasibility of implementing sharding in GitLab.
Conflicting Features
| Feature | Conflict |
|---|---|
| Audit Events | 75% of the data is not associated with a namespace. |
| Runners | Can be created for a project or group and then assigned to another. |
| Merge Requests | Can be created from forks from a different namespace |
| Admin Console | Configures instance wide behavior. Examples kubernetes clusters, runners, etc |
| Issue_Links | Issue can relate issues across namespaces |
| Management (Ownership) of Kubernetes clusters | These can move across namespaces |
| Multi project pipelines | https://docs.gitlab.com/ee/ci/multi_project_pipelines.html |
| User dashboards | Issues and MRs search by assignee, author, etc |
| Search by user across namespace | Similar to dashboards |
| Global snippets | |
| Instance wide audit log | |
| User profile | Contribution log, events, etc |
| Todos | |
| Group -> Group sharing | https://docs.gitlab.com/ee/user/group/#sharing-a-group-with-another-group |
| Group - movement | You can transfer a group or a project 'anywhere': https://docs.gitlab.com/ee/user/project/settings/#transferring-an-existing-project-into-another-namespace / https://docs.gitlab.com/ee/user/group/#transferring-groups |
| Issue - movement | You can also move an issue across any two projects, as long as you have sufficient permissions: https://docs.gitlab.com/ee/user/project/issues/managing_issues.html#moving-issues |
Easily Shardable Features (by Namespace)
| Feature | Conflict |
|---|