Test reference architecture with Praefect database colocated with GitLab database
In https://docs.gitlab.com/ee/administration/gitaly/praefect.html#postgresql we say:
> Note: do not store the GitLab application database and the Praefect database on the same PostgreSQL server if using Geo. The replication state is internal to each instance of GitLab and should not be replicated.
@francispotter and @dtacheny are trying to propose a reference architecture with Geo and Gitaly Cluster to a number of potential customers, but adding a separate PostgreSQL cluster for Praefect complicates matters.
As mentioned in https://gitlab.com/gitlab-org/gitlab/-/issues/214986#note_349559248, I think it's possible to colocate the Praefect logical database (`praefect_production`) with the GitLab logical database `gitlabhq_production` in the same cluster. The main downside is that it generates more and unnecessary network traffic and database load for all the Geo PostgreSQL replicas, but that may be less of a concern than managing more nodes.
@grantyoung I think we'll want to do performance testing with the Praefect database colocated to validate that there aren't any issues with doing this. I think this is mostly a configuration and documentation issue to get going. The layout would look like:
```plantuml
package "Geo Primary" {
database "Primary DB" {
[gitlabhq_production]
[praefect_production]
}
}
package "Geo Secondary" {
database "Replicated DB" {
[gitlabhq_production] --> [gitlabhq_production (read-only)]
[praefect_production] --> [praefect_production (read-only)]
}
database "Tracking DB" {
[gitlabhq_geo_production]
[praefect_production (read-write)]
}
}
```
Adding @tpazitny (for Quality scheduling input), @nhxnguyen (mostly FYI for Geo), @abrandl (for database input)
issue