Architecture of how to run a stateful application on Kubernetes

Context

As part of the Gitlab.com on Kubernetes Epic and the Kubernetes Migration WG we got to the point that most of the components are already migrated to Kubernetes (excluding Databases and HAProxy, Console). While Redis on deployments on Kubernetes is in progress

Problem

On the 2022-06-13 Kubernetes Migration WG call we have been discussing the possible migration of Gitaly on Kubernetes (overarching Epic.

We currently don't have any Stateful service running on our Kubernetes installation. We don't currently know what we would exactly need, on the Kubernetes world, to run a stateful service with data persisted on top of Kubernetes.

E.g: Stateful sets, how do we persist data, dedicated volumes, network identity, etc.

The Redis deployments on Kubernetes effort has some similarities with this since it is also a Stateful service. For the production readiness review of Redis on Kubernetes, Scalability already went through some investigations that could be useful for this exercise.

Goal of this issue

Sketch a high-level architecture of what would be the best components setup to run a Stateful service (like Gitaly on Kubernetes). This will be the first iteration. It will also serve as a set of requirements/guide (tasks) that every team developing/owning a stateful application needs to consider to have the application successfully running on Kubernetes.