First iteration at a CRD for sharding: SGShardedCluster

I'd like to propose working on a new CRD, tentatively named SGShardedCluster to support automatic creation and management of sharded clusters in StackGres.

A sharded cluster is a set of clusters, which are architecturally independent of each other, but are logically linked by a given sharding technology that treats them and queries them, potentially, as a single, logical, single cluster. There is no in-core sharding solution, but several sharding technologies outside of Postgres core. Our aim is to potentially support several or all of them.

It is a non goal of this issue to develop a new sharding solution or solve problems or limitations of existing sharding solutions; but instead to leverage them and create a level of abstraction in the CRD that allows for very simple creation and management of sharded clusters.

Since the process of creating a generic CRD to support this use case is hard, and may require further iterations; it is a goal of this issue to discuss and implement a first, basic version of it.

Proposed guidelines for SGShardedCluster:

  • The CRD should be marked as alpha for this issue and upcoming release, so we can iterate it freely.
  • The CRD should not generate any resource-consumption resource in Kubernetes (e.g. Pods) but rather delegate this to other Kubernetes or SG CRDs. Essentially, the job of the controller of this CRD is to generate other SGClusters (and possibly Secrets, Services, etc).
  • The CRD should make room for more than one future sharding technology.
  • For this first version, it is proposed that Citus is used for sharding.