Second iteration at a CRD for sharding: SGShardedCluster

I'd like to propose some changes to the SGShardedCluster implemented in target_version1.5.0-beta1.

We need to add much more configuration from the SGCluster to the SGShardedCluster for both coordinator and shards. In particular:

  • scheduling
  • metadata
  • extensions
  • ssl
  • replication
  • customVolumes and customContainers
  • postgresServices for any coordinator, primary coordinator and any primary shard
  • disableConnectionPooling, disableMetricsExporter and disablePostgresUtil
  • resources
  • credentials
  • managementPolicy
  • managedSql
  • distributedLogs
  • backups

More or less all those are straightforward changes and could be implemented easily with just checking the spec is applied as is to the generated SGCluster.

It is a non goal of this issue to add support for replicas service, backups, initialData and replicateFrom. Those developments have to go in separate standalone issues since they require to provide a PoC, some testing and, maybe, some support on other CRD (like SGBackup).

Another proposal we need to address is to provide specific configuration overrides for individual sharded clusters. The proposal to achieve that is to create a new section called .spec.shards.overrides. It will be an array of object where each object will allow any field already available in the .spec.shards section with the only exception of the clusters field. A new field index of each object in the array of the new field .spec.shards.overrides will indicate the shard to which the specified configuration overrides will apply. This section, if not specified, will indicate that all shards have the same configurations as specified in the .spec.shards section.

apiVersion: stackgres.io/v1alpha1
kind: SGShardedCluster
spec:
  postgres: {} # same as SGCluster.spec.postgres
  replication: {} # same as SGCluster.spec.replication but without role and groups
  postgresServices:
    coordinator:
      any: {} # same as SGCluster.spec.postgresServices.* but without customPorts
      primary: {} # same as SGCluster.spec.postgresServices.* but without customPorts
      customPorts: {} # same as SGCluster.spec.postgresServices.*.customPorts
    shards:
      anyPrimary: {} # same as SGCluster.spec.postgresServices.* but without customPorts
      customPorts: {} # same as SGCluster.spec.postgresServices.*.customPorts
  configurations:
    credentials: {} # same as SGCluster.spec.configurations.credentials
    backups: # Same as SGCluster.spec.configurations.backups but without the .path and with .paths 
      - paths: [ <string> ] # set and updated automatically by the operator
  metadata: {} # same as SGCluster.spec.metadata
  coordinator:
    instances: 2
    managedSql: {} # same as SGCluster.spec.managedSql
    replication: {} # same as SGCluster.spec.replication override SGShardedCluster.spec.replication
    configurations: {} # same as SGCluster.spec.configurations excluding backups and credentials
    pods: {} # same as SGCluster.spec.pods
    metadata: {} # same as SGCluster.spec.metadata override SGShardedCluster.spec.metadata
  shards:
    cluster: 2
    instancesPerCluster: 2
    sgInstanceProfile: size-s
    managedSql: {} # same as SGCluster.spec.managedSql
    replication: {} # same as SGCluster.spec.replication override SGShardedCluster.spec.replication
    configurations: {} # same as SGCluster.spec.configurations excluding backups and credentials
    pods: {} # same as SGCluster.spec.pods
    metadata: {} # same as SGCluster.spec.metadata override SGShardedCluster.spec.metadata
    overrides: # every thing under here override SGShardedCluster.spec.shards (do not include .clusters and .overrides)
      index: 0
      instancesPerCluster: 3
      sgInstanceProfile: size-l
Edited by Matteo Melli