SGShardedCluster: the SGPostgresConfig referenced by a query router override is ignored and the workers one is used instead

Summary

In an SGShardedCluster using the citus sharding technology, an entry of spec.workers.overrides[] with type: queryRouter is only partially applied:

  • the fields that end up directly in the generated SGCluster spec (sgInstanceProfile, pods, scheduling, configurations.sgPoolingConfig, ...) are honoured;
  • configurations.sgPostgresConfig is ignored: the per query router SGPostgresConfig generated by the operator is derived from spec.workers.configurations.sgPostgresConfig.

Deleting the generated SGPostgresConfig makes the operator regenerate it, still as a copy of the workers one, which confirms that the wrong source is used when the resource is built, not that the resource is stale.

Root cause

There are two override lookups for query routers, using two different index conventions:

  • StackGresShardedClusterSpec.getQueryRoutersOverrides() filters the overrides by type: queryRouter and shifts their index by spec.coordinator.queryRouterIndexOffset (default 1024). It is used by StackGresShardedClusterForUtil.getBaseQueryRouterCluster(), which is why every field that is copied into the generated SGCluster spec works as expected.
  • StackGresShardedClusterSpec.getPlainOverrides() does not filter by type and does not apply the offset. It is used by ShardedClusterWorkersClustersContextAppender.getQueryRoutersClusters(), which then matches the overrides against the global query router index (queryRouterIndexOffset + i). No override can ever match, so ShardedClusterWorkersPostgresConfigContextAppender.findPostgresConfig() falls back to spec.workers.configurations.sgPostgresConfig, and CitusShardedClusterQueryRouterPostgresConfig materializes that fallback as the per query router SGPostgresConfig.

The same broken lookup also feeds the resolved SGPoolingConfig and SGInstanceProfile of the query routers in the reconciliation context. Those are not visibly broken because the generated SGCluster spec carries the referenced names directly, but the context values are wrong, which affects validation and anything else relying on them.

Related defect

Since getPlainOverrides() neither filters by type nor applies the offset, an override with type: queryRouter and index N is also returned with index N and is therefore matched as the override of worker N by ShardedClusterWorkersClustersContextAppender.getWorkersClusters(). A query router override can then leak into the SGPostgresConfig, SGPoolingConfig and SGInstanceProfile resolved for the worker with the same index. To be confirmed with a test.

Expected behaviour and design decision

Beyond fixing the lookup, we should decide what the default Postgres configuration of a query router is when no override is provided. Query routers do not run the same workload as the workers, and inheriting the workers configuration (or the coordinator one) is arguably not a good default for either choice. Options:

  1. Minimum fix: honour overrides[].configurations.sgPostgresConfig for entries with type: queryRouter.
  2. Add a dedicated section for query routers (e.g. under spec.coordinator) holding their own configurations, used as the default for all query routers, so that a user does not need to declare one override per query router just to change their Postgres configuration.

Actionables

  • Use getQueryRoutersOverrides() (or an index aware equivalent) in ShardedClusterWorkersClustersContextAppender.getQueryRoutersClusters().
  • Make the worker lookup type aware so that a query router override can not be matched as a worker override (either by filtering in getPlainOverrides() or by replacing it with getWorkersOverrides() plus getQueryRoutersOverrides()).
  • Decide and implement the default source of the query routers configurations, and apply the same decision to sgPoolingConfig and sgInstanceProfile.
  • Add unit tests in ShardedClusterWorkersPostgresConfigContextAppenderTest, ShardedClusterWorkersPoolingConfigContextAppenderTest and ShardedClusterWorkersInstanceProfileContextAppenderTest covering a query router override and the index collision with the worker of the same plain index.
  • Add a test on CitusShardedClusterQueryRouterPostgresConfig asserting the generated SGPostgresConfig comes from the overridden SGPostgresConfig.
  • Extend the citus sharded cluster e2e spec with a query router override that references its own SGPostgresConfig.
  • State, in the SGShardedCluster CRD field descriptions and in the sharded cluster administration guide, that the index of an override of type queryRouter is 0 based within the query routers, and what the default configurations of a query router are.