Deploy CHProxy as part of each CR to enforce limits and access control

Summary

This MR adds proxy configuration to ClickHouse CRD and deploys CHProxy using the specified settings.

The proxy settings in this MR define:

• Proxy users and their mapping to ClickHouse users with (see .spec.users[].toUser)
• Proxy listeners. This MR only supports HTTP.
• Number of proxy replicas

With this settings the Operator uses a Deployment with the specified number of replicas to deploy CHProxy instances and configures them to use the backend headless Services that are defined per node/StatefulSet. For unassigned values it tries to uses default values or fall back to CHProxy defaults.

Testing

Grab the latest image SHA for this branch from ch-operator image Job, e.g. this and use the following to deploy the Operator from this branch:

SHA=sha256:6243e39d093b6b8f771b98288828470d67fa87f720bbd841b8924df0f237981a
IMG=registry.gitlab.com/gitlab-org/opstrace/opstrace/clickhouse-operator@${SHA} 

# Install CRD
make install

# Deploy Operator
make deploy

Use the following ClickHouse CR (and Secrets) to test:

apiVersion: v1
kind: Secret
metadata:
  name: example-users
  namespace: clickhouse
type: Opaque
stringData:
  admin-password: !ns3cure
---
apiVersion: v1
kind: Secret
metadata:
  name: example-proxy-users
  namespace: clickhouse
type: Opaque
stringData:
  password: s3cret
---
apiVersion: clickhouse.gitlab.com/v1alpha1
kind: ClickHouse
metadata:
  name: example
  namespace: clickhouse
spec:
  deploy:
    image: clickhouse/clickhouse-server:22.4.2.1
    replicas: 3
    storageSize: 5Gi
    affinity: {}
  adminUsers:
  - name: admin
    secretKeyRef:
      name: example-users
      key: admin-password
  proxy:
    enabled: true
    users:
    - name: example
      passwordSecret:
        name: example-proxy-users
        key: password
      # add user settings here

Limitations

The following CHProxy features is not included in this MR:

• HTTPS listener support
• Caching support
• Parameter filtering
• Heartbeat configuration
• Deployment details for CHProxy, e.g. image name, tag, version.

Related issues

Closes #1646 (closed)