Skip to content

Support replicate from instances with layout not compatible with SGCluster

Problem to solve

In some cases replicate from break when replicating from a Postgres instance with a different layout than the one provided for the SGCluster.

We want to support such kind of scenario by allowing users to configure custom commands to apply after the base backup or the restore of the database.

Further details

Replicate from can replicate from any Postgres instance by using streaming replication with restore from pg_basebackup or using WAL shipping with restore from an SGBackup.

In both cases Patroni requires a layout that does not break the operation. In particular for:

  • Configuration files that changes properties that can break the filesystem layout, the network layout or the logs layout.
  • Tablespaces that may break the filesystem layout.

Proposal

Allow to configure the:

  • Replicate from external instance in order to use custom method:
kind: SGCluster
spec:
  replicateFrom:
    instance:
      external:
        customRestoreMethod:
          command: <string> # A command to be run to restore the leader replica. Mutually exclusive with `script` field.
          script: <string> # A shell script to be run to restore the leader replica. Mutually exclusive with `command` field.
          no_leader: <boolean> # If set to `true` allows Patroni to call the replica creation method even if there is no running leader or replicas. In that case, an empty string will be passed in a connection string. This is useful for restoring the formerly running cluster from the binary backup.
          keep_data: <boolean> # If set to `true`, will instruct Patroni to not clean PGDATA folder before calling restore.
          no_params: <boolean> # If set to `true`, restricts passing parameters to custom command or script.
          parameters: # A list of optional parameters to pass to the command or script.
            <string>: <string>
  • Restore from an SGBackup using a custom method:
kind: SGBackup
spec:
  status:
    customRestoreMethod:
      command: <string> # A command to be run to restore the leader replica. Mutually exclusive with `script` field.
      script: <string> # A shell script to be run to restore the leader replica. Mutually exclusive with `command` field.
      no_leader: <boolean> # If set to `true` allows Patroni to call the replica creation method even if there is no running leader or replicas. In that case, an empty string will be passed in a connection string. This is useful for restoring the formerly running cluster from the binary backup.
      keep_data: <boolean> # If set to `true`, will instruct Patroni to not clean PGDATA folder before calling restore.
      no_params: <boolean> # If set to `true`, restricts passing parameters to custom command or script.
      parameters: # A list of optional parameters to pass to the command or script.
        <string>: <string>

This will also open the door to restore any kind of backups including a pg_dump by crafting a custom SGBackup.

Testing

  • [] The custom method is used correctly for replicating from an external instance.
  • [] The custom method is used correctly for restoring from a custom SGBackup.

Acceptance Criteria

  • [] Implement the feature
  • [] Document the feature
  • [] Test the feature