Major upgrade fails with Operator 1.6.0 and Postgres extensions installed
Summary
I've upgraded the Stackgres Operator to v1.6.0 (via Helm). Everything went fine. I restarted all Postgres clusters maintained by the operator afterwards.
Now I'm trying to perform a major upgrade from Postgres 14 to 15. I've tried 14.6 to 15.5, this didn't work. I tried a Minor upgrade from 14.6 to 14.9, which was successful, and then a Major upgrade from 14.9 to 15.5. Same errors.
I tried it both with Hard link files
enabled and disabled. After failure the rollback finished successfully.
Current Behaviour
Major upgrade won't work with Postgres extensions installed.
Steps to reproduce
- Create a SGCluster
- Intall extensions, e.g.
timescale_tsl
orpg_stat_kcache
- Perform a major upgrade from Postgres 14.x to 15.x
Install extensions via the SGCluster manifest:
postgres:
extensions:
- name: timescaledb_tsl
publisher: com.ongres
version: 2.11.0
- name: pg_stat_kcache
publisher: com.ongres
version: 2.2.1
Expected Behaviour
Upgrade successful.
Possible Solution
It seems something goes wrong when installing the extensions in the new Postgres version. It successfully downloads the extension, but fails when installing them.
Environment
- StackGres version: v1.6.0
- Kubernetes version: v1.26.7
Relevant logs and/or screenshots
See attachment for full logs from the postgres pod and the upgrade job pod.
In short (from the postgres pod):
Detected extension com.ongres/timescaledb_tsl for version 2.11.0[pg15/x86_64/linux]
Download extension com.ongres/timescaledb_tsl for version 2.11.0[pg15/x86_64/linux]
Downloading com.ongres/timescaledb_tsl for version 2.11.0[pg15/x86_64/linux] from https://extensions.stackgres.io/postgres/repository/com.ongres/x86_64/linux/timescaledb_tsl-2.11.0-pg15-build-6.22.tar
Verify extension com.ongres/timescaledb_tsl for version 2.11.0[pg15/x86_64/linux]
Install extension com.ongres/timescaledb_tsl for version 2.11.0[pg15/x86_64/linux]
SGCluster pg-test-cluster.pg-test-cluster-postgres: install of extension com.ongres/timescaledb_tsl for version 2.11.0[pg15/x86_64/linux] failed on pod pg-test-cluster-postgres-0: java.io.UncheckedIOException: java.nio.file.NoSuchFileException: /var/lib/postgresql/relocated/15.5/6.29/usr/lib/postgresql/15.5/lib/timescaledb-2.11.0.so18bddf9dfc0.tmp
...JAVA STACKTRACE...
Detected extension com.ongres/pg_stat_kcache for version 2.2.1[pg15/x86_64/linux]
Download extension com.ongres/pg_stat_kcache for version 2.2.1[pg15/x86_64/linux]
Downloading com.ongres/pg_stat_kcache for version 2.2.1[pg15/x86_64/linux] from https://extensions.stackgres.io/postgres/repository/com.ongres/x86_64/linux/pg_stat_kcache-2.2.1-pg15-build-6.18.tar[major-upgrade-job-logs-2023-11-20_11_38_04.txt](/uploads/1fb8c95cd0f36d8e385c540813a742dd/major-upgrade-job-logs-2023-11-20_11_38_04.txt)
[postgres-pod-0-logs-2023-11-20_11_55_50.txt](/uploads/5afc622dc27601e00532473bc474e98b/postgres-pod-0-logs-2023-11-20_11_55_50.txt)
Verify extension com.ongres/pg_stat_kcache for version 2.2.1[pg15/x86_64/linux]
Install extension com.ongres/pg_stat_kcache for version 2.2.1[pg15/x86_64/linux]
SGCluster pg-test-cluster.pg-test-cluster-postgres: install of extension com.ongres/pg_stat_kcache for version 2.2.1[pg15/x86_64/linux] failed on pod pg-test-cluster-postgres-0: java.io.UncheckedIOException: java.nio.file.NoSuchFileException: /var/lib/postgresql/relocated/15.5/6.29/usr/lib/postgresql/15.5/lib/timescaledb-2.11.0.so18bddf9e005.tmp
...JAVA STACKTRACE...
Notice that it's actually weird that even for the pg_stat_kcache it failes because .tmp file from timescaledb doesn't exists.