Files don't get deleted in S3 backend
I added a fileserver to my existing Psono installation. That worked fine, I can upload files without issues. However, when I delete a file, it disappears from Psono, but is not deleted from the backing S3 bucket.
Bucket content before upload:
2022-07-07 16:03:58 736 1b/21/fb/16/1b21fb1620bc0dd97eae00c3adc86695acf6449c23fa1165cb468e63202f0700874fe989b902cc76aa27f0cb1e6c40400063ed185aacdbbefdb9f095420ad640
2022-07-07 15:51:29 736 32/d4/b7/99/32d4b7999522e16d39fed7390597c08d221e3cd2595bb0a995ff28c0f3534f19f25297f95731bff7b845edd1131b292c1eaf9253651aea42679d435796b3f5a2
2022-07-07 15:57:40 736 c3/0d/2e/a6/c30d2ea66d7e97eacea8f1fa01557bcb3f41f2585ad097c4e48700ecaafd5bcaba65588ec31e8d4f1d53c7087d6cf80affe5c73e50638595e3a02d3518da14f4
Bucket content after upload:
2022-07-08 07:35:52 421237 15/b3/7f/d4/15b37fd4e61a2e630040c36d104712880179385b075cd824e346b32c0b2da3bd9ce934a7badfaae28bc37fb8f9eb5e167a55fb315faae8e428c12d8587bcc16c
2022-07-07 16:03:58 736 1b/21/fb/16/1b21fb1620bc0dd97eae00c3adc86695acf6449c23fa1165cb468e63202f0700874fe989b902cc76aa27f0cb1e6c40400063ed185aacdbbefdb9f095420ad640
2022-07-07 15:51:29 736 32/d4/b7/99/32d4b7999522e16d39fed7390597c08d221e3cd2595bb0a995ff28c0f3534f19f25297f95731bff7b845edd1131b292c1eaf9253651aea42679d435796b3f5a2
2022-07-07 15:57:40 736 c3/0d/2e/a6/c30d2ea66d7e97eacea8f1fa01557bcb3f41f2585ad097c4e48700ecaafd5bcaba65588ec31e8d4f1d53c7087d6cf80affe5c73e50638595e3a02d3518da14f4
Bucket content after deleting the file:
2022-07-08 07:35:52 421237 15/b3/7f/d4/15b37fd4e61a2e630040c36d104712880179385b075cd824e346b32c0b2da3bd9ce934a7badfaae28bc37fb8f9eb5e167a55fb315faae8e428c12d8587bcc16c
2022-07-07 16:03:58 736 1b/21/fb/16/1b21fb1620bc0dd97eae00c3adc86695acf6449c23fa1165cb468e63202f0700874fe989b902cc76aa27f0cb1e6c40400063ed185aacdbbefdb9f095420ad640
2022-07-07 15:51:29 736 32/d4/b7/99/32d4b7999522e16d39fed7390597c08d221e3cd2595bb0a995ff28c0f3534f19f25297f95731bff7b845edd1131b292c1eaf9253651aea42679d435796b3f5a2
2022-07-07 15:57:40 736 c3/0d/2e/a6/c30d2ea66d7e97eacea8f1fa01557bcb3f41f2585ad097c4e48700ecaafd5bcaba65588ec31e8d4f1d53c7087d6cf80affe5c73e50638595e3a02d3518da14f4
Running testconfig doesn't show an issue either:
Success: SECRET_KEY configuration found.
Success: PRIVATE_KEY configuration found.
Success: PUBLIC_KEY configuration found.
Success: SERVER_URL configuration found.
Success: SERVER_PUBLIC_KEY configuration found.
Success: CLUSTER_ID configuration found.
Success: CLUSTER_PRIVATE_KEY configuration found.
Success: HOST_URL configuration found.
Success: SHARDS configuration found.
- Success: SERVER_URL format seems to be correct
- Success: Your SERVER_URL port is an integer.
- Success: Your SERVER_URL port is in the correct range.
- Success: Host resolved
- Success: Host and port exist and firewall seems to allow connections.
Testing shard 9ba2f826-38fb-491f-99d6-7304b73ece22:
- Success: Required property read present
- Success: Required property write present
- Success: Required property delete present
- Success: Required property engine present
- Success: property engine has the right type
- Success: Required property class present
- Success: Required property kwargs present
- Success: Test file successfully created with user unknown.
- Success: Stored test file successfully deleted with user unknown.
- Success: Connection to server and authorization successful.
Successfully completed. (Don't forget to restart your fileserver if you have modified your settings.yaml.)
It's also not a permission issue since I can delete that file using the shard's AWS credentials:
AWS_PROFILE=psono aws s3 rm s3://XXXXXXXXXXXX-psono-files/bb/71/1a/d1/bb711ad18343d6f4f2d6d51f5bc279a61c5f64ffbebabc19cb101f105cb59f74bb466cbe98aaa6e3bcc21e05454ce05443e6606ae17dbc91054043c88e307b3c
delete: s3://XXXXXXXXXXXX-psono-files/bb/71/1a/d1/bb711ad18343d6f4f2d6d51f5bc279a61c5f64ffbebabc19cb101f105cb59f74bb466cbe98aaa6e3bcc21e05454ce05443e6606ae17dbc91054043c88e307b3c
Versions: client: 2.0.6 server: 2.0.7 fileserver: 1.0.4