backup-utility backup fails with <registry file> not writable: Operation not permitted
Summary
I assume this is the correct location to raise this issue, since the problematic file is versioned in the CNG-repo and thus k8s-specific.
Basically the issue is that backups of our S3-backed registry fail on our instance due to permission issues.
I've briefly looked into the issue and it appears that the cause is that somehow filesystem attributes were stored with the S3 registry files, which now causes issues since the gitlab-toolbox runs as non-root and thus cannot apply those attributes during sync as described in this SO answer.
Steps to reproduce
- Set up a gitlab instance with the registry stored in a ceph-s3 bucket (others might or might not produce similar issues)
- Push images to that registry
- Run
backup-utility
within thegitlab-toolbox
pod.
If above steps are not enough to reproduce the issue you might need to first do a omnibus install, push the images there and then migrate those to s3 first, before migrating the gitlab instance from omnibus to charts.
Configuration used
global:
hosts:
domain: example.com
https: true
appConfig:
contentSecurityPolicy:
enabled: true
report_only: false
enableUsagePing: true
enableSeatLink: true
enableImpersonation: true
applicationSettingsCacheSeconds: 60
usernameChangingEnabled: true
issueClosingPattern:
defaultTheme: 2
defaultProjectsFeatures:
issues: true
mergeRequests: true
wiki: false
snippets: false
builds: true
containerRegistry: false
object_store:
enabled: true
proxy_download: false
connection:
secret: gitlab-object-store
lfs:
enabled: true
proxy_download: false
bucket: lfs
artifacts:
enabled: true
proxy_download: false
bucket: artifacts
uploads:
enabled: true
proxy_download: false
bucket: uploads
packages:
enabled: true
proxy_download: false
bucket: packages
externalDiffs:
enabled: true
proxy_download: false
bucket: external-diffs
terraformState:
enabled: true
bucket: terraform-state
ciSecureFiles:
enabled: true
bucket: ci-secure-files
dependencyProxy:
enabled: true
bucket: dependency-proxy
backups:
bucket: backup
tmpBucket: tmp
psql:
host: gitlab-pg
password:
secret: gitlab-postgres
key: password
ingress:
enabled: true
configureCertmanager: false
class: "nginx"
provider: "nginx"
tls:
secretName: example-com-tls
annotations:
nginx.ingress.kubernetes.io/proxy-body-size: "0"
nginx.ingress.kubernetes.io/proxy-buffering: "off"
nginx.ingress.kubernetes.io/proxy-request-buffering: "off"
grafana:
enabled: false
registry:
bucket: registry
minio:
enabled: false
shell:
port: 22
pages:
enabled: false
host: static.example.com
objectStore:
proxy_download: false
bucket: pages
gitlab:
webservice:
minReplicas: 2
maxReplicas: 2
sidekiq:
minReplicas: 1
maxReplicas: 1
mailroom:
hpa:
minReplicas: 1
maxReplicas: 1
spamcheck:
minReplicas: 1
maxReplicas: 1
kas:
minReplicas: 1
maxReplicas: 1
pages:
hpa:
minReplicas: 1
maxReplicas: 1
toolbox:
replicas: 1
nodeSelector:
kubernetes.io/hostname: node1
backups:
objectStorage:
config:
secret: gitlab-backup-store
key: storage.config
registry:
storage:
secret: gitlab-registry-store
key: config
maintenance:
readonly:
enabled: false
hpa:
minReplicas: 1
maxReplicas: 1
cpu:
targetAverageUtilization: 75
behavior:
scaleDown:
stabilizationWindowSeconds: 300
gitlab-runner:
install: false
certmanager:
install: false
nginx-ingress:
enabled: false
postgresql:
install: false
prometheus:
install: false
Current behavior
$ backup-utility --skip repositories2023-02-05 20:12:30 UTC -- Dumping main_database ...
Dumping PostgreSQL database gitlabhq_production ... pg_dump: warning: could not find where to insert IF EXISTS in statement "-- *not* dropping schema, since initdb creates it
"
[DONE]
2023-02-05 20:12:55 UTC -- Dumping main_database ... done
2023-02-05 20:12:55 UTC -- Dumping ci_database ... [DISABLED]
2023-02-05 20:12:55 +0000 -- Deleting backup and restore lock file
Dumping registry ...
[Error] WARNING: /srv/gitlab/tmp/registry/docker/registry/v2/blobs/sha256/01/012755b0a08b199db672bbfcc4d107bc1ee2f60c0a48564e8076e7b71daf6ef1/data not writable: Operation not permitted
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
An unexpected error has occurred.
Please try reproducing the error using
the latest s3cmd code from the git master
branch found at:
https://github.com/s3tools/s3cmd
and have a look at the known issues list:
https://github.com/s3tools/s3cmd/wiki/Common-known-issues-and-their-solutions-(FAQ)
If the error persists, please report the
following lines (removing any private
info as necessary) to:
s3tools-bugs@lists.sourceforge.net
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
Invoked as: /usr/local/bin/s3cmd --stop-on-error --delete-removed --exclude tmp/builds/* sync s3://registry/ /srv/gitlab/tmp/registry/
Problem: <class 'PermissionError: [Errno 1] Operation not permitted: b'/srv/gitlab/tmp/registry/docker/registry/v2/blobs/sha256/01/012755b0a08b199db672bbfcc4d107bc1ee2f60c0a48564e8076e7b71daf6ef1/data'
S3cmd: 2.3.0
python: 3.8.16 (default, Jan 30 2023, 20:54:49)
[GCC 10.2.1 20210110]
environment LANG=C.UTF-8
Traceback (most recent call last):
File "/usr/local/bin/s3cmd", line 3286, in <module>
rc = main()
File "/usr/local/bin/s3cmd", line 3183, in main
rc = cmd_func(args)
File "/usr/local/bin/s3cmd", line 2001, in cmd_sync
return cmd_sync_remote2local(args)
File "/usr/local/bin/s3cmd", line 1567, in cmd_sync_remote2local
ret, seq, size_transferred = _download(remote_list, seq, remote_count + update_count, size_transferred, dir_cache)
File "/usr/local/bin/s3cmd", line 1533, in _download
raise e
File "/usr/local/bin/s3cmd", line 1520, in _download
os.lchown(deunicodise(dst_file),uid,gid)
PermissionError: [Errno 1] Operation not permitted: b'/srv/gitlab/tmp/registry/docker/registry/v2/blobs/sha256/01/012755b0a08b199db672bbfcc4d107bc1ee2f60c0a48564e8076e7b71daf6ef1/data'
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
An unexpected error has occurred.
Please try reproducing the error using
the latest s3cmd code from the git master
branch found at:
https://github.com/s3tools/s3cmd
and have a look at the known issues list:
https://github.com/s3tools/s3cmd/wiki/Common-known-issues-and-their-solutions-(FAQ)
If the error persists, please report the
above lines (removing any private
info as necessary) to:
s3tools-bugs@lists.sourceforge.net
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
creation of working directory of registry failed
Expected behavior
$ backup-utility --skip repositories2023-02-05 20:12:30 UTC -- Dumping main_database ...
Dumping PostgreSQL database gitlabhq_production ... pg_dump: warning: could not find where to insert IF EXISTS in statement "-- *not* dropping schema, since initdb creates it
"
[DONE]
2023-02-05 20:12:55 UTC -- Dumping main_database ... done
2023-02-05 20:12:55 UTC -- Dumping ci_database ... [DISABLED]
2023-02-05 20:12:55 +0000 -- Deleting backup and restore lock file
Dumping registry ... done
......
Versions
- Chart: 6.8.1
- Platform:
- Self-hosted: Kubernetes (deployed via kubespray)
- Kubernetes: (
kubectl version
)- Client: v1.26.1
- Server: v1.25.6
- Helm: (
helm version
)- Client: v3.11.0
- Server: N/A
Relevant logs
(Please provide any relevant log snippets you have collected, using code blocks (```) to format)