Skip to content

Server Side backup on the self-managed S3 object storage failed

Everyone can contribute. Help move this issue forward while earning points, leveling up and collecting rewards.

Hi GitLab users,

I have a test 3k-Architecture with 3 gitay repository nodes on the openstack. The deployment is done by GitLab Environment Toolkits (GET), successfully. I can create projects and groups. Few test users are created as well. GitLab version is v16.5.1 (latest release)

To make a server side backup, I have configured accordingly for gitaly environment to access self-managed S3 storage system.

gitaly['env'] = {
  'AWS_ACCESS_KEY_ID' => "xxxxxxxxxxx",
  'AWS_SECRET_ACCESS_KEY' => "xxxxxxxxxxxxxxxxx"
}
gitaly['configuration'] = {
  backup: {
    go_cloud_url: 's3://gitlab-dev?endpoint=gitlab.s3.scc.kit.edu:9021&disableSSL=true&s3ForcePathStyle=true'
  }
}

In GET, a custom configuration is involved

To avoid an ambiguity from customize configuration on GET, I have configured above gitaly environment part directly gitlab.rb in each gitaly node (1,2,3), and also into the gitlab-rails (1,2,3) node. And I have started to backup on my test system.

In gitlab-rails-1 node, I can complete backup for repository on my local disk.

gitlab-backup create --trace

output_backup_local.txt

But server side backup is broken, due to misconfiguration or rpc protocol error from gitaly node.

gitlab-backup create REPOSITORIES_SERVER_SIDE=true --trace

output_backup_server_side.txt

In the documentation, it is not required to use any region, becasue self-managed S3 does not running with regional configuration.

In our productive system, we have already attached s3 with below configurtion, it works properly, but S3 configuration in the gitaly environment cannot be interpreted in the backup task correctly.

gitlab_rails['object_store']['connection'] = {
  provider: "AWS",
  aws_access_key_id: "xxxxxxxxxxxxxxxx",
  aws_secret_access_key: "xxxxxxxxxxxxxxxxxxxxxx",
  endpoint: "https://gitlab.s3.scc.kit.edu:9021",
  path_style: true
}

From each server node (gitaly 1,2,3 and gitlab-rails 1,2,3) I can access to S3 with same endpoint usign python boto3 or s3cmd.
It is not the network issue because all ports are simply open while I am testing.

Q1) In the output of server side backup, this error message MissingRegion: could not find region configuration come out. Do I need to setup also a region for self managed S3 case?

Q2) How do I know correctly, one of gitaly node(dedicated backup task) can access to S3 with gitaly["env"] configratuion? (not general access from server)

Does anyone have some experience server side backup with self-managed S3?

Best regards,

Donghee

Edited by 🤖 GitLab Bot 🤖