Skip to content

Cloud native backup does not support deployments in AWS with IMDSv1 disabled

Summary

Customer raised a ticket to troubleshoot cloud-native backups that weren't working. GitLab team members can read more in the ticket.

The reason is that they've disabled IMDSv1, and s3cmd doesn't seem to support IMDSv2. AWS EKS best practice advises disabling IMDSv1 for nodes and pods.

Potential fix: use Rails for Helm backups - #1127

This is one of a number of issues around GitLab support for deployments with only IMDSv2 enabled. For more information see the description and comments in: gitlab-org/gitlab#334160

What is IMDSv2

  • Instance Metadata Service Version 2
  • IMDS is the AWS API that's available at 169.254.169.254
  • One use case is obtaining credentials in an environment that uses IAM.

s3cmd

looking at the s3cmd code, it makes an HTTP connection to 169.254.169.254, and then:

request('GET', "/latest/meta-data/iam/security-credentials/")

This will return JSON payload, and from that AccessKeyId, SecretAccessKey, and Token can be extracted.

However, it looks like s3cmd is only using IMDSv1, because the steps documented for using IMDSv2 are:

  • obtain a session token with: PUT "http://169.254.169.254/latest/api/token"
  • include that token in GET requests to the instance metadata service

Steps to reproduce

  • Deploy GitLab to EKS
  • Disable IMDSv1
  • Using IAM, attempt to back up GitLab

Configuration used

(Please provide a sanitized version of the configuration used wrapped in a code block (```yaml))

(Paste sanitized configuration here)

Current behavior

Backups don't work with IMDSv2 only

Expected behavior

Customers can disable IMDSv1 per AWS recommendations and backups still work

Versions

GitLab 13.11.5-ee

  • Chart: 4.11.5
  • Platform:
    • Cloud: EKS

Relevant logs

(Please provide any relevate log snippets you have collected, using code blocks (```) to format)

Edited by Ben Prescott_