Support S3 server side encryption in CI cloud native job logs

Previously if S3 server side encryption were required to store CI artifacts via a bucket policy, the final migration to object storage would fail, leaving the job log unmigrated.

To fix this, we need to send the right headers (e.g. x-amz-server-side-encryption) to the object storage backend via Fog.

Relates to #273498 (closed)

Testing

1. Set up an S3 bucket with this policy that rejects any uploads that don't have the right server side encryption headers:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "VisualEditor0",
            "Effect": "Allow",
            "Action": [
                "s3:PutObject",
                "s3:GetObject",
                "s3:AbortMultipartUpload",
                "s3:DeleteObject"
            ],
            "Resource": "arn:aws:s3:::stanhu-s3-workhorse-testing/*"
        },
        {
            "Sid": "DenyIncorrectEncryptionHeader",
            "Effect": "Deny",
            "Action": "s3:PutObject",
            "Resource": "arn:aws:s3:::stanhu-s3-workhorse-testing/*",
            "Condition": {
                "StringNotEquals": {
                    "s3:x-amz-server-side-encryption": "AES256"
                }
            }
        },
        {
            "Sid": "DenyUnEncryptedObjectUploads",
            "Effect": "Deny",
            "Action": "s3:PutObject",
            "Resource": "arn:aws:s3:::stanhu-s3-workhorse-testing/*",
            "Condition": {
                "Null": {
                    "s3:x-amz-server-side-encryption": "true"
                }
            }
        }
    ]
}

2. Enable CI live job traces: Feature.enable(:ci_enable_live_trace)
3. Create a simple CI job:

image: alpine:latest

test:
  script:
    - date
    - echo "hello" > test.txt
    - apk add curl
    - curl -o guide.txt http://www.textfiles.com/programming/cguide_3.txt
    - cat guide.txt
  artifacts:
    paths:
      - test.txt

4. Configure GitLab for object storage:

gitlab_rails['object_store']['connection'] = {
  'provider' => 'AWS',
   'region' => 'us-west-2',
  'use_iam_profile' => true
}
gitlab_rails['object_store']['storage_options'] = { 'server_side_encryption' => 'AES256' }

5. Run the CI job. Before this merge request, this error would be seen in Sidekiq:

Before

{
  "severity": "WARN",
  "time": "2020-11-12T21:34:38.282Z",
  "error_class": "Excon::Error::Forbidden",
  "error_message": "Expected(200) <=> Actual(403 Forbidden)\nexcon.error.response\n  :body          => \"<?xml version=\\\"1.0\\\" encoding=\\\"UTF-8\\\"?>\\n<Error><Code>AccessDenied</Code><Message>Access Denied</Message><RequestId>6246E8900FE3422F</RequestId><HostId>y8nNy7TgayMAcZMTzJZ9B5t3u1lSZiDuvFRYGaOHOMHiaNDJE4U1TCrt23u+DK6fUEKNK7UXk50=</HostId></Error>\"\n  :cookies       => [\n  ]\n  :headers       => {\n    \"Connection\"       => \"close\"\n    \"Content-Type\"     => \"application/xml\"\n    \"Date\"             => \"Thu, 12 Nov 2020 21:33:56 GMT\"\n    \"Server\"           => \"AmazonS3\"\n    \"x-amz-id-2\"       => \"y8nNy7TgayMAcZMTzJZ9B5t3u1lSZiDuvFRYGaOHOMHiaNDJE4U1TCrt23u+DK6fUEKNK7UXk50=\"\n    \"x-amz-request-id\" => \"6246E8900FE3422F\"\n  }\n  :host          => \"stanhu-s3-workhorse-testing.s3-us-west-2.amazonaws.com\"\n  :local_address => \"172.30.0.215\"\n  :local_port    => 37564\n  :path          => \"/tmp/builds/104/chunks/0.log\"\n  :port          => 443\n  :reason_phrase => \"Forbidden\"\n  :remote_ip     => \"52.218.244.49\"\n  :status        => 403\n  :status_line   => \"HTTP/1.1 403 Forbidden\\r\\n\"\n",
  "context": "Job raised exception",
  "class": "Ci::BuildTraceChunkFlushWorker",
  "args": [
    "93"
  ],
  "retry": 3,
  "queue": "pipeline_background:ci_build_trace_chunk_flush",
  "version": 0,
  "queue_namespace": "pipeline_background",
  "jid": "5839366323d57a8f50f07c02",
  "created_at": "2020-11-12T21:33:56.783Z",
  "meta.user": "root",
  "meta.project": "root/simple-ci2",
  "meta.root_namespace": "root",
  "meta.caller_id": "/api/:version/jobs/:id/trace",
  "meta.feature_category": "continuous_integration",
  "correlation_id": "b8871dc7a4df9f803c9cea9fc9708406",
  "enqueued_at": "2020-11-12T21:34:37.084Z",
  "failed_at": "2020-11-12T21:33:57.131Z",
  "retry_count": 0,
  "db_count": 2,
  "db_write_count": 0,
  "db_cached_count": 0,
  "error_backtrace": [
    "app/models/ci/build_trace_chunks/fog.rb:18:in `set_data'",
    "app/models/ci/build_trace_chunk.rb:250:in `unsafe_set_data!'",
...

After:

irb(main):020:0> b = Ci::Build.find(106)
=> #<Ci::Build id: 106, status: "success", finished_at: "2020-11-12 21:41:16", trace: nil, created_at: "2020-11-12 21:41:00", updated_at: "2020-11-1...
irb(main):021:0> b.job_artifacts.last.file.path
=> "85/27/8527a891e224136950ff32ca212b45bc93f69fbb801c3b1ebedac52775f99e61/2020_11_12/106/153/job.log"