Unable to upload to s3 object storage

Hello,

I am using gitlab CE version and so far using local storage but now i thought of using amazon s3 storage.

so enabled it using

gitlab_rails['uploads_object_store_enabled'] = true
 gitlab_rails['uploads_object_store_remote_directory'] = "uploads"
 gitlab_rails['uploads_object_store_connection'] = {
   'provider' => 'AWS',
   'region' => 'us-east-1',
   'aws_access_key_id' => 'xyz',
   'aws_secret_access_key' => 'xyz'
 }

and i ran command gitlab-rake "gitlab:uploads:migrate:all"

but it always throws error in background jobs (i re-tried many times)

Expected(200) <=> Actual(403 Forbidden) excon.error.response :body => "<?xml version=\"1.0\" encoding=\"UTF-8\"?>\n<Error><Code>AllAccessDisabled</Code><Message>All access to this object has been disabled</Message><RequestId>0385CF6FF3C4667D</RequestId><HostId>zR4y4x3gBnTMgiwEgAre34YqJ/B9v1wgyFqkldWM+Q1/AJstkl7zsbJ1Z+AV5FfvJmmP9oEnHGU=</HostId></Error>" :cookies => [ ] :headers => { "Connection" => "close" "Content-Type" => "application/xml" "Date" => "Fri, 10 May 2019 07:36:13 GMT" "Server" => "AmazonS3" "x-amz-id-2" => "zR4y4x3gBnTMgiwEgAre34YqJ/B9v1wgyFqkldWM+Q1/AJstkl7zsbJ1Z+AV5FfvJmmP9oEnHGU=" "x-amz-request-id" => "0385CF6FF3C4667D" } :host => "uploads.s3.amazonaws.com" :local_address => "a.b.c.d" :local_port => 35514 :path => "/%40hashed/1e/47/1e472b39b105d349bcd069c4a711b44a2fffb8e274714bb07ecfff69a9a7f67b/0b8f6857987eefa3d081d2f93a108ebc/Entrepreneur-mock-1_1024x1024.jpg" :port => 443 :reason_phrase => "Forbidden" :remote_ip => "a.b.c.d" :status => 403 :status_line => "HTTP/1.1 403 Forbidden\r\n"

this is first time I am using amazon s3, so i just created a bucket and for permissions etc, i just choose default values, grabbed keys and entered into gitlab.rb file.

can anyone tell me, what i am doing wrong? or is there any known bug exists in CE (unlikely)

thanks in advance.