Gitlab On Helm - Incorrect Artifiact External Minio config causes pipelines to hang for 5 minutes before moving to next stage

Everyone can contribute. Help move this issue forward while earning points, leveling up and collecting rewards.

Summary

When using the helm chart with external minio, having an invalid connection config causes:

  • Pipelines succeed but stay "running" for 5 minutes before continuing to next stage.
  • pipeline_background:ci_build_trace_chunk_flush Sidekiq job fails with:
"Excon::Error::NotFound: Expected(200) <=> Actual(404 Not Found) excon.error.response :body => "<html>\r\n<head><title>404 Not Found</title></head>\r\n<body>\r\n<center><h1>404 Not Found</h1>..."

Steps to reproduce

  • Deploy gitlab as usual with helm
  •         artifacts:
        enabled: true
        proxy_download: true
        bucket: gitlab-artifacts
        connection:
          secret: objectstore-config
          key: config

``

  • Have the "config key" equal to the b64 of:
provider: AWS
region: eu-west-1
aws_access_key_id: <access key>
aws_secret_access_key: <secret key>
endpoint: "https://minio.example.com:443"
# The problem was that I did not have "path_style" set

Example Project

What is the current bug behavior?

  • Pipelines will log "suceeded" but then stay running for 5 minutes before continuing
  • pipeline_background:ci_build_trace_chunk_flush jobs fail with 404 errors

What is the expected correct behavior?

  • I would expect this problem with uploading the artifacts to be logged in the runner/sidekiq.

Relevant logs and/or screenshots

Excon::Error::NotFound: Expected(200) <=> Actual(404 Not Found) excon.error.response :body => "<html>\r\n<head><title>404 Not Found</title></head>\r\n<body>\r\n<center><h1>404 Not Found</h1>...

Results of GitLab environment info

Gitlab on Kubernetes with chart: 4.5.1


System information
System:		
Current User:	git
Using RVM:	no
Ruby Version:	2.6.6p146
Gem Version:	2.7.10
Bundler Version:1.17.3
Rake Version:	12.3.3
Redis Version:	unknown
Git Version:	unknown
Sidekiq Version:5.2.9
Go Version:	unknown

GitLab information
Version:	13.5.1
Revision:	
Directory:	/srv/gitlab
DB Adapter:	PostgreSQL
DB Version:	11.9
URL:		https://gitlab..
HTTP Clone URL:	https://gitlab../some-group/some-project.git
SSH Clone URL:	git@gitlab..:some-group/some-project.git
Using LDAP:	no
Using Omniauth:	no

GitLab Shell
Version:	13.11.0
Repository storage paths:
- default: 	/var/opt/gitlab/repo
GitLab Shell path:		/home/git/gitlab-shell
Git:		/usr/bin/git

Results of GitLab application Check

Expand for output related to the GitLab application check 
Running /home/git/gitlab-shell/bin/check
gitlab-shell self-check failed
Try fixing it:
Make sure GitLab is running;
Check the gitlab-shell configuration file:
sudo -u git -H editor /home/git/gitlab-shell/config.yml
Please fix the error above and rerun the checks.


Checking GitLab Shell ... Finished### Possible fixes



Checking Gitaly ...


Gitaly: ... default ... OK


Checking Gitaly ... Finished


Checking Sidekiq ...


Sidekiq: ... Running? ... no
Try fixing it:
sudo -u git -H RAILS_ENV=production bin/background_jobs start
For more information see:
doc/install/installation.md in section "Install Init Script"
see log/sidekiq.log for possible errors
Please fix the error above### Possible fixes

 and rerun the checks.

Checking Sidekiq ... Finished


Checking Incoming Email ...


Incoming Email: ... Reply by email is disabled in config/gitlab.yml


Checking Incoming Email ... Finished


Checking LDAP ...


LDAP: ... LDAP is disabled in config/gitlab.yml


Checking LDAP ... Finished


Checking GitLab App ...


Git configured correctly? ... no
Trying to fix error automatically. ...Failed
Try fixing it:
sudo -u git -H "/usr/bin/git" config --global core.autocrlf "input"
For more information see:
doc/install/installation.md in section "GitLab"
Database config exists? ... yes
All migrations up? ... yes
Database contains orphaned GroupMembers? ... no
GitLab config exists? ... yes
GitLab config up to date? ... yes
Log directory writable? ... yes
Tmp directory writable? ... yes
Uploads directory exists? ... yes
Uploads directory has correct permissions? ... yes
Uploads directory tmp has correct permissions? ... skipped (no tmp uploads folder yet)
Init script exists? ... no
Try fixing it:
Install the init script
For more information see:
doc/install/installation.md in section "Install Init Script"
Please fix the error above and rerun the checks.
Init script up-to-date? ... can't check because of previous errors
Projects have namespace: ...
2/1 ... yes
3/2 ... yes
3/3 ... yes
3/4 ... yes
3/6 ... yes### Possible fixes



3/7 ... yes
3/8 ... yes
3/9 ... yes
3/11 ... yes
3/12 ... yes
Redis version >= 4.0.0? ... yes
Ruby version >= 2.5.3 ? ... yes (2.6.6)
Git version >= 2.24.0 ? ... no
Your git bin path is "/usr/bin/git"
Try fixing it:
Update your git to a version >= 2.24.0 from Unknown
Please fix the error above and rerun the checks.
Git user has default SSH configuration? ... yes
Active users: ... 2
Is authorized keys file accessible? ... skipped (authorized keys not enabled)
GitLab configured to store new projects in hashed storage? ... yes
All projects are in hashed storage? ... yes


Checking GitLab App ... Finished


Checking GitLab subtasks ... Finished
Edited by 🤖 GitLab Bot 🤖