CI/CD Pipeline artifacts. / Can't verify CSRF token authenticity. (No such file or directory @ realpath_rec - /tmp/gitlab/metadata.gz)
Summary
CI/CD Pipeline / gitlab-runner fails to upload artifacts to coordinator.
WARNING: Uploading artifacts to coordinator... failed id=3654 responseStatus=500 Internal Server Error status=500 Internal Server Error token=[REDACTED]
WARNING: Retrying... error=invalid argument
then i found this in the 'production.log':
Started POST "/api/v4/jobs/3654/artifacts?artifact_format=zip&artifact_type=archive&expire_in=5+day" for 10.43.0.5 at 2019-11-04 17:08:43 +0000
Processing by Gitlab::RequestForgeryProtection::Controller#index as HTML
Parameters: {"file.sha1"=>"0f266e31565d5c0443dddef68840ac100e663075", "file.sha256"=>"bc6c0f0214d07207077b3db7d8cfea1c861ba3ce7962953864ba16ee6bb13041", "file.sha512"=>"a3e2096010efe1edd3cfe9f52d2e7d3cb2debd9652eb3396bfa7ca634a517814a09953e06bb1aad66ed01e43b82e1716e4f50ec61ff6ec5a9538b3418cfc8931", "file.etag"=>"f48dd42528caf44e075090dbcbe2fd5c-1", "file.remote_url"=>"https://webserver:port/gitlab-artifacts/tmp/uploads/1572887323-30-0012-5115-4065563960a4e99f45ef772380f0c0c0?X-Amz-Expires=15300&X-Amz-Date=20191104T170843Z&X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=[REDACTED]t&X-Amz-SignedHeaders=host&X-Amz-Signature=[REDACTED]", "file.size"=>"3061", "file.md5"=>"6d631ab24fc9b1bd274f1d7368102c97", "file.name"=>"artifacts.zip", "file.remote_id"=>"1572887323-30-0012-5115-4065563960a4e99f45ef772380f0c0c0", "metadata.sha512"=>"5251529bcea2202569f4640d0deec7ee882e719ade5dfa0b9b11957f99c6dd0b6b22a985b5a74e3587114e448f433a94a3152e08da708eaaade8ce02ba257d3a", "metadata.name"=>"metadata.gz", "metadata.path"=>"/tmp/gitlab/metadata.gz792269306", "metadata.size"=>"191", "metadata.md5"=>"e0521f3425bc4f36552478e16d6e5cbe", "metadata.sha1"=>"6a423daec7d5799bf0f033c18aac3da4ee3f33ea", "metadata.sha256"=>"eab624ef6f62c24fc31d36bc1904409a4861c24c6b397123c1afecb5a9d37abb", "artifact_format"=>"zip", "artifact_type"=>"archive", "expire_in"=>"5 day"}
Can't verify CSRF token authenticity.
This CSRF token verification failure is handled internally by `GitLab::RequestForgeryProtection`
Unlike the logs may suggest, this does not result in an actual 422 response to the user
For API requests, the only effect is that `current_user` will be `nil` for the duration of the request
Completed 422 Unprocessable Entity in 0ms (ActiveRecord: 0.0ms | Elasticsearch: 0.0ms)
Errno::ENOENT (No such file or directory @ realpath_rec - /tmp/gitlab/metadata.gz792269306):
/srv/gitlab/lib/uploaded_file.rb:52:in `realpath'
/srv/gitlab/lib/uploaded_file.rb:52:in `from_params'
Seems very similar to other 'CSRF token authenticity' issues. but clearing the runner token's did nothing for me.
i encountered this issue after upgrading ( backup -> restore ) , from 12.3.0 to 12.5.0.
I am using the helm charts, for both gitlab && gitlab runnner. This is the output from gitlab:check:
git@gitlab-task-runner-5fbb6f8f68-94sjn:/scripts/bin$ ./gitlab-rake gitlab:check SANITIZE=true
WARNING: This version of GitLab depends on gitlab-shell 10.2.0, but you're running Unknown. Please update gitlab-shell.
Checking GitLab subtasks ...
Checking GitLab Shell ...
GitLab Shell: ... GitLab Shell version >= 10.2.0 ? ... FAIL. Please update gitlab-shell to 10.2.0 from Unknown
Running /home/git/gitlab-shell/bin/check
gitlab-shell self-check failed
Try fixing it:
Make sure GitLab is running;
Check the gitlab-shell configuration file:
sudo -u git -H editor /home/git/gitlab-shell/config.yml
Please fix the error above and rerun the checks.
Checking GitLab Shell ... Finished
Checking Gitaly ...
Gitaly: ... default ... OK
Checking Gitaly ... Finished
Checking Sidekiq ...
Sidekiq: ... Running? ... no
Try fixing it:
sudo -u git -H RAILS_ENV=production bin/background_jobs start
For more information see:
doc/install/installation.md in section "Install Init Script"
see log/sidekiq.log for possible errors
Please fix the error above and rerun the checks.
Checking Sidekiq ... Finished
Checking Incoming Email ...
Incoming Email: ... Reply by email is disabled in config/gitlab.yml
Checking Incoming Email ... Finished
Checking LDAP ...
LDAP: ... LDAP is disabled in config/gitlab.yml
Checking LDAP ... Finished
Checking GitLab App ...
Git configured correctly? ... no
Trying to fix error automatically. ...Failed
Try fixing it:
sudo -u git -H "/usr/bin/git" config --global core.autocrlf "input"
For more information see:
doc/install/installation.md in section "GitLab"
Database config exists? ... yes
All migrations up? ... yes
Database contains orphaned GroupMembers? ... no
GitLab config exists? ... yes
GitLab config up to date? ... yes
Log directory writable? ... yes
Tmp directory writable? ... yes
Uploads directory exists? ... yes
Uploads directory has correct permissions? ... yes
Uploads directory tmp has correct permissions? ... skipped (no tmp uploads folder yet)
Init script exists? ... no
Try fixing it:
Install the init script
For more information see:
doc/install/installation.md in section "Install Init Script"
Please fix the error above and rerun the checks.
Init script up-to-date? ... can't check because of previous errors
Projects have namespace: ...
30/1 ... yes
<snip>
0/383 ... yes
63/384 ... yes
Redis version >= 2.8.0? ... no
Try fixing it:
Update your redis server to a version >= 2.8.0
For more information see:
gitlab-public-wiki/wiki/Trouble-Shooting-Guide in section sidekiq
Please fix the error above and rerun the checks.
Ruby version >= 2.5.3 ? ... yes (2.6.3)
Git version >= 2.22.0 ? ... no
Your git bin path is "/usr/bin/git"
Try fixing it:
Update your git to a version >= 2.22.0 from Unknown
Please fix the error above and rerun the checks.
Git user has default SSH configuration? ... yes
Active users: ... 15
Is authorized keys file accessible? ... skipped (authorized keys not enabled)
Checking GitLab App ... Finished
Checking GitLab subtasks ... Finished
seems like secrets.yml / gitlab_secrets.json has not changed either.
Any idea on how i move forward from here?, what is causing this?