Skip to content

Call to Internal API fails with Gitlab Cloud Hybrid Reference Architecture Implemented

Everyone can contribute. Help move this issue forward while earning points, leveling up and collecting rewards.

Summary

This appears to be gitlab server configuration problem.

In this setup all interactions between the Gitlab Webservice and the Gitaly Server fail with an unauthorized (401) code generated from the call to the /internal/allowed API call.

API calls from the command line to the backend fail also with the Tokens all checked multiple times. The call generated by the front end webservice to gitaly appears to be missing the auth token completely.

Steps to reproduce

  1. Log into Gitlab UI
  2. Create a project
  3. Click the Add README button
  4. Add text to README File
  5. Click Commit Button and add Commit comment
  6. Complete the commit
### Example Project

What is the current bug behavior?

  1. Modal Dialogue appears with "401: Unauthorized"
  2. No commits occur

Command line invocations of git also fail with precheck fail error.

What is the expected correct behavior?

One should see the README file correctly committed.

Relevant logs and/or screenshots

Sample body from generated API call captured from logs:

{
  "time": "2022-02-25T17:52:10.910Z",
  "severity": "INFO",
  "duration_s": 0.0016,
  "db_duration_s": 0,
  "view_duration_s": 0.0016,
  "status": 401,
  "method": "POST",
  "path": "/api/v4/internal/allowed",
  "params": [
    {
      "key": "action",
      "value": "git-receive-pack"
    },
    {
      "key": "gl_repository",
      "value": "project-13"
    },
    {
      "key": "project",
      "value": "/var/opt/gitlab/git-data/repositories/@hashed/3f/db/3fdba35f04dc8c462986c992bcf875546257113072a909c162f7e470e581e278.git"
    },
    {
      "key": "changes",
      "value": "0000000000000000000000000000000000000000 ae6787f6c00a1419276412f129ef2ae2e1212db3 refs/heads/main\n"
    },
    {
      "key": "protocol",
      "value": "web"
    },
    {
      "key": "env",
      "value": "{\"GIT_ALTERNATE_OBJECT_DIRECTORIES_RELATIVE\":[\"objects\"],\"GIT_OBJECT_DIRECTORY_RELATIVE\":\"../../../../+gitaly/tmp/quarantine-dd250b70515d6831-3710779876\"}"
    },
    {
      "key": "user_id",
      "value": "2"
    }
  ],
  "host": "gitlab.trythisout.org",
  "remote_ip": "192.168.251.64, 192.168.59.199",
  "ua": "gitaly/14.7.0",
  "route": "/api/:version/internal/allowed",
  "queue_duration_s": 0.023461,
  "db_count": 1,
  "db_write_count": 0,
  "db_cached_count": 0,
  "db_replica_count": 0,
  "db_primary_count": 1,
  "db_replica_cached_count": 0,
  "db_primary_cached_count": 0,
  "db_replica_wal_count": 0,
  "db_primary_wal_count": 0,
  "db_replica_wal_cached_count": 0,
  "db_primary_wal_cached_count": 0,
  "db_replica_duration_s": 0,
  "db_primary_duration_s": 0.003,
  "cpu_s": 0.021289,
  "mem_objects": 6183,
  "mem_bytes": 724304,
  "mem_mallocs": 1837,
  "mem_total_bytes": 971624,
  "pid": 32,
  "correlation_id": "01FWS05Z44Z9Q19ZXBTBFPNSX1",
  "meta.caller_id": "POST /api/:version/internal/allowed",
  "meta.remote_ip": "192.168.59.199",
  "meta.feature_category": "source_code_management",
  "meta.client_id": "ip/192.168.59.199",
  "content_length": "505",
  "request_urgency": "default",
  "target_duration_s": 1
}

Log entries from gitaly log:

{"correlation_id":"01FVDHM75X6DR82SH8D9JJ7R3X","duration_ms":24,"error":"401 Unauthorized","level":"error","method":"POST","msg":"Internal API error","status":401,"time":"2022-02-08T20:50:28.839Z","url":"https://gitlab.trythisout.org/api/v4/internal/allowed"}
{"correlation_id":"01FVDHM75X6DR82SH8D9JJ7R3X","error":"GitLab: 401 Unauthorized","grpc.meta.auth_version":"v2","grpc.meta.client_name":"gitlab-web","grpc.meta.deadline_type":"regular","grpc.meta.method_type":"client_stream","grpc.method":"UserCommitFiles","grpc.request.deadline":"2022-02-08T20:51:23.090","grpc.request.fullMethod":"/gitaly.OperationService/UserCommitFiles","grpc.service":"gitaly.OperationService","grpc.start_time":"2022-02-08T20:50:28.790","level":"warning","msg":"stopping transaction because pre-receive hook failed","peer.address":"145.40.82.1:32372","pid":26030,"remote_ip":"192.168.59.211","span.kind":"server","system":"grpc","time":"2022-02-08T20:50:28.839Z","username":"bendorman22"}
{"branch_name":"bWFpbg==","correlation_id":"01FVDHM75X6DR82SH8D9JJ7R3X","error":"update reference: GitLab: 401 Unauthorized","force":false,"grpc.meta.auth_version":"v2","grpc.meta.client_name":"gitlab-web","grpc.meta.deadline_type":"regular","grpc.meta.method_type":"client_stream","grpc.method":"UserCommitFiles","grpc.request.deadline":"2022-02-08T20:51:23.090","grpc.request.fullMethod":"/gitaly.OperationService/UserCommitFiles","grpc.service":"gitaly.OperationService","grpc.start_time":"2022-02-08T20:50:28.790","level":"error","msg":"user commit files failed","peer.address":"145.40.82.1:32372","pid":26030,"remote_ip":"192.168.59.211","repository_relative_path":"@hashed/3f/db/3fdba35f04dc8c462986c992bcf875546257113072a909c162f7e470e581e278.git","repository_storage":"default","span.kind":"server","start_branch_name":"bWFpbg==","start_repository_relative_path":"@hashed/3f/db/3fdba35f04dc8c462986c992bcf875546257113072a909c162f7e470e581e278.git","start_repository_storage":"default","start_sha":"","system":"grpc","time":"2022-02-08T20:50:28.839Z","username":"bendorman22"}
{"branch_name":"bWFpbg==","correlation_id":"01FVDHM75X6DR82SH8D9JJ7R3X","diskcache":"f76f99d8-a839-45d3-86d2-5238901dfc38","force":false,"grpc.meta.auth_version":"v2","grpc.meta.client_name":"gitlab-web","grpc.meta.deadline_type":"regular","grpc.meta.method_type":"client_stream","grpc.method":"UserCommitFiles","grpc.request.deadline":"2022-02-08T20:51:23.090","grpc.request.fullMethod":"/gitaly.OperationService/UserCommitFiles","grpc.service":"gitaly.OperationService","grpc.start_time":"2022-02-08T20:50:28.790","level":"info","msg":"diskcache state change","peer.address":"145.40.82.1:32372","pid":26030,"remote_ip":"192.168.59.211","repository_relative_path":"@hashed/3f/db/3fdba35f04dc8c462986c992bcf875546257113072a909c162f7e470e581e278.git","repository_storage":"default","span.kind":"server","start_branch_name":"bWFpbg==","start_repository_relative_path":"@hashed/3f/db/3fdba35f04dc8c462986c992bcf875546257113072a909c162f7e470e581e278.git","start_repository_storage":"default","start_sha":"","system":"grpc","time":"2022-02-08T20:50:28.840Z","username":"bendorman22"}
{"branch_name":"bWFpbg==","command.count":5,"command.cpu_time_ms":29,"command.inblock":0,"command.majflt":0,"command.maxrss":147552,"command.minflt":3465,"command.oublock":24,"command.real_time_ms":21,"command.system_time_ms":1,"command.user_time_ms":27,"correlation_id":"01FVDHM75X6DR82SH8D9JJ7R3X","force":false,"grpc.code":"OK","grpc.meta.auth_version":"v2","grpc.meta.client_name":"gitlab-web","grpc.meta.deadline_type":"regular","grpc.meta.method_type":"client_stream","grpc.method":"UserCommitFiles","grpc.request.deadline":"2022-02-08T20:51:23.090","grpc.request.fullMethod":"/gitaly.OperationService/UserCommitFiles","grpc.request.payload_bytes":452,"grpc.response.payload_bytes":26,"grpc.service":"gitaly.OperationService","grpc.start_time":"2022-02-08T20:50:28.790","grpc.time_ms":50.663,"level":"info","msg":"finished streaming call with code OK","peer.address":"145.40.82.1:32372","pid":26030,"remote_ip":"192.168.59.211","repository_relative_path":"@hashed/3f/db/3fdba35f04dc8c462986c992bcf875546257113072a909c162f7e470e581e278.git","repository_storage":"default","span.kind":"server","start_branch_name":"bWFpbg==","start_repository_relative_path":"@hashed/3f/db/3fdba35f04dc8c462986c992bcf875546257113072a909c162f7e470e581e278.git","start_repository_storage":"default","start_sha":"","system":"grpc","time":"2022-02-08T20:50:28.840Z","username":"bendorman22"}

Output of checks

Results of GitLab environment info

Expand for output related to GitLab environment info 

(For installations with omnibus-gitlab package run and paste the output of:
`sudo gitlab-rake gitlab:env:info`)

(For installations from source run and paste the output of:
`sudo -u git -H bundle exec rake gitlab:env:info RAILS_ENV=production`)

Results of GitLab application Check

With the Cloud Hybrid Architecture, Applications from Gitlab Omnibus are installed on a linux server: Postgres, Redis, and Gitaly/Gitlab. Gitlab Webserver, Workhorse and Gitlab SSH are deployed on a Kubernetes cluster. The gitlab check seems to work only partially on both since in either case it expects all the components to be running on the same platform.

The first output is from the toolbox pod on the cluster. It incorrectly reports that Sidekiq is not running (it is, in a different pod).

Running gitlab-rake on the backend server (with partial omnibus installation) fails to connect to the server

root@gap-backend1:~# gitlab-rake gitlab:check SANITIZE=true
Checking GitLab subtasks ...

rake aborted!
ActiveRecord::ConnectionNotEstablished: could not connect to server: Connection refused
        Is the server running on host "0.0.0.0" and accepting
        TCP/IP connections on port 5432?
Expand for output related to the GitLab application check 
git@gitlab-toolbox-58d7c8b6fb-pxxxz:/$ gitlab-rake gitlab:check SANITIZE=true


Checking GitLab subtasks ...


Checking GitLab Shell ...


GitLab Shell: ... GitLab Shell version >= 13.23.2 ? ... OK (13.23.2)
Running /home/git/gitlab-shell/bin/check
gitlab-shell self-check failed
Try fixing it:
Make sure GitLab is running;
Check the gitlab-shell configuration file:
sudo -u git -H editor /home/git/gitlab-shell/config.yml
Please fix the error above and rerun the checks.


Checking GitLab Shell ... Finished


Checking Gitaly ...


Gitaly: ... default ... OK


Checking Gitaly ... Finished


Checking Sidekiq ...


Sidekiq: ... Running? ... no
Try fixing it:
sudo -u git -H RAILS_ENV=production bin/background_jobs start
For more information see:
doc/install/installation.md in section "Install Init Script"
see log/sidekiq.log for possible errors
Please fix the error above and rerun the checks.


Checking Sidekiq ... Finished


Checking Incoming Email ...


Incoming Email: ... Reply by email is disabled in config/gitlab.yml


Checking Incoming Email ... Finished


Checking LDAP ...


LDAP: ... LDAP is disabled in config/gitlab.yml


Checking LDAP ... Finished


Checking GitLab App ...


Database config exists? ... yes
All migrations up? ... yes
Database contains orphaned GroupMembers? ... no
GitLab config exists? ... yes
GitLab config up to date? ... yes
Log directory writable? ... yes
Tmp directory writable? ... yes
Uploads directory exists? ... yes
Uploads directory has correct permissions? ... yes
Uploads directory tmp has correct permissions? ... skipped (no tmp uploads folder yet)
Systemd unit files or init script exist? ... no
Try fixing it:
Install the Service
For more information see:
doc/install/installation.md in section "Install the Service"
Please fix the error above and rerun the checks.
Systemd unit files or init script up-to-date? ... can't check because of previous errors
Projects have namespace: ...
2/1 ... yes
4/2 ... yes
4/3 ... yes
4/4 ... yes
4/5 ... yes
Redis version >= 5.0.0? ... yes
Ruby version >= 2.7.2 ? ... yes (2.7.5)
Git user has default SSH configuration? ... yes
Active users: ... 3
Is authorized keys file accessible? ... skipped (authorized keys not enabled)
GitLab configured to store new projects in hashed storage? ... yes
All projects are in hashed storage? ... yes


Checking GitLab App ... Finished


Checking GitLab subtasks ... Finished


(For installations with omnibus-gitlab package run and paste the output of:
sudo gitlab-rake gitlab:check SANITIZE=true)


(For installations from source run and paste the output of:
sudo -u git -H bundle exec rake gitlab:check RAILS_ENV=production SANITIZE=true)


(we will only investigate if the tests are passing)


The main error to report is that on commits, Gitlab UI returns a 401 Unauthorized error from the


~

Possible fixes

Edited by 🤖 GitLab Bot 🤖