Webservice logging on wrong severity levels
Summary
I noticed that our GitLab instance on GKE is logging quite a bit. Diving into the Log Explorer results showed that the webservice is logging a lot of data on "error" level which don't seem like errors based on the content.
An example of one of the log messages which is logged as ERROR
but has no discerning information to indicate it is actually an error is included below.
Seems more suited for info
level or even debug
?
Steps to reproduce
Nothing in particular, this seems to be "just happening" out of the box on our chart-installed Gitlab.
What is the current bug behavior?
Logging appears to be wrongly using error
level rather than info
or debug
What is the expected correct behavior?
No logging of things on error
level that are not errors.
Relevant logs and/or screenshots
{
insertId: "xntzy89hupz0uinp"
jsonPayload: {
action: "git_upload_pack"
component: "gitlab"
controller: "Repositories::GitHttpController"
correlation_id: "01HE7NEJ0Y1BPJYZDKKRER8RXE"
cpu_s: 0.025587
db_cached_count: 1
db_ci_cached_count: 0
db_ci_count: 0
db_ci_duration_s: 0
db_ci_replica_cached_count: 0
db_ci_replica_count: 0
db_ci_replica_duration_s: 0
db_ci_replica_wal_cached_count: 0
db_ci_replica_wal_count: 0
db_ci_wal_cached_count: 0
db_ci_wal_count: 0
db_count: 14
db_duration_s: 0.01135
db_main_cached_count: 1
db_main_count: 14
db_main_duration_s: 0.012
db_main_replica_cached_count: 0
db_main_replica_count: 0
db_main_replica_duration_s: 0
db_main_replica_wal_cached_count: 0
db_main_replica_wal_count: 0
db_main_wal_cached_count: 0
db_main_wal_count: 0
db_primary_cached_count: 1
db_primary_count: 14
db_primary_duration_s: 0.012
db_primary_wal_cached_count: 0
db_primary_wal_count: 0
db_replica_cached_count: 0
db_replica_count: 0
db_replica_duration_s: 0
db_replica_wal_cached_count: 0
db_replica_wal_count: 0
db_write_count: 1
duration_s: 0.02912
format: null
mem_bytes: 774144
mem_mallocs: 3990
mem_objects: 14174
mem_total_bytes: 1341104
meta.caller_id: "Repositories::GitHttpController#git_upload_pack"
meta.client_id: "ip/REDACTED"
meta.feature_category: "source_code_management"
meta.project: "REDACTED"
meta.remote_ip: "REDACTED"
meta.root_namespace: "REDACTED"
method: "POST"
params: [
0: {
key: "repository_path"
value: "REDACTED.git"
}]
path: "/REDACTED.git/git-upload-pack"
pid: 1482
rate_limiting_gates: [
]
redis_calls: 1
redis_duration_s: 0.000439
redis_read_bytes: 99
redis_repository_cache_calls: 1
redis_repository_cache_duration_s: 0.000439
redis_repository_cache_read_bytes: 99
redis_repository_cache_write_bytes: 54
redis_write_bytes: 54
remote_ip: "REDACTED"
request_urgency: "default"
status: 200
subcomponent: "production_json"
target_duration_s: 1
ua: "git/2.34.1"
user_id: 7
username: "argocd"
view_duration_s: 0.00021
worker_id: "puma_0"
}
labels: {
compute.googleapis.com/resource_name: "REDACTED"
k8s-pod/app: "webservice"
k8s-pod/chart: "webservice-7.5.1"
k8s-pod/environment: "production"
k8s-pod/gitlab_com/webservice-name: "default"
k8s-pod/heritage: "Helm"
k8s-pod/pod-template-hash: "5f7565789"
k8s-pod/release: "gitlab"
}
logName: "projects/REDACTED/logs/stderr"
receiveTimestamp: "2023-11-02T09:22:25.978948044Z"
resource: {
labels: {
cluster_name: "REDACTED"
container_name: "webservice"
location: "europe-west4"
namespace_name: "gitlab"
pod_name: "gitlab-webservice-default-5f7565789-9fcwc"
project_id: "REDACTED"
}
type: "k8s_container"
}
severity: "ERROR"
timestamp: "2023-11-02T09:22:23.939Z"
}
Another one, a readiness check which succeeds yet is logged on error level.
{
"insertId": "jlqhhdww1b26uzf9",
"jsonPayload": {
"db_main_cached_count": 0,
"db_replica_count": 0,
"mem_total_bytes": 177080,
"correlation_id": "1992f393-21c2-4735-8b16-0205e37c2326",
"db_main_replica_wal_count": 0,
"db_main_replica_cached_count": 0,
"db_ci_duration_s": 0,
"db_main_wal_count": 0,
"db_primary_wal_cached_count": 0,
"rate_limiting_gates": [],
"db_main_duration_s": 0,
"mem_objects": 2936,
"controller": "HealthController",
"db_primary_cached_count": 0,
"view_duration_s": 0.00021,
"db_main_count": 0,
"params": [],
"db_ci_replica_wal_count": 0,
"db_ci_replica_wal_cached_count": 0,
"db_ci_count": 0,
"format": "*/*",
"db_ci_replica_duration_s": 0,
"pid": 1484,
"db_write_count": 0,
"method": "GET",
"db_ci_cached_count": 0,
"db_main_replica_duration_s": 0,
"db_count": 0,
"db_primary_count": 0,
"path": "/-/readiness",
"db_replica_cached_count": 0,
"db_primary_duration_s": 0,
"worker_id": "puma_1",
"db_main_replica_count": 0,
"subcomponent": "production_json",
"db_main_wal_cached_count": 0,
"db_replica_duration_s": 0,
"component": "gitlab",
"db_ci_replica_cached_count": 0,
"mem_mallocs": 540,
"db_replica_wal_cached_count": 0,
"db_ci_replica_count": 0,
"db_ci_wal_count": 0,
"cpu_s": 0.006767,
"db_ci_wal_cached_count": 0,
"duration_s": 0.00137,
"db_primary_wal_count": 0,
"mem_bytes": 59640,
"db_main_replica_wal_cached_count": 0,
"db_cached_count": 0,
"db_replica_wal_count": 0,
"action": "readiness",
"db_duration_s": 0,
"status": 200
},
"resource": {
"type": "k8s_container",
"labels": {
"container_name": "webservice",
"namespace_name": "gitlab",
"pod_name": "gitlab-webservice-default-5f7565789-9fcwc",
"cluster_name": "REDACTED",
"location": "europe-west4",
"project_id": "REDACTED"
}
},
"timestamp": "2023-11-02T09:22:31.117Z",
"severity": "ERROR",
"labels": {
"k8s-pod/pod-template-hash": "5f7565789",
"k8s-pod/heritage": "Helm",
"k8s-pod/chart": "webservice-7.5.1",
"k8s-pod/app": "webservice",
"k8s-pod/environment": "production",
"k8s-pod/release": "gitlab",
"k8s-pod/gitlab_com/webservice-name": "default",
"compute.googleapis.com/resource_name": "REDACTED"
},
"logName": "projects/REDACTED/logs/stderr",
"receiveTimestamp": "2023-11-02T09:22:35.883130138Z"
}
Output of checks
Results of GitLab environment info
Expand for output related to GitLab environment info
(For installations with omnibus-gitlab package run and paste the output of: `sudo gitlab-rake gitlab:env:info`) (For installations from source run and paste the output of: `sudo -u git -H bundle exec rake gitlab:env:info RAILS_ENV=production`)
Results of GitLab application Check
Expand for output related to the GitLab application check
(For installations with omnibus-gitlab package run and paste the output of:
sudo gitlab-rake gitlab:check SANITIZE=true
)(For installations from source run and paste the output of:
sudo -u git -H bundle exec rake gitlab:check RAILS_ENV=production SANITIZE=true
)(we will only investigate if the tests are passing)