GitLab Environment Monitoring: No support for standard nginx metrics
Summary
In the last days I read through some documentations of GitLab to realize monitoring of the nginx for my application.
I do not use kuberenetes. I do use GitLab in an Docker Container.
So I setup an own Promotheus Server Docker Container. I connected my project. And I setup an nginx with vts Extension. But it did not work.
I dove into GitLab sources and found out that all the nginx Labels do not match, in no way.
Steps to reproduce
docker-compose.yml
version: '3'
services:
webserver:
build: docker-nginx
depends_on:
- metrics
volumes:
- ./src:/www
ports:
- 80:80
links:
- php
metrics:
image: sophos/nginx-vts-exporter
environment:
- NGINX_STATUS=http://webserver/status/format/json
- METRICS_ENDPOINT=/metrics
- METRICS_ADDR=:9931
- METRICS_NS=nginx
docker-nginx/Dockerfile
FROM alpine:3.5
MAINTAINER Oliver Lippert <oliver@lipperts-web.de>
RUN apk --update add \
bash \
git
# vts
RUN mkdir /vts
RUN cd /vts && git clone git://github.com/vozlt/nginx-module-vts.git
# nginx
ENV NGINX_VERSION nginx-1.13.8
RUN apk --update add \
openssl-dev \
pcre-dev \
zlib-dev \
wget \
build-base
RUN mkdir -p /tmp/src
WORKDIR /tmp/src
RUN wget http://nginx.org/download/${NGINX_VERSION}.tar.gz
RUN tar -zxvf ${NGINX_VERSION}.tar.gz
WORKDIR /tmp/src/${NGINX_VERSION}
RUN ./configure \
--with-http_ssl_module \
--with-http_gzip_static_module \
--prefix=/etc/nginx \
--http-log-path=/var/log/nginx/access.log \
--error-log-path=/var/log/nginx/error.log \
--sbin-path=/usr/local/sbin/nginx \
--add-module=/vts/nginx-module-vts
RUN make
RUN make install
WORKDIR /
RUN apk del build-base
RUN rm -rf /tmp/src
RUN rm -rf /var/cache/apk/*
# Timezone
RUN apk --update add tzdata
RUN rm -rf /var/cache/apk/*
ENV TIMEZONE "Europe/Berlin"
RUN cp /usr/share/zoneinfo/${TIMEZONE} /etc/localtime
RUN echo "${TIMEZONE}" > /etc/timezone
# Automail
COPY nginx.conf /etc/nginx/conf/nginx.conf
CMD ["nginx", "-g", "daemon off;"]
nginx.conf
worker_processes 1;
error_log /dev/stderr info;
pid /var/run/nginx.pid;
events {
worker_connections 1024;
}
http {
vhost_traffic_status_zone;
include mime.types;
default_type application/octet-stream;
sendfile on;
access_log /var/log/nginx/access.log;
keepalive_timeout 3000;
server {
listen 80;
root /www/public;
index index.html index.htm;
server_name localhost;
client_max_body_size 100m;
error_page 500 502 503 504 /50x.html;
location / {
try_files $uri $uri/;
}
location = /50x.html {
root /var/lib/nginx/html;
}
location /status {
vhost_traffic_status_display;
vhost_traffic_status_display_format html;
}
location /metrics {
proxy_pass http://metrics:9931/metrics;
}
}
}
Example Project
Sorry, as I don't know how to run Docker from gitlab.com I can't create a Project here.
What is the current bug behavior?
GitLab does not find any nginx metrics.
What is the expected correct behavior?
Throughput You search for nginx_responses_total and status_code but it is nginx_server_requests and code (I get the data by querying sum(rate(nginx_server_requests{server_zone!="", server_zone!="_", environment="pre.automail.lipperts-web.de"}[2m])) by (code)*)
Latency You search for nginx_upstream_response_msecs_avg but it is nginx_server_requestMsec (I get the data by querying avg(nginx_server_requestMsec{environment="pre.automail.lipperts-web.de"}))
HTTP Error Rate Same as two above (I get the data by querying sum(rate(nginx_server_requests{code="5xx", environment="pre.automail.lipperts-web.de"}[2m])))
Relevant logs and/or screenshots
Here's the metrics Output:
# HELP nginx_server_bytes request/response bytes
# TYPE nginx_server_bytes counter
nginx_server_bytes{direction="in",host="*"} 33849
nginx_server_bytes{direction="in",host="localhost"} 33849
nginx_server_bytes{direction="out",host="*"} 510080
nginx_server_bytes{direction="out",host="localhost"} 510080
# HELP nginx_server_cache cache counter
# TYPE nginx_server_cache counter
nginx_server_cache{host="*",status="bypass"} 0
nginx_server_cache{host="*",status="expired"} 0
nginx_server_cache{host="*",status="hit"} 0
nginx_server_cache{host="*",status="miss"} 0
nginx_server_cache{host="*",status="revalidated"} 0
nginx_server_cache{host="*",status="scarce"} 0
nginx_server_cache{host="*",status="stale"} 0
nginx_server_cache{host="*",status="updating"} 0
nginx_server_cache{host="localhost",status="bypass"} 0
nginx_server_cache{host="localhost",status="expired"} 0
nginx_server_cache{host="localhost",status="hit"} 0
nginx_server_cache{host="localhost",status="miss"} 0
nginx_server_cache{host="localhost",status="revalidated"} 0
nginx_server_cache{host="localhost",status="scarce"} 0
nginx_server_cache{host="localhost",status="stale"} 0
nginx_server_cache{host="localhost",status="updating"} 0
# HELP nginx_server_connections nginx connections
# TYPE nginx_server_connections gauge
nginx_server_connections{status="accepted"} 11
nginx_server_connections{status="active"} 3
nginx_server_connections{status="handled"} 11
nginx_server_connections{status="reading"} 0
nginx_server_connections{status="requests"} 141
nginx_server_connections{status="waiting"} 1
nginx_server_connections{status="writing"} 2
# HELP nginx_server_info nginx info
# TYPE nginx_server_info gauge
nginx_server_info{hostName="afb23a4185e9",nginxVersion="1.13.8"} 776
# HELP nginx_server_requestMsec average of request processing times in milliseconds
# TYPE nginx_server_requestMsec gauge
nginx_server_requestMsec{host="*"} 0
nginx_server_requestMsec{host="localhost"} 0
# HELP nginx_server_requests requests counter
# TYPE nginx_server_requests counter
nginx_server_requests{code="1xx",host="*"} 0
nginx_server_requests{code="1xx",host="localhost"} 0
nginx_server_requests{code="2xx",host="*"} 124
nginx_server_requests{code="2xx",host="localhost"} 124
nginx_server_requests{code="3xx",host="*"} 0
nginx_server_requests{code="3xx",host="localhost"} 0
nginx_server_requests{code="4xx",host="*"} 15
nginx_server_requests{code="4xx",host="localhost"} 15
nginx_server_requests{code="5xx",host="*"} 0
nginx_server_requests{code="5xx",host="localhost"} 0
nginx_server_requests{code="total",host="*"} 139
nginx_server_requests{code="total",host="localhost"} 139
# HELP nginx_upstream_bytes request/response bytes
# TYPE nginx_upstream_bytes counter
nginx_upstream_bytes{backend="172.18.0.2:9931",direction="in",upstream="::nogroups"} 12072
nginx_upstream_bytes{backend="172.18.0.2:9931",direction="out",upstream="::nogroups"} 58230
nginx_upstream_bytes{backend="172.18.0.4:9000",direction="in",upstream="::nogroups"} 11385
nginx_upstream_bytes{backend="172.18.0.4:9000",direction="out",upstream="::nogroups"} 152168
# HELP nginx_upstream_requestMsec average of request processing times in milliseconds
# TYPE nginx_upstream_requestMsec gauge
nginx_upstream_requestMsec{backend="172.18.0.2:9931",upstream="::nogroups"} 0
nginx_upstream_requestMsec{backend="172.18.0.4:9000",upstream="::nogroups"} 0
# HELP nginx_upstream_requests requests counter
# TYPE nginx_upstream_requests counter
nginx_upstream_requests{backend="172.18.0.2:9931",code="1xx",upstream="::nogroups"} 0
nginx_upstream_requests{backend="172.18.0.2:9931",code="2xx",upstream="::nogroups"} 54
nginx_upstream_requests{backend="172.18.0.2:9931",code="3xx",upstream="::nogroups"} 0
nginx_upstream_requests{backend="172.18.0.2:9931",code="4xx",upstream="::nogroups"} 0
nginx_upstream_requests{backend="172.18.0.2:9931",code="5xx",upstream="::nogroups"} 0
nginx_upstream_requests{backend="172.18.0.2:9931",code="total",upstream="::nogroups"} 54
nginx_upstream_requests{backend="172.18.0.4:9000",code="1xx",upstream="::nogroups"} 0
nginx_upstream_requests{backend="172.18.0.4:9000",code="2xx",upstream="::nogroups"} 0
nginx_upstream_requests{backend="172.18.0.4:9000",code="3xx",upstream="::nogroups"} 0
nginx_upstream_requests{backend="172.18.0.4:9000",code="4xx",upstream="::nogroups"} 15
nginx_upstream_requests{backend="172.18.0.4:9000",code="5xx",upstream="::nogroups"} 0
nginx_upstream_requests{backend="172.18.0.4:9000",code="total",upstream="::nogroups"} 15
# HELP nginx_upstream_responseMsec average of only upstream/backend response processing times in milliseconds
# TYPE nginx_upstream_responseMsec gauge
nginx_upstream_responseMsec{backend="172.18.0.2:9931",upstream="::nogroups"} 0
nginx_upstream_responseMsec{backend="172.18.0.4:9000",upstream="::nogroups"} 0
# HELP nginx_vts_exporter_build_info A metric with a constant '1' value labeled by version, revision, branch, and goversion from which nginx_vts_exporter was built.
# TYPE nginx_vts_exporter_build_info gauge
nginx_vts_exporter_build_info{branch="HEAD",goversion="go1.9.4",revision="8aa2881c7050d9b28f2312d7ce99d93458611d04",version="0.10.3"} 1
Output of checks
Results of GitLab environment info
Expand for output related to GitLab environment info
System information System: Current User: git Using RVM: no Ruby Version: 2.3.6p384 Gem Version: 2.5.2.2 Bundler Version:1.16.1 Rake Version: 12.3.0 Redis Version: 3.0.6 Git Version: 2.16.2 Sidekiq Version:5.0.5 Go Version: unknown
GitLab information Version: 10.6.3 Revision: 753d851 Directory: /home/git/gitlab DB Adapter: postgresql URL: https://lw-scm.de HTTP Clone URL: https://lw-scm.de/some-group/some-project.git SSH Clone URL: ssh://git@lw-scm.de:10022/some-group/some-project.git Using LDAP: no Using Omniauth: no
GitLab Shell Version: 6.0.4 Repository storage paths:
- default: /home/git/data/repositories Hooks: /home/git/gitlab-shell/hooks Git: /usr/bin/git
Results of GitLab application Check
Expand for output related to the GitLab application check
Checking GitLab Shell ...
GitLab Shell version >= 6.0.4 ? ... OK (6.0.4) Repo base directory exists? default... yes Repo storage directories are symlinks? default... no Repo paths owned by git:root, or git:git? default... yes Repo paths access is drwxrws---? default... yes hooks directories in repos are links: ... (all) ok Running /home/git/gitlab-shell/bin/check Check GitLab API access: OK Redis available via internal API: OK
Access to /home/git/.ssh/authorized_keys: OK gitlab-shell self-check successful
Checking GitLab Shell ... Finished
Checking Sidekiq ...
Running? ... yes Number of Sidekiq processes ... 1
Checking Sidekiq ... Finished
Checking Reply by email ...
IMAP server credentials are correct? ... yes Init.d configured correctly? ... yes MailRoom running? ... yes
Checking Reply by email ... Finished
Checking LDAP ...
LDAP is disabled in config/gitlab.yml
Checking LDAP ... Finished
Checking GitLab ...
Git configured correctly? ... yes Database config exists? ... yes All migrations up? ... yes Database contains orphaned GroupMembers? ... no GitLab config exists? ... yes GitLab config up to date? ... yes Log directory writable? ... yes Tmp directory writable? ... yes Uploads directory exists? ... yes Uploads directory has correct permissions? ... yes Uploads directory tmp has correct permissions? ... yes Init script exists? ... yes Init script up-to-date? ... yes Projects have namespace: ... (all) yes Redis version >= 2.8.0? ... yes Ruby version >= 2.3.5 ? ... yes (2.3.6) Git version >= 2.9.5 ? ... yes (2.16.2) Git user has default SSH configuration? ... yes Active users: ... 10
Checking GitLab ... Finished
Possible fixes
https://gitlab.com/gitlab-org/gitlab-ce/blob/master/config/prometheus/additional_metrics.yml#L100