Logging
Logging is an important part of monitoring and debugging production environments. It's a huge topic. Below is a proposal for the iterating on the subject.
- Pod Logging: https://gitlab.com/gitlab-org/gitlab-ee/issues/4752
- Application Logging: https://gitlab.com/gitlab-org/gitlab-ee/issues/3711
- Distributed Logging
Pod logging starts simple and exposes logs for each pod associated with a project's environment, most likely through the deploy board for the environment. This is helpful, and helps us deprecate usage of the Kubernetes dashboard.
Application logging aggregates logs across all pods for an project environment so you can get a holistic picture of your application. This is common on platforms like Heroku. Filter and searching become very important to make these useful, especially if application and services are intermingled (like Postgres).
Distributed logging is specifically advantageous for microservices where an "application" is actually comprised of several services, so to get a true picture of how your application is doing, you need to aggregate across multiple projects. Structured logging and conventions for noting request IDs help correlate logs across services for the same request. e.g. a user hits the front end, which then hits the backend API, which then hits several backing services. If each service outputs the same request ID in their logs, then it's possible to get insight into the logs for a complete request.
Because of the ambiguity of an "application" being a single project or multiple projects, it might be more correct to call the middle iteration "project logging", but "application logging" is a more user-friendly term and the existence of "distributed logging" makes it clear that there's a difference.