Skip to content

Add jsonlog.so from pg_plugins

Gregory Stark requested to merge add-jsonlog into master

Working towards structured logging throughout the gitlab.com infrastructure we want to add this module for Postgres: https://gitlab.com/gitlab-com/infrastructure/issues/3444

The current remote-syslog and logstash based tooling for getting Postgres logs into an ELK stack works especially poorly for Postgres as Postgres generates multi-line logs for many things and most notably for the statement context for errors. Logstash doesn't have a good story for multiline logs (there's an awkward regexp based solution but it's unreliable and insecure).

We're trying to standardize on JSON format for all components of gitlab and brings a number of benefits including being able to handle multi-line logs cleanly. It also makes it easier to filter and aggregate logs in Kibana which would be extremely useful for gitlab.com and any customers using an ELK stack.

There is a module to handle this available for Postgres in Michael Paquier's pg_plugins which is on github at https://github.com/michaelpq/pg_plugins which is a collection of miscellaneous modules that are useful for testing, monitoring, and debugging specific use cases. We don't have an immediate need for any of the other modules.

jsonlog itself is enabled simply by adding it to shared_preload_libraries which will load the .so from dynamic_library_path which is defaults to the compiled-in directory ($libdir) /opt/gitlab/embedded/postgresql/9.6.5/lib (which can be verified with /opt/gitlab/embedded/postgresql/9.6.5/bin/pg_config --libdir).

So what I want to do is build the jsonlog.so module from github and include it in the omnibus package. That should be sufficient to allow users to enable it and configure it appropriately.

Then in production we'll just add it to shared_libraries_preload and set other variables above to allow the logs to be imported using td-agent into our ELK stack. That work will be done using the gitlab.com chef recipes.

Edited by Gregory Stark

Merge request reports