Do not log Sentry exceptions by default

From this slack thread.

At the moment, we provide some ways to capture an exception manually:

  • Gitlab::ErrorTracking.track_and_raise_exception
  • Gitlab::ErrorTracking.track_and_raise_for_dev_exception
  • Gitlab::ErrorTracking.track_exception
  • Gitlab::ErrorTracking.log_exception

By default, any event captured by the aforementioned methods has a corresponding line of log in exceptions.json file. The logs from this file are ingested by fluentd. Eventually, they are available in Kibana, indistinguishable with other logs of the same type. In the discussion, we found problems with this kind of logs:

  • Exception logs have a different set of fields. As the logs are mixed into normal application logs, they create confusion and noises for people who depend on Kibana. Plus, an engineer can specify literally anything in the metadata, the log is not very well-structured. This is an example:

image

  • Most of the exceptions are eventually sent to Sentry. Sentry captures, classifies, and collect information better than Kibana, simply because they are built for different purposes. So, if an event is already captured, the log is not necessary.
  • A significant amount of exceptions are rescued, captured, the re-raise. They are then handled again by an upper layer. That leads to log duplication. The point is, it's impossible to tell an exception is logged once or twice.

Proposal

  • Turn off the exception log by default on non-local environments.
  • Gitlab::ErrorTracking.log_exception is preserved for intentional log purpose.