Skip to content

High volume of Sentry errors

On our Sentry instance, I noticed a high volume of traffic mostly coming from the gitlab-shared-runners project (https://sentry.gitlab.net/gitlab/shared-gitlab-runners/) that's impacting our ability to log events in real-time. The top two events:

image

*errors.errorString: Problem while reading command output
  File "/builds/gitlab-org/gitlab-runner/.gopath/src/gitlab.com/gitlab-org/gitlab-runner/helpers/sentry/log_hook.go", line 45, in Fire
  File "/builds/gitlab-org/gitlab-runner/.gopath/src/gitlab.com/gitlab-org/gitlab-runner/vendor/github.com/sirupsen/logrus/hooks.go", line 28, in Fire
  File "/builds/gitlab-org/gitlab-runner/.gopath/src/gitlab.com/gitlab-org/gitlab-runner/vendor/github.com/sirupsen/logrus/entry.go", line 246, in fireHooks
  File "/builds/gitlab-org/gitlab-runner/.gopath/src/gitlab.com/gitlab-org/gitlab-runner/vendor/github.com/sirupsen/logrus/entry.go", line 224, in log
  File "/builds/gitlab-org/gitlab-runner/.gopath/src/gitlab.com/gitlab-org/gitlab-runner/vendor/github.com/sirupsen/logrus/entry.go", line 268, in Log
  File "/builds/gitlab-org/gitlab-runner/.gopath/src/gitlab.com/gitlab-org/gitlab-runner/vendor/github.com/sirupsen/logrus/entry.go", line 359, in Logln
  File "/builds/gitlab-org/gitlab-runner/.gopath/src/gitlab.com/gitlab-org/gitlab-runner/vendor/github.com/sirupsen/logrus/entry.go", line 388, in Errorln
  File "/builds/gitlab-org/gitlab-runner/.gopath/src/gitlab.com/gitlab-org/gitlab-runner/helpers/docker/machine_command.go", line 54, in watch
*errors.errorString: Failed to process runner
  File "/builds/gitlab-org/gitlab-runner/.gopath/src/gitlab.com/gitlab-org/gitlab-runner/helpers/sentry/log_hook.go", line 45, in Fire
  File "/builds/gitlab-org/gitlab-runner/.gopath/src/gitlab.com/gitlab-org/gitlab-runner/vendor/github.com/sirupsen/logrus/hooks.go", line 28, in Fire
  File "/builds/gitlab-org/gitlab-runner/.gopath/src/gitlab.com/gitlab-org/gitlab-runner/vendor/github.com/sirupsen/logrus/entry.go", line 246, in fireHooks
  File "/builds/gitlab-org/gitlab-runner/.gopath/src/gitlab.com/gitlab-org/gitlab-runner/vendor/github.com/sirupsen/logrus/entry.go", line 224, in log
  File "/builds/gitlab-org/gitlab-runner/.gopath/src/gitlab.com/gitlab-org/gitlab-runner/vendor/github.com/sirupsen/logrus/entry.go", line 268, in Log
  File "/builds/gitlab-org/gitlab-runner/.gopath/src/gitlab.com/gitlab-org/gitlab-runner/vendor/github.com/sirupsen/logrus/entry.go", line 297, in Error
  File "/builds/gitlab-org/gitlab-runner/.gopath/src/gitlab.com/gitlab-org/gitlab-runner/commands/multi.go", line 279, in processRunners

@steveazz @tmaczukin Can we just squelch these events?