Skip to content

Rescue other errors when checking redis' connectivity

David Dieulivol requested to merge ddieulivol-redis_checks_rescue_other_errors into master

Context

When checking Redis connectivity in https://gitlab.com/gitlab-org/build/CNG/blob/master/gitlab-rails/scripts/lib/checks/redis.rb, the following exceptions are sometimes raised:

  • Errno::EALREADY: Operation already in progress - connect(2) for 10.0.13.23:6379 (see GCP logs)
  • __connect_nonblock': Operation now in progress - connect(2) would block (IO::EINPROGRESSWaitWritable) (see GCP logs)

Those exceptions are not being correctly caught at the moment. This is because we only catch RuntimeError exceptions, and Errno::* errors and IO::EINPROGRESSWaitWritable are not children errors of RuntimeError (source). This results in a forceful exit of the Checks::Redis class.

What does this MR do?

We also catch SystemCallError exceptions, because it will catch both of the exceptions mentioned above:

irb> Errno::EALREADY < SystemCallError
true

irb> IO::EINPROGRESSWaitWritable < SystemCallError
true

Related issues

gitlab-org/quality/engineering-productivity/review-apps-broken-incidents#30 (closed)

Checklist

See Definition of done.

For anything in this list which will not be completed, please provide a reason in the MR discussion

Required

  • Merge Request Title, and Description are up to date, accurate, and descriptive
  • MR targeting the appropriate branch
  • MR has a green pipeline on GitLab.com

Expected (please provide an explanation if not completing)

  • Test plan indicating conditions for success has been posted and passes
  • Documentation created/updated
  • Integration tests added to GitLab QA
  • The impact any change in container size has should be evaluated
Edited by Hossein Pursultani

Merge request reports