Temporarily enable rbtrace on api-01-sv-gprd.c.gitlab-production.internal

Production Change - Criticality 3 C3

Change Objective	In order to better understand the behaviour of ruby threads, and ruby redis connections (https://gitlab.com/gitlab-com/gl-infra/infrastructure/-/issues/9420), I'd like to enable rbtrace on a single api node. This will allow getting backtraces of all ruby threads via rbtrace --backtraces.
Change Type	ConfigurationChange
Services Impacted	api
Change Team Members	@igorwwwwwwwwwwwwwwwwwwww
Change Severity	C3
Change Reviewer or tested in staging	@hphilipps
Dry-run output	-
Due Date	2020-03-24 09:00:00 UTC
Time tracking	-

Detailed steps for the change

• $ knife node attribute set api-01-sv-gprd.c.gitlab-production.internal . '{ "omnibus-gitlab": { "gitlab_rb": { "gitlab-rails": { "env": { "ENABLE_RBTRACE": "true" } } } } }'
• $ ssh api-01-sv-gprd.c.gitlab-production.internal
  • $ sudo chef-client
  • $ ps faux | grep puma
  • $ sudo /opt/gitlab/embedded/bin/ruby /opt/gitlab/embedded/bin/rbtrace -p <pid> --backtraces
• monitoring
  • api: overview
  • host stats

Rollback steps

• $ EDITOR=vim knife node edit api-01-sv-gprd.c.gitlab-production.internal # remove ENABLE_RBTRACE
• $ ssh api-01-sv-gprd.c.gitlab-production.internal sudo chef-client

Changes checklist

• Detailed steps and rollback steps have been filled prior to commencing work
• Person on-call has been informed prior to change being rolled out