Skip to content

Log threads in rspec_run_time

Jennifer Li requested to merge jennli-log-threads-and-open-files-count into master

What does this MR do and why?

As a part of the corrective actions mentioned in #462928 (closed), let's print the number of threads in rspec_run_time throughout rspec execution to help us troubleshoot thread leaking problems.

Tested the output in https://gitlab.com/gitlab-org/gitlab/-/jobs/6927949980#L833 by reverting our fix to the base_exporter_spec

# [RSpecRunTime] RSpec elapsed time: 6.43 seconds. Current RSS: ~1189M. Threads: 7. Open files: 0. load average: 1.18 0.65 0.59 2/304 390. Although later I have learned from Stan's comment that thread leaking by itself didn't cause our timeout, so one can argue that this is not entirely helpful for the issue in question. Still, I think this is a good data to keep track of in the jobs.

On a side note, I noticed that the number of threads can actually be much higher, such as in https://gitlab.com/gitlab-org/gitlab/-/jobs/6927949718:

# [RSpecRunTime] RSpec elapsed time: 22 minutes 23.81 seconds. Current RSS: ~1462M. Threads: 26. Open files: 0. load average: 1.60 1.14 0.99 1/488 8249

and no evidence suggests that they are ever being cleared at the end of the job execution so this might be interesting to watch for.

Please ignore the open files output because I realized our image doesn't have lsof so I will leave this out of this MR.

MR acceptance checklist

Please evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.

Screenshots or screen recordings

Screenshots are required for UI changes, and strongly recommended for all other merge requests.

Before After

How to set up and validate locally

Numbered steps to set up and validate the change are strongly suggested.

Edited by Jennifer Li

Merge request reports