2016-10-31: Sidekiq queues processed slow today

Today around 12:00 UTC time Sidekiq queues grew for quite some time, resulting in merge requests not being merged, pipelines not running, etc.:

image

This is a summary of what happened today and what we did:

image

  1. ~08:00 UTC: The queue grew as Sidekiq was dominated by PostReceive jobs. This is the job that runs after a user pushes to a repository.
  2. ~17:00 UTC: There was a short gap while we deployed 8.13.2.
  3. ~18:34 UTC: We moved the gitlab-ce repository back to NFS in an attempt to improve the situation.
  4. ~19:45 UTC: We attempted to change the Sidekiq concurrency but accidentally made an error in the Redis IP.
  5. ~20:00 UTC: The Redis config change was made and the number of Sidekiq workers was increased from 25 to 50.
  6. ~20:00 - 23:00 UTC: We attempted to trace the cause of the slowness and trim the queue via manual actions (e.g. removing duplicate ProjectCacheWorker jobs).
  7. 00:00 UTC: Seeing that Sidekiq seemed to process PostReceive tasks quickly with just one thread, we lowered the number of threads from 50 to 4. This seemed to bring the queue back down to a normal level.

Why would lowering the number of threads from to 50 to 4 help? Several ideas:

  1. Each worker machine runs Sidekiq, unicorn, and a CephFS client. Each machine is a 16-core machine, but all this load makes adding more threads counterproductive:

    image

  2. We experimented by running another Sidekiq process with a single thread on worker1 via:

    PATH=/opt/gitlab/bin:/opt/gitlab/embedded/bin:/bin:/usr/bin LD_PRELOAD=/opt/gitlab/embedded/lib/libjemalloc.so BUNDLE_GEMFILE=/opt/gitlab/embedded/service/gitlab-rails/Gemfile /opt/gitlab/embedded/bin/bundle exec sidekiq -q post_receive -t 4 -c 1 -r /opt/gitlab/embedded/service/gitlab-rails -e production

    The result was surprising: PostReceive jobs were finishing relatively fast (< 60 s), whereas when we experimented with 10 or 25 threads they started taking around 200-300 s. Since the system was already fully loaded, it's not surprising adding more threads would cause a problem. However, we would expect to see all processes subject to the same issue, and the single Sidekiq process did not seem affected by the load. With an increasing number of threads, it's possible there is something going on with Sidekiq, the Ruby Global Interpreter Lock (GIL), libgit2, etc. that causes slow performance. We have some strace data that we can examine here (https://gitlab.slack.com/files/stanhu/F2WFY0QTD/worker1-sidekiq-strace-with-c10.txt).

Other things we observed:

  1. PostReceive tasks appear to be slow in expiring the cache, which requires iterating all branches on the filesystem for the repo:

    image

  2. This is further reflected in all the slow timings in PostReceive#perform:

    image

  3. This suggests a problem could be with the filesystem. Upon further investigation, @pcarranza plotted the CephFS OSD latency. It's relatively high, and if this is happening, this could be blocking filesystem operations. We suspect a problem inherit with using high-performance Azure networked drives:

image

Next steps:

  1. We should consider separating Sidekiq workers from unicorn worker machines.

  2. The Sidekiq API/gitlab-monitor does not seem to be reporting the queue sizes for post_receive or email_receiver for some reason. We need to fix that.

  3. We need to investigate further why a single process on a loaded machine still seems to operate smoothly even on a loaded system. Perhaps look at the strace. Could there be something preventing threads from running in an optimal manner?

  4. CephFS OSD latency does NOT seem acceptable for running a production CephFS cluster in the Azure cloud. The performance problem is just going to get worse with more load. We also learned that Azure does not support jumbo frames. We discussed alternatives today:

    1. If we really want to run CephFS, we should consider running on bare metal since that is recommended by RedHat. This means we have to consider a provider such as Rackspace or Softlayer, or do something on our own.
    2. We should consider buying ourselves some time by spinning up NFS servers, sharding our repos there, and then implementing backups with those servers.