Skip to content

WIP: Speed up showing repository tree

Hiroyuki Sato requested to merge hiroponz/gitlab-ce:speed-up-repository-tree into master

What does this MR do?

Speed up showing repository tree.

Benchmark

The following is a benchmark script.

require "benchmark/ips"

Benchmark.ips do |x|
  x.config(time: 10, warmup: 2)

  project = Project.find_with_namespace("gitlab-org/gitlab-ce")
  repository = project.repository
  commit = repository.commit("master")
  tree = repository.tree(commit.id, nil)

  x.report("new") do
    logs = repository.logs_tree(commit.id)
  end

  x.report("old") do
    contents = []
    contents.push(*tree.trees)
    contents.push(*tree.blobs)
    contents.push(*tree.submodules)

    logs = contents.to_a.map do |content|
      last_commit = repository.last_commit_for_path(commit.id, content.name)
      {
        file_name: content.name,
        commit: last_commit
      }
    end
  end

  x.compare!
end

And loads the script in the rails console.

[1] pry(main)> load "tmp/tree_logs_benchmark.rb"
Calculating -------------------------------------
                 new   103.000  i/100ms
                 old     1.000  i/100ms
-------------------------------------------------
                 new    999.825  (±13.7%) i/s -      9.785k
                 old      0.393  (± 0.0%) i/s -      4.000

Comparison:
                 new:      999.8 i/s
                 old:        0.4 i/s - 2544.45x slower

It is about 2500x faster to get tree logs from the gitlab-ce repository if the tree logs are cached.

Are there points in the code the reviewer needs to double check?

Cache key includes<commit sha>. So it is not necessary to expire the cache manually.

  • cache key: logs_tree/<commit sha>/<path>
  • cache value: compressed <content name>/<commit sha of content>/<committed date (unix time)>/<commit subject>\n[...]

Why was this MR needed?

It is very slow to show the repository tree view.

Does this MR meet the acceptance criteria?

Merge request reports