Use Gitaly as a git backend

Indexing already works! See TODOs

How to run:

make && ELASTIC_CONNECTION_INFO='{"url":["http://localhost:9200"]}' GITALY_CONNECTION_INFO='{"address": "unix:/Users/valery/gdk-ee/gitaly.socket", "storage": "default"}' ./bin/gitlab-elasticsearch-indexer 7 "gitlab-org/gitlab-shell.git"

Remote Gitaly:

make && ELASTIC_CONNECTION_INFO='{"url":["http://localhost:9200"]}' GITALY_CONNECTION_INFO='{"address": "tcp://localhost:9999", "storage": "default"}' ./bin/gitlab-elasticsearch-indexer 7 "gitlab-org/gitlab-shell.git"

You can also specify a secret token.

TODO:

• Base structure for Gitaly
• Commit indexing
• Blob indexing
• Make sure zero-hashes are handled correctly
• GitLab integration
  • GitLab side https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/7434
  • Indexer side
  • Switching between Gitaly and native git clients (We decided to remove go-git implementation instead)
• Implementing token auth
• Test indexing against the big repositories
• Remote Gitaly
• Early review comments
• Fix test suite
• Fix tests
• Fix integration test
• Fix CI
• Fix: FromSHA commit is not indexed
• Rebase the branch
• Pre-production manual testing (staging?)
• Refactoring after early review
• Sort out COPIED operation and check the switch more carefully
• Cache open channels?
• Get it to work with Go 1.10
• Fix how we index the HEAD commit, we should look it up before indexing any blobs

Issue - #4 (closed)