Learning Gitaly - Mike Lockhart
module-name: "Gitaly"
area: "Product Knowledge"
gitlab-group: "Create:Gitaly"
maintainers:
- bcarranza
- wchandler
Introduction
This training module is intended to provide Support Engineers with a better understanding of Gitaly.
Goals of this training module
At the end of this module, you should be able to:
- answer Gitaly-related tickets (or have a good idea of how to start troubleshooting Gitaly-related tickets/problems)
- demonstrate an understanding of the role Gitaly plays in GitLab
- support peers with questions about Gitaly
General Timeline and Expectations
- This issue should take you TBD to complete.
Reminders:
- This is a Public issue; don't include anything confidential.
- You should do stage 0 first and the final stage last. Outside of that, you can perform tasks from the stages in any order you like.
Stage 0: Create Your module
-
Create an issue using this template by making the Issue Title: "Learning Gitaly - " -
Add yourself as the assignee -
Consider setting a milestone and/or a due date to help motivate yourself! -
Use the GitLab Environment Toolkit (GET) to deploy the 3k reference architecture - This environment will be needed for Stage 2.
Consider using the Time Tracking functionality so that the estimated length for the module can be refined.
Stage 0.5: Fundamental Concepts
-
Done with Stage 0.5
Optional.
Goal This stage is intended to help ensure that people completing the module have a familiarity with fundamental concepts that are key to understanding Gitaly and the role it plays.
Stage 1. Gitaly in theory
-
Done with Stage 1
Goal Understand the basics of what Gitaly does and how it fits into the GitLab application
-
Review the GitLab architecture diagram, pay attention to where Gitaly sits in relation to other GitLab components -
Read The road to Gitaly v1.0 to understand the problems that Gitaly was introduced to solve -
Read the Gitaly docs for administrators -
Review Gitaly Basics slides -
Read about Gitaly cluster -
Read the Gitaly Cluster design documents -
Watch GitLab Support Gitaly Cluster Deep Dive (~56 minutes) -
Read the Gitaly developers guide -
Read the beginner's guide to Gitaly contributions -
Watch Using strace
to understand GitLab (~40 minutes) -
Watch the Gitaly for Ruby devs Deep Dive from May 2019 (~40 minutes) -
Optional but recommended: watch the How Gitaly fits into GitLab playlist -
Watch How Gitaly fits into GitLab: Episode 1 – Gitaly client (~56 minutes) -
Watch How Gitaly fits into GitLab: Episode 2 - Git SSH (~55 minutes) -
Watch How Gitaly fits into GitLab: Episode 3 - Git push (~58 minutes) -
Watch How Gitaly fits into GitLab: Episode 4 - Git HTTP - (~58 minutes) -
Watch How Gitaly fits into GitLab: Episode 5 - Merge Requests across Forks (~64 minutes) -
Watch How Gitaly fits into GitLab: Episode 6 - Creating Git commits on behalf of Git users (~68 minutes)
-
-
Read about CVE-2020-13353, a security vulnerability impacting Gitaly and take a look at the fix that was implemented. A brief summary of the vulnerability:
When importing repos via URL, one time use git credentials were persisted beyond the expected time windows in Gitaly 1.79.0 or above.
Stage 2. Gitaly in practice
-
Done with Stage 2
Goal Get hands on with Gitaly operations
In Stage 0, you should have used the GitLab Environment Toolkit (GET) to deploy the 3k reference architecture. This environment will be useful for the tasks in this stage.
-
Run the gitlab:gitaly:check
Rake task, familiarize yourself with the output. -
Use the API to schedule a repository storage move for a project between shards in the environment you spun up using GET -
Observe the impact of Gitaly timeouts -
Create an intentionally large commit (1GB or so) -
Decrease the Gitaly timeouts -
Observe that the page fails to load -
Increase the Gitaly timeouts -
If necessary, increase puma['worker_timeout']
orunicorn['worker_timeout]
to be at least 3 seconds longer than the Gitaly timeout and reconfigure. -
If using Puma, GITLAB_RAILS_RACK_TIMEOUT
must also be set to a value greater than your Gitaly time out using gitlab_rails['env']
-
-
Observe that the page loads successfully
-
-
Follow the Gitaly authentication token rotation process, document the process, put notes about what you did, what you encountered, etc below.
Stage 3. Gitaly at GitLab and beyond
-
Done with Stage 3
Goal Learn about the current status and future direction of Gitaly at GitLab
-
Review the Gitaly category vision epic and associated issues that you find interesting -
Familiarize yourself with the Gitaly runbooks used for emergencies on gitlab.com
-
Review the March 2021 GitLab Support Deep Dive on Gitaly Cluster issue -
Watch one of the recordings from the March 2021 GitLab Support Deep Dive on Gitaly Cluster (60 mins) -
Troubleshooting Gitaly High CPU -
Gitaly error rate is too high -
Gitaly Queueing -
Consider scheduling a coffee chat with someone from the Gitaly group -
Read the Category Direction page for Gitaly -
Read Gitaly at Scalingo: Explaining the complete redesign of how we host your application git repositories -
Read How a fix in Go 1.9 sped up our Gitaly service by 30x -
Optional Review GitLab Forum posts about Gitaly and provide an answer if possible.
📋 Interesting Issues
-
Review each of the issues called out on the Category Direction page for Gitaly. Choose a few that are interesting to you, record them here and enable Notifications so you can follow the progress of the issue. -
Browse through the issues in the gitlab-org/gitaly repository. Choose a few that are interesting to you, record them here and enable Notifications so you can follow the progress of the issue.
Issue List
- Prune large blobs (gitlab-org/git#10)
- Unable to view the commit history of files due ... (gitlab-org/gitaly#5426 - closed)
- Introduce and use slog (gitlab-org/gitaly#5405)
- Change the default replication factor (gitlab-org/gitaly#5155)
Gitaly and NFS
-
Review the Remove Gitaly support for NFS epic and associated issues -
Review GitLab 14.0 Support Readiness - Removal of support for Gitaly and NFS -
Read about Gitaly and NFS:
From GitLab 14.0, enhancements and bug fixes for NFS for Git repositories will no longer be considered and customer technical support will be considered out of scope.
Stage 4. Demonstrate Knowledge
-
Done with Stage 4
Goal Reinforce what you learned in other stages through tickets, customer calls and documentation updates
-
Answer 5 Gitaly-related tickets 🎟 and paste the links below. Collaborate with a colleague or an expert if you don't have all of the answers.- Ticket: https://gitlab.zendesk.com/agent/tickets/318931 (with Will and Diana)
- Ticket: https://gitlab.zendesk.com/agent/tickets/338936 (escalated ticket, with a bunch of Gitaly engineers, Andrew C, Michael L)
- Ticket: https://gitlab.zendesk.com/agent/tickets/401487 (escalated ticket, gitaly bugs and patches, Gerardo, Michael L)
- Ticket: https://gitlab.zendesk.com/agent/tickets/426745
- Ticket: https://gitlab.zendesk.com/agent/tickets/339956
-
Participate in two customer calls troubleshooting Gitaly. - ticket for call: https://gitlab.zendesk.com/agent/tickets/320374 (again with Diana)
- ticket for call: https://gitlab.zendesk.com/agent/tickets/339911 (with Priyan)
-
Update the docs to include add a new Praefect metric or improve the description of an existing metric -
Review the Gitaly Documentation Needs issue and contribute if possible. Record the links to the MRs below. -
Optional but recommended: submit docs MRs to improve the Gitaly documentation and post the links below. - Docs:
- Docs:
Stage 5. Final Stage
-
Have your manager review this issue. -
Manager: Have a Gitaly expert in Support review 5-7 of the tickets from Stage 4 and report back to you on whether sufficient Gitaly knowledge has been demonstrated. -
Manager: schedule a call (or integrate into 1:1) to review how the module went once you have reviewed this issue. -
Submit a MR to update modules
andknowledge_areas
in the Support Team yaml file with this training module's topic. You will now be listed as an expert in Gitaly on Skills by Person page.
🎉 Congratulations! You made it, and are well-prepared to assist customers and colleagues with Gitaly-related tickets. 🎉
If you think of any improvements to this module, please submit an MR! The file is located in an issue template in the 'support-training` repository.