Jason Young - Support Engineering Onboarding
Welcome to your Self-managed Support onboarding issue!
As you work through this onboarding issue, please remember that this issue is non-confidential and is public. With that said, please do not add any confidential data such as customer names, logs, etc. to this issue. Linking to the appropriate Zendesk ticket is OK!
We believe that people often learn best and fastest by diving into (relatively easy) tickets and learning by doing. However, this has to be balanced with making time to learn some of the tougher topics. You should start tackling stages 0, 1, 2, and 3 as early as your first week, though you are not likely to complete all items for at least several weeks. The expectation is therefore that you are sufficiently advanced in your onboarding by the end of your first week that you can answer 2-5 "simple" customer tickets.
Over the following weeks, your manager will set higher targets for the number and difficulty of tickets to be tackled, you should always save about 2-3 hours per day to spend on working through this boot camp. Some days it will be less, some days it will be more, depending on your working style and ticket load; however an average of 2-3 hours per day should allow you to complete everything through to the end of stage 6 within about four weeks.
We need to keep iterating on this checklist so please submit MR's for any improvements that you can think of. The file is located in an issue template in the 'support-training` repository If an item does not start with a role or someone else's name, it's yours to do.
The goal of this entire checklist: To set a clear path for training a Gitlab Support Engineer
Manager Tasks pre-Stage 0
-
Open Support Engineering onboarding issue -
Open Support Engineering access request issue -
Add new team member to: - regional call
-
Schedule 1:1 meeting -
Create 1:1 doc -
Schedule 1st day
-
-
Add new team member to calendars: -
GitLab Support -
Support Time Off
-
-
Verify new team member has updated BambooHR with their primary and other citizenships -
Verify if new team member is a US citizen living in the US -
(If they are) add them to the Federal Zendesk instance as Staff
-
-
Check Stage 0 for access request checklist, and other manager tasks.
@kevenhughes
Your trainer: Stage 0: Complete general onboarding to have all necessary accounts and permissions
-
Done with Stage 0
Typically completed within the first week
-
General Onboarding Checklist: this should have been created for you on the People Ops Employment issue tracker when you were hired. -
Finish every item on that list starting with New team member:
until you are waiting for someone to answer a question or do something before you can continue that list.
-
-
Ensure that as part of general onboarding, you and your manager check in that you have been added to all relevant Engineering communications channels. Access can be requested via a Single Person Access Request. Relevant channels include: -
Email list (Google Groups) - Google Drive shared drives (auto access via Google Groups)
- Support
- Engineering
-
1password Support Vault -
In ZenDesk as Staff in Support Group -
Slack - Join
#support-team-chat
,#support_gitlab-com
(optional),#support_self-managed
,#support-managers
- Added to
@support-dotcom
and/or@support-selfmanaged
group - Added to
@support-team-amer
,@support-team-emea
or@support-team-apac
group
- Join
-
GitLab groups and aliases: -
Manager: add team member to gitlab-com/support
asMaintainer
-
Manager: add team member to regional group gitlab-com/support/amer
,gitlab-com/support/apac
orgitlab-com/support/emea
asOwner
-
-
-
Start Stage 1 here, as well as the first steps of Stage 2 and Stage 3. -
Check back daily on what was blocking you on your General Onboarding Checklist until that list is completely done, then focus on this one. -
Team Calls: To keep up with the changes and happenings within the Support team, we have team calls every week. Every member of the support team is encouraged to join so they can also state their opinions during the call. -
Read up on the calls the Support Team uses to sync up and make sure you have the ones that pertain to you on your calendar. -
Identify agendas for those meetings, and read through a few past meetings in the document. -
Verify that you have a 1:1 scheduled with your manager and you have access to the agenda for that meeting.
-
-
Introduce yourself to your team! -
Write an entry with your name, location, the date you started, a quick blurb about your experience, personal interests and what drew you to GitLab support in the Support Week in Review doc before Friday. -
Manager: Make sure to introduce the team member in the Engineering Week-in-Review document and copy the introduction over to Support Week in Review! -
Manager: If the new team member is in a region that does a regional team meeting, talk to the new team member about doing an introduction at the next regional team meeting, and add it to the agenda.
-
-
Now, also format and send the introduction post you just created to the #support-team-chat
Slack channel. Welcome to multi-modal communication, key to effective communication at GitLab!
-
Stage 1: Become familiar with git and GitLab basics
-
Done with Stage 1
Typically started in the first week, completed by the end of the second week
If you are already comfortable with using Git and GitLab and you are able to retain a good amount of information by just watching or reading through, that is okay. But if you see a topic that is completely new to you, stop the video and try it out for yourself before continuing.
-
Let your manager know you're ready to be assigned a trainer. -
Add your Primary Citizenship and Additional Citizenship(s) to the fields in BambooHR on the Personal tab if you haven't already. -
Manager: if they are US Citizens based in the US, add them to the Federal ZD instance.
-
-
Just quickly check on your Zendesk account to make sure that is ready for you when you need it. -
Add a profile picture to your Zendesk account -
Let your manager know if you were not able to create an account in Zendesk for some reason. -
Under your profile in Zendesk, it should read Support Staff
. If it readsLight Agent
, inform your manager. -
Get familiar with our Support Workflows & Statement of Support -
Request access to Chatops -
Watch this video about gitlab debug techniques and architecture overview
Go over these topics in GitLab University:
-
Under the topic of Version Control and Git -
About Version Control -
Try Git -
Explore Git internals and go back to it from time to time to learn more about how Git works
-
-
Under the topic of GitLab Basics -
All the GitLab Basics that you don't feel comfortable with. If you get stuck, see the linked videos under GitLab Basics in GitLab University -
GitLab Flow -
Take a look at how the different GitLab versions compare
-
-
Any of these topics that you don't feel comfortable with in the user training we use for our customers: env_setup.md
,feature_branching.md
,stash.md
,git_log.md
. For the rest of the topics inuser training
, just do a quick read over the file names so you start remembering where to find them. -
Get familiar with the services GitLab offers -
The differences between CE and EE (Core, Starter, Premium, Ultimate)
-
-
Get familiar with the different teams in-charge of every stage in the DevOps cycle and what they are responsible for. This will assist you in adding the right labels when creating issues and escalating in the right Slack channel.
Stage 2. Installation and Administration basics.
-
Done with Stage 2
Typically started in the first week, completed by the end of the third week
Goals
- Get your development machine set up.
- Familiarize yourself with the codebase.
- Be prepared to reproduce issues that our users encounter.
- Be comfortable with the different installation options of GitLab and configure LDAP.
- Have an installation available for reproducing customer bug reports.
-
Install this masterpiece created by Anton Smith, you'll thank yourself later for doing this. -
Check back on your Zendesk account to see if you are an Support Staff
yet. -
Manager: Add team member to the GitLab Support
Google Calendar.
-
Add team member to the GitLab Support
Google Cloud Project withSupport Permissions
.
Set up your development machine
-
Install the GDK. If you like Docker (especially if you're running Linux on your desktop - you might like to consider GitLab Compose Kit instead.) -
Add OpenLDAP to your GDK installation.
Installations
You will keep one installation continually updated on Digital Ocean (managed through terraform), just like many of our clients, but you need to choose where you would like to test other installations. (TODO: We need to list some benefits of each choice here.)
-
Manager: Add new team member to the dev-resources project. -
Use terraform to create a new test instance. -
Clone the dev-resources project. -
Read up on how to create a new instance. -
Create a new file and name it <your-full-name>.tf
in a new branch. You can copy another team member's file and modify it or you can create your own from scratch. If you have any issues or questions ask in the#support_self-managed
channel. -
After you've created it, open up a merge request and if the tests pass then merge it.
-
-
Set up your test environments. -
Choose between Local VM's or Digital Ocean for your preferred test environment, and note it in a comment below. -
Perform each of the following Installation Methods on the test environment you chose above: -
Install via Omnibus -
Populate with some test data: User account, Project, Issue -
Backup using our Backup rake task -
Install via Docker on your local machine. -
Restore a backup to your Docker VM using our Restore rake task -
Install GitLab from Source. Installation from source is not common but will give you a greater understanding of the components that we employ and how everything fits together.
-
-
With your manager or another Support team member, manually generate 3 'self managed' licenses at https://license.gitlab.com/ one for each of Starter, Premium and Ultimate. Suggest around 100 users with no expiry. Keep the licenses securely in 1Password. Use the licenses on your test instance when replicating customer issues. By having the correct license you will ensure feature parity with what the customer is seeing. -
The services below are part of the many services used by our customers. Installing these services and learning how they work with GitLab will prepare you for Stage 3 and give you a solid foundation for when you're ready to resolve tickets on your own.
-
GitLab Runner -
Elastic Search -
Install on a VM or on a separate instance on Terraform -
Make sure you [install the JVM](I used Docker instead, because Java) (https://www.elastic.co/guide/en/elasticsearch/reference/current/setup.html#jvm-version)- The steps outlined on Digital Ocean should help you configure your instance's IP
-
Connect Elastic Search to your GitLab instance and index your repositories
-
-
Kubernetes -
Login to https://console.cloud.google.com/ with your work email -
Create a cluster under the project gitlab-internal
. Remember to remove it when you're done! -
You will also need to install the gcloud SDK
-
Create a project on your self-managed instance, and then create a file named .gitlab-ci.yml
: -
Use the yaml docs as a resource to build a basic script -
Make sure your project has access to your Kubernetes cluster under Operations -> Kubernetes -
Make sure your project has registered runners under your project's Settings -> CI/CD -> Runners -
Pat yourself on the back; you're now ready to reproduce some of our common customer tickets
-
-
-
You can check this box but this one never stops as long as you are a Support Engineer for GitLab: Keep this installation up-to-date as patch and version releases become available, just like our customers would. -
Ask as many questions as you can think of in the #support_self-managed
chat channel
Stage 3. Customer Interaction through Zendesk
-
Done with Stage 3
Typically started in the first week, and completed by the end of the fourth week
Goals
- Have a good understanding of ticket flow in Zendesk and how to interact with our various channels.
- See some common issues that our customers face and how to resolve them.
Initial Zendesk training
Zendesk is our Support Center and our main communication line with our customers. We communicate with customers through several other channels too, see the support handbook for the full list. Set aside around 40 minutes for completion of this section.
-
Complete Zendesk Agent training -
Navigate to Zendesk university and register. You'll receive an email with information on accessing the Zendesk course -
Complete the "Zendesk Suite Overview: Support" course (approx. 10 min)
-
-
Review additional Zendesk resources -
UI Overview -
Updating Tickets -
Working w/ Tickets (Read Avoiding Agent Collisions carefully). -
Using Macros (Try the Self-Managed > Pre customer call
in one of the tickets) -
Language Translation using Unbabel
-
Learn about the Support process
-
Perform 15 one-hour pairing sessions with Support Engineers. Contact each in Slack to find a time that works, then create a Calendar event, inviting that person. When it's time for the pairing session, create a new Support Pairing project Issue and use the Ticket Pairing template for the call. Have at least 3 calls with a Support GitLab.com member. -
call with Keven; issue link: 384 -
call with Keven; issue link: 425 -
call with Cynthia; issue link: 445 -
call with Keven; issue link: 448 -
call with Blair; issue link: 480 -
call with Aric; issue link: 481 -
call with Muhamed; issue link: 504 -
call with Julianne; issue link: 506 -
call with Diana; issue link: 511 -
call with Keven; issue link: 515 -
call with Ben/Chris/VladB; issue link: 552 -
call with Rehab; issue link: 583 -
call with Catalin; issue link: 584 -
call with Nathan; issue link: 593 -
call with Greg; issue link: 594
-
-
Dive into our ZenDesk support process by reading how to handle tickets. -
Learn about our Support Modes of Work. - Here you will learn about our four rotating roles so that people know what to do and have variation in their work.
-
Learn about the hot queue. -
Learn about the Zendesk SLA settings and Breach Alerts. -
Start getting real world experience by handling real tickets, all the while gaining further experience with GitLab. -
First, learn about our Support Channels. -
Proceed on to Regular email support tickets. - Here you will find tickets from our GitLab EE Customers and GitLab CE Users
- Tickets here are extremely varied and often very technical
- You should be prepared for these tickets, given the knowledge gained from the previous steps of your training
- Tickets in Zendesk contain a wealth of knowledge from past interactions. Often times, the issue has occurred at least once and a solution has been provided, or an issue created on the customer's behalf. Reading through past tickets can aide in the learning process, getting you up to speed quicker.
- Look for errors, keywords, etc. to search Zendesk like so:
"keyword or error"
.
- Look for errors, keywords, etc. to search Zendesk like so:
-
Zendesk Support search reference
-
-
Check out your colleagues' responses. -
Hop on to the #zd-self-managed-feed
channel in Slack and see the tickets as they come in and are updated. -
Read through about 20 old tickets that your colleagues have worked on and their responses.
-
-
Watch the GitLab Debugging Techniques: A Support Engineering Perspective video on GitLab Unfiltered (YouTube). This dives into some common issues that customers might run into.
Learn about the Escalation process for Issues
Some tickets need specific knowledge or a deep understanding of a particular component and will need to be escalated to a Senior Support Engineer or a Developer.
-
Read about creating issues and issue prioritization. -
Take a look at the GitLab.com Team page to find the resident experts in their fields.
Learn about raising issues and fielding feature proposals
-
Understand what's in the pipeline at GitLab as well as proposed features: Direction Page. -
Practice searching issues and filtering using labels to find existing feature proposals and bugs. -
If raising a new issue always provide a relevant label and a link to the relevant ticket in Zendesk. -
Add customer labels for those issues relevant to our subscribers. -
Take a look at the existing issue templates to see what is expected (look at the comments in the markup for details.) -
Raise issues for bugs in a manner that would make the issue easily reproducible. A Developer or a contributor may work on your issue. -
Schedule a 45 minute call with your trainer where you share your screen with them while you answer tickets on Zendesk, and they give you feedback and answer your questions. The goal of this call is for your trainer to pass on tactical knowledge about the ticket handling process. Repeat this step a few times if it would be beneficial to you.
Congratulations. You now have your Zendesk Wings!
From now on you can spend most of your work time answering tickets in Zendesk. Try
to set aside 2 hours per day to make it through Stage 4-6 of this list. Never
hesitate to ask as many questions as you can think of in the #support_self-managed
chat channel.
Stage 4. Customer Calls
-
Done with Stage 4
Typically started in week 2 or 3, and completed by the end of week 4.
Goal Gain experience in scheduling and participating in calls with customers.
Look at the GitLab Support
Google Calendar to find customer calls you can listen
in on. Contact the person leading the call to check if it is okay for you to jump
in on the call, and if they could stay on with you for a few minutes after the call,
so you can ask them a few questions about the things you didn't understand. Also, ask
them to ask you a few questions to make sure you understand the points they want to
highlight from the call.
-
Start arranging to pair on calls with other Support Engineers. Aim to cover two of each type of call. Comment on this issue with the type of call you were in, who it was with, and the link to the relevant ticket (if one exists).
Note: Circumstances ended up occurring where I just field calls on my own without reverse shadowing
-
Manager: invite the new team member to the group Calendly account (Account->Users->New User) -
Read our Calendly page, and change it from the GitLab setup to how Support use it, including: -
Set up Zoom integration on your Calendly account by going to calendly integrations and following the instructions for Zoom (make sure to update the calendly event's location to use "Zoom") -
Set the title of your calendly events to include 'support' so the zap works, and install the browser plugin.
-
-
Calculate your desired work hours in UTC time and ping @dstanley to add you to the calendly Team Support Call event with that info. -
Read Customer Call process documentation
Stage 5. Gathering Diagnostics
-
Done with Stage 5
Typically done around the third week.
Goal Understand the gathering of diagnostics for GitLab instances
-
Learn about the GitLab checks that are available -
Learn about additional Support Tools
Stage 6. On Call Duty
-
Done with Stage 6
Note: I will be taking on CMOC Duties, not Customer Emergencies. First shift will be November 16, 2020.
Typically to be completed within 8 weeks of joining.
Goal Aim to become a fully-fledged Support Engineer!
-
Read about on-call duty: -
GitLab's on-call guide. On that page, the Customer Emergency On-Call Rotation Section is what applies to Support Engineers, NOT the Production Team section. -
GitLab Support's On-Call Guide contains even more information specific to handling emergencies in Support.
-
-
Schedule time with your trainer to have them show you how to respond to Customer Emergencies. -
Ping your manager to add you to PagerDuty. -
Manager: add the Engineer to the rotation with at least 6 weeks of lead time before their first on-call shift.
-
-
Sign up on PagerDuty with the link that will be emailed to you, and install the app on your phone. -
Familiarize yourself with the interface and the functionality.
-
-
Configure your personal notification rules in PagerDuty under "My Profile" > "Notification Rules" -
Currently, customer emergency escalation policy is set to 10 minutes. That means if you do not respond to the notification within this period, the emergency will escalate to the rest of the team. Make sure your personal notification rules take this into account. -
Remember to update this accordingly when your details changed
-
-
Use this PagerDuty guide to subscribe to your on-call schedule. -
In the weeks leading up to your first on-call shift, shadow the on-call engineers who are handling customer emergencies. -
ZD Ticket: ___ -
Repeat this as many times as you are able.
-
-
After your first on-call shift, add the "On-call champion!" moniker to your entry on the team page and assign the MR to your manager.
Stage 7. Make the path better for those who come after.
You probably noticed that there was something broken or non-ideal in the onboarding process. You might have brought it up to your manager. They likely told you to look at Stage 7 of your onboarding issue.
-
Make an improvement to the Support Engineer Bootcamp issue.
Stage 8. Optional Advanced GitLab Topics
Discuss with your training manager if you should stop here and close your issue or continue. Also, discuss which of the advanced topics should be followed. Do not do all of them as they might not be relevant to what customers need right now and can be a significant time investment.
These are some of GitLab's more advanced features. You can make use of GitLab.com to understand the features from an end-user perspective and then use your own instance to understand setup and configuration of the feature from an Administrative perspective
-
Set up and try Git LFS -
Get to know the GitLab API, its capabilities and shortcomings -
Learn how to migrate from SVN to Git -
Set up GitLab CI -
Create your first GitLab Page -
Learn about using Sentry to track errors: -
Learn about using Kibana to track errors: -
Advanced Gitlab debugging and log collection