Michelle Almendarez - GitLab-com SaaS Basics
module-name: "GitLab-com SaaS Basics"
area: "Customer Service"
maintainers:
- cynthia
Introduction
This training module provides Support Engineers with the basics of answering GitLab.com (SaaS) product related tickets, for both new and existing Support team members who are looking to start working on SaaS tickets.
Goals
At the end of this module, you should be able to:
- use chatops to look up information.
- answer GitLab.com questions.
General Timeline and Expectations
- This module should take you 2 weeks to complete.
- Read about our Support Onboarding process, the page also shows you the modules under the GitLab.com SAAS Support Basics Pathway.
Stage 0: Create Your Module
-
Create an issue using this template by making the Issue Title: <module title> - <your name>
. -
Add yourself and your trainer as the assignees. -
Set milestones, if applicable, and a due date to help motivate yourself! -
Submit a Slack, Google Groups, 1Password Vaults or Groups Access Request to be added to the support-dotcom
Slack Group (not CHANNEL). This can only be done by a Slack Admin in our IT department. In theSystem name
area, specify as follows: Issue 23345
- [ ] System name: Slack Group support-dotcom
- Justification for this access: I am completing the SaaS Learning Module.
Consider using the Time Tracking functionality so that the estimated length for the module can be refined.
Stage 1: Overview and Chatops
NOTE
In Stage 1 you will be using an Auditor account which is a read-only account, and does not have admin access. If you have any issues completing tasks, please pair with a support team member that has admin access. To find someone with admin access, look for a team member that has 50% or more SaaS focus.
-
Done with Stage 1
-
Read the GitLab.com Overview page. -
You should have already received an Auditor account for GitLab.com during the onboarding process. If not, follow up on your onboarding issue, or create a new GitLab.com account and submit a new access request. Example access request. - If your regular GitLab account has the following details:
- username:
awesome_user7
- email:
awesomeuser@gitlab.com
- username:
- Your auditor account should be:
- username:
awesome_user7-auditor
- email:
awesomeuser+auditor@gitlab.com
- username:
- If your regular GitLab account has the following details:
Setup the chatops access as follows:
-
Read the first two sections: the Chatops intro and how it works. -
If you don't already have it, get Chatops access. - Note: Use the "Google" sign in on the ops instance to create an account if needed.
-
Read about commonly used chatops commands. - Practice using chatops by finding your own user and one of the support groups. You can direct message chatops by finding 'GitLab Chatops' under Apps in Slack.
-
What's your user ID? ID: malmendarez ; malmendarez-auditor -
What's the group ID for gitlab-com/support
? ID: 2573511 -
Find a feature flag to search for and get its current status. Take a screenshot and post it in a comment to the issue. Please blur out any user/group-identifiable information or don't include one with scoping.
-
-
Take a look at our improving chatops for Support epic. Consider contributing to one of the issues in the future. As more of these functions become available, the more we can move cases out of the admin-only section. -
(Optional) Watch this Getting Started with Chatops in Support video (~35 minutes) to understand how to get started with contributing to chatops in GitLab Support.
-
Stage 2: GitLab.com Accounts
Note: Stage 2 and 3 are reversible. For those who want a more "structured" introduction to procedural tickets, start here. If you're already familiar with GitLab and are used to troubleshooting self-managed instances, consider doing Stage 3 first.
See also Stage 4 for a place to record pairing sessions.
-
Done with Stage 2
Keep in mind that only GitLab team members are admins, and that accounts are bound by our Terms of Use as covered in the overview.
Bottom Line: Never share information that a user would not have access to even if it's about another user within their organization, except for Enterprise Users. You can still suggest possible solutions based on the description of the problem (and even what you find out about the account), but definitive information cannot be shared.
Triaging
Many common tickets are triaged to other places.
-
Personal Data Access and Account Deletion requests: review the workflow and the introduction page to understand how to process these types of requests. Note that there are macros in Zendesk for these cases. -
Subpoenas and other legal (court) requests. See Subpoenas and other requests for information - Note: If the customer request is not tied to an active court case or other legal matter, it may fall under one of the other cases below.
-
Read over the example cases that we redirect to Security. -
Copyright (DMCA) Removal Requests has its own email.
-
Email Receiving
One of the main differences of working with GitLab.com is that we receive many requests from free users because we are the administrators of GitLab.com. One of the most common cases is not receiving confirmation, password, or other system notifications from GitLab.
-
Read the Confirmation Emails workflow. -
Answer at least 1 ticket with an email receiving workflow: Zendesk 425102 -
Briefly look over Real Time Blocklist Delisting for when GitLab emails are being marked as spam.
Other Account Issues
-
Take a look at the list of GitLab.com workflows or read the admin only section (Stage 4) and make note what other common cases we take care of, but require admin access.
Remember: When triaging tickets and you're unsure if a user is asking about a Self-managed instance or GitLab.com, use the GitLab User Lookup app in right ticket sidebar to see if the user exists.
Stage 3: GitLab.com Architecture and Troubleshooting
-
Done with Stage 3
Keep in mind:
- As previously noted, GitLab.com users do not have admin accounts. Please do not send any doc links that are self-managed only (check badging at the top of the page or section for FREE/STARTER/PREMIUM/ULTIMATE ONLY) or anything from the
administration
section. - Troubleshooting problems on SaaS and Self-managed are the same or similar. The main difference is where and how we gather the necessary information.
Architecture
- Review materials relevant to GitLab.com Architecture
-
Read about the GitLab architecture up to and including the Simplified component overview
section.- Note: You're not expected to remember everything, but to get a general sense of GitLab's architecture. Feel free to read the details on any component based on your interest.
-
Give the Production architecture page a brief lookover. - Note: Mostly, you'll want a broad understanding of the "Current", "Service", and "Network Architecture" sections.
-
Give the GitLab.com Setting docs page a brief review to know what settings we publicize. - Note: Instance level settings are split between admin dashboard, gprd-base settings (equivalent of
gitlab.rb
), and secrets (which only infra has access to).
- Note: Instance level settings are split between admin dashboard, gprd-base settings (equivalent of
-
Searching logs and filing issues
- Learn about finding and filing errors. Note: Use your dev account to sign into Kibana and Sentry.
-
Using Kibana. - Note: Typical usage is
rails
andsidekiq
logs.HAproxy
and Cloudflare log access is restricted. Ask in the #production Slack channel if necessary.
- Note: Typical usage is
-
Searching Kibana, Sentry, and filing GitLab issues from Sentry. -
As an exercise, visit a non-existing page on GitLab.com, such as gitlab.com/gitlab-com/doesntexist. Search Kibana for the 404 error using your username. Add a screenshot of the relevant portion of the error you find in Kibana as a comment to this issue. -
Search Sentry using the correlation_id
. You may not find anything and the search is not reliable, but take a screenshot of your search and results and add it as a comment to this issue anyway. -
Exercise: Read through the import workflow, then search for an example import error. -
Importing customer projects is the only current case of "getting around" a bug or lack of feature. Read the rest of the import criteria to learn when we do these as a courtesy.
-
- If an issue can only be resolved through rails console access, you can file a "GitLab.com Console Escalation" issue.
-
Recommended: Pair on ticket(s) that require use of Kibana/Sentry. Alternatively, wait until you have your first ticket that requires Kibana and/or Sentry.
Places to reproduce
The best way to troubleshoot is to try to reproduce it.
-
You have access to 2 GitLab.com groups: one for each plan tier gitlab-gold
(Ultimate),gitlab-silver
(Premium).- Since these are private groups, you can not link them in tickets for customers to view.
- If you find out that you do not have access to one of these groups, ask for help in #support_gitlab-com or #support_operations Slack channel.
Specific cases
Because GitLab is the steward of GitLab.com, we sometimes need to troubleshoot some of our integrations. We have workflows for some cases. You don't need to read these in detail now, but keep in mind that these are available:
IP blocks and log requests
-
At times, the infra team blocks users due to alerts of possible abuse. Learn about sending notices. - Note: If needed, ping the CMOC to send these out.
-
Read about identifying the cause of IP blocks. You won't find a block, but try doing a search by IP of your own IP to see what gets logged in the rails
log. -
When responding to users about possible blocks, you'll need to keep in mind what information you're allowed to provide. See the log and audit requests workflow for more information.
Security issues
Security's focus is instance wide security breaches and vulnerabilities. They do however take care of some account related cases (covered in previous stage).
-
Commonly, users file commit associations as a "security" issue. Follow the investigate commits on determining whether it's a security incident.
Stage 4: Keeping up to date
-
Done with Stage 4
Keeping up to date and asking questions:
-
Check the "Support Week in Review" doc for recent GitLab.com [SaaS]
updates/issues. -
Make sure you're in the #support_gitlab-com Slack channel where you can ask questions about tickets, process, etc. Also check the pinned messages in that channel for recent changes.
Stage 5: GitLab.com User Accounts work and admin access (with manager approval)
If you are not doing this stage, skip to the last section.
-
Done with Stage 5
-
Speak to your manager about whether you will be handling "SaaS Account" tickets. If yes, do this section. If not, don't. See the support training handbook page for more information. -
Typically at week 3-4 at GitLab, you and your manager should check in that you're ready for admin access. If you are, complete GitLab.com Admin access Training Module. Note: We recommend submitting the access request approximately 1 week before you will be actively using it. (Gitlab Admin Issue)[#3315 (closed)]
Internal Requests
- Update notification settings to Watch which will send you notifications for all activity, or Custom for new issues in the following project:
-
internal requests. Alternatively, given there are a lot of different requests, only subscribe to the following labels: - Admin Escalation
- DEWR (Dotcom Escalation Weekly Report)
-
- For internal requests, you only need to work on the requests that relate specifically to GitLab.com. Many of the other issues as labelled are specific to other areas (such as Console or Licensing).
-
Check out the list of templates. -
Read the Servicing Internal Requests workflow which covers most internal issues. -
(Optional as these are rare) Link to issue:
-
-
User Accounts
Aside from the ones covered in Stage 2, the most common user requests have to do with disabling 2FA and name squatting. However, most of these workflows apply in other cases as well.
-
When making any changes to a user account, make sure to use an admin note. -
For all 2FA disabling requests, use the Account Ownership Verification. This workflow may be used for some other cases as well where we want to verify the ownership. -
Link to an account ticket requiring verification: Zendesk Ticket 426651 / Zendesk Ticket 423136
-
-
Name Squatting Requests apply to groups and users. Included here since it is often for users. -
Except in the cases above, or wherever else there is a clear ask of the action (including impersonating the user), always ask for permission to take action.
Other Common Requests
-
reCaptcha complaints are often due to actions on a specific issue by a specific user being marked as spam. Learn how to mark as ham. - Note: The spam logs are not searchable, so requests have be made almost immediately after receiving the recaptcha prompt.
-
Reinstating blocked accounts when users say they they've been mistakenly blocked. -
(Optional) Link to a ticket about blocked account:
-
-
Personal Identifiable Information Removal Requests are uncommon, but read the workflow to understand Support's responsibility.
Data Requests
-
Discuss with your manager if you should be added to the group of Support team members working on Data requests. - If no, skip this section.
- If you're unsure, you can post in the
#support_gitlab-com
channel to shadow a Support team member specifically on an Account Deletion request before making a decision.
- If yes,
-
submit a merge request to the support team yaml, changing works_account_deletion
totrue
. Assign to your manager for approval. -
Submit an access request for "Contributor" access to the Personal Data Access Request shared Drive, tagging @lyle
for approval and provisioning.
-
Personal Data Access and Account Deletion Requests are created in the Account Deletion and Other Requests project tracker. Once you've been assigned your first issue, follow the workflow to work on it. Pair with someone else who works on these requests as needed.
-
Link to issue:
Stage 6: GitLab.com console (with manager approval)
Depending on your prior experience and readiness, discuss with your manager whether to open the module listed below. Your manager may decide to have you pick this up later once you have more familiarity with GitLab.
-
Open a module on GitLab.com Console.
Final Stage
-
Manager: schedule a call (or integrate into 1:1) to review how the module went once you have reviewed this issue. -
Please submit MRs for this Issue Template with any improvements that you have noticed. -
Submit an MR to update your focuses
slug in the Support Team yaml file with this training module's topic and the percentage of your time you are dedicated to this Area of Focus. The settings will display on the Areas of Focus page. MR 49d68315
NOTE: if you completed Stage 5 (user accounts + admin access) section, you should be working on SaaS Account tickets and admin-related internal requests, and your Area of Focus should be "SaaS 50%" or higher.