Skip to content

GitLab-com SaaS Basics - Simone Susino

module-name: "GitLab-com SaaS Basics"
area: "Customer Service"
maintainers:
  - ralfaro

Introduction

This training module provides Support Engineers with the basics of answering GitLab.com (SaaS) product related tickets, for both new and existing Support team members who are looking to start working on SaaS tickets.

Goals

At the end of this module, you should be able to:

  • use chatops to look up information.
  • answer GitLab.com questions.

General Timeline and Expectations

  • This module should take you 2 weeks to complete.
  • Read about our Support Onboarding process, the page also shows you the modules under the GitLab.com SAAS Support Basics Pathway.

Stage 0: Create Your Module

  1. Create an issue using this template by making the Issue Title: <module title> - <your name>.
  2. Add yourself and your trainer as the assignees.
  3. Set milestones, if applicable, and a due date to help motivate yourself!
  4. Submit a Slack, Google Groups, 1Password Vaults or Groups Access Request to be added to the support-dotcom Slack Group (not CHANNEL). This can only be done by a Slack Admin in our IT department. In the System name area, specify as follows:
- [ ] System name: Slack Group support-dotcom
    - Justification for this access: I am completing the SaaS Learning Module.

Consider using the Time Tracking functionality so that the estimated length for the module can be refined.

Stage 1: Overview and Chatops

NOTE

In Stage 1 you will be using an Auditor account which is a read-only account, and does not have admin access. If you have any issues completing tasks, please pair with a support team member that has admin access. To find someone with admin access, look for a team member that has 50% or more SaaS focus.

  • Done with Stage 1

  1. Read the GitLab.com Overview page.

  2. Your auditor account for GitLab.com should have been already provisioned for you during the onboarding process. If not, please check with your manager, onboarding buddy or the internal IT support team.

    • If your regular GitLab account has the following details:
      • username:awesome_user7
      • email: awesomeuser@gitlab.com
    • Your auditor account should be:
      • username: awesome_user7-auditor
      • email: awesomeuser+auditor@gitlab.com

    Note: If your username is different from your email prefix then your email is +auditor@gitlab.com

Setup the chatops access as follows:

  1. Read the first two sections: the Chatops intro and how it works.
  2. Check if you have Chatops access via your onboarding. If you don't already have it, get Chatops access.
    • Note: Use the "Google" sign in on the ops instance to create an account if needed.
  3. Read about commonly used chatops commands.
  4. Practice using chatops by finding your own user and one of the support groups. You can direct message chatops by finding 'GitLab Chatops' under Apps in Slack.
    • What's your user ID? ID: 29866052
    • What's the group ID for gitlab-com/support? ID: 2573511
    • Find a feature flag to search for and get its current status. Take a screenshot and post it in a comment to the issue. Please blur out any user/group-identifiable information or don't include one with scoping.
  5. Take a look at our improving chatops for Support epic. Consider contributing to one of the issues in the future. As more of these functions become available, the more we can move cases out of the admin-only section.

Stage 2: GitLab.com Admin Access (with manager approval)

Note: Stage 2 and 3 are reversible. For those who want a more "structured" introduction to procedural tickets, start here. If you're already familiar with GitLab and are used to troubleshooting self-managed instances, consider doing Stage 3 first.

Note

If you have already requested GitLab.com admin access as part of GitLab-com Saas Account Basics, then you can skip requesting admin access

  • Done with Stage 2

Request your Admin Access

  1. Speak to your manager about whether you will be handling "SaaS Account" tickets. If yes, do this section. If not, don't. See the support training handbook page for more information.
  2. Typically at week 3-4 at GitLab, you and your manager should check in that you're ready for admin access. If you are, complete GitLab.com Admin access Training Module. Note: We recommend submitting the access request approximately 1 week before you will be actively using it.

Keep in mind that only GitLab team members are admins, and that accounts are bound by our Terms of Use as covered in the overview.

Bottom Line: Never share information that a user would not have access to even if it's about another user within their organization, except for Enterprise Users. You can still suggest possible solutions based on the description of the problem (and even what you find out about the account), but definitive information cannot be shared.

Triaging

Many common tickets are triaged to other places.

  1. Personal Data Access and Account Deletion requests: review the workflow and the introduction page to understand how to process these types of requests. Note that there are macros in Zendesk for these cases.
  2. Subpoenas and other legal (court) requests. See Subpoenas and other requests for information
    • Note: If the customer request is not tied to an active court case or other legal matter, it may fall under one of the other cases below.
  3. Read over the example cases that we redirect to Security.
    1. Copyright (DMCA) Removal Requests has its own email.

Confirm Your Accounts

At this stage you should have 3 different GitLab.com accounts

  1. Your User account you recived on day 1 - username, username@gitlab.com
  2. Your Auditor account that was setup during your onboarding - username-auditor, username+auditor@gitlab.com
  3. Your Admin account (pending manager approval) that was setup during this module - username-admin, username+admin@gitlab.com

Note: If your username is different from your email prefix then your email is +"*"@gitlab.com

Stage 3: GitLab.com Architecture and Troubleshooting

  • Done with Stage 3

Keep in mind:

  1. As previously noted, GitLab.com users do not have admin accounts. Please do not send any doc links that are self-managed only (check badging at the top of the page or section for FREE/STARTER/PREMIUM/ULTIMATE ONLY) or anything from the administration section.
  2. Troubleshooting problems on SaaS and Self-managed are the same or similar. The main difference is where and how we gather the necessary information.

Architecture

  1. Review materials relevant to GitLab.com Architecture
    1. Read about the GitLab architecture up to and including the Simplified component overview section.
      • Note: You're not expected to remember everything, but to get a general sense of GitLab's architecture. Feel free to read the details on any component based on your interest.
    2. Give the Production architecture page a brief lookover.
      • Note: Mostly, you'll want a broad understanding of the "Current", "Service", and "Network Architecture" sections.
    3. Give the GitLab.com Setting docs page a brief review to know what settings we publicize.
      • Note: Instance level settings are split between admin dashboard, gprd-base settings (equivalent of gitlab.rb), and secrets (which only infra has access to).

Searching logs and filing issues

  1. Learn about finding and filing errors. Note: Use the Okta option to sign in to Kibana and Sentry.
    1. Using Kibana.
    2. Searching Kibana, Sentry, and filing GitLab issues from Sentry.
    3. As an exercise, visit a non-existing page on GitLab.com, such as gitlab.com/gitlab-com/doesntexist. Search Kibana for the 404 error using your username. Add a screenshot of the relevant portion of the error you find in Kibana as a comment to this issue.
    4. Search Sentry using the correlation_id. You may not find anything and the search is not reliable, but take a screenshot of your search and results and add it as a comment to this issue anyway.
    5. Exercise: Read through the Export workflow.
    6. Perform an example: Export a personal project on GitLab.com, and search for the log entries related to export.
  2. Recommended: Pair on ticket(s) that require use of Kibana/Sentry. Alternatively, wait until you have your first ticket that requires Kibana and/or Sentry.

Places to reproduce

The best way to troubleshoot is to try to reproduce it.

  1. You have access to 2 GitLab.com personal namespaces, one for each plan tier. _premium_group, and _ultimate_group.
    • Since these are private groups, you can not link them in tickets for customers to view.
    • If you find out that you do not have access to one of these groups, ask for help in the #support_operations Slack channel.

Specific cases

Because GitLab is the steward of GitLab.com, we sometimes need to troubleshoot some of our integrations. We have workflows for some cases. You don't need to read these in detail now, but keep in mind that these are available:

  1. Custom domain verification on GitLab.com
  2. Service Desk Troubleshooting

IP blocks and log requests

  1. At times, the infra team blocks users due to alerts of possible abuse. Learn about sending notices.
    • Note: If needed, ping the CMOC to send these out.
  2. Read about identifying the cause of IP blocks. You won't find a block, but try doing a search by IP of your own IP to see what gets logged in the rails log.
  3. When responding to users about possible blocks, you'll need to keep in mind what information you're allowed to provide. See the log and audit requests workflow for more information.

Security issues

Security's focus is instance wide security breaches and vulnerabilities. They do however take care of some account related cases (covered in previous stage).

  1. Commonly, users file commit associations as a "security" issue. Follow the investigate commits on determining whether it's a security incident.

GitLab.com incidents

Incidents are anomalous conditions that result in — or may lead to — service degradation or outages. You will learn more about GitLab.com incidents in the On-Call Basics module, so you can skip this section if you're already doing that module.

If you notice an influx of very similar SaaS tickets in a short period of time, first check status.gitlab.com and the #incident-management channel to see if an incident has been declared. If one hasn't been declared, you may need to do so.

  1. Consider subscribing to status.gitlab.com to stay aware of active incidents.
  2. Read about how to report an incident.
  3. Give the Tracking Incidents workflow a brief review to understand how to handle tickets related to a declared incident.

Stage 4: GitLab.com User Accounts work

If you are not doing this stage, skip to the last section.

  • Done with Stage 4

Note

If you are working on GitLab.com user account tickets (SaaS Account), we recommend you complete GitLab-com SaaS Account Basics training module. Once done, you can return to this module!

Internal Requests

  1. Update notification settings to Watch which will send you notifications for all activity, or Custom for new issues in the following project:
    1. internal requests. Alternatively, given there are a lot of different requests, only subscribe to the following labels:
  2. For internal requests, you only need to work on the requests that relate specifically to GitLab.com. Many of the other issues as labelled are specific to other areas (such as Console or Licensing).
    1. Check out the list of templates.
    2. Read the Servicing Internal Requests workflow which covers most internal issues.
      1. (Optional as these are rare) Link to issue:

Data Requests

  1. Discuss with your manager if you should be added to the group of Support team members working on Data requests.
    • If no, skip this section.
    • If you're unsure, you can post in the #support_gitlab-com channel to shadow a Support team member specifically on an Account Deletion request before making a decision.
  2. If yes,
    1. submit a merge request to your support team yaml file, changing works_account_deletion to true. Assign to your manager for approval.
    2. Submit an access request for "Contributor" access to the Personal Data Access Request shared Drive, tagging @lyle for approval and provisioning.

Personal Data Access and Account Deletion Requests are created in the Account Deletion and Other Requests project tracker. Once you've been assigned your first issue, follow the workflow to work on it. Pair with someone else who works on these requests as needed. You should complete 1 request and link your issue.

  1. Link to issue:

Stage 5: Keeping up to date

  • Done with Stage 5

Keeping up to date and asking questions:

  1. Keep an eye out for "SaaS focus" marked items in the Support Week in Review (SWIR).
  2. Make sure you're in the #support_gitlab-com Slack channel where you can ask questions about tickets, process, etc. Also check the pinned messages in that channel for recent changes.

Stage 6: GitLab.com console (with manager approval)

Depending on your prior experience and readiness, discuss with your manager whether to open the module listed below. Your manager may decide to have you pick this up later once you have more familiarity with GitLab.

  1. Open a module on GitLab.com Console.

Final Stage

  1. Manager: schedule a call (or integrate into 1:1) to review how the module went once you have reviewed this issue.
  2. Please submit MRs for this Issue Template with any improvements that you have noticed.
  3. Submit an MR to update your focuses slug in your Support Team yaml file with this training module's topic and the percentage of your time you are dedicated to this Area of Focus. The settings will display on the Areas of Focus page.

NOTE: if you completed Stage 4 (user accounts + admin access) section and GitLab-com SaaS Account Basics, you should be working on SaaS Account tickets and admin-related internal requests, and your Area of Focus should be "SaaS 50%" or higher.

Edited by Simone Susino