Skip to content

GitLab-com Basics - Mike Lockhart

module-name: "GitLab-com SaaS Basics"
area: "Customer Service"
maintainers:
  - TBD

Introduction

This checklist provides Support Engineers with the basics of answering GitLab.com (SaaS) product related tickets, for both new and existing Support team members who are looking to start working on SaaS tickets.

Goals of this checklist

At the end of this module, you should be able to:

  • use chatops to look up information.
  • answer GitLab.com questions.

General Timeline and Expectations

  • This issue should take you 2 weeks to complete.
  • Read about our Support Onboarding process, the page also shows you the modules under the GitLab.com SAAS Support Basics Pathway.

Stage 0: Create Your Module

  1. Create an issue using this template by making the Issue Title: -
  2. Add yourself and your trainer as the assignees.
  3. Set milestones, if applicable, and a due date to help motivate yourself!

Consider using the Time Tracking functionality so that the estimated length for the module can be refined.

Stage 1: Overview and Chatops

  • Done with Stage 1
  1. Read the GitLab.com Overview page.
  2. If you have not already received an Auditor account for GitLab.com, create a new GitLab.com account that will serve as your admin account using the username and email address format of username-auditor and username+auditor@gitlab.com, confirm it via email, and then enable 2FA on it. Example access request.
    • If your regular GitLab account has the following details:
      • username:awesome_user7
      • email: awesomeuser@gitlab.com
    • Your auditor account should be:
      • username: awesome_user7-auditor
      • email: awesomeuser+auditor@gitlab.com

Regardless of whether you receive admin access later, you should set up chatops access.

  1. Read the first two sections: the Chatops intro and how it works.
  2. If you don't already have it, get Chatops access.
    • Note: Use the "Google" sign in on the ops instance to create an account if needed.
  3. Read about commonly used chatops commands.
  4. Practice using chatops by finding your own user and one of the support groups. You can direct message chatops by finding 'GitLab Chatops' under Apps in Slack.
    • What's your user ID? 5791080
    • What's the group ID for gitlab-com/support? ID: 2573511
    • Find a feature flag to search for and get its current status. Take a screenshot and post it in a comment to the issue. Please blur out any user/group-identifiable information or don't include one with scoping.
  5. Take a look at our improving chatops for Support epic. Consider contributing to one of the issues in the future. As more of these functions become available, the more we can move cases out of the admin-only section.

Stage 2: GitLab.com Accounts

Note: Stage 2 and 3 are reversible. For those who want a more "structured" introduction to procedural tickets, start here. If you're already familiar with GitLab and are used to troubleshooting Self-managed, consider doing Stage 3 first.

See also Stage 4 for a place to record pairing sessions.

  • Done with Stage 2

Keep in mind that only GitLab team members are admins, and that accounts are bound by our Terms of Use as covered in the overview.

Bottom Line: Never share information that a user would not have access to even if it's about another user within their organization, except for Enterprise Users. You can still suggest possible solutions based on the description of the problem (and even what you find out about the account), but definitive information cannot be shared.

Triaging

Many common tickets are triaged to other places.

  1. Account Deletion and Data requests: Read only the Overview piece for each workflow so you can see where these should be routed to. There are also macros in ZenDesk for these cases.
    1. Data Access Requests (typically GDPR Article 15)
    2. Account Deletion (typically GDPR Driven)
  2. Subpoenas and other legal (court) requests. See Subpoenas and other requests for information
    • Note: If the customer request is not tied to an active court case or other legal matter, it may fall under one of the other cases below.
  3. Read over the example cases to move to Security.
    1. Copyright (DMCA) Removal Requests has its own email. Instead of redirecting the user to email the dmca account, the ticket can be moved to the Security queue.

Email Receiving

One of the main differences of working with GitLab.com is that we receive many requests from free users because we are the administrators of GitLab.com. One of the most common cases is not receiving confirmation, password, or other system notifications from GitLab.

  1. Read the Confirmation Emails workflow.
  2. [-] Answer at least 1 ticket with an email receiving workflow: (insert ticket link here)
  3. Briefly look over Real Time Blocklist Delisting for when GitLab emails are being marked as spam.

Other Account Issues

  1. Take a look at the list of GitLab.com workflows or read the admin only section (Stage 4) and make note what other common cases we take care of, but require admin access.

Remember: When triaging tickets and you're unsure if a user is asking about a Self-managed instance or GitLab.com, use chatops to check if the user exists. The API searches for primary email and secondary emails starting 13.7 as described in this issue.

Stage 3: GitLab.com Architecture and Troubleshooting

  • Done with Stage 3

Keep in mind:

  1. As previously noted, GitLab.com users do not have admin accounts. Please do not send any doc links that are self-managed only (check badging at the top of the page or section for CORE/STARTER/PREMIUM/ULTIMATE ONLY) or anything from the administration section.
  2. Troubleshooting problems on SaaS and Self-managed are the same or similar. The main difference is where and how we gather the necessary information.

Architecture

  1. Review materials relevant to GitLab.com Architecture
    1. Read about the GitLab architecture up to and including the Simplified component overview section.
      • Note: You're not expected to remember everything, but to get a general sense of GitLab's architecture. Feel free to read the details on any component based on your interest.
    2. Give the Production architecture page a brief lookover.
      • Note: Mostly, you'll want a broad understanding of the "Current", "Service", and "Network Architecture" sections.
    3. Give the GitLab.com Setting docs page a brief review to know what settings we publicize.
      • Note: Instance level settings are split between admin dashboard, gprd-base settings (equivalent of gitlab.rb), and secrets (which only infra has access to).

Searching logs and filing issues

  1. Learn about finding and filing errors. Note: Use your dev account to sign into Kibana and Sentry.
    1. Using Kibana.
      • Note: Typical usage is rails and sidekiq logs. HAproxy and CloudFlare logs are only available to infra team. Ask in the #production Slack channel if necessary.
    2. Searching Kibana, Sentry, and filing GitLab issues from Sentry.
    3. As an exercise, visit a non-existing page on GitLab.com, such as gitlab.com/gitlab-com/doesntexist. Search Kibana for the 404 error using your username. Add a screenshot of the relevant portion of the error you find in Kibana as a comment to this issue.
    4. Search Sentry using the correlation_id. You may not find anything and the search is not reliable, but take a screenshot of your search and results and add it as a comment to this issue anyway.
    5. Exercise: Read through the import workflow, then search for an example import error.
    6. Importing customer projects is the only current case of "getting around" a bug or lack of feature. Read the rest of the import criteria to learn when we do these as a courtesy.
  2. If an issue can only be resolved through rails console access, you can file a "GitLab.com Console Escalation" issue.

Places to reproduce

The best way to troubleshoot as always is to try to reproduce it.

  1. Remember that you have access to 3 groups for each plan tier gitlab-gold, gitlab-silver, and gitlab-bronze.
    • Since these are public, you can also link them in tickets as long as you make the project public as well.
  2. You can check the Auto DevOps project to see what a typical job for various parts of the pipeline may be. It's on a semi-monthly schedule to catch what's changed pre and post release.

Specific cases

Because GitLab is the steward of GitLab.com, we sometimes need to troubleshoot some of our integrations. We have workflows for some cases. You don't need to read these in detail now, but keep in mind that these are available:

  1. Custom domain verification on GitLab.com
  2. Service Desk Troubleshooting

IP blocks and log requests

  1. At times, the infra team blocks users due to alerts of possible abuse. Learn about sending notices.
    • Note: If needed, ping the CMOC to send these out.
  2. Read about identifying the cause of IP blocks. You won't find a block, but try doing a search by IP of your own IP to see what gets logged in the rails log.
  3. When responding to users about possible blocks, you'll need to keep in mind what information you're allowed to provide. See the log and audit requests workflow for more information.

Security issues

Security's focus is instance wide security breaches and vulnerabilities. They do however take care of other account related cases (covered in previous stage).

  1. Commonly, users often file commit associations as a "security" issue. Follow the investigate commits on determining whether it's a security incident.

Congratulations! You made it, and can now help customers with GitLab.com SAAS product related tickets.

Please also submit MRs for any improvements that you can think of! The file is located in an issue template in the 'support-training` repository.

Edited by Mike Lockhart | GitLab