GitLab Dedicated Basics - Amaury Yacksmith

module-name: "GitLab Dedicated Basics"
area: "Customer Service"
maintainers:
  - bcarranza
  - lyle
  - mbadeau

Introduction

This training module provides Support Engineers with the basics of answering GitLab Dedicated product related tickets, for both new and existing Support team members who are looking to start working on these tickets.

Goals

At the end of this module, you should be able to:

• Understand the differences between GitLab SaaS, Self-managed and Dedicated
• Answer GitLab Dedicated questions
• Know where to get additional help

General Timeline and Expectations

• This module should take you 1 week to complete.
• Read about our Support Onboarding process

Stage 0: Create Your Module

1. Create an issue using this template by making the Issue Title: <module title> - <your name>.

2. Add yourself and your trainer as the assignees.

3. Set milestones, if applicable, and a due date to help motivate yourself!

4. Update your Support Team yaml file to indicate that you've started learning this knowledge area:

   knowledge_areas:
   - name: GitLab Dedicated
     level: 1

Consider using the Time Tracking functionality so that the estimated length for the module can be refined.

Stage 1: Overview - What is GitLab Dedicated?

• Done with Stage 1

1. Read the GitLab Dedicated Overview page.
2. Watch a recording on GitLab Unfiltered by Fabian Zimmer explaining how GitLab Dedicated differs from self-managed and GitLab.com as an offering. You can also review the presentation.
3. The GitLab Dedicated Engineer On Call (GDEOC) will be your main point of contact for infrastructure-related issues. Read the GDEOC on-call runbook to learn more about the GDEOC's responsibilities.
4. Newly created requests for help (RFHs) are assigned automatically to the GDEOC.
5. Read the GitLab Dedicated product category direction page
6. Read the GitLab Dedicated Docs page
   1. Note: Not all features are available and there are some features that are unavailable.
7. Some features are not available to all GitLab offerings (Saas, Self-managed, GitLab Dedicated). Briefly review our style guide for the product tier badges.
   1. View the top of this GitLab Dedicated docs page to see how the product tier badges are structed in the documentation. Note: If there isn't a tier badge present, it is assumed that the feature in question is available to all offerings.

Stage 2: GitLab Dedicated Architecture and Troubleshooting

• Done with Stage 2

Keep in mind:

1. Only GitLab Support Engineers and SREs do have access to logs, while GitLab Dedicated users do not have access to real-time logging or the underlying infrastructure. Please do not send any doc links that are self-managed only and that talk about server-side configuration (check badging at the top of the page or section for FREE/STARTER/PREMIUM/ULTIMATE ONLY).
   1. We should not expect customers to have logs for their GitLab Dedicated instance. Customers can request application logs be forwarded to an S3 bucket but these are not in real-time and not all customers have requested log forwarding.
   • Neither GitLab Support Engineers nor SREs have access to the administration UI (admin settings pages). Customers have access to their administration UI and are the only people that can manage users, groups, etc.
     1. SREs are unable to access instances, for example the GitLab Rails console, except in the case of emergencies. This procedure is called breaking glass and has a minimum set of requirements.
   • Troubleshooting problems on Dedicated, SaaS, and Self-managed are the same or similar. The main difference is where and how we gather the necessary information.
2. Bookmark the SaaS, Self-Managed and Dedicated Troubleshooting tables for more information about the differences and similarities between these platforms.
3. Bookmark the Dedicated Info Portal for checking maintenance window and release upgrade dates.

Architecture

1. Switchboard is the SSOT for mapping customer codenames to customer names and external URLs. You can read more about the history of this in the historical Temporary SSOT: Dedicated Customer Codenames Mapping issue.
2. Review materials relevant to GitLab Dedicated Architecture
   1. Read about the GitLab Dedicated architecture.
      • Note: You're not expected to remember everything, but to get a general sense of GitLab Dedicated's architecture. Feel free to read the details on any component based on your interest.
   2. GitLab Dedicated runs a modified Cloud Native Hybrid Reference Architecture. Review the Changes from Reference Architecture page. (This may be brief, if you are already familiar with the GitLab reference architecture).
   • Configs are managed by the Environment Automation team and presented to the customer via Switchboard.
     1. Visit Switchboard to view the configuration for a specific tenant. Read about accessing Switchboard. You should have access as a Baseline Entitlement, if you don't have access, you will need to open an Access Request
     2. Read about how customers can perform configuration changes in Switchboard.

Searching logs

• Access to logs is through credentials stored in the 1Password Vault.
• Watch a recording on GitLab Unfiltered by Craig Miskell demonstrating a live use of monitoring metrics and logs to debug a performance usage spike on a Dedicated tenant, with the goal of showing generalized techniques that can be used in many scenarios.

Accessing OpenSearch mini exercise

We leverage AWS OpenSearch (an AWS fork of Elasticsearch) for logging.

1. Access Opensearch by looking in the 1Password Gitlab Dedicated - Support Vault for the opensearch entries and access one of the URLs listed there.
2. Select "Global" tenant
3. Choose "Discover" at the sidebar under OpenSearch Dashboards
4. On the next screen, you should see logs. Make sure that index called gitlab-* is selected.
5. Only application logs can be shared with customers, not the internal Kubernetes logs.

Accessing Grafana mini exercise

Grafana can give you information on platform status such as possible downtime.

1. Access Grafana by looking in the 1Password Gitlab Dedicated - Support Vault for the grafana entries and access one of the URLs listed there.
2. Click the four boxes (dashboards) icon
3. Review the dashboards available. For this exercise, choose the Triage dashboard to see an overview
4. Grafana graphs cannot be shared with customers and are internal use only

Getting Help

NOTE: The GitLab Dedicated team is currently at its capacity and unable to answer requests in Slack.

• General questions about GitLab Dedicated can go to the #support_gitlab-dedicated channel.

If you need attention from the GitLab Dedicated team, open an issue in the issue tracker, do not ping the team or members of the team unless it's an absolute emergency.

Filing a Support Request (Request for Help)

These Customer Support Requests are the highest non-paging priority for Dedicated SREs.

• If an issue can only be resolved through rails console access, or needs additional troubleshooting from an SRE you can raise a GitLab Dedicated Support Request.

  • Direct link for opening an issue with the "Support Request" issue template: https://gitlab.com/gitlab-com/request-for-help/-/issues/new?issuable_template=SupportRequestTemplate-GitLabDedicated
  • For some requests (for example creation of a private link) there might be a more specific issue template available; feel free to browse through the available templates for inspiration, but you can always default to the "Support Request" template if unsure.
  • If you pick a more specific issue template, please make sure that the following labels are set:

  /label "Dedicated::Support Request"
  /confidential
  /label "team::GitLab Dedicated"
  /label "workflow-infra::Ready"

• Ping the appropriate team members in the issue to request a status update

• As a last resort in urgent situations, you can reach out in the #g_dedicated-team Slack channel to draw attention to the request

Emergencies + GitLab Dedicated

Escalating an Emergency issue

Emergencies from GitLab Dedicated will come through the Customer Emergencies On-call Rotation as with other emergency types.

1. Read the definition of an emergency in the GDEOC runbook
2. The GitLab Dedicated Infrastructure team has a 24/7 PagerDuty rotation: GitLab Dedicated Platform Escalation. You can manually create a PD Incident using the Dedicated Platform Service or use the Slack command /pd trigger and choose "Dedicated Platform Service" as the Impacted Service to escalate an emergency to an SRE after initial triage and analysis.

GitLab Dedicated CMOC

The Communications Manager on Call (CMOC) also acts as the GDCMOC (GitLab Dedicated Communications Manager on Call).

1. Be aware of How to engage the GDCMOC section of the handbook.

Places to reproduce

When you need to reproduce an issue:

Troubleshooting GitLab Features

1. You have access to 2 GitLab.com groups: one for each plan tier <your_gitlab_username>_ultimate_group (Ultimate), <your_gitlab_username>_premium_group (Premium).
   • This should be your primary method of troubleshooting as it will quickly identify if it's a problem with GitLab, the product, or GitLab Dedicated, the platform.
   • Since these are private groups, you can not link them in tickets for customers to view.
   • If you find out that you do not have access to one of these groups, ask for help in #support_gitlab-com or #support_operations Slack channel.

Troubleshooting platform or reference architecture

1. As with Self-managed troubleshooting, you have access to testing environments. For GitLab Dedicated specifically, you'll likely be testing in AWS with the appropriate GitLab Dedicated Architecture. Remember, you can also file an issue with Dedicated SREs.
   • Note: Recreating an entire GitLab Dedicated setup is an arduous task and is not recommended in most circumstances. For problems that could be platform specific it is recommended to file an issue with the GitLab Dedicated Environment Automation Team.

Security issues

The Gitlab Infrastructure Security team focuses on instance wide security breaches and vulnerabilities. If you suspect a security issue, engage the Security Incident Response Team.

Stage 3: Keeping up to date

• Done with Stage 3

Keeping up to date and asking questions:

1. Set up a pairing or coffee chat with GitLab Dedicated Support Stable Counterparts (within your region or outside of it) - Optional
2. Check the Support Week in Review for recent GitLab.com [Dedicated] updates/issues.
3. Make sure you're in the #f_gitlab_dedicated Slack channel where you can ask questions about the product itself, tickets, process, etc. Also check the pinned messages in that channel for recent changes.
4. Consider joining the GitLab Dedicated pod in the #support_gitlab-dedicated channel on Slack.
5. Consider joining #g_dedicated-team to interact with Dedicated SREs directly for any questions about a specific instance or Dedicated behavior.
6. Other Slack channels to consider joining include:

• #g_dedicated-environment-automation-team
• #g_dedicated-switchboard-team
• #f_hosted_runners_for_dedicated
• #sme-dedicated

Final Stage

1. Manager: schedule a call (or integrate into 1:1) to review how the module went once you have reviewed this issue.

2. Please submit MRs for this Issue Template with any improvements that you have noticed.

3. Submit an MR to update your knowledge_areas slug in your Support Team yaml file with this training module's topic.

   knowledge_areas:
   - name: GitLab Dedicated
     level: 2