Adam Mulvany - GitLab Dedicated training
module-name: "GitLab Dedicated Basics"
area: "Customer Service"
maintainers:
- bcarranza
- lyle
- mbadeau
Introduction
This training module provides Support Engineers with the basics of answering GitLab Dedicated product related tickets, for both new and existing Support team members who are looking to start working on these tickets.
Goals
At the end of this module, you should be able to:
- Understand the differences between GitLab SaaS, Self-managed and Dedicated
- Answer GitLab Dedicated questions
- Know where to get additional help
General Timeline and Expectations
- This module should take you 1 week to complete.
- Read about our Support Onboarding process
Stage 0: Create Your Module
-
Create an issue using this template by making the Issue Title: <module title> - <your name>
. -
Add yourself and your trainer as the assignees. -
Set milestones, if applicable, and a due date to help motivate yourself! -
Update your Support Team yaml file to indicate that you've started learning this knowledge area: knowledge_areas: - name: GitLab Dedicated level: 1
Consider using the Time Tracking functionality so that the estimated length for the module can be refined.
Stage 1: Overview - What is GitLab Dedicated?
-
Done with Stage 1
-
Read the GitLab Dedicated Overview page. -
The GitLab Dedicated Engineer On Call (GDEOC) will be your main point of contact for infrastructure-related issues. Read the GDEOC on-call runbook to learn more about the GDEOC's responsibilities. -
Read the GitLab Dedicated product category direction page -
Read the GitLab Dedicated Docs page -
Note: Not all features are available and there are many features that are unavailable.
-
-
Some features are not available to all GitLab offerings (Saas, Self-managed, GitLab Dedicated). Briefly review our style guide for the product tier badges. - View the top of this GitLab dedicated docs page to see how the product tier badges are structed in the documentation. Note: If there isn't a tier badge present, it is assumed that the feature in question is available to all offerings.
Stage 2: GitLab Dedicated Architecture and Troubleshooting
-
Done with Stage 2
Keep in mind:
- Only GitLab Support Engineers and SREs do have access to logs, while GitLab Dedicated users do not have access to real-time logging or the underlying infrastructure. Please do not send any doc links that are self-managed only and that talk about server-side configuration (check badging at the top of the page or section for FREE/STARTER/PREMIUM/ULTIMATE ONLY).
- We should not expect customers to have logs for their GitLab Dedicated instance. Customers can request application logs be forwarded to an S3 bucket but these are not in real-time and not all customers have requested log forwarding.
- Neither GitLab Support Engineers nor SREs have access to the administration UI (admin settings pages). Customers have access to their administration UI and are the only people that can manage users, groups, etc.
- SREs are unable to access instances, for example the GitLab Rails console, except in the case of emergencies. This procedure is called breaking glass and has a minimum set of requirements.
- Troubleshooting problems on Dedicated, SaaS, and Self-managed are the same or similar. The main difference is where and how we gather the necessary information.
Architecture
-
🔖 Bookmark the temporary SSOT for mapping customer numbers to customer names - Review materials relevant to GitLab Dedicated Architecture
-
Read about the GitLab Dedicated architecture. - Note: You're not expected to remember everything, but to get a general sense of GitLab Dedicated's architecture. Feel free to read the details on any component based on your interest.
-
GitLab Dedicated runs a modified Cloud Native Hybrid Reference Architecture. Review the Changes from Reference Architecture page. (This may be brief, if you are already familiar with the GitLab reference architecture).
-
- Configs are managed by the Environment Automation team and presented to the customer via Switchboard.
-
Visit Switchboard to view the configuration for a specific tenant. Read about accessing Switchboard. You should have access as a Baseline Entitlement, if you don't have access, you will need to open an Access Request -
Read about how customers can perform configuration changes in Switchboard.
-
Searching logs
Access to logs is through credentials stored in the 1Password Vault.
-
Access Opensearch by looking in the 1Password Gitlab Dedicated - Support
Vault for theopensearch
entries and access one of the URLs listed there.-
Accessing Opensearch mini exercise - Select "Global" tenant
- Choose "Discover" at the sidebar under OpenSearch Dashboards
- On the next screen, you should see logs. Make sure that index called
gitlab-*
is selected. - Only application logs can be shared with customers, not the internal Kubernetes logs.
-
-
Access Grafana by looking in the 1Password Gitlab Dedicated - Support
Vault for thegrafana
entries and access one of the URLs listed there.- Grafana can give you information on platform status such as possible downtime.
-
Accessing Grafana mini exercise - Click the four boxes (dashboards) icon
- Review the dashboards available. For this exercise, choose the Triage dashboard to see an overview
- Grafana graphs cannot be shared with customers and are internal use only
Getting Help
NOTE: The GitLab Dedicated team is currently at its capacity and unable to answer requests in Slack.
-
General questions about GitLab Dedicated can go to the #spt_pod_dedicated channel.
If you need attention from the GitLab Dedicated team, open an issue in the issue tracker, do not ping the team or members of the team unless it's an absolute emergency.
Filing a Support Request (Request for Help)
These Customer Support Requests are the highest non-paging priority for Dedicated SREs.
-
If an issue can only be resolved through rails console access, or needs additional troubleshooting from an SRE you can raise a GitLab Dedicated Support Request. - Direct link for opening an issue with the "Support Request" issue template: https://gitlab.com/gitlab-com/gl-infra/gitlab-dedicated/team/-/issues/new?issuable_template=support_request
- For some requests (for example creation of a private link) there might be a more specific issue template available; feel free to browse through the available templates for inspiration, but you can always default to the "Support Request" template if unsure.
- If you pick a more specific issue template, please make sure that the following labels are set:
/label "Dedicated::Support Request" /confidential /label "team::GitLab Dedicated" /label "workflow-infra::Ready"
-
Ping the appropriate team members in the issue to request a status update -
As a last resort in urgent situations, you can reach out in the #g_dedicated-team Slack channel to draw attention to the request
Emergencies + GitLab Dedicated
Escalating an Emergency issue
Emergencies from GitLab Dedicated will come through the Customer Emergencies On-call Rotation as with other emergency types.
-
Read the definition of an emergency in the GDEOC runbook -
The GitLab Dedicated Infrastructure team has a 24/7 PagerDuty rotation: GitLab Dedicated Platform Escalation. You can manually create a PD Incident using the Dedicated Platform Service or use the Slack command /pd trigger
and choose "Dedicated Platform Service" as the Impacted Service to escalate an emergency to an SRE after initial triage and analysis.
GitLab Dedicated CMOC
The Communications Manager on Call (CMOC) also acts as the GDCMOC (GitLab Dedicated Communications Manager on Call).
-
Read the How to Perform GitLab Dedicated CMOC Duties page. -
Browse through the GitLab Dedicated CMOC to get an idea of the expectations for the person acting as the GDCMOC.
Places to reproduce
When you need to reproduce an issue:
Troubleshooting GitLab Features
-
You have access to 2 GitLab.com groups: one for each plan tier <your_gitlab_username>_ultimate_group
(Ultimate),<your_gitlab_username>_premium_group
(Premium).- This should be your primary method of troubleshooting as it will quickly identify if it's a problem with GitLab, the product, or GitLab Dedicated, the platform.
- Since these are private groups, you can not link them in tickets for customers to view.
- If you find out that you do not have access to one of these groups, ask for help in
#support_gitlab-com
or#support_operations
Slack channel.
Troubleshooting platform or reference architecture
-
As with Self-managed troubleshooting, you have access to testing environments. For GitLab Dedicated specifically, you'll likely be testing in AWS with the appropriate GitLab Dedicated Architecture. Remember, you can also file an issue with Dedicated SREs. - Note: Recreating an entire GitLab Dedicated setup is an arduous task and is not recommended in most circumstances. For problems that could be platform specific it is recommended to file an issue with the GitLab Dedicated Environment Automation Team.
Security issues
The Gitlab Infrastructure Security team focuses on instance wide security breaches and vulnerabilities. If you suspect a security issue, engage the Security Incident Response Team.
Stage 3: Keeping up to date
-
Done with Stage 3
Keeping up to date and asking questions:
-
Check the Support Week in Review for recent GitLab.com [Dedicated]
updates/issues. -
Make sure you're in the #f_gitlab_dedicated
Slack channel where you can ask questions about the product itself, tickets, process, etc. Also check the pinned messages in that channel for recent changes. -
Consider joining the GitLab Dedicated pod in the #spt_pod_dedicated channel on Slack. -
Consider joining #g_dedicated-team
to interact with Dedicated SREs directly for any questions about a specific instance or Dedicated behavior.
Final Stage
-
Manager: schedule a call (or integrate into 1:1) to review how the module went once you have reviewed this issue. -
Please submit MRs for this Issue Template with any improvements that you have noticed. -
Submit an MR to update your knowledge_areas
slug in your Support Team yaml file with this training module's topic.knowledge_areas: - name: GitLab Dedicated level: 2