Realignment from Monitor:APM to Fulfillment, Telemetry and Verify
Note
This reorganization is an adjustment to the degree we allocate investment across our Product groups, it is not a cost-savings effort.
Transparency
This issue was originally authored in a private project because it is discussing a planned reorganization. We have since moved it to a public project as a confidential issue to ensure transparency amongst all GitLab team-members. it later became non-confidential in-line with our transparency value.
Background
TL;DR
From @sfwgitlab
We made a tentative decision in e-group today to reduce our investment in Monitor by roughly half, and transfer those people to fulfillment and possibly telemetry. Verify also has acute needs for additional people if Fulfillment/Telemetry don't need them all.
More Context
As a Product organization we have been shifting to concentrating our R&D investment into product scopes where we have shown the ability to drive active usage quickly. This is an important shift, as it allows us to be more data driven and therefore effective in our pursuit of the increasingly competitive complete DevOps platform market we find ourselves in. In looking across our current R&D investments, our outsized investment in Monitor stage capabilities was not the most effective use of our investment dollars - the current usage number growth don't support such a high investment in a Horizon 3 opportunity.
The E-Group decided where to best re-allocate that investment - and high impact initiatives of:
- Fulfillment - where our online ordering system is broken, causing customer pain and loss revenue
- Telemetry - where our ability to be data driven and experiment focused inhibits the effectiveness of all our Product investment
- Verify - where we're increasingly seeing the criticality of landing and expanding to CI as part of our sales and customer success journeys
were rightfully deemed as priorities.
Current Monitor Stage Staffing
Team | BE IC | FE IC | FE EM | BE EM | PM | UX | SDET |
---|---|---|---|---|---|---|---|
Monitor APM | 3 | 4 | 0.5 | 1 | 1 | 1 | 0.5 |
Monitor Health | 4 | 4 | 0.5 | 1 | 1 | 1 | 0.5 |
TOTAL | 7 | 8 | 1 | 2 | 2 | 2 | 1 |
Requests from Priority Teams
Team (by Priority) | BE IC | FE IC | FE EM | BE EM | PM | UX | SDET |
---|---|---|---|---|---|---|---|
1. Fulfillment | +2 | +2 | 0 | 0 | 0 | 0 | 0 |
2. Telemetry | +2 | 0 | 0 | 0 | 0 | 0 | 0 |
3. Verify | +2 | +2 | 0 | +1 | +1 | +1 | 0 |
TOTAL | 7 | 3 | 0 | 0 | 1 | 2 | 1 |
What's Changing
Overall Adjustments
Team | BE IC | FE IC | FE EM | BE EM | PM | UX | SDET |
---|---|---|---|---|---|---|---|
Monitor:APM | -3 | -4 | -.5 | -1 | -1 | -1 | -.5 |
Fulfillment | +2 | +2 | 0 | 0 | 0 | 0 | 0 |
Telemetry | +1 | 0 | 0 | 0 | 0 | 0 | 0 |
Verify | 0 | +2 | 0 | 0 | +1 | +1 | 0 |
Create:Knowledge & Create:Editor | 0 | 0 | 0 | +1 | 0 | 0 | 0 |
Monitor:Monitor* | 0 | 0 | +.5 | 0 | 0 | 0 | +.5 |
TOTAL | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
*The shared team members between Monitor:Health and Monitor:APM (SDET and FEM) will remain with the Monitor:Monitor Group.
DRI for Priority Teams
Team | DRI |
---|---|
Growth:Fulfillment FE | Chris Baus |
Growth:Fulfillment BE | Chris Baus |
Growth:Telemetry | Jerome Ng |
Verify:CI |
Timeline and Tasks
Monday - 2020-08-17
-
Update background, timeline and tasks and add transparency content - @kencjohnston -
Review issue and approve content
Tuesday - 2020-08-18
-
Inform Monitor EMs - @sgoldstein -
Inform @dhershkovitch - @kencjohnston @kbychu -
Inform Sarah - @kbychu -
Inform Monitor UX - @clenneville -
Inform Monitor SET - @meks
Wednesday - 2020-08-19
-
Confirm all directly effected ICs have been informed - @sgoldstein -
Move issue to GitLab-com/Product project at noon Pacific time - @sgoldstein -
Announce in Slack channels with link to issue @sfwgitlab @edjdev -
send invites for scheduled AMAs - @sgoldstein
Thursday - 2020-08-20
-
APAC and EMEA AMA Meetings including overview of realignment process and team pitch section for Fulfillment, Telemetry and Verify - @sgoldstein (@kencjohnston and others in attendance) -
Agenda: https://docs.google.com/document/d/1nWKTpDJO-nBN7EtoJr9_GKAxebQO1ycOJ8FsNb0M5YI/edit# - @sgoldstein -
Scheduled 8am Pacific (EMEA Friendly) and 4pm Pacific (APAC Friendly) - @sgoldstein -
EMEA meeting recording: https://youtu.be/x8iGac3POl4 -
APAC meeting recording: not publishing (meeting was lightly attended so refer to EMEA recording and agenda)
-
Friday - 2020-08-21
-
Collect team member preferences from BE and FE Engineers who have two positions to consider (Hiring EM makes final decision in cases where more engineers are interested than positions available) - @sgoldstein
Monday - 2020-08-24
-
Team preferences from BE and FE are finalized. Legal and People Ops review of all transfers.
Week of 2020-08-24
-
Team members wrap up work in current role
Week of 2020-08-31
-
Team members start in new roles with announcements of the change made using normal team change announcement communication.
Week of 2020-09-07
-
Make this issue public (transparency) @kencjohnston -
Create Retro issue to solicit feedback on this process @bmarnane www-gitlab-com#9056
Additional Details
New Monitor Product Group Details
Structure and Team
Existing Monitor:Health Group remains entirely the same
Scope and Priorities
High priority categories will remain those from Monitor:Health, primarily Alert & Incident Management. In the short term, there will be little additional improvements to current Monitor:APM categories(metrics, logging or tracing) or current APM assigned workflows(instrumentation and GitLab Self-Monitor). The triage workflow would remain a focus for the remaining team and as existing Monitor:Health categories mature we'll expect to build out required APM capabilities for completing the triage workflow.
Monitor Rationale
- We'll reduce our investment in Monitor by half as a result of low Product Performance indicators from our investment
- In order to minimize disruption, we'll opt to keep one group whole and expand their scope to include all of Monitor. Due to the Monitor:Health group showing better current performance and trends than Monitor:APM - we'll opt to keep Monitor:Health whole.
Note - Monitor:APM's GMAU of unique users is using sessions, so is an over-count for APM, while Monitor:Health's PI for count of projects is a significant undercount when it comes to users receiving value. We can conservatively estimate that each project creating incidents results in 100 sessions
worth of value. This makes Monitor:Health 4,400 sessions
(and trending up) and Monitor:APM 631 sessions
(and trending flat).
Full details of the rationale between Monitor:APM and Monitor:Health are available below.
Priority Product Group Requests and Rationale
Growth:Fulfillment Group
from @hilaqu We want to ask for 4 additional engineers allocated to Fulfillment/Business integration team: 3 backend engineers & 1 front end engineer. We can do with 2 BE and 2 FE
Click here to see rationale...
Currently, fulfillment teams have to support both business priorities & workflow improvements and they have to constantly tradeoff. These additional engineers will help accelerate progress on portal/infrastructure improvements: purchase/renew/provisioning features and user experience. This will allow fulfillment team to prioritize work that will reduce L&R tickets to improve support efficiency, and improve portal and process to increase sales efficiency.We should create a dedicated set of folks within Fulfillment focused on efficiency improvements for internal teams and ticket reduction by improving self-serve options in the portal and information in the GitLab instance. This would consist of 3 engineers (2 BE + 1 FE). This team would work with the business stakeholders to identify, prioritize, and create a roadmap of the issues we are going to work on and sync with them regularly for feedback and updates on priorities based on what we're seeing in the field.
1 additional BE engineer should go to the Business Integrations team to allow us to increase stabilization efforts that will also reduce ticket volume and headaches for internal teams and customers, in addition to improving overall speed of development and velocity on the Fulfillment and Growth teams. More context in this comment.
We may also want to consider an additional PM if we go this route because three teams with mostly independent streams and areas will be a lot for a single PM to handle.
Growth:Telemetry Group
from @hilaqu We want to ask for 2 BE for telemetry.
Click here to see rationale...
As Jerome mentioned, currently, telemetry has only 2 BE, we have been borrowing Alper from conversion for past 2 quarters. With 2 BE, even just to finish the most immediate roadmap of scaling usage ping and improving Snowplow collection framework, it will take 13 milestones - this will make it very challenging for product teams to implement their PIs and metrics.from @bmarnane (via Jerome)
The 3 main focus areas for Telemetry are
- Scaling and maintaining Usage Ping
- Improving tracking capabilities of collection framework
- Instrumenting tracking for Product, Sales, and Customer Success
The details are provided in https://docs.google.com/document/d/1qYHNpgFIMb6Cl1N5SVF7imtF8twrUrsfYgrMHl9YS6U along with an estimated roadmap totalling 24 milestones of unplanned work for a single engineer. Additional headcount, if available, will accelerate our efforts. I'll defer to @hilaqu for where the specific items fit in order or priority.
Verify:CI Group
Expand to see the rationale (of the original request)...
The Verify:CI group recently split the product scope between two PMs (one Director of Product is filling the role of IC PM). The current team includes 10 Developers (7 Backend Developers and 3 Frontend Developers) plus a FEM and a BEM. We should ensure we provide the supporting personnel to complete that split. Namely:
- Add one BE developer to Verify:CI
- Add one BEM to Verify:CI
- Create Two Verify Groups
- Verify:CI:Authoring with 5 developers (4 backend + 1 frontend)
- Verify:CI:Interactions with 6 developers (4 backend + 2 frontend)
- Add one Product Manager to Verify:CI:Authoring
- Add one Product Designer to Verify:CI:Authoring
Monitor:Monitor Group
Expand to see full details of rationale...
It's important that we have a heavy focus on Product Performance. In this case the Monitor stage has low SMAU relative (within the Ops Section) to other stages, especially given the investment of two full product groups in the Monitor stage scope.
As a result we need to consider Monitor stage investment for re-deployment to other scopes.
Next we need to determine which part of the Monitor stage's scope (by group) is must successful in order to be a minimally disruptive as possible. If we fully combined scope and priority immediatley and created a mixed team we'd have significant output slow-down versus keeping one team whole and their short-term focus stable and adding the other groups scope as a medium term focus.
Again we place a heavy emphasis on Usage Drivers results here.
Usage Driver
As noted above - Performance indicators indicate higher usage and better trends for Monitor:Health over Monitor:APM.
Secondary Concern - Dogfooding
While it is useful to take a pure external user view, the Monitor stage is a Horizon 3 investment and in the "Not used at GitLab Inc" stage lifecycle. As a result we should as a secondary evaluating criteria consider dogfooding potential another input to this decision. I asked Steve Loyd (VP of Infrastructure) the question in Slack.
Usage Driver Result
Group | Usage Score |
---|---|
APM | |
Health |
Other components
When considering our investment allocation we take a more robust evaluation appraoch that includes Usage Drivers, ASP Driver and SAM approach. While that is useful, given these are existing efforts we should primarily focus on our Usage Drivers for this decsision. I consider the ASP and SAM drivers as tertiary at best.
- ASP (Average Selling Price) Driver
- SAM (Serviceable Addressable Market) Driver
ASP Driver
This evaluation is pretty straightforward. Current tiering of features with Incident Management, Alert Management and Status Pages in paid tiers for Health, and only Tracing (which is moving to core) in paid tiers for APM. The tier strategy for the Monitor stage also highlights future tier value being added more rapidly in Health group categories.
ASP Driver Result
Group | Usage Score |
---|---|
APM | |
Health |
SAM Driver
This evaluation is more difficult. The market CAP and TAM for companies (like DataDog - $26.95B) we'd compete with in Monitor:APM categories is an order of magnitude larger than the TAM for companies (like PagerDuty - $2.49B) we'd compete with in Monitor:Health. However, I'd rate our serviceability of that market (Steve's dogfooding response above gives some weight to that serviceability) as significantly higher and increasing in a shorter time frame.
I think the evaluation is too subjective to justify an obvious result here, so I'm going to mark it as a pass.
SAM Driver Result
Group | Usage Score |
---|---|
APM | |
Health |
Result
Using our product investment allocation framework yields the following results.
Group | Usage Score | Revenue Driver | SAM Driver |
---|---|---|---|
APM | |||
Health |
This points us to Monitor:Health being the team to keep in tact and to focus on Monitor:Health categories in the short term. The evaluation does point to the closeness in long-term return from both investments and so ensuring the team continues to build our APM category capabilities in the medium term will be critical. This will result in:
- Opting for a multi-year vision where we can enclose traditional APM and Observability providers by being the place developers go to instrument those providers, and operators go to collaborate on response to their alerts before investing to capture that market directly.
- Play to our strengths by taking advantage of opportunities to add value by connecting to other stages of GitLab first.
FAQ
Please ask questions in the comments below and ping @sgoldstein or @kencjohnston. As we answer common ones we'll add them here.