FY21-Q2 Infra KR: Work with CI/CD - Verify team to implement service-level spend metrics => 100%, work planned, implementation to be done in Q3
End Goals
-
Scheduled work to label runners for better billing insight
- Runner Managers to be labeled in all CI projects (and each runner with a runner_manager label) so we can validate at least the cost by runner manager in GCP billing console.
-
Submit Data team request for needed data sources
What's expected of the team:
The team working with the Infrastructure Analyst needs to be an active participant in the process for things to be successful. The team is expected to know or learn how to use Periscope to own their kpi and dashboard after it has been defined and the MVC query has been built.
A good example of this type of partnership is what we have had with the quality team, so if you have questions about this, I'd recommend asking @meks or @kwiebers about their work with building out avg cost/pipeline for gitlab-org/gitlab metric.
Overview of Process
Stage 1 - Understanding of Cost Drivers
Infrastructure analyst will set up meeting with relevant parties to go over what infrastructure the team is currently running and will work together to discover what are the main cost drivers for this team
Status
-
Cost Driver Identified
Stage 2 - Defining the KPI
Team will work with infrastructure analyst to define what the KPI will be that will best capture the main cost drivers that the team's actions can affect.
Status
-
KPI initial definition discussed -
KPI first iteration definition defined -
KPI in handbook
Stage 3 - Gathering the data
If needed, request will be submitted to data team to bring in the relevant data into the data warehouse
Status
-
Identify data needed -
Submit data issue to get data into DW if needed -
Create base dbt model for relevant data
First step is getting the service level tracking in place for registry which is being done by creating a new postgres DB to contain the registry data
Next step is to submit a data requests to bring in this data into Data Warehouse (Snowflake) so it can be combined with the rest of the .com data
Stage 4 - Building the business logic and MVC query
Infra Analyst will work with the team to build out the minimum viable query for the KPI definition and explain the business logic. From here, additional support may be provided, but the team is expected to be the main owner of the SQL and business logic after this point.
Status
-
DBT logic built for kpi -
MVC query for KPI built
Stage 5 - Continuous Monitoring
In the future, there could be feature or architecture changes that would necessitate changes to this kpi. The team will be responsible for going through the normal kpi change process to make these changes, ideally these definition changes should be reviewed by the infrastructure analyst and any questions about the business logic and cost impact can be asked to the infra analyst during this process.