Note: The API Vision is in draft and very much a _work-in-progress_. Priorities defined below are subject to change. ## Vision By the end of 2023, we will guide and enable GitLab to adopt and reinforce an `API First` and `GraphQL First` approach to development. Through a combination of best practices, policies, API standards, and tooling, we'll enable developers to build features in a way that increases velocity, transparency, accessibility/usability of data by developers across the many personas of API users that not only use, but _depend_, on our APIs - our internal GitLab users, our customers, Technology Partners, and 3rd party maintainers. ## Why is this important? At least 30-40% of our API Usage is generated - not by GitLab - but by our API Users, it's critical that we understand that features delivered without APIs do not exist for these users and this represents a large subset of undervalued Product usage across the GitLab Product. Customers leverage our APIs to automate workflows and integrate into internal (as well as 3rd party tools where GitLab hasn't yet provided a solution). Partners are heavy API users as they are focused on using our APIs/webhooks to establish integrations and capture more usage for their tools, as well as resolve workflow challenges for mutual customers. Third party maintainers of client libraries and SDKs take on a role of a power user as they provide an intermediary surface area for end users of our APIs (such as customers/partners). If GitLab's features are not supported by an API, they can't resolve issues for their "customers". They are stuck in the middle and unable to resolve pain points for their users, and become a pass through to GitLab. Lastly, internal GitLab users _also_ depend on GitLab's APIs. Our frontend team heavily depends on GQL, the gitlab~21094707 team leverages APIs/webhooks to establish native integrations, and cross-product collaboration enables optimization or usability improvements based on our APIs. Thus, even internally we are suppressing innovation when our features are not exposed via API. To this point, our APIs have been largely developed in a decentralized way, dependent on collaboration across teams. While the collaboration is excellent, our customers do sense a lack of unity in our APIs. By developing a centralized set of standards and processes, and by focusing more investment in our API infrastructure, maintenance, and support, we can take full advantage of the _leverage_ that our APIs can create for GitLab. ## Prioritization Based on what we've learned through our API Working Group, via our API User Survey, Customer Interviews, Industry Research, and internal dialog, we've identified a hierarchy of needs for the API User, or a `DX Hierarchy`: <img src="/uploads/4a57a1c779d0ca908fb990d6cd15898a/image.png" height="300"> This chart, while not comprehensive, we hope will provide some guidance in the way we organize our priorities. Internally, we'll also need to balance the customer perspective with architectural, technological, and business constraints. One of the key areas we must make a decision is around our delivery of features through one or more API Styles. When it comes to REST vs GraphQL, there are tradeoffs to where we focus our efforts, where we add value, and where we open ourselves to risk or impact velocity. While we learned that most customers expect and are familiar with REST, we also found that most users of our APIs don't have a _strong_ preference. More important is that GitLab is consistent in providing _an_ API for each of our features. That said, a GraphQL first requirement or reinforcement is not taken lightly. Some customers/users would be required to change their approach to support APIs where they've used our REST APIs in the past, while others are reticent to use GraphQL for lack for perceived value (if not just familiarity). Ultimately, this means for GitLab, REST cannot be entirely deprioritized. While GQL will be our first step towards delivering a feature, we'll work to provide solutions for supporting REST endpoints based on our GQL resources (`R>GQL` until I see a better proposal) and define process for providing consistency between our APIs. #### 1. API-First & GraphQL First - ~"api:first" (or possibly ~"api:coverage" 1. Experiment with approaches to create REST endpoints based on GraphQL resources, so we may solve gaps in parity and prioritize GQL without impacting velocity. (https://gitlab.com/gitlab-org/gitlab/-/issues/363795) 1. Implement a process to enable GitLab developers to define GQL endpoints as a requirement for all new features. 1. Define process, standards, and best practice for contributing new endpoints that helps us self-document gaps and expose them transparently. 1. Improve documentation/support for internal users building new APIs. 1. Establish an MR Reviewer process and checklist for reviewers to govern the process. On `3`, we'll establish a new YAML/configuration file (or any better solution) to allow us to automatically register new APIs (be it in GraphQL or REST), and document the owners, map the endpoint/resources to their respective Product Group (DRI), Product Category, and Product Feature. This will allow us to see transparently what our coverage is across GitLab and better analyze/prioritize where to invest first. #### 2. API Consistency - ~"api:consistency" 1. Identify all gaps (starting with the [Gap Analysis](https://gitlab.com/gitlab-org/gitlab/-/issues/363615#note_980209922)) 1. Document gaps by creating issues and labeling 2. Prioritize gaps with GQL first 3. Address REST Gaps at the Stage level, based on usage, performance, security, and customer priority 4. Utilize R>GQL #### 3. Automated Documentation & Documentation Improvements ~"api:docs" By automating our documentation, we can address a common pain point expressed by many users - that our actual state of our APIs can drift from our documentation, leaving our users confused and sometimes frustrated. Automated documentation mutually improves the quality of life for our internal developers by requiring fewer manual steps and reducing the margin of error. 1. Update our OpenAPI Spec to match the current state of our APIs 2. Develop tooling that will build an OpenAPI Spec automatically as new REST endpoints are added 3. Add more behavioral context (for both REST & GraphQL) - expose possible error codes, expected response bodies/parameters, expected ranges, etc #### 4. Code Samples & Support Systems ~"api:samples" As our APIs become more reliable and consistent - and, as users come to expect new features to start with GQL APIs - we'll want to provide more robust and interactive components to educate users of the GQL API. We'll want to better show and explain the value of our GQL API, highlight use cases and opportunities for users to leverage GQL in impactful ways, and for us to demonstrate using our APIs with samples. These efforts will help us improve usability of our APIs generally (benefiting users of 3rd party libraries across all languages, CLIs, regardless of the persona, internal or external). By focusing first on general usability improvements (vs investing entirely on a particular client library for example), we'll raise the quality of life across the board. > The most helpful enhancement API producers can make is to provide better examples in the documentation (61%), followed by sample code (56%), and standardization (54%). (Postman’s 2021 State of API - https://www.postman.com/state-of-api/) 1. Code Samples / Tutorials for most common use cases 2. Updated GQL Video Series / Workshop 3. Make it simple for API users to report gaps and automatically generate labeled issues for us to track/address 4. Developer Portal Enhancements ## Low Hanging Fruit ~"api:fruit" Our priority list details the order of priority based on the customer feedback and insights. They should drive the order of prioritization largely, but as we identify any small boulders that could have a high impact, we may increase the priority for those efforts. Low Hanging Fruit should be able to be achieved in 1-2 milestones. ## Out of Scope (for now) For the sake of a succinct and targeted Vision, we've identified a few efforts that for now will remain out of scope. This does not mean they cannot be reintroduced in the future, but we'll maintain focus on the top priority issues before we consider reintroducing them. 1. Officially Supporting Third-party Libraries or SDKs - While there are opportunities to reach a large audience that depends on a single library and add value for those users, the foundation of our API infrastructure affects all API users and should be prioritized. We'll track this effort and explore it separately, so we may keep a tight focus in our Vision. 2. Rails controller endpoints - For our API Vision, we'll be focusing primarily on the experience and value that extends to both internal/external users versus cases that are exclusively adding value internally. 3. APIv5 - There appear to be many opportunities to explore that will solve for customers now, without new architecture. For any new API version, we should define a process to concretely identify and validate the cases that would necessitate a large scale breaking change. ## Data / Insights - [API Usage Survey Findings (internal) ](https://docs.google.com/presentation/d/1jyVsk5JrCFo_4C7mxgLQ--N74JqGgVonx6zs4XMhbg8/edit?usp=sharing) - [User Agent Analysis (internal)](https://log.gprd.gitlab.net/app/dashboards#/view/ee792100-cfc7-11ec-afaf-2bca15dfbf33?_g=h@b255a67) - [Go-GitLab Usage (internal)](https://log.gprd.gitlab.net/app/discover#/?_g=(filters:!(),refreshInterval:(pause:!t,value:0),time:(from:now-15m,to:now))&_a=h@de208e7) - [500 Errors by Feature (internal)](https://log.gprd.gitlab.net/app/dashboards#/view/197f49e0-7ae3-11ec-9dd2-93d354bef8e7?_g=h@c823129) - [Rack Attack Rate Limiting (internal)](https://log.gprd.gitlab.net/app/dashboards#/view/e1442eb0-837f-11ec-a649-b7cbb8e4f62e?_g=(filters%3A!()%2CrefreshInterval%3A(pause%3A!t%2Cvalue%3A0)%2Ctime%3A(from%3Anow-24h%2Fh%2Cto%3Anow))) - [We've been communicating](https://gitlab.com/gitlab-com/www-gitlab-com/-/blob/be2393b449cec60e6df6215bd7ac9793120069b1/source/handbook/engineering/index.html.md#graphql-first) that GitLab is `GraphQL-first`. Our approach in this proposal would mean we're consistent in this communication, but exploring ways to ensure GitLab is consistent in practice and that we're enabling developers to comply in a way that best solves for customers without affecting velocity. ## Further Exploration / TODOs - Capture data around User Agents to better understand (objectively) which tools are being used to access our APIs (e.g. 3rd party libraries, API Clients, or otherwise) - Q: What % of namespaces/accounts utilize our APIs? - Q: What % of our API usage is internal vs external? - How many issues or MRs are completed each milestone? Explore based on current `api` labels. - Are there opportunities for GitLab to solve these challenges in differentiated ways only we can? - Compare GitLab vs other APIs for insights - https://gitlab.com/groups/gitlab-org/-/epics/5842 - consolidate/align GQL Blueprint and API Vision