Investigate and develop tooling for OpenAPI v3 API documentation

Everyone can contribute. Help move this issue forward while earning points, leveling up and collecting rewards.

• Close this issue

Background

Currently, GitLab maintains API documentation in multiple formats:

• Markdown files that are manually updated (https://docs.gitlab.com/api/api_resources/)
• OpenAPI v2 specification (automatically generated via rake task) - (https://gitlab.com/gitlab-org/gitlab/-/blob/master/doc/api/openapi/openapi_v2.yaml)
• A manually created and incomplete OpenAPI v3 file (https://gitlab.com/gitlab-org/gitlab/-/blob/master/doc/api/openapi/openapi.yaml)

Related epic: Document the REST API in OpenAPI V2 (&8926).

This creates several challenges:

• Manual updates to Markdown docs are error-prone and time-consuming
• OpenAPI v2 has limitations around documenting feature flags and deprecation notices
• We lack a single source of truth for API documentation

Proposal

We should investigate building tooling to generate OpenAPI v3 documentation that would eventually allow us to:

1. Move away from manually maintained Markdown API docs
2. Properly document feature flags and deprecation notices
3. Have a single, automatically generated source of truth for API documentation

The above is in line with Why use OpenAPI and Decisions, point 2 from latest CTO Summit.

Opportunity for Early Action

While a dedicated API team is being established, there's significant value in starting this work now:

1. Technical Foundation Benefits:

• By the time the API team forms, they would have a modern documentation infrastructure ready
• The team could focus on API strategy and development rather than documentation tooling
• Reduced technical debt from the start of the API team's formation

2. Time-to-Market Benefits:

• Get ahead of documentation challenges before they compound with new API development
• Enable faster API development cycles with automated documentation
• Reduce friction in API development process before the new team starts their work

3. Knowledge Benefits:

• Build institutional knowledge about our API documentation needs
• Better inform API team staffing needs based on learned complexities
• Develop clear patterns and practices that can guide the API team's work

Benefits

• Reduced maintenance burden by eliminating manual documentation updates
• Improved documentation accuracy through automated generation
• Better developer and user experience with machine-readable API specs
• Ability to properly document feature flags and deprecation notices
• Potential for improved API client generation and testing tools
• Alignment with modern API documentation standards
• Foundational work completed before API team forms, accelerating their effectiveness

Detailed benefits copied from CTO Summit doc

Providing a strongly-typed, well-structured and machine readable specification for our REST API would provide many benefits.

• It encourages an API-first approach to building GitLab.
• API clients can be generated from OpenAPI specifications, reducing the workload on maintaining clients for many different programming languages (Go, Python, Javascript, Ruby, etc). Many languages provide an OpenAPI client generator, reducing the need to hand-code these client libraries, which often lag behind the API interface.
• Fuzz-testing of the API is possible using tools designed specifically to ingest API endpoints for this purpose.
• Some AI Agents can parse OpenAPI specs and use the API endpoints described by the spec as part of AI workflows.
• OpenAPI allows documentation to be generated, instead of being maintained manually by hand. This leads not only to much more consistent, versioned documentation, but also allows for greater interactivity in the documentation: for example, automatically generating client code for various user-selected languages, or providing a browser-based interface for interacting with the endpoints.
• OpenAPI has the concept of overlays, which could assist in progressive translation of API documentation.
• OpenAPI allows for extensions to allow additional information to be annotated onto the end-point. This could include lifecycle information (when an API is due to be deprecated, etc). This ensures consistent documentation, and also allows downstream benefits, such as automated warnings when deprecated APIs are being used.
• Extensions could also be used to express GitLab-specific attributes such as Tier (“Free”, “Premium” etc) and Offering (“GitLab.com”, “GitLab Self-Managed”, “GitLab Dedicated”)
• OpenAPI provides a much stronger, typed interface than the current markdown/text based description. This leads to less ambiguity, and would also lead to better automated validation on our API guarantees (CI jobs could validate API changes according to rules, etc).
• We can use the Open API specs to validate consistency between GraphQL and REST API fields.
• OpenAPI specs can be used to generate quality & performance tests.
• Deprecations can be annotated on endpoints rather than managed through unstructured markdown in documentation. This can be automatically added to OpenAPI specs, documentation, and used to build automated monitoring of the usage of deprecated endpoints. This can be used to notify users they’re using APIs due to be removed from the product.

Implementation Steps

Phase 1: Investigation & Spike (2-3 weeks)

• Evaluate current grape-swagger gem limitations for v3 support
• Research potential approaches:
  • Contributing to grape-swagger to add v3 support
  • Building a new tool specifically for GitLab's needs
• Document technical requirements for v3 support including:
  • Feature flag documentation
  • Deprecation notices
  • Other GitLab-specific metadata
• Create proof of concept for chosen approach

Phase 2: Development (timing TBD based on spike)

• Develop chosen solution (either grape-swagger extension or new tool)
• Add support for GitLab-specific metadata (feature flags, deprecation notices)
• Create CI job to automatically generate v3 documentation
• Implement validation to ensure documentation completeness

Phase 3: Migration Planning

• Work with Technical Writing team to:
  • Map current Markdown documentation to OpenAPI v3 structure
  • Identify any gaps in automated generation
  • Plan phased migration approach
• Create documentation for maintaining API docs in new system
• Develop plan for gradual migration from Markdown to OpenAPI

Next Steps

1. Engineering, perhaps Engineering Productivity team but could also be a dedicated contributor from any team, to evaluate feasibility and resource requirements for spike
2. Technical Writing team to assess collaboration requirements for migration
3. Define timeline and resource allocation based on priorities

Questions

• What is the preferred approach - extending grape-swagger or building new tool?
• How do we want to handle the transition period with multiple documentation sources?
• What validation would we need to ensure documentation quality?

Strategic Advantages of Early Implementation

1. Risk Reduction:

   • Address known documentation challenges before they impact the new API team
   • Validate approaches before larger-scale API initiatives begin
   • Identify potential roadblocks early in the process

2. Resource Efficiency:

   • Parallel development while API team is being formed
   • Leverage existing knowledge in Engineering Productivity team
   • Avoid competition for resources once API team begins their primary work

3. Future-Proofing:

   • Establish modern documentation practices from the start
   • Create scalable solutions that support future API growth
   • Build with future API team needs in mind

@timzallmann @jeromezng @vincywilson @rymai @susantacker @sselhorn would you be interested in exploring this initiative to improve our API documentation infrastructure? Starting this work now would provide significant advantages for the upcoming API team and our overall API strategy.

/cc @Andyschoenen @aqualls @.luke