Migrate Pipelines Page from REST to GraphQL for Performance Optimization
Everyone can contribute. Help move this issue forward while earning points, leveling up and collecting rewards.
Problem Statement
The pipelines page (/pipelines) suffers from significant performance issues, with server response times reaching 10+ seconds in worst cases. The current REST endpoint (pipelines.json) returns excessive data that isn't used by the UI, causing:
- Over-fetching of pipeline data and relationships
- Expensive preloading operations in
Ci::Pipeline::Preloader - Loading all downstream pipelines when only 3 are displayed
- Redundant serialization of manual actions that aren't immediately needed
Proposed Solution
Migrate the pipelines page from REST API to GraphQL to enable precise data fetching and improve performance.
Requirements
Phase 1: GraphQL Query Development
- Create GraphQL query that fetches only data displayed in the pipelines UI
- Implement pagination support matching current REST behavior
- Add filtering capabilities (branch, status, etc.)
- Ensure query supports both project and group-level pipeline views
Phase 2: Frontend Migration
- Replace REST API calls with GraphQL queries in Vue components
- Implement lazy loading for manual actions (show button, fetch on click)
- Limit downstream pipeline loading to maximum 3 items
- Maintain existing UI/UX behavior and functionality
Phase 3: Performance Optimization
- Add necessary preloads to GraphQL resolvers to avoid N+1 queries
- Optimize database queries for the new GraphQL schema
- Implement caching strategies where appropriate
Phase 4: Rollout & Cleanup
- Feature flag the GraphQL implementation
- Gradual rollout with performance monitoring
- Deprecate unused REST endpoint preloads (breaking change for 18.0)
- Remove legacy REST endpoint usage
Success Criteria
Performance Targets
- Reduce p95 response time by at least 50%
- Eliminate 10+ second response times in p99 cases
- Reduce database query count for pipeline loading
Functional Requirements
- All existing pipeline page functionality preserved
- No regression in user experience
- Backward compatibility maintained during transition
Technical Considerations
GraphQL Schema Requirements
query PipelinesList($projectPath: ID!, $first: Int, $after: String) { project(fullPath: $projectPath) { pipelines(first: $first, after: $after) { nodes { id iid status ref sha createdAt updatedAt user { id name avatarUrl } # Only essential fields for list view hasManualActions # Boolean instead of full manual actions downstreamPipelines(first: 3) { # Limit to displayed count nodes { id status project { name fullPath } } } } pageInfo { hasNextPage endCursor } } } }
Implementation Notes
- Follow patterns established in successful Jobs page GraphQL migration
- Consider removing non-essential UI elements identified in spike (commit author avatars, latest ref indicators)
- Implement progressive enhancement for manual actions loading
Dependencies
- GraphQL infrastructure (already available)
- Feature flag system for gradual rollout
- Performance monitoring tools
Risks & Mitigations
-
Risk: GraphQL queries may initially be slower due to missing optimizations
- Mitigation: Implement necessary preloads and database optimizations before rollout
-
Risk: Breaking changes for API consumers
- Mitigation: Maintain REST endpoint until 18.0, provide migration guide
Definition of Done
- GraphQL implementation performs better than REST equivalent
- All pipeline page functionality works with GraphQL
- Performance metrics show measurable improvement
- Feature flag allows safe rollout
- Documentation updated for API changes