Draft: Generate REST API from GraphQL

What does this MR do and why?

• generates statically REST API code from GraphQL fields/types (this is generated by running rake gitlab:graphql:generate_rest), in this POC generated file is in lib/api/wrapper.rb
• this rake task parses Gitlab's GraphQL schema and for each field having a rest attribute
  • it defines a REST endpoint including optional and required parameters taken from the field's type definition (example: !116112 (diffs))
  • it defines set of object's attributes/fields which are returned by default in REST API response (only "inexpensive" fields are returned), expensive fields can be fetched too by using "include_" parameter (e.g. include_author). Parimary decision maker for whether a field is included is its complexity.
• generated API endpoints are marked as "alpha", the reason is different deprecation policy in GraphQL vs REST (#363795 (comment 1324190547))
• opt-in approach is used for deciding which REST endpoints to generate - this means this approach can be easily combined with classic way of manual REST endpoints definition -> we could use it for defining selective (new) endpoints in REST v4 (w/o making any breaking changes in current endpoints)
• pagination - wrapper passes "pagination: :offset" context setting, then resolvers/fields can check for this context value if offset pagination should be used instead. A limitation is that this is currently handled per-resolvers/per-field, but we could investigate if we could introduce this on "lower" connection layer instead. Wrapper also does replacement of offset params (page, per_page) with cursor-base params.
• the idea is this rake task becomes part of our CI so we can easily check if a) GraphQL endpoint is updated (e.g. new field is added) but REST endpoint is not update, and b) we can easily review how exactly the resulting API looks like - generated code is part of the MR which updates GraphQL API. So this would work similarly to our GraphQL's schema documentation rake task.

In this POC endpoints are generated for following new work items endpoints:

• get all work items in a project: /api/v4/projects/6/work_items/1?include_author=true&include_widgets=true&first=2
• get single work item: /api/v4/projects/6/work_items/1?include_author=true&include_widgets=true
• update a work item: PUT /work_items/1?title=updated1

I used work item endpoints because this is related to #368055

Related to #363795

Performance measurements

Measured on a local GDK deployment, using customized Gtilab's GPT tool - gitlab-org/quality/performance!500 (closed) - although this is not relevant for "absolute" numbers, it's still sufficient for "relative" measurements when comparing API performance with each other. issues endpoints was used for general performance comparison - this is well known endpoint which we have represented both in REST and GraphQL.

General thoughts

Comapring performance of "getting 100 issues in REST vs GraphQL" is problematic because it compares endpoints doing different things (e.g. because both provide different set of fields which massively impacts performance). Related to #369097 (comment 1302313192).

Naming

api_graphql_projects_issues - requests against "native" GraphQL API
api_v4_projects_issues - requests against "native" REST API
api_v4_projects_issues_gql - requests against REST wrapper

Results

Note: most interesting columns are TTFB (10 requests per second was used during testing so RPS result is limited by this).

• Getting list of issues with default set of fields

NAME                        | RPS  | RPS RESULT       | TTFB AVG   | TTFB P90            | REQ STATUS     | RESULT
----------------------------|------|------------------|------------|---------------------|----------------|-----------------
api_graphql_projects_issues | 10/s | 0.46/s (>8.00/s) | 16418.79ms | 19956.92ms (<500ms) | 100.00% (>99%) | FAILED²
api_v4_projects_issues      | 10/s | 5.23/s (>8.00/s) | 1479.39ms  | 2163.79ms (<500ms)  | 100.00% (>99%) | FAILED²
api_v4_projects_issues_gql  | 10/s | 0.78/s (>8.00/s) | 10189.42ms | 12507.07ms (<500ms) | 100.00% (>99%) | FAILED²

Here we can see massive difference between getting list of issues through native REST vs Graphql. Surprising is better performance of wrapped API vs native graphql but I was getting different results here so might be measurement deviation.

• Getting list of issues with "normalized" set of fields - same subset of fields present both in REST and GraphQL and excluded fields which are not optimized/slow in GraphQL (e.g. because of missing preload): "closedAt", "confidential", "createdAt", "description", "dueDate", "healthStatus", "id", "iid", "moved", "projectId", "severity", "state", "title", "type", "updatedAt"

NAME                        | RPS  | RPS RESULT       | TTFB AVG  | TTFB P90           | REQ STATUS     | RESULT
----------------------------|------|------------------|-----------|--------------------|----------------|-----------------
api_graphql_projects_issues | 10/s | 5.06/s (>8.00/s) | 1512.94ms | 2307.40ms (<500ms) | 100.00% (>99%) | FAILED²
api_v4_projects_issues      | 10/s | 9.84/s (>8.00/s) | 118.57ms  | 130.82ms (<500ms)  | 100.00% (>99%) | Passed
api_v4_projects_issues_gql  | 10/s | 5.05/s (>8.00/s) | 1550.48ms | 2148.86ms (<500ms) | 100.00% (>99%) | FAILED²

Here we can still see massive difference between REST vs both GraphQLs, but GraphQL is noticably faster than with default set of fields (IOW there is for sure room to improve default set of fields being returned). Also we can see similar timings for native GraphQL vs REST wrapper. IOW there is not too much overhead added with the wrapper.

• Getting only iid and id fields:

NAME                        | RPS  | RPS RESULT       | TTFB AVG | TTFB P90           | REQ STATUS     | RESULT
----------------------------|------|------------------|----------|--------------------|----------------|-----------------
api_graphql_projects_issues | 10/s | 8.42/s (>8.00/s) | 858.28ms | 1232.80ms (<500ms) | 100.00% (>99%) | FAILED²
api_v4_projects_issues      | 10/s | 9.95/s (>8.00/s) | 93.64ms  | 96.26ms (<500ms)   | 100.00% (>99%) | Passed
api_v4_projects_issues_gql  | 10/s | 7.84/s (>8.00/s) | 945.60ms | 1299.37ms (<500ms) | 100.00% (>99%) | FAILED²

REST is still ~10 times faster than GraphQL

• Getting id and iid fields with authorization disabled in issue_type.rb and with commented out #before_connection_authorization

NAME                        | RPS  | RPS RESULT       | TTFB AVG | TTFB P90          | REQ STATUS     | RESULT
----------------------------|------|------------------|----------|-------------------|----------------|----------------
api_graphql_projects_issues | 10/s | 9.52/s (>8.00/s) | 131.97ms | 135.23ms (<500ms) | 100.00% (>99%) | Passed
api_v4_projects_issues      | 10/s | 9.95/s (>8.00/s) | 99.97ms  | 124.59ms (<500ms) | 100.00% (>99%) | Passed
api_v4_projects_issues_gql  | 10/s | 9.87/s (>8.00/s) | 132.95ms | 140.57ms (<500ms) | 100.00% (>99%) | Passed

Now we are getting much more similar numbers both in REST and GraphQL.

• Getting single issue with same subset of fields: "closedAt", "confidential", "createdAt", "description", "dueDate", "healthStatus", "id", "iid", "moved", "projectId", "severity", "state", "title", "type", "updatedAt"

NAME                              | RPS  | RPS RESULT        | TTFB AVG | TTFB P90          | REQ STATUS     | RESULT
----------------------------------|------|-------------------|----------|-------------------|----------------|----------------
api_graphql_projects_issues_issue | 10/s | 10.02/s (>8.00/s) | 43.99ms  | 59.01ms (<500ms)  | 100.00% (>99%) | Passed
api_v4_projects_issues_issue      | 10/s | 10.01/s (>8.00/s) | 56.29ms  | 66.74ms (<1500ms) | 100.00% (>99%) | Passed
api_v4_projects_issues_issue_gql  | 10/s | 10.03/s (>8.00/s) | 56.00ms  | 71.03ms (<1500ms) | 100.00% (>99%) | Passed

Here we can see similar timings for all three APIs, GraphQL doesn't suffer from per-resource permission check here (as permission check is done for getting single issue both in GraphQL and REST).

To sum up these measurements:

• major contributors to slower GraphQL requests are:
  • per-resource permission checking - this is important when getting multiple resources, performance difference might be different for other resources (I assume for issues this is expensive)
  • exposing some fields which are not optimized in GraphQL (yet) - I think this can be mitigated in our GraphQL (mainly by fixing N+1 issues)
• I don't think our GraphQL API itself is slower than REST API - as shown above if I would remove extra authorization and fields, then timings are similar in all APIs
• it's important to be very careful about default set of fields being exposed in the wrapped API
• there is not much performance degradation caused by REST wrapper compared to "native" GraphQL

Screenshots or screen recordings

Screenshots are required for UI changes, and strongly recommended for all other merge requests.

How to set up and validate locally

Numbered steps to set up and validate the change are strongly suggested.

MR acceptance checklist

This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.

• I have evaluated the MR acceptance checklist for this MR.