"Create new issue" API shows a low throughput

Summary

With our increased performance testing efforts we've started to identify slow areas in GitLab and raising them as issues. One such area is the Create a new project issue API where we've found its time to respond significantly slows down with increasing RPS. It's especially crucial for the current 50k architecture reference: https://gitlab.com/gitlab-org/quality/performance/issues/102

Results against 10k environment:

Environment:    10k (12.3.5-ee 9dbaa740018)  
Scenario:       60s_200rps  
Date:           2019-10-11  
Run Time:       1692.72s (Start: 01:42:19 UTC, End: 02:10:32 UTC)  
  
NAME                                                     | DURATION | P95        | RPS                  | SUCCESS RATE   | RESULT  
---------------------------------------------------------|----------|------------|----------------------|----------------|-------  
api_v4_new_issues                                        | 60.0s    | 6627.33ms  | 52.13/s (>16.00/s)   | 100.00% (>95%) | Passed

Grafana dashboard results: http://34.73.165.75/-/grafana/d/tVdO8YIWz/server-performance?orgId=1&from=1570816860000&to=1570817760000 (To be able to see it, please sign up)

Results against 50k environment:

Environment:    50k (12.3.5-ee 9dbaa740018)
Scenario:       60s_1000rps
Date:           2019-10-10
Run Time:       2303.62s (Start: 13:40:27 UTC, End: 14:18:50 UTC)
NAME                                                     | DURATION | P95        | RPS                  | SUCCESS RATE   | RESULT
---------------------------------------------------------|----------|------------|----------------------|----------------|-------
api_v4_new_issues                                        | 60.0s    | 25602.59ms | 45.58/s (>80.00/s)   | 100.00% (>95%) | Failed

Grafana: http://35.196.33.84/-/grafana/d/tVdO8YIWz/server-performance?orgId=1&refresh=10s

Steps to reproduce

  1. Check out the Performance Toolkit
  2. Run the specific test with the run-k6 command. For example against the 10k environment you would run this following from the project root: ACCESS_TOKEN="" ./run-k6 -e environments/10k.json -s scenarios/60s_200rps.json -t tests/standalone/api_v4_new_issues.js, where ACCESS_TOKEN is a valid GitLab Personal Access Token for the specified environment(10k in this case). The token should come from a User that has admin access for the project(s) to be tested and have API and read_repository permissions.
  3. If you're seeking to run the test against your own environment the Toolkit's documentation has details on how to achieve this.

What is the current bug behavior?

The API throughput is low under RPS > 40 and duration > 60s.

What is the expected correct behavior?

That the API response time is improved.

Edited by Nailia Iskhakova