Skip to content

API caching for the Releases endpoint [RUN ALL RSPEC] [RUN AS-IF-FOSS]

Catalin Irimie requested to merge cat-api-releases-caching into master

NOTE: This is an urgent MR in the context.

What does this MR do?

Caches the output JSON for performance, similar to !54975 (merged).

Cache invalidation happens on project & user & permissions, important as the user can have hierarchical access through shared groups, and some of the data is hidden depending on this.

Feature flag:

api_caching_releases

GPT 10k tests on the endpoint, and api_json logs:

Uncached - feature flag turned off:

NAME                     | RPS   | RPS RESULT          | TTFB AVG  | TTFB P90            | REQ STATUS     | RESULT
-------------------------|-------|---------------------|-----------|---------------------|----------------|----------------
api_v4_projects_releases | 200/s | 127.87/s (>64.00/s) | 1408.92ms | 2378.86ms (<4000ms) | 100.00% (>99%) | Passed
root@cat-gpt-10k-gitlab-rails-1:~# zcat /var/log/gitlab/gitlab-rails/api_json.log.1.gz | grep '^{' | jq -r 'select(."route" == "/api/:version/projects/:id/releases") | [.db_count, .db_duration_s] | @tsv' | head -n 5
83      0.11487
83      0.113
83      0.11209
81      0.11357
83      0.10504

Cached - feature flag turned on:

NAME                     | RPS   | RPS RESULT         | TTFB AVG | TTFB P90          | REQ STATUS     | RESULT
-------------------------|-------|--------------------|----------|-------------------|----------------|----------------
api_v4_projects_releases | 200/s | 197.8/s (>64.00/s) | 83.14ms  | 88.76ms (<4000ms) | 100.00% (>99%) | Passed
root@cat-gpt-10k-gitlab-rails-1:~# cat /var/log/gitlab/gitlab-rails/api_json.log | grep '^{' | jq -r 'select(."route" == "/api/:version/projects/:id/releases") | [.db_count, .db_duration_s] | @tsv' | tail -n 5
18      0.02428
18      0.02245
18      0.02295
18      0.02311
18      0.02407

Manual QA

Performed manual QA locally and confirmed that the cache is correctly generated in Redis.

Execute the following request:

curl -H 'Private-token: [REDACTED]' "http://local.gitlab.test:8181/api/v4/projects/35/releases"

Response is correctly generated:

[{"name":"Test","tag_name":"v0.1","description":"a","description_html":"\u003cp data-sourcepos=\"1:1-1:1\" dir=\"auto\"\u003ea\u003c/p\u003e","created_at":"2021-05-03T02:27:08.559Z","released_at":"2021-05-03T02:27:08.559Z","author":{"id":1,"name":"Administrator","username":"root","state":"active","avatar_url":"https://www.gravatar.com/avatar/e64c7d89f26bd1972efa854d13d7dd61?s=80\u0026d=identicon","web_url":"http://local.gitlab.test:8181/root"},"commit":{"id":"73ad9a9c13f7a885860425f272a28c2dc52d6d0c","short_id":"73ad9a9c","created_at":"2021-04-29T01:49:40.000+00:00","parent_ids":["6f515815f7a6c33fdc603c309bd6f7500896bea9"],"title":"Update .gitlab-ci.yml","message":"Update .gitlab-ci.yml","author_name":"Administrator","author_email":"admin@example.com","authored_date":"2021-04-29T01:49:40.000+00:00","committer_name":"Administrator","committer_email":"admin@example.com","committed_date":"2021-04-29T01:49:40.000+00:00","web_url":"http://local.gitlab.test:8181/root/dora-metrics/-/commit/73ad9a9c13f7a885860425f272a28c2dc52d6d0c"},"upcoming_release":false,"commit_path":"/root/dora-metrics/-/commit/73ad9a9c13f7a885860425f272a28c2dc52d6d0c","tag_path":"/root/dora-metrics/-/tags/v0.1","assets":{"count":7,"sources":[{"format":"zip","url":"http://local.gitlab.test:8181/root/dora-metrics/-/archive/v0.1/dora-metrics-v0.1.zip"},{"format":"tar.gz","url":"http://local.gitlab.test:8181/root/dora-metrics/-/archive/v0.1/dora-metrics-v0.1.tar.gz"},{"format":"tar.bz2","url":"http://local.gitlab.test:8181/root/dora-metrics/-/archive/v0.1/dora-metrics-v0.1.tar.bz2"},{"format":"tar","url":"http://local.gitlab.test:8181/root/dora-metrics/-/archive/v0.1/dora-metrics-v0.1.tar"}],"links":[{"id":3,"name":"google-3","url":"https://google-3.com","direct_asset_url":"https://google-3.com","external":true,"link_type":"other"},{"id":2,"name":"google-2","url":"https://google-2.com","direct_asset_url":"https://google-2.com","external":true,"link_type":"other"},{"id":1,"name":"google-1","url":"https://google-1.com","direct_asset_url":"https://google-1.com","external":true,"link_type":"other"}]},"evidences":[{"sha":"2cc4554ec19ef23185261fd556f93f5c22f1a97b9a0d","filepath":"http://local.gitlab.test:8181/root/dora-metrics/-/releases/v0.1/evidences/2.json","collected_at":"2021-05-03T02:27:08.990Z"}],"_links":{"self":"http://local.gitlab.test:8181/root/dora-metrics/-/releases/v0.1","edit_url":"http://local.gitlab.test:8181/root/dora-metrics/-/releases/v0.1/edit"}}]

Confirmed the expected cache key is generated:

local.gitlab.test:6379> KEYS *releases*
1) "cache:gitlab:releases/5-20210503023215859318:user:{}"
2) "cache:gitlab:releases/5-20210503023215859318:user:{1}"

Confirmed the expected TTL is set:

local.gitlab.test:6379> TTL cache:gitlab:releases/5-20210503023215859318:user:{1}
(integer) 233

Does this MR meet the acceptance criteria?

Conformity

Availability and Testing

Security

If this MR contains changes to processing or storing of credentials or tokens, authorization and authentication methods and other items described in the security review guidelines:

  • Label as security and @ mention @gitlab-com/gl-security/appsec
  • The MR includes necessary changes to maintain consistency between UI, API, email, or other methods
  • Security reports checked/validated by a reviewer from the AppSec team
Edited by Shinya Maeda

Merge request reports