Skip to content

Use Mutex and limit threads by pool size in GPT Generator

Nailia Iskhakova requested to merge 486-threads-improvement into main

Update Generator to use Mutex and sync thread counts with pool size. With this update Generator will create GPT_GENERATOR_POOL_SIZE threads and each of them will send multiple requests. Previous configuration was not efficient as it tried to create threads for each project/group.

Changes:

  • Fixes active connection limitation issue for projects - #486 (closed)
  • Fixes Generator Group count issue if pool size wasn't divided correctly when a custom pool size was used - #485 (closed)
  • Bumps Generator version to 1.0.23

As can be seen from the charts below, active connection now increases correctly (grafana):

10k graphs before and after the change

Screenshot_2021-12-03_at_22.47.24

Screenshot_2021-12-03_at_22.49.58

Before update - 2 minutes 57 seconds with incorrect 40 groups instead of 50:

Local test - 50 groups x 10 projects with pool size 20
GPT Data Generator v1.0.22 - opinionated test data for the GitLab Performance Tool
Checking that GitLab environment 'http://localhost' is available, supported and that provided Access Token works...
Environment and Access Token check complete - URL: http://localhost, Version: 14.6.0-pre 49ec55172c1
Creating group gpt_5_old
Creating group gpt_5_old/many_groups_and_projects
Creating 50 groups with name prefix 'gpt-subgroup-' under parent group 'gpt_5_old/many_groups_and_projects'
........................................
Checking for existing projects under groups...
........................................
Creating 10 projects each under 40 subgroups with name prefix 'gpt-project-'
................................................................................................................................................................................................................................................................................................................................................................................................................
<-> Horizontal data: successfully generated after 2 minutes 57 seconds!
█ GPT data generation finished after 2 minutes 57 seconds.

After update - 2 minutes 30 seconds with correct 50 groups:

Local test - 50 groups x 10 projects with pool size 20 -
GPT Data Generator v1.0.23 - opinionated test data for the GitLab Performance Tool
Checking that GitLab environment 'http://localhost' is available, supported and that provided Access Token works...
Environment and Access Token check complete - URL: http://localhost, Version: 14.6.0-pre 49ec55172c1
Creating group gpt_5_new
Creating group gpt_5_new/many_groups_and_projects
Creating 50 groups with name prefix 'gpt-subgroup-' under parent group 'gpt_5_new/many_groups_and_projects'
..................................................
Checking for existing projects under groups...
..................................................
Creating 10 projects each under 50 subgroups with name prefix 'gpt-project-'
....................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................
<-> Horizontal data: successfully generated after 2 minutes 30 seconds!
█ GPT data generation finished after 2 minutes 30 seconds.

█ Logs: results/generate-gpt-data_localhost_2021-12-03_231315.log

Closes #486 (closed) #485 (closed)

Merge request reports