Skip to content

Use Mutex and limit threads by pool size in GPT Generator

Update Generator to use Mutex and sync thread counts with pool size. With this update Generator will create GPT_GENERATOR_POOL_SIZE threads and each of them will send multiple requests. Previous configuration was not efficient as it tried to create threads for each project/group.

Changes:

  • Fixes active connection limitation issue for projects - #486 (closed)
  • Fixes Generator Group count issue if pool size wasn't divided correctly when a custom pool size was used - #485 (closed)
  • Bumps Generator version to 1.0.23

As can be seen from the charts below, active connection now increases correctly (grafana):

10k graphs before and after the change

Screenshot_2021-12-03_at_22.47.24

Screenshot_2021-12-03_at_22.49.58

Before update - 2 minutes 57 seconds with incorrect 40 groups instead of 50:

Local test - 50 groups x 10 projects with pool size 20
GPT Data Generator v1.0.22 - opinionated test data for the GitLab Performance Tool
Checking that GitLab environment 'http://localhost' is available, supported and that provided Access Token works...
Environment and Access Token check complete - URL: http://localhost, Version: 14.6.0-pre 49ec55172c1
Creating group gpt_5_old
Creating group gpt_5_old/many_groups_and_projects
Creating 50 groups with name prefix 'gpt-subgroup-' under parent group 'gpt_5_old/many_groups_and_projects'
........................................
Checking for existing projects under groups...
........................................
Creating 10 projects each under 40 subgroups with name prefix 'gpt-project-'
................................................................................................................................................................................................................................................................................................................................................................................................................
<-> Horizontal data: successfully generated after 2 minutes 57 seconds!
█ GPT data generation finished after 2 minutes 57 seconds.

After update - 2 minutes 30 seconds with correct 50 groups:

Local test - 50 groups x 10 projects with pool size 20 -
GPT Data Generator v1.0.23 - opinionated test data for the GitLab Performance Tool
Checking that GitLab environment 'http://localhost' is available, supported and that provided Access Token works...
Environment and Access Token check complete - URL: http://localhost, Version: 14.6.0-pre 49ec55172c1
Creating group gpt_5_new
Creating group gpt_5_new/many_groups_and_projects
Creating 50 groups with name prefix 'gpt-subgroup-' under parent group 'gpt_5_new/many_groups_and_projects'
..................................................
Checking for existing projects under groups...
..................................................
Creating 10 projects each under 50 subgroups with name prefix 'gpt-project-'
....................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................
<-> Horizontal data: successfully generated after 2 minutes 30 seconds!
█ GPT data generation finished after 2 minutes 30 seconds.

█ Logs: results/generate-gpt-data_localhost_2021-12-03_231315.log

Closes #486 (closed) #485 (closed)

Merge request reports

Loading