Skip to content

Batch insert CI profiling spec data

Stan Hu requested to merge sh-store-rspec-profiling-csv into master

Instead of inserting a row after each example to an external database, we save the CI profiling reports into the rspec_profiling directory and insert the data in the update-tests-metadata CI stage. This should make each spec run faster and also reduce the number of PostgreSQL connections needed by concurrent CI builds.

scripts/insert-rspec-profiling-data also inserts one file at a time via the PostgreSQL COPY command for faster inserts. created_at and updated_at are stored in the CSV when the example runs.

Note that CI statistics will only be inserted if the build is successful. update-tests-metadata doesn't run otherwise. We might miss some slow test runs when the build is red.

Sample run: https://gitlab.com/gitlab-org/gitlab-ee/pipelines/51379201

Closes https://gitlab.com/gitlab-org/gitlab-ee/issues/10154

Edited by Stan Hu

Merge request reports