WIP: New Gitlab QA API test for #52703 (!27092) · Merge requests · GitLab.org / GitLab FOSS

The source project of this merge request has been removed.

Grant Young requested to merge (removed):qa-52703-project-archive-compare into master Apr 07, 2019

Commits a new Gitlab QA API test for #52703 (closed) that checks all archive types of different user projects with the same name aren't the same by checksum. This ensures a different user can't download another's project if they happen to have the same project name.

Several enhancements and additions had to be made to enable this test:

The Runtime::API::Client class has been updated to create new users when selected as well as handle multiple API clients for different users at the same time. Current iteration only allowed for one client to exist with the existing project default user and would always return that when called subsequent times.
Custom download method was created for the test as the current Airborne gem used for the API doesn't give the option to download files. The gem uses rest-clientto perform it's API calls, which does allow for file downloads if instructed. Test utilizes this in a custom method to get the archives. This could be promoted in the future to be project wide but keeping it isolated for now so other tests aren't impacted.

Performance and testing

Test passing:

bin/qa Test::Instance::All http://10.0.2.15:3000 -- qa/specs/features/api/3_create/repository/project_archive_compare_spec.rb
Run options: exclude {:orchestrated=>true}

Randomized with seed 34456

Create
  Compare archives of different user projects with the same name and check they're different
    user1 creates, downloads and gets checksums of project archives
    user2 creates, downloads and gets checksums of project archives
    compare the archives and check they are different

Top 3 slowest examples (107.89 seconds, 99.0% of total time):
  Create Compare archives of different user projects with the same name and check they're different user1 creates, downloads and gets checksums of project archives
    55.18 seconds ./qa/specs/features/api/3_create/repository/project_archive_compare_spec.rb:16
  Create Compare archives of different user projects with the same name and check they're different user2 creates, downloads and gets checksums of project archives
    52.71 seconds ./qa/specs/features/api/3_create/repository/project_archive_compare_spec.rb:25
  Create Compare archives of different user projects with the same name and check they're different compare the archives and check they are different
    0.00036 seconds ./qa/specs/features/api/3_create/repository/project_archive_compare_spec.rb:34

Finished in 1 minute 49 seconds (files took 7.38 seconds to load)
3 examples, 0 failures

Randomized with seed 34456

Test failing by force (specifically made the checksum of the zip archive the same):

bin/qa Test::Instance::All http://10.0.2.15:3000 -- qa/specs/features/api/3_create/repository/project_archive_compare_spec.rb
Run options: exclude {:orchestrated=>true}

Randomized with seed 37010

Create
  Compare archives of different user projects with the same name and check they're different
    user1 creates, downloads and gets checksums of project archives
    user2 creates, downloads and gets checksums of project archives
WARN: Screenshot could not be saved. `page.current_path` is empty.

1st Try error in ./qa/specs/features/api/3_create/repository/project_archive_compare_spec.rb:34:
expected {"tar" => "778f099611ba4db436df73d51136aad4", "tar.bz2" => "1a57b23b0527b77c106c45132150274b", "tar.gz" => "58edcd2f3a7479c7e860b967e0c618e4", "zip" => "1628ee897146732bf562d371032e2b7e"} not to include {"zip" => "1628ee897146732bf562d371032e2b7e"}
Diff:
@@ -1,5 +1,5 @@
-"tar" => "8833b311af9ea342cc48a175c94f27c9",
-"tar.bz2" => "5f11fa7cd4c75ece4dedf09d71fe4446",
-"tar.gz" => "fd137e9441336c20c59f4a85e2bd5ddd",
+"tar" => "778f099611ba4db436df73d51136aad4",
+"tar.bz2" => "1a57b23b0527b77c106c45132150274b",
+"tar.gz" => "58edcd2f3a7479c7e860b967e0c618e4",
 "zip" => "1628ee897146732bf562d371032e2b7e",


RSpec::Retry: 2nd try ./qa/specs/features/api/3_create/repository/project_archive_compare_spec.rb:34
WARN: Screenshot could not be saved. `page.current_path` is empty.

2nd Try error in ./qa/specs/features/api/3_create/repository/project_archive_compare_spec.rb:34:
expected {"tar" => "778f099611ba4db436df73d51136aad4", "tar.bz2" => "1a57b23b0527b77c106c45132150274b", "tar.gz" => "58edcd2f3a7479c7e860b967e0c618e4", "zip" => "1628ee897146732bf562d371032e2b7e"} not to include {"zip" => "1628ee897146732bf562d371032e2b7e"}
Diff:
@@ -1,5 +1,5 @@
-"tar" => "8833b311af9ea342cc48a175c94f27c9",
-"tar.bz2" => "5f11fa7cd4c75ece4dedf09d71fe4446",
-"tar.gz" => "fd137e9441336c20c59f4a85e2bd5ddd",
+"tar" => "778f099611ba4db436df73d51136aad4",
+"tar.bz2" => "1a57b23b0527b77c106c45132150274b",
+"tar.gz" => "58edcd2f3a7479c7e860b967e0c618e4",
 "zip" => "1628ee897146732bf562d371032e2b7e",

RSpec::Retry: 3rd try ./qa/specs/features/api/3_create/repository/project_archive_compare_spec.rb:34
WARN: Screenshot could not be saved. `page.current_path` is empty.
    compare the archives and check they are different (FAILED - 1)

Failures:

  1) Create Compare archives of different user projects with the same name and check they're different compare the archives and check they are different
     Failure/Error: expect(@archive_checksums[:user1]).not_to include(@archive_checksums[:user2])
     
       expected {"tar" => "778f099611ba4db436df73d51136aad4", "tar.bz2" => "1a57b23b0527b77c106c45132150274b", "tar.gz" => "58edcd2f3a7479c7e860b967e0c618e4", "zip" => "1628ee897146732bf562d371032e2b7e"} not to include {"zip" => "1628ee897146732bf562d371032e2b7e"}
       Diff:
       @@ -1,5 +1,5 @@
       -"tar" => "8833b311af9ea342cc48a175c94f27c9",
       -"tar.bz2" => "5f11fa7cd4c75ece4dedf09d71fe4446",
       -"tar.gz" => "fd137e9441336c20c59f4a85e2bd5ddd",
       +"tar" => "778f099611ba4db436df73d51136aad4",
       +"tar.bz2" => "1a57b23b0527b77c106c45132150274b",
       +"tar.gz" => "58edcd2f3a7479c7e860b967e0c618e4",
        "zip" => "1628ee897146732bf562d371032e2b7e",
       
     # ./qa/specs/features/api/3_create/repository/project_archive_compare_spec.rb:36:in `block (3 levels) in <module:QA>'
     # ./spec/spec_helper.rb:38:in `block (2 levels) in <top (required)>'
     # ./qa/specs/runner.rb:37:in `perform'
     # ./qa/scenario/template.rb:8:in `block in perform'
     # ./qa/scenario/template.rb:6:in `tap'
     # ./qa/scenario/template.rb:6:in `perform'
     # ./qa/scenario/template.rb:31:in `perform'
     # ./qa/scenario/template.rb:8:in `block in perform'
     # ./qa/scenario/template.rb:6:in `tap'
     # ./qa/scenario/template.rb:6:in `perform'
     # ./qa/scenario/bootable.rb:26:in `launch!'

Top 3 slowest examples (110.41 seconds, 99.1% of total time):
  Create Compare archives of different user projects with the same name and check they're different user1 creates, downloads and gets checksums of project archives
    55.46 seconds ./qa/specs/features/api/3_create/repository/project_archive_compare_spec.rb:16
  Create Compare archives of different user projects with the same name and check they're different user2 creates, downloads and gets checksums of project archives
    54.83 seconds ./qa/specs/features/api/3_create/repository/project_archive_compare_spec.rb:25
  Create Compare archives of different user projects with the same name and check they're different compare the archives and check they are different
    0.11578 seconds ./qa/specs/features/api/3_create/repository/project_archive_compare_spec.rb:34

Finished in 1 minute 51.42 seconds (files took 6.16 seconds to load)
3 examples, 1 failure

Failed examples:

rspec ./qa/specs/features/api/3_create/repository/project_archive_compare_spec.rb:34 # Create Compare archives of different user projects with the same name and check they're different compare the archives and check they are different

Randomized with seed 37010

Finally, all API tests passing due to Runtime::API::Client change:

gitlab-qa Test::Instance::Any gitlab/gitlab-ce:52703 http://10.0.2.15:3000 -- qa/specs/features/api
Running test suite `Test::Instance::All` for gitlab-ce
Docker shell command: `docker run -t --rm --net=bridge --volume /var/run/docker.sock:/var/run/docker.sock:z --volume /tmp/gitlab-qa/gitlab-qa-run-2019-04-07-21-46-09-fb81b7d2/gitlab-ce-qa-24a542c7:/home/qa/tmp:z --name gitlab-ce-qa-24a542c7 gitlab/gitlab-ce-qa:52703 Test::Instance::All http://10.0.2.15:3000 -- qa/specs/features/api`
Run options: exclude {:orchestrated=>true}

Randomized with seed 15867

Manage
  Users API
    GET /users
    GET /users/:username with a valid username
    GET /users/:username with an invalid username

Create
  Compare archives of different user projects with the same name and check they're different
    user1 creates, downloads and gets checksums of project archives
    user2 creates, downloads and gets checksums of project archives
    compare the archives and check they are different

API basics
  user creates a project with a file and deletes them afterwards

Top 7 slowest examples (186.91 seconds, 99.7% of total time):
  Create Compare archives of different user projects with the same name and check they're different user1 creates, downloads and gets checksums of project archives
    55.98 seconds ./qa/specs/features/api/3_create/repository/project_archive_compare_spec.rb:16
  Create Compare archives of different user projects with the same name and check they're different user2 creates, downloads and gets checksums of project archives
    51.05 seconds ./qa/specs/features/api/3_create/repository/project_archive_compare_spec.rb:25
  Manage Users API GET /users
    39.67 seconds ./qa/specs/features/api/1_manage/users_spec.rb:12
  API basics user creates a project with a file and deletes them afterwards
    39.66 seconds ./qa/specs/features/api/3_create/repository/files_spec.rb:12
  Manage Users API GET /users/:username with an invalid username
    0.33281 seconds ./qa/specs/features/api/1_manage/users_spec.rb:27
  Manage Users API GET /users/:username with a valid username
    0.21492 seconds ./qa/specs/features/api/1_manage/users_spec.rb:18
  Create Compare archives of different user projects with the same name and check they're different compare the archives and check they are different
    0.00032 seconds ./qa/specs/features/api/3_create/repository/project_archive_compare_spec.rb:34

Top 3 slowest example groups:
  API basics
    39.66 seconds average (39.66 seconds / 1 example) ./qa/specs/features/api/3_create/repository/files_spec.rb:4
  Create
    35.68 seconds average (107.03 seconds / 3 examples) ./qa/specs/features/api/3_create/repository/project_archive_compare_spec.rb:7
  Manage
    13.61 seconds average (40.82 seconds / 3 examples) ./qa/specs/features/api/1_manage/users_spec.rb:4

Finished in 3 minutes 7.5 seconds (files took 51.2 seconds to load)
7 examples, 0 failures

Randomized with seed 15867

WIP: New Gitlab QA API test for #52703

Performance and testing

Merge request reports