Skip to content

[CE] Add e2e test comparing archives of identically named user projects

Grant Young requested to merge grantyoung-52703-project-archive-compare-qa into master

Commits a new Gitlab QA API test that implements #52703 (closed). It checks that all archive types of separate identically named user projects aren't the same by checksum. This ensures a different user can't download another's project if they happen to have the same project name.

EE MR: https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/14322

Several enhancements and additions had to be made to enable this test:

  • Runtime::API::Client has been refactored to allow for the generation of an api_client for a specified user from Resource::User along with allowing multiples of these to exist at the same time in memory. It's also been further improved to handle multiple successive personal_access_token calls by clearing up any previously logged in browser instances.
  • Resource::ApiFabricator now allows passthrough of an existing api_client from Runtime::API::Client as well as a user from Resource::User to attach the client too.
  • Resource::Project can now optionally create a standalone project for a user instead of always creating one in a separate Sandbox and Group. The class was also improved to allow specific project names instead of always adding a random uuid if required.
  • Resource::User.fabricate_or_use can now be called without specifying a username or password and just let both be generated automatically.
  • Support::API.get method has been improved so it can now download responses into temp files. This is optional and is delivered via the rest-client gem's raw_response flag. Note that calling get without specifically referencing or including Support::API will switch to using the Airborne gem and will act differently.

Performance and testing

Test passing:

Run options: exclude {:orchestrated=>true}

Randomized with seed 4294

Create
  Compare archives of different user projects with the same name and check they're different
    download archives of each user project then check they are different

Top 1 slowest examples (3.08 seconds, 1.6% of total time):
  Create Compare archives of different user projects with the same name and check they're different download archives of each user project then check they are different
    3.08 seconds ./qa/specs/features/api/3_create/repository/project_archive_compare_spec.rb:23

Finished in 3 minutes 9.6 seconds (files took 13.29 seconds to load)
1 example, 0 failures

Randomized with seed 4294

Test failing by force (specifically made the checksum of the zip archive the same):

Run options: exclude {:orchestrated=>true}

Randomized with seed 51524

Create
  Compare archives of different user projects with the same name and check they're different

1st Try error in ./qa/specs/features/api/3_create/repository/project_archive_compare_spec.rb:23:
expected {"tar" => "dcde80208366064c1495f4c4a68550f8", "tar.bz2" => "4de0a803301998cffacb571ee64aaf56", "tar.gz" => "9c03fe54dbd8911ddeabd123df90069a", "zip" => "f174c812da7f639f44bd204b9a450a42"} not to include {"zip" => "f174c812da7f639f44bd204b9a450a42"}
Diff:
@@ -1,5 +1,5 @@
-"tar" => "d6c5c61a55a088b0409b520761f9222c",
-"tar.bz2" => "b24586e30ea67d61882d7981378cc914",
-"tar.gz" => "d1fa7f31c4b4b85bf327641d4c39748a",
+"tar" => "dcde80208366064c1495f4c4a68550f8",
+"tar.bz2" => "4de0a803301998cffacb571ee64aaf56",
+"tar.gz" => "9c03fe54dbd8911ddeabd123df90069a",
 "zip" => "f174c812da7f639f44bd204b9a450a42",


RSpec::Retry: 2nd try ./qa/specs/features/api/3_create/repository/project_archive_compare_spec.rb:23
WARN: Screenshot could not be saved. `page.current_path` is empty.

2nd Try error in ./qa/specs/features/api/3_create/repository/project_archive_compare_spec.rb:23:
expected {"tar" => "dcde80208366064c1495f4c4a68550f8", "tar.bz2" => "4de0a803301998cffacb571ee64aaf56", "tar.gz" => "9c03fe54dbd8911ddeabd123df90069a", "zip" => "f174c812da7f639f44bd204b9a450a42"} not to include {"zip" => "f174c812da7f639f44bd204b9a450a42"}
Diff:
@@ -1,5 +1,5 @@
-"tar" => "d6c5c61a55a088b0409b520761f9222c",
-"tar.bz2" => "b24586e30ea67d61882d7981378cc914",
-"tar.gz" => "d1fa7f31c4b4b85bf327641d4c39748a",
+"tar" => "dcde80208366064c1495f4c4a68550f8",
+"tar.bz2" => "4de0a803301998cffacb571ee64aaf56",
+"tar.gz" => "9c03fe54dbd8911ddeabd123df90069a",
 "zip" => "f174c812da7f639f44bd204b9a450a42",

RSpec::Retry: 3rd try ./qa/specs/features/api/3_create/repository/project_archive_compare_spec.rb:23
WARN: Screenshot could not be saved. `page.current_path` is empty.
    download archives of each user project then check they are different (FAILED - 1)

Failures:

  1) Create Compare archives of different user projects with the same name and check they're different download archives of each user project then check they are different
     Failure/Error: expect(archive_checksums[:user1]).not_to include(archive_checksums[:user2])
     
       expected {"tar" => "dcde80208366064c1495f4c4a68550f8", "tar.bz2" => "4de0a803301998cffacb571ee64aaf56", "tar.gz" => "9c03fe54dbd8911ddeabd123df90069a", "zip" => "f174c812da7f639f44bd204b9a450a42"} not to include {"zip" => "f174c812da7f639f44bd204b9a450a42"}
       Diff:
       @@ -1,5 +1,5 @@
       -"tar" => "d6c5c61a55a088b0409b520761f9222c",
       -"tar.bz2" => "b24586e30ea67d61882d7981378cc914",
       -"tar.gz" => "d1fa7f31c4b4b85bf327641d4c39748a",
       +"tar" => "dcde80208366064c1495f4c4a68550f8",
       +"tar.bz2" => "4de0a803301998cffacb571ee64aaf56",
       +"tar.gz" => "9c03fe54dbd8911ddeabd123df90069a",
        "zip" => "f174c812da7f639f44bd204b9a450a42",
       
     # ./qa/specs/features/api/3_create/repository/project_archive_compare_spec.rb:39:in `block (3 levels) in <module:QA>'
     # ./spec/spec_helper.rb:38:in `block (2 levels) in <top (required)>'
     # ./qa/specs/runner.rb:37:in `perform'
     # ./qa/scenario/template.rb:8:in `block in perform'
     # ./qa/scenario/template.rb:6:in `tap'
     # ./qa/scenario/template.rb:6:in `perform'
     # ./qa/scenario/template.rb:31:in `perform'
     # ./qa/scenario/template.rb:8:in `block in perform'
     # ./qa/scenario/template.rb:6:in `tap'
     # ./qa/scenario/template.rb:6:in `perform'
     # ./qa/scenario/bootable.rb:26:in `launch!'

Top 1 slowest examples (7.74 seconds, 4.3% of total time):
  Create Compare archives of different user projects with the same name and check they're different download archives of each user project then check they are different
    7.74 seconds ./qa/specs/features/api/3_create/repository/project_archive_compare_spec.rb:23

Finished in 3 minutes 1.7 seconds (files took 8.34 seconds to load)
1 example, 1 failure

Failed examples:

rspec ./qa/specs/features/api/3_create/repository/project_archive_compare_spec.rb:23 # Create Compare archives of different user projects with the same name and check they're different download archives of each user project then check they are different

Randomized with seed 51524

Finally, all API tests passing due to Runtime::API::Client and Resource class changes:

Run options: exclude {:orchestrated=>true}

Randomized with seed 39025

Create
  Compare archives of different user projects with the same name and check they're different
    download archives of each user project then check they are different

API basics
  user creates a project with a file and deletes them afterwards

Manage
  Users API
    GET /users/:username with a valid username
    GET /users
    GET /users/:username with an invalid username

Top 5 slowest examples (95.15 seconds, 35.8% of total time):
  API basics user creates a project with a file and deletes them afterwards
    49.14 seconds ./qa/specs/features/api/3_create/repository/files_spec.rb:12
  Manage Users API GET /users/:username with a valid username
    42.75 seconds ./qa/specs/features/api/1_manage/users_spec.rb:18
  Create Compare archives of different user projects with the same name and check they're different download archives of each user project then check they are different
    2.5 seconds ./qa/specs/features/api/3_create/repository/project_archive_compare_spec.rb:23
  Manage Users API GET /users
    0.54686 seconds ./qa/specs/features/api/1_manage/users_spec.rb:12
  Manage Users API GET /users/:username with an invalid username
    0.21449 seconds ./qa/specs/features/api/1_manage/users_spec.rb:27

Top 3 slowest example groups:
  Create
    173.33 seconds average (173.33 seconds / 1 example) ./qa/specs/features/api/3_create/repository/project_archive_compare_spec.rb:7
  API basics
    49.14 seconds average (49.14 seconds / 1 example) ./qa/specs/features/api/3_create/repository/files_spec.rb:4
  Manage
    14.51 seconds average (43.52 seconds / 3 examples) ./qa/specs/features/api/1_manage/users_spec.rb:4

Finished in 4 minutes 26 seconds (files took 8.65 seconds to load)
5 examples, 0 failures

Randomized with seed 39025
Edited by Grant Young

Merge request reports