[CE] Add e2e test comparing archives of identically named user projects
Commits a new Gitlab QA API test that implements #52703 (closed). It checks that all archive types of separate identically named user projects aren't the same by checksum. This ensures a different user can't download another's project if they happen to have the same project name.
EE MR: https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/14322
Several enhancements and additions had to be made to enable this test:
-
Runtime::API::Client
has been refactored to allow for the generation of anapi_client
for a specifieduser
fromResource::User
along with allowing multiples of these to exist at the same time in memory. It's also been further improved to handle multiple successivepersonal_access_token
calls by clearing up any previously logged in browser instances. -
Resource::ApiFabricator
now allows passthrough of an existingapi_client
fromRuntime::API::Client
as well as auser
fromResource::User
to attach the client too. -
Resource::Project
can now optionally create a standalone project for a user instead of always creating one in a separateSandbox
andGroup
. The class was also improved to allow specific project names instead of always adding a random uuid if required. -
Resource::User.fabricate_or_use
can now be called without specifying a username or password and just let both be generated automatically. -
Support::API.get
method has been improved so it can now download responses into temp files. This is optional and is delivered via the rest-client gem's raw_response flag. Note that callingget
without specifically referencing or includingSupport::API
will switch to using the Airborne gem and will act differently.
Performance and testing
Test passing:
Run options: exclude {:orchestrated=>true}
Randomized with seed 4294
Create
Compare archives of different user projects with the same name and check they're different
download archives of each user project then check they are different
Top 1 slowest examples (3.08 seconds, 1.6% of total time):
Create Compare archives of different user projects with the same name and check they're different download archives of each user project then check they are different
3.08 seconds ./qa/specs/features/api/3_create/repository/project_archive_compare_spec.rb:23
Finished in 3 minutes 9.6 seconds (files took 13.29 seconds to load)
1 example, 0 failures
Randomized with seed 4294
Test failing by force (specifically made the checksum of the zip archive the same):
Run options: exclude {:orchestrated=>true}
Randomized with seed 51524
Create
Compare archives of different user projects with the same name and check they're different
1st Try error in ./qa/specs/features/api/3_create/repository/project_archive_compare_spec.rb:23:
expected {"tar" => "dcde80208366064c1495f4c4a68550f8", "tar.bz2" => "4de0a803301998cffacb571ee64aaf56", "tar.gz" => "9c03fe54dbd8911ddeabd123df90069a", "zip" => "f174c812da7f639f44bd204b9a450a42"} not to include {"zip" => "f174c812da7f639f44bd204b9a450a42"}
Diff:
@@ -1,5 +1,5 @@
-"tar" => "d6c5c61a55a088b0409b520761f9222c",
-"tar.bz2" => "b24586e30ea67d61882d7981378cc914",
-"tar.gz" => "d1fa7f31c4b4b85bf327641d4c39748a",
+"tar" => "dcde80208366064c1495f4c4a68550f8",
+"tar.bz2" => "4de0a803301998cffacb571ee64aaf56",
+"tar.gz" => "9c03fe54dbd8911ddeabd123df90069a",
"zip" => "f174c812da7f639f44bd204b9a450a42",
RSpec::Retry: 2nd try ./qa/specs/features/api/3_create/repository/project_archive_compare_spec.rb:23
WARN: Screenshot could not be saved. `page.current_path` is empty.
2nd Try error in ./qa/specs/features/api/3_create/repository/project_archive_compare_spec.rb:23:
expected {"tar" => "dcde80208366064c1495f4c4a68550f8", "tar.bz2" => "4de0a803301998cffacb571ee64aaf56", "tar.gz" => "9c03fe54dbd8911ddeabd123df90069a", "zip" => "f174c812da7f639f44bd204b9a450a42"} not to include {"zip" => "f174c812da7f639f44bd204b9a450a42"}
Diff:
@@ -1,5 +1,5 @@
-"tar" => "d6c5c61a55a088b0409b520761f9222c",
-"tar.bz2" => "b24586e30ea67d61882d7981378cc914",
-"tar.gz" => "d1fa7f31c4b4b85bf327641d4c39748a",
+"tar" => "dcde80208366064c1495f4c4a68550f8",
+"tar.bz2" => "4de0a803301998cffacb571ee64aaf56",
+"tar.gz" => "9c03fe54dbd8911ddeabd123df90069a",
"zip" => "f174c812da7f639f44bd204b9a450a42",
RSpec::Retry: 3rd try ./qa/specs/features/api/3_create/repository/project_archive_compare_spec.rb:23
WARN: Screenshot could not be saved. `page.current_path` is empty.
download archives of each user project then check they are different (FAILED - 1)
Failures:
1) Create Compare archives of different user projects with the same name and check they're different download archives of each user project then check they are different
Failure/Error: expect(archive_checksums[:user1]).not_to include(archive_checksums[:user2])
expected {"tar" => "dcde80208366064c1495f4c4a68550f8", "tar.bz2" => "4de0a803301998cffacb571ee64aaf56", "tar.gz" => "9c03fe54dbd8911ddeabd123df90069a", "zip" => "f174c812da7f639f44bd204b9a450a42"} not to include {"zip" => "f174c812da7f639f44bd204b9a450a42"}
Diff:
@@ -1,5 +1,5 @@
-"tar" => "d6c5c61a55a088b0409b520761f9222c",
-"tar.bz2" => "b24586e30ea67d61882d7981378cc914",
-"tar.gz" => "d1fa7f31c4b4b85bf327641d4c39748a",
+"tar" => "dcde80208366064c1495f4c4a68550f8",
+"tar.bz2" => "4de0a803301998cffacb571ee64aaf56",
+"tar.gz" => "9c03fe54dbd8911ddeabd123df90069a",
"zip" => "f174c812da7f639f44bd204b9a450a42",
# ./qa/specs/features/api/3_create/repository/project_archive_compare_spec.rb:39:in `block (3 levels) in <module:QA>'
# ./spec/spec_helper.rb:38:in `block (2 levels) in <top (required)>'
# ./qa/specs/runner.rb:37:in `perform'
# ./qa/scenario/template.rb:8:in `block in perform'
# ./qa/scenario/template.rb:6:in `tap'
# ./qa/scenario/template.rb:6:in `perform'
# ./qa/scenario/template.rb:31:in `perform'
# ./qa/scenario/template.rb:8:in `block in perform'
# ./qa/scenario/template.rb:6:in `tap'
# ./qa/scenario/template.rb:6:in `perform'
# ./qa/scenario/bootable.rb:26:in `launch!'
Top 1 slowest examples (7.74 seconds, 4.3% of total time):
Create Compare archives of different user projects with the same name and check they're different download archives of each user project then check they are different
7.74 seconds ./qa/specs/features/api/3_create/repository/project_archive_compare_spec.rb:23
Finished in 3 minutes 1.7 seconds (files took 8.34 seconds to load)
1 example, 1 failure
Failed examples:
rspec ./qa/specs/features/api/3_create/repository/project_archive_compare_spec.rb:23 # Create Compare archives of different user projects with the same name and check they're different download archives of each user project then check they are different
Randomized with seed 51524
Finally, all API tests passing due to Runtime::API::Client
and Resource
class changes:
Run options: exclude {:orchestrated=>true}
Randomized with seed 39025
Create
Compare archives of different user projects with the same name and check they're different
download archives of each user project then check they are different
API basics
user creates a project with a file and deletes them afterwards
Manage
Users API
GET /users/:username with a valid username
GET /users
GET /users/:username with an invalid username
Top 5 slowest examples (95.15 seconds, 35.8% of total time):
API basics user creates a project with a file and deletes them afterwards
49.14 seconds ./qa/specs/features/api/3_create/repository/files_spec.rb:12
Manage Users API GET /users/:username with a valid username
42.75 seconds ./qa/specs/features/api/1_manage/users_spec.rb:18
Create Compare archives of different user projects with the same name and check they're different download archives of each user project then check they are different
2.5 seconds ./qa/specs/features/api/3_create/repository/project_archive_compare_spec.rb:23
Manage Users API GET /users
0.54686 seconds ./qa/specs/features/api/1_manage/users_spec.rb:12
Manage Users API GET /users/:username with an invalid username
0.21449 seconds ./qa/specs/features/api/1_manage/users_spec.rb:27
Top 3 slowest example groups:
Create
173.33 seconds average (173.33 seconds / 1 example) ./qa/specs/features/api/3_create/repository/project_archive_compare_spec.rb:7
API basics
49.14 seconds average (49.14 seconds / 1 example) ./qa/specs/features/api/3_create/repository/files_spec.rb:4
Manage
14.51 seconds average (43.52 seconds / 3 examples) ./qa/specs/features/api/1_manage/users_spec.rb:4
Finished in 4 minutes 26 seconds (files took 8.65 seconds to load)
5 examples, 0 failures
Randomized with seed 39025
Edited by Grant Young