Screenshot diffing in GitLab - Meta
Screenshot diffing(visual regression testing) in GitLab meta issue,
- CI View for Selenium, https://gitlab.com/gitlab-org/gitlab-ee/issues/6061 -> https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/6051
- PoC CI view issue, https://gitlab.com/gitlab-org/gitlab-ee/issues/6278
- Capture Selenium Artifact during CI, https://gitlab.com/gitlab-org/gitlab-ee/issues/6178
gitlab-ce
Fork -
Create a personalgitlab-ce
fork to test with- Might not be necessary
Run GitLab rspec tests through the GitLab Selenium Server/proxy
-
Run GitLab rspec tests through the GitLab Selenium Server/proxy -
Run GitLab rspec tests through the GitLab Selenium Server/proxy by default and on GitLab CI
Fingerprint each test
-
Fingerprint each test by their nested describe
/it
name so we can easily comparemaster
to the feature branch- We will probably need to send a request to the GitLab Selenium server/proxy in a before all hook with the test name
- Capyabara/rspec shares a Selenium session across multiple tests.
- See https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/17672
spec/support/capybara.rb
RSpec.configure do |config|
config.before(:example, :js) do |example|
# POST "/session/#{Capybara.current_session.driver.browser.capabilities['webdriver.remote.sessionid']}/gitlab-meta" with name of spec
#req.body = {
# description: example.metadata[:full_description],
# location: example.metadata[:location],
#}.to_json
end
end
This would add a log entry in selenium-logs.json
{
"gitlab": [
{
"id": "a10d9a192e-1520453590211",
"type": null,
"path": "/session/c69eb6fd-8df2-4d9c-a704-dea48fa921e3/gitlab-meta"
"info": {
"specName": "Create notes on issues mentioning issue on a private project"
}
},
],
...
}
Screenshot diffing
-
With the combined results, diff images from the latest master
build with the MR and create some sort of report - https://github.com/wswebcreation/wdio-image-comparison-service
Flesh out GitLab Selenium proxy/server
After we have a PoC for all of the above points, we need to come back to the GitLab Selenium proxy and flesh out all of the aspects and make sure things are properly tested.
-
Add more screenshot trigger criteria, #4 -
Run against someone else's existing project Selenium tests - QA sanity test, #2 -
Ensure everything has test coverage
CI views
-
Create a job-level CI view for Selenium screenshots, https://gitlab.com/gitlab-org/gitlab-ce/issues/37791 -
Create a PoC CI view for Selenium screenshots, https://gitlab.com/gitlab-org/gitlab-ee/issues/6278 -> https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/6051
-
-
Create a pipeline-level CI view for Selenium screenshots - Create
selenium
job that collects and combines allselenium/
artifacts across all other jobs into it's own artifacts- For reference, see
update-tests-metadata
, example. This job already combinesknapsack/gitlab-ee/rspec-pg_node_xx_xx_report.json
across jobs intoknapsack/gitlab-ee/rspec_report-master.json
. Seescripts/merge-reports
- For reference, see
- Create
-
Create a pipeline-level CI view for visual regression from selenium screenshots CI view - Here is the general issue around CI views, https://gitlab.com/gitlab-org/gitlab-ce/issues/35379
Auto DevOps
-
During the Auto DevOps test stage (where Selenium tests would run) - Startup GitLab Selenium Proxy (and if necessary backing Selenium server/chromedriver)
- Define
SELENIUM_REMOTE_URL: http://localhost:4545/wd/hub
(points at GitLab Selenium Proxy) - Define
GITLAB_TARGET_SELENIUM_REMOTE_URL: http://localhost:4444/wd/hub
(points at backing Selenium server)
-
It would make things a lot more slick if we had Docker networks connected between services, gitlab-runner!1041 (closed)
Edited by Eric Eastwood