DISCUSS: Improvements to Gitlab QA, simplify on-boarding for new Quality / Test Automation engineers

First of all, I wanted to say that the team did a great job implementing the framework. After following instructions and getting things up I was able to run most the tests and getting 18 examples, 3 failures on my first try.

These are the notes that I have written down after going through all the projects while on-boarding. Meaning the 4 projects to get productive with GitLab QA test development.

  • GDK
  • GitLab QA
  • GitLab CE
  • GitLab EE

The following are items that we should consider at the minimum for the near term.

  1. We don't have a place that documents all the test scenarios that can be run on a given instance.
    • We have Local Instance (seeded via GDK) / dev / staging / canary / production
    • I assume we can't run all the test suites against all of these instances. The local instance being the smallest set and as we move up to production environments more and more test can be run. e.g. LDAP, Email notification, Upgrade, Github integration and etc.
  2. Running gitlab-qa Test::Instance::Any CE|EE nightly|latest http://your.instance.gitlab on a local GDK instance does not work.
  • I was running ~/gitlab/gitlab-org/gitlab-qa./bin/qa Test::Instance::Any CE Latest http://localhost:3001 This seems to be spinning up a docker instance then fails. I didn't fully setup docker yet but I was expecting the test to run on my GDK instance with that address. Docker shell command: `docker run -t --rm --net=bridge --volume /var/run/docker.sock:/var/run/docker.sock:z --volume /tmp/gitlab-qa/screenshots:/home/qa/tmp:z --name gitlab-specs-1523057037 gitlab/gitlab-ce-qa:Latest Test::Instance http://localhost:3001` Unable to find image 'gitlab/gitlab-ce-qa:Latest' locally docker: Error response from daemon: manifest for gitlab/gitlab-ce-qa:Latest not found.
  1. We need more clarity on what parameters are needed and optional. This needs to be part of the CLI behavior. Most of the time the users get a generic output like the one below.
/Users/janedoe/.rvm/gems/ruby-2.2.1/gems/gitlab-qa-0.5.0/exe/gitlab-qa:6:in `const_get': no implicit conversion of nil into String (TypeError)
...
/Users/janedoe/.rvm/gems/ruby-2.2.1/gems/gitlab-qa-0.5.0/lib/gitlab/qa/release.rb:67: syntax error, unexpected '.'(SyntaxError) ...ch(CUSTOM_GITLAB_IMAGE_REGEX)&.[](:tag) || DEFAULT_TAG

We should ensure the user can be self-sufficient and can get past this basic first hurdle. Updating the output of the CLI command can help with this. The scenarios can include

  • no arguments
  • wrong GitLab param instance used (EE|CE|Latest|Any)
  1. In addition to using GitLab QA via gem install, it seems like we can also use it locally with bin/setup and bin/qa. However, I found out that the commands are not equal. This is a bit confusing. We should make this command consistent.
    • In the CE project we can execute ./bin/qa Test::Instance http://localhost:3001. *, For example, I was running ~/gitlab/gitlab-org/gitlab-ce/qa ./bin/qa Test::Instance http://localhost:3001 and I was able to get some tests to spin up.
    • However running this via GitLab QA fails
      • ~/gitlab/gitlab-org/gitlab-qa ./bin/qa Test::Instance http://localhost:3001 returns ./bin/qa:8:in <main>: undefined method perform for Gitlab::QA::Scenario::Test::Instance:Module (NoMethodError)
    • There is also an issue requesting us to improve the documentation, we should try to define whats the MVP we can improve and complete this as part of the Q2 OKR
  2. On-boarding should point to both https://gitlab.com/gitlab-org/gitlab-ce/tree/master/qa and also the readme in the GitLab QA project. I made the mistake of going straight to GitLab QA first and had to go back to read about GDK and QA dirs in the main projects. Ideally, we should just have a dedicated onboarding page that lists the proper onboarding sequence.
  1. Running selenium tests with browser driver installation, we should add homebrew install chromedriver to the steps and have this be included in the install. Going through https://sites.google.com/a/chromium.org/chromedriver/getting-started and having our users setup multiple paths is not ideal. We should be opinionated on how to install and make this as easy as possible.
  • Also we might want to consider using a ready-made package with a local selenium server. I used to use https://www.npmjs.com/package/selenium-standalone This took care of all the browsers driver installation (chrome,safari,firefox,etc) The framework just needs to communicate via json wire protocol to http://localhost:4444/wd/hub and it simplifies the framework. Since this is the same interface as Browserstack and Saucelabs.
  1. At first start, it felt like there are many components to gitlab qa and it seems complex. I like that we are trying to reduce overall complexity. I see that we have #71 (closed) which is a good starting point.
  • We should try to solicit feedback from external users as much as we can. E.g. FE teams and other BE teams (Frontend & Backend)
  1. Documenting which users to use on which environment explicitly
  • In the CE readme we default to root this is where I made the most progress
  • We also have seeded users in https://gitlab.com/gitlab-org/gitlab-ce/blob/master/db/fixtures/development/05_users.rb
    • I tried using user#1 / 12345678 but the test does not run and the results are not the same as using root
  • We can also use our own personal users? This will not work if its not setup before hand.
    • Are there steps to create personal users on staging and other envs during on-boarding? Should we add a step for this?
  • Ideally we would want to work towards a shared test data model that can also test data integrity (e.g. these projects with these users should not change no matter how many migration scripts have run / a static MR should never change state or timestamp) The data should be the same on all environments so we can run a given minimum set of regressions tests in all environments.

  1. Let's consider implementing a HTML summary report. The command line test report and headless is great but when we start climbing to 100+ tests this is unwieldy. This also helps with triaging failure in CI. See the basic example below (this was done via html templating engine and a basic css file)Screen_Shot_2018-04-06_at_4.50.03_PM

    • Each suite is expandable
    • Each failed test has its own result page
    • Each failed scenario / assertion is catagorized (failed assertion or failed test infrastructure)
    • Screen_Shot_2018-04-06_at_4.50.03_PM

  2. Document and implement selenium helpers for more advance actions. Goal here is to make the contributing of more complex scenarios and building the page objects for those as easy as possible.

  • Drag & drop
  • Scroll to (element / top / bottom) #107 (closed)
  • Get a list of records from a list view
  • Get a 2D array data from a table view
  • Finding an item from a list, test develop can call this and assert on the result.

This is important because engineers in the Edge/Quality team will need to juggle multiple GitLab projects as their standard toolsets, simplifying and making this easy to understand is crucial for the team (and new team members, we have an OKR to hire 2 engineers in Q2!) I have started on a few of these around GDK setup. But I am barely scratching the surface.

I would like to use this issue as a starting point to some discussions brainstorming. Then let's prioritize and branching off with creating/linking issues to get things moving. Comments appreciated!

cc/ @stanhu @bjgopinath @rymai @rspeicher @chriscool @godfat @grzesiek

Edited by Mek Stittri