Make assertions on e2e tests more specific than the complete page
In many cases, in the browser_ui
tests, expectations are based on the complete web page, instead of on more specific components or elements.
E.g., If you expect that a certain text is displayed in the top navbar, the assertion could explicitly inform that. It could be something like expect(page.header.navbar).to have_content('some expected content')
.
Instead, what we do in many test cases is something like this: expect(page).to have_content('some expected comment)
.
Problems with too general assertions
There are problems associated with making assertions too general. Some that I can think of now are:
- In the second example above, if the text 'some expected content' is displayed in the footer, that test still would pass, but this would be a false positive since there is a real bug, because a content that should be in the top navbar is actually in the footer.
- It's difficult, if not impossible, to say exactly where you expected that particular text to be displayed if you don't explicitly tell the web driver to do so.
Advantages of more specific assertions
By making more specific assertions we gain:
- More reliable tests since we avoid the above mentioned false positive results.
- Tests easier to debug during failures.
- The ability to evolve the test framework to be less coupled (with smaller page objects composed by small and reusable components).
Proposal
For all the GUI tests, we could follow an architecture where page objects are simple representations of web pages, and they have no more information than their relative URLs, and the components that they are composed by. Components are then responsible for the definition of elements and their methods, but because they are components, they are smaller, easier to maintain, reuse, etc. This would simplify making assertions more specific.
An example where we are already using an approach close to the proposed one, except for the assertions, is the [SubMenus](https://gitlab.com/gitlab-org/gitlab-ce/tree/master/qa/qa/page/project/sub_menus)
module of the Project
page object. Every file in this directory is what I'm calling a component.
In the [Menu](https://gitlab.com/gitlab-org/gitlab-ce/blob/master/qa/qa/page/project/menu.rb)
page object, you can see the usage of the SubMenus
module.
Cc @gl-quality