A feature test to validate QA selectors

Background

This is a follow up from #58 (closed) and an alternative approach to https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/16109

We want to have qa:selectors test which could detect if there's any view changes which would break QA tests for every CE/EE pipeline, because this test should be cheap while QA tests are expensive which we don't want to run for each CE/EE pipeline.

This is like a lint for QA tests.

The problem

During the implementation of https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/16464, I realized that we even need to text or regexp matches against Vue templates, which makes me feel this is going to be a mess in the future.

We currently ended up with:

view 'app/assets/javascripts/deploy_keys/components/key.vue' do
  element :key_title, /class=".*title.*"/
  element :key_title_field, '{{ deployKey.title }}'
end

Where we're trying to match against:

<strong class="title">
  {{ deployKey.title }}
</strong>

This is going to be inaccurate because the following template would also pass:

<strong class="title">
  I am just another title
</strong>

<div class="keyTitle">
  {{ deployKey.title }}
</div>

However the QA test would certainly fail because the structure is changed.

Feature tests

I think the underlying problem is the dispersion between Capybara and regular expression match. We could solve the dispersion by using Capybara directly, without involving any QA test.

To do so, we could define a map between URLs and selectors, and have a feature test which visits each URLs and try to run the selectors. It just need to verify it could find something, and it doesn't have to really check anything.

The code could look like:

class DeployKeys < Page::Base
  verify_selectors do
    let(:deploy_key) { create(:deploy_key, project: project) }
    let(:namespace) { create(name: 'namepsace') }
    let(:project) { create(:project, name: 'project', namespace: namespace) }

    path { "/#{namespace.name}/#{project.name}/settings/repository" }

    check do
      expand_deploy_keys # how to implement this?

      find_key_title
      find_key_value
      find_add_key
      has_key_title?(deploy_key.name)
    end
  end

  def fill_key_title(title)
    find_key_title.fill(with: title)
  end

  def fill_key_value(key)
    fill_key_value.fill(with: key)
  end

  def add_key
    find_add_key.click
  end

  def has_key_title?(title)
    page.within('.deploy-keys') do
      page.find('.title', text: title)
    end
  end

  private

  def find_key_title
    find('deploy_key_title')
  end

  def find_key_value
    find('deploy_key_key')
  end

  def find_add_key
    find('Add key')
  end
end

Of course, this is just a rough idea. I am not familiar with Capybara to know if this would work, and there are a number of other questions:

  • Where are we going to run those codes? It should be in CE/EE feature tests, but we're writing the code in QA, which seems wrong. If this is ever going to work, the feature tests would need to load QA codes
  • It's like writing another test, which is too much work

The idea behind that is trying to extract all the possible selectors we're going to use, and try to run them in a reasonable and fast way.

Related issues/discussions

/cc @mkozono @brodock @grzesiek @rymai