UI Tests – How do we keep them from being brittle?

There has been a lot of talk about the value of tests that drive the user interface. Some people in the industry that I have a lot of respect for have gone as far as to say that you should not create these type of tests. You should instead create tests that access the application code directly under the UI. Part of their reasoning is that tests that hit the UI are brittle and therefore high maintenance.

And yet UI tests are very valuable. When a user describes the behavior of a system they usually do so in terms of the user interface. Also, acceptance tests that provide examples of behavior for a story almost always present this behavior from the end users point of view.

This post is the first in a series that will describe the techniques I use to make my UI tests agile. These techniques have been developed while coaching teams in very diverse technologies and platforms and as a result are “field tested”. The tools I will use for all of the examples are ruby and cucumber.

The application under test

We’ll be writing scripts against the depot application. It is a simple rails application that accompanies the wonderful book titled Agile Web Development with Rails. If you don’t own this book you should go buy it now. Trust me, it is good!

The depot application is a web based storefront in which you can purchase books. You will not see any of the code from the depot application in this article. Instead, you will see my cucumber features that test some of the functionality of the bookstore.

What’s in your shopping cart

In the depot application you are presented with a list of books that can add to your shopping cart. We will begin by writing tests for the shopping cart page. Here is an example of the cart with three books in it:

Shopping Cart

Shopping cart from depot application

As you can see, the shopping cart has a quantity, description, unit price, and a subtotal for each book. It also has a total for the cart. Here are three simple Scenarios that test the page:


Feature: Purchasing books

  Background:
    Given I am on the shopping page

  Scenario: Verify cart with one book
    When I purchase "Pragmatic Project Automation"
    Then I should see "1" in the quantity for line "1"
    And I should see "Pragmatic Project Automation" in the description for line "1"
    And I should see "29.95" in the each for line "1"
    And I should see "29.95" in the total for line "1"
    And I should see "29.95" in the cart total

  Scenario: verify cart with two books
    When I purchase "Pragmatic Project Automation"
    And I continue shopping
    And I purchase "Pragmatic Project Automation"
    Then I should see "2" in the quantity for line "1"
    And I should see "Pragmatic Project Automation" in the description for line "1"
    And I should see "29.95" in the each for line "1"
    And I should see "59.90" in the total for line "1"
    And I should see "59.90" in the cart total

  Scenario: verify cart with two different books
    When I purchase "Pragmatic Project Automation"
    And I continue shopping
    And I purchase "Pragmatic Version Control"
    Then I should see "1" in the quantity for line "1"
    And I should see "Pragmatic Project Automation" in the description for line "1"
    And I should see "29.95" in the each for line "1"
    And I should see "29.95" in the total for line "1"
    And I should see "1" in the quantity for line "2"
    And I should see "Pragmatic Version Control" in the description for line "2"
    And I should see "28.50" in the each for line "2"
    And I should see "28.50" in the total for line "2"
    And I should see "58.45" in the cart total
 

Here are the corresponding step definitions:

  
BOOKS = {
  "Pragmatic Project Automation" => 1,
  "Pragmatic Unit Testing" => 2,
  "Pragmatic Version Control" => 3
}

def cart_item(row, column)
  @browser.table(:index=>1)[2+row][column].text
end

Given /^I am on the shopping page$/ do
  @browser.goto 'http://localhost:3000/store'
end

When /^I purchase "([^\"]*)"$/ do |book|
  @browser.button(:value=>"Add to Cart", :index=>BOOKS[book]).click
end

When /^I continue shopping$/ do
  @browser.link(:text => 'Continue shopping').click
end

Then /^I should see "([^\"]*)" in the quantity for line "([^\"]*)"$/ do |quantity, line|
    cart_item(line.to_i, 1).should == quantity
end

Then /^I should see "([^\"]*)" in the description for line "([^\"]*)"$/ do |desc, line|
    cart_item(line.to_i, 2).should == desc
end

Then /^I should see "([^\"]*)" in the each for line "([^\"]*)"$/ do |each, line|
    cart_item(line.to_i, 3).should == "$#{each}"
end

Then /^I should see "([^\"]*)" in the total for line "([^\"]*)"$/ do |total, line|
    cart_item(line.to_i, 4).should == "$#{total}"
end

Then /^I should see "([^\"]*)" in the cart total$/ do |total|
  @browser.cell(:class=>'total-cell').text.should == "$#{total}"
end

With Watir you access the elements in a table using rows and columns. The cart_item method that begins on line seven provides a convenient way of getting the text for a row/column. On line eight I am adding 2 to the row to account for the two row header on the table.

The verification methods on line 23 through 37 use the cart_item method to get the value from the table data and compare it with the expected result. Each method knows its’ column and uses the line passed in from the feature to identify the row. The method beginning on line 39 that checks the total amount of the cart is able to access the cell directly since the page provides a unique class.

What’s wrong with this approach?

Whenever I speak about testing at conferences and user groups I usually ask the attendees how many have seen test automation efforts fail. Usually over half of the attendees raise their hand. As I probe further asking why the effort failed the top answer is brittle tests that take too much effort to maintain.

Historically a lot of test automation efforts have used record/playback tools. The problem with this category of tool is that they tightly couple the activities of the tests to the application under test. Whenever the application changes many tests fail and the tester usually has to change a lot of code in a lot of places. In my opinion it is fine for a lot of tests to fail when the application changes but it is not okay to have to make changes in a lot of places. A tester should have to change the test code in one place only and it should be obvious where to make that change.

From what I have seen, the way most people write cucumber step definitions suffers from the same problems as the record/playback approach to automation. If you look at many of the examples of writing cukes on the web they show webrat code in the step definitions accessing the elements on the page directly. While this might work on a small project when you scale it up to a large project it falls apart for the same reasons mentioned above. The examples I have above are fairly simple and yet if a designer decides to change the layout of the page the tester will have to look through all of the code to see what is effected. While this might be okay for a small project imagine a project where there are over one hundred features and many hundreds of steps.

In software development we already have solutions of this type of tight coupling. It is called encapsulation and abstraction layers. What would happen if we try to apply these design techniques to our tests?

Enter Page Objects

The first concept I’d like to introduce is called a Page Object. Simply put, it is a class that holds all of the details about the elements on a web page. Here is a page object representing the shopping cart page above:

  
class ShoppingCartPage

  QUANTITY_COLUMN = 1
  DESCRIPTION_COLUMN = 2
  EACH_COLUMN = 3
  TOTAL_COLUMN = 4
  HEADER_OFFSET = 2

  def initialize(browser)
    @browser = browser
  end

  def checkout
    @browser.link(:text => 'Checkout').click
  end

  def continue_shopping
    @browser.link(:text => 'Continue shopping').click
  end

  def cart_total
    @browser.cell(:class => 'total-cell').text
  end

  def quantity_for_line(line_number)
    cart_data_for_line(line_number)[QUANTITY_COLUMN].text
  end

  def description_for_line(line_number)
    cart_data_for_line(line_number)[DESCRIPTION_COLUMN].text
  end

  def each_for_line(line_number)
    cart_data_for_line(line_number)[EACH_COLUMN].text
  end

  def total_for_line(line_number)
    cart_data_for_line(line_number)[TOTAL_COLUMN].text
  end

  private

  def cart_data_for_line(line_number)
    @browser.table(:index => 1)[HEADER_OFFSET+line_number]
  end
end

While we’re at it let’s create a page object for the catalog page as well:

  
class CatalogPage

  BOOK_MAPPING = {
      "Pragmatic Project Automation" => 1,
      "Pragmatic Unit Testing (C#)" => 2,
      "Pragmatic Version Control" => 3
  }

  def initialize(browser)
    @browser = browser
  end

  def visit
    @browser.goto 'http://localhost:3000/store'
  end

  def purchase_book(name = 'Pragmatic Project Automation')
    @browser.button(:value => "Add to Cart", :index => BOOK_MAPPING[name]).click
  end
end

And with these changes our step definitions now look like this:

  
Given /^I am on the shopping page$/ do
  @catalog = CatalogPage.new(@browser)
  @catalog.visit
end

When /^I purchase "([^\"]*)"$/ do |book|
  @catalog.purchase_book(book)
  @shopping_cart = ShoppingCartPage.new(@browser)
end

When /^I continue shopping$/ do
  @shopping_cart.continue_shopping
end

Then /^I should see "([^\"]*)" in the quantity for line "([^\"]*)"$/ do |quantity, line|
  @shopping_cart.quantity_for_line(line.to_i).should == quantity
end

Then /^I should see "([^\"]*)" in the description for line "([^\"]*)"$/ do |desc, line|
  @shopping_cart.description_for_line(line.to_i).should == desc
end

Then /^I should see "([^\"]*)" in the each for line "([^\"]*)"$/ do |each, line|
  @shopping_cart.each_for_line(line.to_i).should == "$#{each}"
end

Then /^I should see "([^\"]*)" in the total for line "([^\"]*)"$/ do |total, line|
  @shopping_cart.total_for_line(line.to_i).should == "$#{total}"
end

Then /^I should see "([^\"]*)" in the cart total$/ do |total|
  @shopping_cart.cart_total.should == "$#{total}"
end

Why Page Objects?

Wow, that is a lot more code than the original example. Why would somebody go through the extra effort to write this code?

In a small example like the one we have in this post it is hard to see the benefit of this additional work. In a larger application to benefit becomes obvious. If a designer changes the order of the columns on the shopping cart page one would only need to change the corresponding constants at the top of the class. If a developer changes the header so it now takes up three rows of the table the tester would only need to change the corresponding constant. If a designer decided to eliminate the table altogether and replace it with divs the tester would only need to make changes in the shopping cart page.

In all of these cases changes are localized and the tester knows where to go to make the changes. After you are familiar with the page object pattern you can add them fairly quickly. In my experience the savings you get over the lifetime of a project are significantly greater than the costs of creating these early on.

Going forward

As I mentioned earlier, this is the first entry in a series of posts on the topic of making robust UI tests. Please check back here weekly to see new ways you can enhance your tests and make them less brittle. We will continue to enhance our page objects to make them even simpler as well as apply patterns to our features to make them more useful to the whole team.

If you want to follow along and write the code from the blog entries you can download a starter project here. Just unzip the file and follow along by adding each of the examples included in the posts. The finished code can be found here.

Update: The code for this series is now available in github. You can access it here. There is a branch for each checkpoint in this series with the latest on master.

25 thoughts on “UI Tests – How do we keep them from being brittle?

  1. Pingback: Tweets that mention UI Tests – How do we keep them from being brittle? | CheezyWorld -- Topsy.com

  2. Completely agree with you that Page Object pattern is really very helpful and may reduce pain of support UI functional tests. This concept can be extended even more to Widget Object, because pages consist of similar components: data grids, forms, lists, etc. The main goal is to create and support application level DSL (domain specific language). Initially it is hard task, but after some time, when all main components are created and application specific issues are solved, adding new tests or building new pages become easy task. Very important note here that using dynamic languages makes code more compact and readable. That is why I suggest to use Groovy or Ruby for functional UI tests.

    Page Object pattern may be used with every testing tool, but some tools go even further and automate page objects building. One of such tools is Tellurium. It is based on the Page Object concept and has Firefox plug-in for automated generation. You just run it and click elements of the page, then edit gathered data in the special UI forms and Groovy class for the page is generated. Only operations available for page should be added manually. It also has ability to group elements and generate locators on the fly for you. Execution part of Tellurium is based on Selenium, so it may be used with any browser.

    Thank you for sharing your experience! Waiting for next parts.

  3. Mikalai, please check back here over the next few weeks. This is the first of six posts that will continue to build out the testing framework as well as introduce a DSL I developed while working with a few teams. I hope you find the future posts as interesting and helpful.

  4. Pingback: Code coverage with unit & integration tests « Actively Lazy

  5. Pingback: UI Tests – Part Two | CheezyWorld

  6. Pingback: A Smattering of Selenium #32 « Official Selenium Blog

  7. Pingback: UI Tests – Introducing a simple DSL | CheezyWorld

    • This is the first of a series of six posts on this topic. The second and third are already posted on this site. Hopefully there will be plenty for you to read and enjoy.

  8. Pingback: UI Tests – Default Data | CheezyWorld

  9. Pingback: UI Tests – putting it all together | CheezyWorld

  10. Pingback: Transforming My Cukes | CheezyWorld

  11. I’ve learned a lot from reading this series. Thank you so much!

    But (had to be one, didn’t it ;)), if you use the PageObject to encapsulate the interaction to each page – will not that make you write a lot of similar step definitions such as:
    – Given I am on the home page
    – Then I should be on the home page
    – When I visit the home page
    – Given I am on the customer page
    – Then I should be on the customer page
    – When I visit the customer page
    – etc.

    I love the PageObject concepts but I cannot see how to get around this repeating of step definitions. Before I wrote one step definition:
    “I am on the (.*) page”
    and navigated with my browser emulator based on the sent-in name.

    But that is not good either since I then hardwire the name of the page to the navigation.

    So my question is; is repeating the step definitions like above not so bad or should I do it some other way?

    • I do not try to reuse the “Given I am on the Home page” steps. Instead I use these steps as the place to create my page objects. The step definitions are typically two lines long:

      @home_page = HomePage.new(@browser)
      @home_page.visit

      In some cases you have to travel through a page or two to get to where you wish to test. In those cases I like to have a step like “Given I have traversed to the Customer Address page” and then use that step to submit all preceding pages with default data so I end up on the correct page. The step definition might look like this:

      home_page = HomePage.new(@browser)
      detail_page = home_page.goto_customer_detail
      @customer_address_page = detail_page.provide_customer_details

      I do not typically have steps like “Then I should be on the Home page” Instead I focus on the actions that took me there. I structure them so they will fail if I do not end up on the appropriate page. If I am landing on a page and displaying some message then I think it is reasonable to verify the message is displayed correctly.

      Hope this helps.

  12. Pingback: marcusoft.net: Clean up your steps–use page objects in SpecFlow step definitions

  13. Hello i am very interested in follow your blog, but first of all i need you to tell me what is necesary as a prerequisite, to begin with this blog?

    • There really are no prerequisites for reading this blog. Many of the posts on this blog are related to automated testing using Ruby and Cucumber. If you do not have this things installed I would start there. I have a post that covers that but it is over a year old but probably still works.

      I will be releasing several chapters from a book I am working on shortly. If you are completely new to all of this perhaps the book would be a good place to start.

  14. Pingback: ASP.NET MVC测试方法与技巧 | chainding

  15. Hello I am very much interested in learning Cucumber. I am new in automation testing so don’t know much about the testing tools. I have installed the following gems on my Linux machine. (I don’t have Windows environment)

    • Installing Rails
    • Installing Ruby
    • Cucumber
    • Capybara
    • Nokogiri
    • Ruby debugger

    If I am missing some gem then please let me know. Now I want to take start from Page objects and I really need your help here. Please guide me what should I do first?

    Let’s suppose I want to automate this test-case using Page Objects, then what things I required here to automate this test-case.

    Feature: Sign-in to Gmail

    Background:
    Given I am on the Gmail

    Scenario: To sign-in to the Gmail
    AND I enter the Username
    AND I enter the password
    Then I click on the Sign in button

  16. Could you please explain me how to test scrolling(Scroll down vertically) functionality in a pop up window.

    Thanks.

  17. There is a typo: the variable names don’t match:

    When /^I purchase “([^\"]*)”$/ do |book|
    @browser.button(:value=>”Add to Cart”, :index=>BOOKS[name]).click
    end

Leave a Reply

Your email address will not be published. Required fields are marked *

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>