Running your Cukes in Jenkins

One question that I am often asked is “How do you run your cucumber scripts?”. This question usually leads to a discussion about what process and software I use to run my features in a regression-like fashion in a team setting. The questioners are usually not interested in how a developer might use autotest (my local tool of choice) to run the cuke/spec loop. They’re not interested in how a developer might run a feature on their local machine to verify they have completed a card. They’re also not interested in how a developer or tester might run the entire suite of features to verify everything still works. They want to know how to schedule the execution of the entire suite of features.

The truth is that I don’t run them. Instead I have a server process run the features for me. I am a strong advocate of having the continuous integration server run the acceptance tests continuously. This post will explain how I do it and hopefully provide you the information you need to do it as well.

Let’s be clear

Before I go into the details let me be clear. I am not proposing that this is the only way you and your team should run your features. I am a proponent of developers running the feature they are working on all of the time (perhaps continuously) and I think testers should run them on their computer very frequently as well. This post is about how I like to run the entire suite of features to form a regression. I’ll start by introducing how I control the running of the features and setup my projects. Next I’ll discuss how this works with Jenkins.

cucumber.yml

First of all I have a cucumber.yml file at the root of my project structure. I use this file to define a set of profiles (a grouping of command-line arguments) that I can easily use when I want to run my features. Cucumber already has a default profile that will be used unless you override it. Here is what that might look like:


default:  --no-source --format pretty --tags ~@not_ready

Everything after the default: label in the entry will be passed to cucumber as arguments when it runs.

You can use profiles to define separate definitions of how you want to run your features. For example, if you have some very slow tests you might want to separate them out into their own run. You would start by placing a @slow tag on those features or scenarios and then create a few profiles like this:


default:  --no-source --format pretty --tags ~@not_ready
slow:  --no-source --format pretty --tags @slow --tags ~@not_ready
fast:  --no-source --format pretty --tags @~slow, ~@not_ready

To use the profiles you simply pass the profile name on the command line. If you wish to run the fast tests you would run:


cucumber -p fast

Rakefile

Rake is a tool that has the ability to execute tasks. Many tools, including cucumber, are distributed with pre-built rake tasks that can be used to control their execution. Jenkins has a plugin that can execute Rakefiles. We use this combination to control the running of our features in Jenkins. We will discuss how to setup Jenkins in the next section. Here we’ll discuss how to write a simple Rakefile. Let’s start by looking at a simple example.

require ‘cucumber’
require ‘cucumber/rake/task’

Cucumber::Rake::Task.new(:features) do |t|
  t.profile = ‘ci’
end

task :default => :features

This rake script creates a new task named features and has it call the ci profile. Next it makes the features task the default task. You can execute rake in the project directory and it will run the features task (the default) using the ci profile created in a cucumber.yml file. We use the Rakefile to specify which profile to run and we use the cucumber.yml file to detail out the profiles.

A more sophisticated Rakefile might create a namespace to contain several tasks.

require ‘cucumber’
require ‘cucumber/rake/task’

namespace :features do
  Cucumber::Rake::Task.new(:fast) do |t|
    t.profile = ‘fast’
  end

  Cucumber::Rake::Task.new(:slow) do |t|
    t.profile = ‘slow’
  end

  task :ci => [:fast, :slow]
end

task :default => :fast

This Rakefile creates a namespace named features, which contains three tasks (fast, slow, and ci). The ci task runs the fast task followed by the slow task. To execute the slow tests you would execute


rake features:slow.

If you want to see what tasks are available from the rake tool you can execute the rake –T command.

Jenkins

The Jenkins configuration couldn’t be simpler. The first thing you need to do is install the Jenkins Rake plugin. Once this is installed you should go to the Configure System page. You will see a new section appear on this page.

It is likely that Jenkins discovered your ruby installation already but if it didn’t, just enter the path to the location where ruby is installed.

Next you need to create the project to run your cukes. Complete the form as you usually would. In the Build section select the Add build step button and then select Invoke Rake. This will open this dialog.

In this section you simply need to type the task you wish to run.

Guidelines I use to run my cukes

How often should you run your features? In my opinion, you should try to run them as close to all of the time as possible. Each time they run they should execute against the latest version of the application.

If it takes 2 hours 30 minutes to run the tests I would schedule them to run every 3 hours. If you have some slow tests you might want to schedule them to run 4 times a day and have the fast test run every hour.

The goal here is to get fast feedback.

17 thoughts on “Running your Cukes in Jenkins

  1. What about cukes that drive a browser (e.g. those that have the @javascript tag)? In that case, I assume that you need to have the CI server running on something the capability of displaying a browser window, correct? Or is there some other cool way that I’m missing?

    • Having a server that can display a browser is one approach. Another approach would be to run your tests headless. There are two ways to do this. The first is to use the remote driver that is a part of webdriver. The second is to use a real browser with a virtual X server. If you use the second approach you’ll want to check out the ‘headless’ gem.

    • We’re running our tests headless on every deploy, and running in a browser (Firefox) once in the early morning (5 am). The advantage to running a browser, is you can save screenshots of failed tests which can help the testers and developers quickly see what has gone wrong. However, since you’re running in a browser, it takes a much longer time (3x longer), so we only take that hit once per day.

      We are running our CI on a Windows server.

    • We run our in-browser Cucumber tests on a Jenkins CI server. We use Cuke4Duke to write the Cucumber tests in Java. We do our browser automation with Selenium 2. The tests are run using an Ant target (we use Ant to do the rest of our build).

      The trick is not running your Jenkins master (or slave) as a service on Windows – just run it from a command line so the browsers runs as normal. You could also use a Selenium 2 grid.

      The great thing is any developer can run “ant ci” on their local machine and do exactly what the Jenkins server would do.

  2. Pingback: A Smattering of Selenium #58 « Official Selenium Blog

  3. Hi all! I’m running a project using Cucumber (with page_object gem), Watir and Jenkins. Right now we’re looking the best way to parallelize test to reduce testing time on multiple virtual machines, with different navigators and so on.

    There is two approaches: to use a tool like parallel_test, Hydra, TestJour or TestBot or organize everything through Jenkins, using paths, tags, etc.

    In order to put effort on the right way… do you consider Jenkins is a good option to make parallel test on multiple machines or should i give a chance to another tool? Suggestions are welcome :)

    Thanks!

    • I don’t use an to run my cukes. Instead, I use rake and the rake jenkins plugin. I have an example of that in the blog post above.

  4. Hi Cheezy,

    Big fan of your blogs and books. High fives for all your hard work and sharing your knowledge with us.

    I am currently working for a client and I don’t have the necessary ‘access rights’ to get the jenkins rake plugin. Do we need the rake plugin for Jenkins? Cant I just simply run rake:task_name within the execute shell command?

  5. Hi all, I am passing different environments at the start of each test or rake and I have a Scenario where I have to hard code 2 different url’s which relate to two different environments. I am trying to pass the environment which I stated at the start of the test. e.g. cucumber test.feature ENVIRONMENT=”uat”. I am trying to do this using if statements. I want to do the following:

    sudo

    if ENVIRONMENT=”uat”
    then
    @browser.link(:id, “test”).click

    elsif
    ENVIRONMENT=”sprint”
    @browser.link(:id, “testing”).click

    end

    end

    my env file passes the following:

    ENV[‘@base_url’] = TRUNK_URL
    @base_url = TRUNK_URL

    case ENV[‘ENVIRONMENT’]

    when “live”
    ENV[‘@base_url’] = LIVE_URL
    @base_url = LIVE_URL
    when “trunk”
    ENV[‘@base_url’] = TRUNK_URL
    @base_url = TRUNK_URL
    when “sprint”
    ENV[‘@base_url’] = SPRINT_URL
    @base_url = SPRINT_URL
    when “dev”
    ENV[‘@base_url’] = DEV_URL
    @base_url = DEV_URL

    end

    • Jamie,

      Take a look at the FigNewton gem. It was designed to solve this type of problem. Let me know if you need help getting it setup.

      -Cheezy

  6. This fails for me. Do we also need cucumber plugin here along with the Rake plugin here to run the cucumber profiles that are defined in my cucumber.yml? Here is an error I get. It seems like either my project/Rake Build definition is wrong or it has issues finding and running the Rakefile. ‘functest’ is the name of my task. Any pointers will help! Thx Cuke.
    ——
    Rake
    [workspace] $ /home/test/.rvm/rubies/ruby-2.0.0-p0/bin/rake –trace -f /home/jenkins/jobs/test/workspace/src/test/webdriver/Rakefile functest
    rake aborted!
    Don’t know how to build task ‘functest’
    /home/jenkins/.rvm/gems/ruby-2.0.0-p247@global/gems/rake-10.1.0/lib/rake/task_manager.rb:49:in `[]’
    /home/jenkins/.rvm/gems/ruby-2.0.0-p247@global/gems/rake-10.1.0/lib/rake/application.rb:148:in `invoke_task’
    /home/jenkins/.rvm/gems/ruby-2.0.0-p247@global/gems/rake-10.1.0/lib/rake/application.rb:106:in `block (2 levels) in top_level’
    /home/jenkins/.rvm/gems/ruby-2.0.0-p247@global/gems/rake-10.1.0/lib/rake/application.rb:106:in `each’
    /home/jenkins/.rvm/gems/ruby-2.0.0-p247@global/gems/rake-10.1.0/lib/rake/application.rb:106:in `block in top_level’
    /home/jenkins/.rvm/gems/ruby-2.0.0-p247@global/gems/rake-10.1.0/lib/rake/application.rb:115:in `run_with_threads’
    /home/jenkins/.rvm/gems/ruby-2.0.0-p247@global/gems/rake-10.1.0/lib/rake/application.rb:100:in `top_level’
    /home/jenkins/.rvm/gems/ruby-2.0.0-p247@global/gems/rake-10.1.0/lib/rake/application.rb:78:in `block in run’
    /home/jenkins/.rvm/gems/ruby-2.0.0-p247@global/gems/rake-10.1.0/lib/rake/application.rb:165:in `standard_exception_handling’
    /home/jenkins/.rvm/gems/ruby-2.0.0-p247@global/gems/rake-10.1.0/lib/rake/application.rb:75:in `run’
    /home/test/.rvm/rubies/ruby-2.0.0-p0/bin/rake:41:in `’
    Build step ‘Invoke Rake’ marked build as failure

Leave a Reply

Your email address will not be published. Required fields are marked *

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>