Running your Cukes in Jenkins

One question that I am often asked is “How do you run your cucumber scripts?”. This question usually leads to a discussion about what process and software I use to run my features in a regression-like fashion in a team setting. The questioners are usually not interested in how a developer might use autotest (my local tool of choice) to run the cuke/spec loop. They’re not interested in how a developer might run a feature on their local machine to verify they have completed a card. They’re also not interested in how a developer or tester might run the entire suite of features to verify everything still works. They want to know how to schedule the execution of the entire suite of features.

The truth is that I don’t run them. Instead I have a server process run the features for me. I am a strong advocate of having the continuous integration server run the acceptance tests continuously. This post will explain how I do it and hopefully provide you the information you need to do it as well.

Let’s be clear

Before I go into the details let me be clear. I am not proposing that this is the only way you and your team should run your features. I am a proponent of developers running the feature they are working on all of the time (perhaps continuously) and I think testers should run them on their computer very frequently as well. This post is about how I like to run the entire suite of features to form a regression. I’ll start by introducing how I control the running of the features and setup my projects. Next I’ll discuss how this works with Jenkins.

cucumber.yml

First of all I have a cucumber.yml file at the root of my project structure. I use this file to define a set of profiles (a grouping of command-line arguments) that I can easily use when I want to run my features. Cucumber already has a default profile that will be used unless you override it. Here is what that might look like:

default:  --no-source --format pretty --tags ~@not_ready

Everything after the default: label in the entry will be passed to cucumber as arguments when it runs.

You can use profiles to define separate definitions of how you want to run your features. For example, if you have some very slow tests you might want to separate them out into their own run. You would start by placing a @slow tag on those features or scenarios and then create a few profiles like this:

default:  --no-source --format pretty --tags ~@not_ready
slow:  --no-source --format pretty --tags @slow --tags ~@not_ready
fast:  --no-source --format pretty --tags @~slow, ~@not_ready

To use the profiles you simply pass the profile name on the command line. If you wish to run the fast tests you would run:

cucumber -p fast

Rakefile

Rake is a tool that has the ability to execute tasks. Many tools, including cucumber, are distributed with pre-built rake tasks that can be used to control their execution. Jenkins has a plugin that can execute Rakefiles. We use this combination to control the running of our features in Jenkins. We will discuss how to setup Jenkins in the next section. Here we’ll discuss how to write a simple Rakefile. Let’s start by looking at a simple example.

require ‘cucumber’
require ‘cucumber/rake/task’

Cucumber::Rake::Task.new(:features) do |t|
  t.profile = ‘ci’
end

task :default => :features

This rake script creates a new task named features and has it call the ci profile. Next it makes the features task the default task. You can execute rake in the project directory and it will run the features task (the default) using the ci profile created in a cucumber.yml file. We use the Rakefile to specify which profile to run and we use the cucumber.yml file to detail out the profiles.

A more sophisticated Rakefile might create a namespace to contain several tasks.

require ‘cucumber’
require ‘cucumber/rake/task’

namespace :features do
  Cucumber::Rake::Task.new(:fast) do |t|
    t.profile = ‘fast’
  end

  Cucumber::Rake::Task.new(:slow) do |t|
    t.profile = ‘slow’
  end

  task :ci => [:fast, :slow]
end

task :default => :fast

This Rakefile creates a namespace named features, which contains three tasks (fast, slow, and ci). The ci task runs the fast task followed by the slow task. To execute the slow tests you would execute

rake features:slow.

If you want to see what tasks are available from the rake tool you can execute the rake –T command.

Jenkins

The Jenkins configuration couldn’t be simpler. The first thing you need to do is install the Jenkins Rake plugin. Once this is installed you should go to the Configure System page. You will see a new section appear on this page.

It is likely that Jenkins discovered your ruby installation already but if it didn’t, just enter the path to the location where ruby is installed.

Next you need to create the project to run your cukes. Complete the form as you usually would. In the Build section select the Add build step button and then select Invoke Rake. This will open this dialog.

In this section you simply need to type the task you wish to run.

Guidelines I use to run my cukes

How often should you run your features? In my opinion, you should try to run them as close to all of the time as possible. Each time they run they should execute against the latest version of the application.

If it takes 2 hours 30 minutes to run the tests I would schedule them to run every 3 hours. If you have some slow tests you might want to schedule them to run 4 times a day and have the fast test run every hour.

The goal here is to get fast feedback.