One question that I am often asked is “How do you run your cucumber scripts?”. This question usually leads to a discussion about what process and software I use to run my features in a regression-like fashion in a team setting. The questioners are usually not interested in how a developer might use autotest (my local tool of choice) to run the cuke/spec loop. They’re not interested in how a developer might run a feature on their local machine to verify they have completed a card. They’re also not interested in how a developer or tester might run the entire suite of features to verify everything still works. They want to know how to schedule the execution of the entire suite of features.
The truth is that I don’t run them. Instead I have a server process run the features for me. I am a strong advocate of having the continuous integration server run the acceptance tests continuously. This post will explain how I do it and hopefully provide you the information you need to do it as well.
Let’s be clear
Before I go into the details let me be clear. I am not proposing that this is the only way you and your team should run your features. I am a proponent of developers running the feature they are working on all of the time (perhaps continuously) and I think testers should run them on their computer very frequently as well. This post is about how I like to run the entire suite of features to form a regression. I’ll start by introducing how I control the running of the features and setup my projects. Next I’ll discuss how this works with Jenkins.
First of all I have a cucumber.yml file at the root of my project structure. I use this file to define a set of profiles (a grouping of command-line arguments) that I can easily use when I want to run my features. Cucumber already has a
default profile that will be used unless you override it. Here is what that might look like:
default: --no-source --format pretty --tags ~@not_ready
Everything after the
default: label in the entry will be passed to cucumber as arguments when it runs.
You can use profiles to define separate definitions of how you want to run your features. For example, if you have some very slow tests you might want to separate them out into their own run. You would start by placing a
@slow tag on those features or scenarios and then create a few profiles like this:
default: --no-source --format pretty --tags ~@not_ready slow: --no-source --format pretty --tags @slow --tags ~@not_ready fast: --no-source --format pretty --tags @~slow, ~@not_ready
To use the profiles you simply pass the profile name on the command line. If you wish to run the fast tests you would run:
cucumber -p fast
Rake is a tool that has the ability to execute tasks. Many tools, including cucumber, are distributed with pre-built rake tasks that can be used to control their execution. Jenkins has a plugin that can execute Rakefiles. We use this combination to control the running of our features in Jenkins. We will discuss how to setup Jenkins in the next section. Here we’ll discuss how to write a simple Rakefile. Let’s start by looking at a simple example.
require ‘cucumber’ require ‘cucumber/rake/task’ Cucumber::Rake::Task.new(:features) do |t| t.profile = ‘ci’ end task :default => :features
This rake script creates a new task named
features and has it call the ci profile. Next it makes the features task the default task. You can execute
rake in the project directory and it will run the
features task (the default) using the
ci profile created in a
cucumber.yml file. We use the Rakefile to specify which profile to run and we use the cucumber.yml file to detail out the profiles.
A more sophisticated Rakefile might create a namespace to contain several tasks.
require ‘cucumber’ require ‘cucumber/rake/task’ namespace :features do Cucumber::Rake::Task.new(:fast) do |t| t.profile = ‘fast’ end Cucumber::Rake::Task.new(:slow) do |t| t.profile = ‘slow’ end task :ci => [:fast, :slow] end task :default => :fast
This Rakefile creates a namespace named
features, which contains three tasks (fast, slow, and ci). The ci task runs the fast task followed by the slow task. To execute the slow tests you would execute
If you want to see what tasks are available from the rake tool you can execute the
rake –T command.
The Jenkins configuration couldn’t be simpler. The first thing you need to do is install the
Jenkins Rake plugin. Once this is installed you should go to the
Configure System page. You will see a new section appear on this page.
It is likely that Jenkins discovered your ruby installation already but if it didn’t, just enter the path to the location where ruby is installed.
Next you need to create the project to run your cukes. Complete the form as you usually would. In the
Build section select the
Add build step button and then select
Invoke Rake. This will open this dialog.
In this section you simply need to type the task you wish to run.
Guidelines I use to run my cukes
How often should you run your features? In my opinion, you should try to run them as close to all of the time as possible. Each time they run they should execute against the latest version of the application.
If it takes 2 hours 30 minutes to run the tests I would schedule them to run every 3 hours. If you have some slow tests you might want to schedule them to run 4 times a day and have the fast test run every hour.
The goal here is to get fast feedback.