Oliver Wehrens

Concordion vs. Cucumber and Java based Acceptance Testing

Oct 13, 2009 - 5 min

If you want your Business Analyst to provide some acceptance tests for your implementation it would be best to make sure you get it in some form your tests can read. That way you can make sure your code does really do that what the customers wants. Two tools to achieve that are Concordion and Cucumber (and cuke4duke). Both taking the approach of Fit a step further. You can find very good documentation for the tools on their web pages or as examples along with the code.

Concordion

Concordion let’s you write the specifications in html. You can instrument them with some Concordion code. Since the instrumentation code is in html, it is hidden when you render it.

<html xmlns:concordion="http://www.concordion.org/2007/concordion"
      xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
      xsi:schemaLocation="http://www.concordion.org/2007/concordion ">
<body>
<p>
  <span concordion:execute="setCurrentTime(#TEXT)">09:00AM</span>
  The greeting for user <span concordion:set="#firstName">Bob</span>
  will be:
  <span concordion:assertEquals="greetingFor(#firstName)">Hello Bob!</span>
</p>
</body>
</html>

Concordion can instrument free text as well as tables to have some sort of data provider for tests. It integrates directly with JUnit (no TestNG support yet). Concordion Specification/Fixture/System The idea is that your specifications is stable. This will not change as much as maybe your code to fulfill the requirements of the customer. Your fixture bridges these two together.

package com.maxheapsize.concordion;
import org.concordion.integration.junit3.ConcordionTestCase;
public class DemoTest extends ConcordionTestCase {

    public String greetingFor(String firstName) {
        return "Hello " + firstName + "!";
    }

    public void setCurrentTime(String time) {
        System.out.println(time);
    }
}

Your java code provides the methods which are called from the specification. At that point you are most likely to use a DSL to build up your business logic. Concordion will assert the expected values.

After you run the unit test you will get a nice looking html page. Concordion output Passed tests will be green, failed red, pretty easy. Pro Concordion:

Instrumention of free text
The final report is nice looking
JUnit support
Easy setup
Pure Java

Con Concordion:

The business analyst needs to know a little bit of html
Usually the programmer needs to change the specification to instrument it
You program parts in html, refactoring support is missing
No TestNG support

Cucumber

Cucumber is a ruby based behaviour driven development tool. Since you can run Ruby code inside the JVM with the help of JRuby it is possible to use it for your Java code. The usually way so far is that you use maven or ant to startup JRuby and the Cucumber ruby code.

In Cucumber you write your specification (or feature) in plain english (or russian, german,lolcode or 36 or so more supported languages). No instrumentation is necessary but your customer has to stick to a certain layout (or better: structure). It follows the well known given-when-then pattern.

  Feature: Demo

  Scenario: A Car with 4 wheels
    Given I have a car
    When I add 4 wheels
    Then I should be able to drive the car

You can write your step definitions (like the fixture in Concordion) with the help of annotations.

package com.maxheapsize.cucumber;

import cuke4duke.Given;
import cuke4duke.Then;
import cuke4duke.When;
import org.junit.Assert;

public class CarFeature {
  private Car car;
  @Given("I have a car")
  public void iHaveACar() {
    car = new Car();
  }

  @When("I add (\\d+) wheels")
  public void iAddWheels(int numberOfWheels) {
    car.setWheels(numberOfWheels);
  }

  @Then("I should be able to drive the car")
  public void iShouldBeAbleToDriveTheCar() {
    Assert.assertTrue(car.canDrive());
  }
}

The regular expressions defined in the annotations must match the feature definitions. Cucumber results You can define information in a table like structure as well to provide more data in an easy way. The given when then structure can be used to have multiple when's or then's. So it is very flexible that way. Pro Cucumber

Feature can be described in plain language
No extra programmer work on he feature required
Instrumentation is directly on the Java code via annotations

Con Cucumber

More complex due to the Ruby/JRuby/Java chain
Business analyst needs to stick to predefined format
The only way (without extra work to write a runner) to run it seems to be within ant or maven, for rapid turn around and TDD I would like to have JUnit/TestNG support of some kind

Verdict

It is easy to get started with both tools. I spent about a couple of hours with each of them and was able to get simple and a bit more complex examples running. They get you going with BDD and acceptance based testing.

I like parts of both concepts. I want to be able to write a feature like in Cucumber. The instrumentation should be directly on my code and not in the part of the customer. The easy setup and integration of Concordion with JUnit (need TestNG here) and the output of it is very nice. So where is the tool which has these features ;-).

So which tools will I use?

I have yet to see the use case where the business analyst writes the (and updates) the acceptance tests. I don’t mind add some extra code to the specification and I know html. I might get started with Concordion for now. You might also check my post about Behavior driven development with EasyB (and vs. TestNG). Do you use any of these tools? Or a different one? What's your expierence? Update The next-gen cuke4duke was released: Cucumber-jvm. Check out my post on that one.