Acceptance Test Driven Development with an internal customer

In my experience and what I hear about problems with Acceptance Driven Development (ATDD) practice, is that it's not easy to have the customer in the same room to create highest abstraction level Test Cases. This is of course a problem to business people and it might be a little bit hard to sell, if you don't have any experience of your own about the topic.

One of the easier way how a corporation can start to practice this is to try it out with internal customers. Usually there should be a situation where the whole product is not created in one place. These kind of relationships are eg. platform vs. application, where platform is created in one part of organization and application(s) in another. It's easier to create Test Cases and get people in the same meeting room when they are close to each others and there are no company barriers.

These activities aim to raise competence and good knowledge about practice inside the company. ATDD of course aims for communication with real customers, but this is a good way to get hands-on.

Robot framework training

I joined with Pekka Klärck and Ismo Aro to do my first Robot training and coaching sessions this October in Athens, Greece. It was a very nice way start coaching Robot usage with such an enthusiastic and eager to learn crowd. Many thanks to everybody who participated!

In the (generally very positive) feedback that we gathered from the sessions two main points of criticism stand out:

1. Two days is a very short time to learn test automation with Robot (I don't think this is in any way Robot specific issue.)
2. People prefer learning by doing exercises over listening to me going on about the theory. Also it was pointed out that it would be nice to have more time for the exercises that we have.

Planning the next training

The first issue is really no surprise. Although the basic syntax and usage of Robot is very easy to learn, learning how to do test automation well in general takes time. Also for learning the more exotic features of Robot Framework, a course of two days is just a too short of a time. When giving Robot training we do usually stay for a few days afterwards to coach and help clients with their real tests. This is a very important part of the training and should be considered a part of the course really. (Maybe next time, I will ask for the feedback after the coaching is done also...)

I do understand that the two day crash course into test automation with robot can be a bit like drinking from a fire hose. We have to think whether it would be better to give Robot training for example first one day, then coaching and some time for the people to try out what they learned and then after a week or so, to give a second day of training with more advanced concepts.

The second point about more hands on training forces us to rethink how to present the more advanced concepts and also what concepts are really necessary in a basic training. It is a lot easier to cover many advanced concepts in one day when you just read them through from slides. To actually let the people try them out, we have to give them some time and therefore also to skip some subjects from the training. But in the end, I guess it is better to teach few things well, than a lot of things too quickly.

This has been an important learning experience and we are looking forward to making the next training even better than this one.

Creating Robot Framework jar distribution

As of Robot Framework 2.5.2, it is also distributed as a single stand alone jar file.

The goal was to create a jar distribution with following properties:

• Standalone executable jar file, i.e. java -jar robotframework.jar mystest.txt would suffice to run tests.
• The exit code of the above command must be the same as if mytests.txt was executed with pybot.
• There would also be a programmatic API to start Robot Framework test execution from any Java program

While there was a wealth of information about Jython-Java interaction available in the net, I could not find a single set of instructions describing all the necessary steps to create the required jar.  Since I was able to do it and since the process is general, I decided to write it down.

First requirement is the Jython standalone jar, which can be created with the Jython installer. Following the instructions in the Jython Book, I created three Java classes and one Python for the interaction. These classes implement the one-to-one Jython object factory described in the Jython book. Source code of these classes is available in Robot Framework source repository.

The java classes had to (of course) be compiled before inserting them into the jar file. Additionally, a MANIFEST.MF file with correct main-class had to be created. Once these were done, the files could be inserted into the Jython jar:

• Robot Framework source code must be placed in Lib directory inside the jar
• The java code must be inserted starting from the root of the and according to package structure
• The MANIFEST.MF has be to in META-INF directory

That's pretty standard stuff expect for one gotcha: Zip file format (and jar files are Zip files) supports multiple entries with same path. Since the Jython jar already contains MANIFEST.MF and since Python's zipfile module does not support overriding or removing entries, the first released version of the jar distribution contained two manifest files. For reasons unknown to me, the one we inserted was effective when the jar file was executed with java, but adding files to the jar file with jar failed because of the duplicate manifest.

This was resolved by unpacking the Jython jar and replacing the MANIFEST.MF with our own on the file system before repackaging. .

All the Python source files were compiled with Jython's compileall module before they were inserted into the jar file. This presumably increases performance slightly.

I will probably do another post about how we finally got the jar distribution uploaded to Maven central, since that's an interesting story too.

Implementing asynchronous Robot Framework keywords

Before tests can be run I have to start all the necessary systems (processes, servers, stuff and things). Let's imagine that there are multiple systems that need to be started. I would like to start them all at the same time and wait until they are all ready to rock 'n' roll before starting my tests (assume that it would take a lot longer if I started all the systems in sequence).

So how to do this with Robot? Basic idea is to have re-usable keywords for starting each system and a keyword for waiting until the systems are ready so that testing can begin (and we don't have to use ugly and unreliable sleeps). Doing this on Robot keyword level makes it possible to have different combinations of systems in different tests.

*** Settings ***
Documentation   Example of using parallel things
Suite Setup     StartSystems
Library         SystemStarterLibrary.py

*** Test Cases ***
... Here should be my tests

*** Keywords ***
StartSystems
   ${SYSTEM1_STARTED}=  Asynchronously Start System 1
   ${SYSTEM2_STARTED}=  Asynchronously Start System 2
   Wait until  ${SYSTEM1_STARTED}  ${SYSTEM2_STARTED}

The way I'm going to implement this is by using python decorator that executes the function that it decorates in a separate thread. The decorated function will return the thread object so that it can be used to implement the waiting functionality.

I'm using this little code for the decorator.

After I've imported that to my SystemStarterLibrary.py I can implement system starter functions as normal functions.

@run_async
def asynchronously_start_system_1():
  .. do stuff to start system 1

@run_async
def asynchronously_start_system_2():
  .. do stuff to start system 2

Now all I need to do is to implement Wait until.

def wait_until(*stuff):
  for something in stuff :
     something.join()

This is kind of OK but it will wait forever if starting of some system will take forever. So it is better to have some timeout that will tricker setup failure after the timeout.

*** Keywords ***
StartSystems
    [Timeout]  5 minutes
    ${SYSTEM1_STARTED}=  Asynch Start System 1
    ${SYSTEM2_STARTED}=  Asynch Start System 2
    Wait until  ${SYSTEM1_STARTED}  ${SYSTEM2_STARTED}

And that should do it.

Hello World Robot Framework library

This is the way I did my first Robot Framework keyword library. It should show the basic steps to add your own python keywords.

Let's do it in a test-driven way!

Failing test case

Add a file called HelloWorld.txt. This is our robot test suite file.

Add following text to the file:

*** Test Cases ***
HelloWorld
   Hello World

After this run pybot HelloWorld.txt - this will execute your Hello World test case.
Output should be something like:

==============================================================================
HelloWorld                                                                   
==============================================================================
HelloWorld                                                            | FAIL |
No keyword with name 'Hello World' found.
------------------------------------------------------------------------------
HelloWorld                                                            | FAIL |
1 critical test, 0 passed, 1 failed
1 test total, 0 passed, 1 failed
==============================================================================

Now we have a failing test case! So we can begin to implement our super cool Hello World keyword.

Keyword file

Add a file called HelloWorld.py to the same directory as our HelloWorld.txt test suite.

Add following text to the file:

def hello_world():
   print "HELLO WORLD!"

Now we have implemented our fine keyword that prints "HELLO WORLD!". Although our test still fails..

Passing test case

We have to import our super cool library to our test suite. Add following lines to the HelloWorld.txt (before test cases):

*** Settings ***
Library         HelloWorld.py

After this run pybot HelloWorld.txt - and watch it PASS:

==============================================================================
HelloWorld                                                                    
==============================================================================
HelloWorld.HelloWorld                                                         
==============================================================================
HelloWorld                                                            | PASS |
------------------------------------------------------------------------------
HelloWorld.HelloWorld                                                 | PASS |
1 critical test, 1 passed, 0 failed
1 test total, 1 passed, 0 failed
==============================================================================
HelloWorld                                                            | PASS |
1 critical test, 1 passed, 0 failed
1 test total, 1 passed, 0 failed
==============================================================================

Thats it.

Installing Robot Framework on Ubuntu

Lately I've been learning to use the robot framework. These are my notes on how to install it.

Installing pybot - normal robot thing

First install python if it's not already installed.


sudo apt-get install python


Then install easy_install and robotframework.


sudo apt-get install python-setuptools
sudo easy_install robotframework


After this you should have pybot (normal robot thing) installed.


pybot --version
== Should output something like ==>
Robot Framework 2.5.4 (Python 2.6.5 on linux2)


Installing jybot - jython version of robot

I'll assume you have done all the things to install pybot so far (it also installed jybot and all you have to do now is to install correct version of jython).

First a word of warning: Ubuntu is still (you should check is this still valid point if your reading this in the future) using old version of Jython that doesn't work with current jybot.

So we have to first download Jython 2.5 (or later) from jython webpage. Follow jythons installation instructions.

Add JYTHON_HOME to PATH so that you can use it.

After this jybot should work.


jybot --version
== Should output something like ==>
Robot Framework 2.5.4 (Jython 2.5.1 on java1.6.0_20)