Here be Robots!

Robot Framework Newsletter, December 2011

2011-12-28T13:36:00.001+02:00

Introduction

Welcome to the second installment of the Robot Framework newsletter. I hope you have survived Christmas, and are looking toward a successful year 2012!

The news

Robot Framework 2.7 in development

The development of Robot Framework 2.7 has proceeded reasonably well and the new rebot implementation is considered done, In the end, we managed to reduce both memory consumption and execution speed of rebot around 50% compared to all previous releases. At the same time, log and report generation when using pybot has also gotten more efficient.

There are still a number of open issues targeted for 2.7, but some of those are not going to make it to the final release. Our goal is to fix the remaining defects and release some sort of alpha after that.

RIDE 0.40 in development

We've also started development of RIDE 0.40, which will contain two major improvements:

A plain text editor mode
Support for aligning columns in test case and keyword tables.

The plain text editor will be enabled by default, and it will allow editing of a whole test case or resource file at a time. The changes are synced between the plain text and structured editor.

If a test case (or a keyword table) contains other than default headers, that table is interpreted as data driven, and the columns will be aligned according to headers when written in plain text format. It is possible to edit the table headers using the plain text editor mentioned above.

In addition, Robot Framework 2.7 fixes some parsing related bugs and inconsistencies that have been blocking some RIDE issues and these will be resolved in RIDE 0.40.

SeleniumLibrary 2.8.1 released.

We recently released 2.8.1 version of SeleniumLibrary. It upgrades the bundled Selenium server to version 2.15 (with support for Firefox 8)

ATDD demo with Robot Framework available.

Pekka Klärck and I presented a session called "Acceptance Test Driven Development Using Robot Framework" at EuroStar conference earlier this year. The demo application is available as a Google Code project.

Upcoming events

Here's a list of upcoming events that are going to feature Robot Framework in one way or another.

Robot Framework Hands-On training at Skills Matter, London

Robot Framework Newsletter, November 2011

2011-11-04T14:07:00.004+02:00

What's this?

I have been thinking recently that the Robot Framework development must seem quite opaque to anyone outside the core team. We occasionally communicate when a development effort of some project is started, but at other times releases just come out of the woods.

To alleviate the lack of communication, I thought that a monthly newsletter would be in order. My intention is shed light on things that have been done in the recent past as well as highlight the things that we are likely to engage in the near future. I also figured that a slightly longer "feature article" would be nice in each newsletter.

And now you are enjoying the very first issue of the said newsletter. Hopefully it won't be the last. I would be grateful for any feedback, as well as suggestions for feature article topics.

The news

Some of these are actually already "olds", but in my opinion important enough to list here anyway.

Robot Framework 2.7 in development

We recently started the development of Robot Framework 2.7. The list of issues initially targeted for 2.7 is quite long, and subject to heavy pruning, probably during next week.

The core team has been working on faster and less resource consuming rebot, and we've been making good progress. Whole rebot was basically written from scratch and it is now integrated, and all tests are passing in the current HEAD.

If you have any contributions towards 2.7, now is the time to act.

Projects moved to GitHub

We've created a GitHub organization for Robot Framework, and have already moved some projects there. The reasoning for this move warrants its own post, but the short story is that we hope to

ease our own work
lower the barrier for contributing

We are likely to continue moving projects to GitHub, although the migration of core framework itself has not yet been scheduled.

New test libraries

Ryan Tomac has released Selenium2Library, which is a drop-in replacement for SeleniumLibrary, but it uses the new Selenium2 WebDriver API instead of the Remote control API used by the old library. Thanks Ryan for this major contribution!

The core team has released first version of Rammbock, which is a generic network protocol test library for Robot Framework. Rammbock is still in its early stages, but shows great promise.

Feature of the month: How Robot Framework development is organized

As mentioned in project pages, Robot Framework was started as an internal project at Nokia Networks (which was later merged with Siemens network business to create NSN), and was open sourced later. It is widely used at NSN, and NSN still funds RF development.

The core team (currently numbering one part-time NSN person and 4,5 externals) is paid by NSN and works at NSN premises in Espoo, Finland. Pekka Klärck, creator of the framework, is part of the core team.

In addition to developing the core framework and RIDE, and maintaining several test libraries, the core team is responsible for NSN-wide training and support of Robot Framework.

Priorities for RF development arise mainly from the internal users, and these needs affect the order in which, for example RF and RIDE releases are made. However, we tend to fix bugs regardless of who opened the bug report. Similarly, generic, useful enhancements of reasonable scope are also often made without direct internal need.

All the core team projects have issue trackers, and even though sometimes neglected for a while, the issue tracker is the most accurate source of information about the scope of any upcoming release.

We do not have any kind of roadmaps, or deadlines for releases. We try to work on only one project at any given time, and after a release is made, next project is chosen based on the priorities at that point. This also means that most of the time, most of the projects are on hold.

All the above applies obviously only to projects that are directly maintained by the core team. There's a growing number of test libraries and other tools maintained by active community members, and those projects have their own governing rules.

If this raises further questions, use comments, or start a thread in mailing list.

Upcoming events

Here's a list of upcoming events that are going to feature Robot Framework in one way or another.

2011-11-07 FinnStar, presentations from CAST and EuroSTAR by Finns, (sold out)
2011-11-24 EuroStar in Manchester, Acceptance Test Driven Development Using Robot Framework
Robot Framework Hands-On training at Skills Matter, London

Monkey Testing: a Technique for Detecting Nasty Bugs

2011-08-20T20:01:00.014+03:00

I'm going to tell about a technique that could help you find those bugs that no one else can find or are classified as "non-reproducible".

Some real life bugs

First let me show you that the technique also works with a real and recent 0.37 version of RIDE by demonstrating some hard to detect bugs that I detected with the tool I made. NOTE: These bugs have been in RIDE for very long time so they should be repeatable also in older versions.

First bug

Open RIDE

Open a keyword or a suite editor

Insert some text to the first row

Select the first row and select move row up from row context menu -- Nothing happens

Undo the last command (Ctrl+Z) -- produces a stack trace either to RIDE log or to command prompt

Second bug

This is a bit trickier but it has an issue in RIDE issue tracker -- unlike the previous one. Most likely because the underlying problem could have more ways to reveal it self than just this one.

Open RIDE

Open a keyword editor

Insert some text to the fifth line

Delete first row

Delete sixth row

Save file

Delete third row -- produces a stack trace either to RIDE log or to command prompt

Monkey testing

So how did I detect those bugs and how did I find a way to reproduce them?

I used a technique called monkey testing. It's name comes from the saying: "a thousand monkeys at a thousand typewriters will eventually type out the entire works of Shakespeare".

The basic idea is to make an automated system that randomly executes actions in the system under test. If the actions are selected in a way that will keep the system under test in a running state the test can just go on forever (or until it finds a bug).

To make the detected bugs reproducible the randomness in the test system must be controlled. Basic method is to control and log the seed value of the used (pseudo) random number generator.

Usually a failing test run will result in a very long trace (a list of executed actions) that needs to be pruned to find out the actions that resulted in the detected error. This pruning can of course be automatic.

You can find the related code from http://code.google.com/p/robotframework-ride/source/browse/#hg%2Frtest.

How do I know (well almost) that there is no more of these bugs around?

Because I have now let those monkeys run for several hours without catching anything. This gives me confidence to say that it is very unlikely that there are bugs that the monkey testing tool could produce and detect. In RIDE I mean that there most likely are no more non-GUI code bugs that will throw exceptions while editing test cases or keywords. And I can still put the monkeys to work to make me even more confident that the bugs that I have detected so far are all there is (I have corrected the bugs so the monkeys will not stumble on them again).

It is very important that a monkey testing tool can produce enough different kinds of actions to make it possible for it to express different types of defecting traces. The tool should also have good enough detection mechanisms so it will catch the defects -- but remember that more complexity means more complexity (the error could be in the tool if the tool is too complex).

In my case with the RIDE the detection mechanism has so far been only to catch exceptions but I've been thinking of taking some basic action related assertions in to use.

If you find this technique useful you could also check out model based testing to make monkeys that handle more complex action sequences and situations.

Super Sized JavaScript

2011-08-01T20:47:00.004+03:00

The most technically challenging improvements in Robot Framework 2.6 are the new logs. They are about 1/10 of the size of the old logs and they are completely generated from very large JSON objects. One of the challenges in the new format is that the log is a single file that includes all the generated JavaScript, HTML and CSS code.

The largest new style log-files so far have been about 100 MB (that is a lot of JavaScript) and I believe that because we are working with such a big JavaScript objects we've encountered many difficulties that others have not yet have to deal with.

Although these files take time to download when loaded from servers, once loaded they work very well. Actually after we had first figured out how to make these almost 100 MB log-files work the reality came in and we had to figure out a way to split those large files to reasonable sized pieces (the method that we used is also explained in this post)..

Here's a list of tricks that we have learned during the process of super sized JavaScript development.

Doing large computations in JavaScript and how to prevent browser from freezing.

One of the first problems we had was that the extra large html/JavaScript files would freeze almost all browsers while expanding all the log elements in logs tree view. The final solution to this problem required some thinking and experimentation as the second point (not putting everything to the event queue) wasn't obvious.

JavaScript engines are one threaded and event based. This means that a big task will freeze everything. You can split your big task with setTimeout function to smaller parts. But if you split your task in too many parts and queue them all to the task queue the browser will again freeze as there is no space for users tasks.

The solution is to have a separate queue for the tasks that are generated during execution and to have only one task in the real event queue in any given time.


function timerTasker() {
    var currentTask = tasksQueue.nextTask();
    currentTask.do();
    tasksQueue.appendAll(currentTask.tasks()); //add tasks generated during current task execution
    if (!tasksQueue.isEmpty()) {
       setTimeout(timerTasker, 0);
    }
}

Lazy domain objects.

Everything should be as lazy as possible when dealing with a huge serialized data. It would be very sad to run out of memory or CPU when generating thousands of useless objects that no one will ever use.

IE9 JavaScript parsing out of memory.

Everything seemed to work ok even with very large files (Over 20 MB) but then we tried them in the IE9 and some of the logs just didn't work. After hard debugging we found that IE9 had a very odd problem that none of the other browsers had.

For some reason IE9 JavaScript parsing with reasonable small sized lists containing nested lists and integers and strings (the total size in our case was "only" 1.5 MB) will run in to out of memory error. This can be prevented by not mixing integers and strings.. In my opinion this could be a bug in IE9 as IE8 doesn't have these problems.

Too Large JavaScript.

After IE9 problem was fixed everything was again OK. Until we finally tried to generate extremely big JavaScript log files. This time the problems were with Firefox. Luckily these were simple problems with easy fixes -- just had to invent a clever way to split our data.

Firefox 4 (and 5) will at some point between 40MB - 80MB start to say that your JavaScript block is too large.. To prevent this use multiple JavaScript blocks instead of just one. The memory errors seem to occur during parsing, thus you can handle extremely large objects but you just have to keep the parser happy.


<script type="text/javascript">
    window.data = [.. [big data subelement] ..];
</script>

-- transform this to: --

<script type="text/javascript">
    window.dataSubElement = [big data subelement];
</script>
<script type="text/javascript">
    window.data = [.. window.dataSubElement ..];
</script>

Loading more JavaScript on the fly + Chrome safety.

100 MB log files were not something that all of our users wanted to work with. So we had to find a way to split the logs. The method had to work also locally. After brainstorming with ideas and spiking with techniqus we finally found a solution that works.

There are many ways to load more JavaScript while a page is open. I think that AJAX request to a server is the most common way but in our case there is no server.

There is a convenient function in jQuery for downloading new scripts ($.getScript) but it doesn't work for local files when using Chrome (@see Chrome issue 40787).

Our solution was to insert new script blocks to dom-tree on the fly (How to Dynamically Insert JavaScript). This works at least in our case.

Robot Framework 2.6 with new logs and reports has been finally released and everything seems to run smoothly (knock on wood). It was very interesting experience (with a lot of unexpected problems) to develop them. Respect to all my teammates!

Robot Framework at XP2011 Conference

2011-03-28T13:18:00.000+03:00

Robot Framework has strong presence at XP2011 conference that is organized in Madrid 10th-13th May, 2011. Janne Härkönen and I, Pekka Klärck, will be present and we organize the following two sessions.

Demo: Acceptance Testing with Robot Framework

Tutorial: Acceptance Test Driven Development (ATDD) with Robot Framework

We are also highly interested to discuss any Robot related topics with the users of the framework and anyone who is interested. If you are coming and want to chat let us know beforehand or just find as wondering around the conference area. If there's more interest, perhaps we can organize an ad-hoc Robot session around a jar of sangria.

Using Jython REPL to explore with SwingLibrary

2011-03-28T10:47:00.000+03:00

Today, I was faced with creating an extension keyword to SwingLibrary, and I wanted (of course) to also make a Robot test case testing that new keyword. With non-trivial applications, Swing tests always require quite a bit of setup, many keywords have to be used before the interesting component is available.

Using Robot Framework to get the setup "right" is quite slow, since the turnaround time (= time to execute the test) is quite long, at least some tens of seconds. There is a somewhat faster method of using the SwingLibrary directly from an interactive Jython session. So I fired up Jython with the dependencies in the CLASSPATH:

CLASSPATH=bin:lib/swinglibrary-1.1.3-jar-with-dependencies.jar jython

And did some exploration:

import SwingLibrary
lib = SwingLibrary()
lib.runKeyword('startApplication', ['com.acme.MyFancyApp'])
lib.runKeyword('selectWindow', ['myMainWindow']) 
lib.runKeyword('listComponentsInContext', [])
...

There are a couple things worth noting here:

It is easiest to use the SwingLibrary keywords using the runKeyword-method, since the actual keywords are not methods of the SwingLibrary class.
When using runKeyword, the arguments are always given in a list and the list has to supplied, even when it is empty.

When the desired workflow is achieved the keywords can be written down in a test case.

Swingin' it

2011-02-17T10:14:00.032+02:00

Here's a really simple demo, or tutorial if you will, about using the Robotframework SwingLibrary.

Download the latest demo and extract it to the desired directory. Oh, and you should have the Robotframework and Jython installed. Open your command line and navigate to the extracted demo directory. Try to run the finnished demo to verify that everything is installed and working ok. Execute:

run_demo.py example

The SwingLibrary jar needs to be included in the classpath in order to be found by the Robotframework. The run_demo.py does this for you and executes the test suite provided as an argument.

Examine the SUT i.e. Todo List Application. Start the application by running the following in the command line at the demo root:


java -cp lib/swinglibrary-1.1.2-SNAPSHOT-jar-with-dependencies.jar \ org.robotframework.swing.testapp.examplesut.TodoListApplication

As you can see, this a really basic todo list Swing application written in Java. Have a look around and close the application when your done.

Open your favourite text editor. Create a new test suite file (e.g. test.txt) and save it to the demo root. Take the SwingLibrary into use in the test suite, type:


*** Settings ***
Library  SwingLibrary

*** Test Cases ***
Test Add Todo Item

(Note! In this plain text format the cell separator is 2 spaces)

Run the test.


run_demo.py example.txt

The test should execute and fail with an error message: "The test case contains no keywords". Ok let's add some and start the application and select it's main window.


*** settings ***
Library  SwingLibrary

*** Test Cases ***
Test Add Todo Item
    Start Test Application

*** User Keywords ***
Start Test Application
    Start Application  org.robotframework.swing.testapp.examplesut.TodoListApplication
    Select Main Window

Run the test again and now the application should start and test still fails.

Ok, progress. Let's move the start-up stuff into Suite Setup. Then insert text "Buy some milk" into the text field, that the developer has named logically "description". While at it, let's push the "Add Todo Item"-button with a logical name "add".


*** settings ***
Library  SwingLibrary
Suite Setup  Start Test Application

*** Test Cases ***
Test Add Todo Item
    Insert Todo Item Buy some milk

*** User Keywords ***
Start Test Application
    Start Application  org.robotframework.swing.testapp.examplesut.TodoListApplication
    Select Main Window

Insert Todo Item ${arg}
    Insert Into Text Field  description  ${arg}
    Push Button  add

Run the test and enjoy the greenness. Note, that the user keyword: Insert Todo Item, uses embedded variable syntax. So the "Buy some milk" is provided to the keyword in the same cell.

Great, but still no testing or assertions. We can easily fix that. Select the first (0th) item from the todolist and read the the selected value into an argument. The we can assert that the value is the same as we expect i.e. "Buy some milk".


*** settings ***
Library  SwingLibrary
Suite Setup  Start Test Application

*** Test Cases ***
Test Add Todo Item
    Insert Todo Item Buy some milk
    Select From List  todolist  0
    ${item}=  Get Selected Value From List  todolist
    Should Be Equal  ${item}  Buy some milk

*** User Keywords ***
Start Test Application
    Start Application  org.robotframework.swing.testapp.examplesut.TodoListApplication
    Select Main Window

Insert Todo Item ${arg}
    Insert Into Text Field  description  ${arg}
    Push Button  add

Run the test and it should pass.

Write tests for the todo item deletion...

Too much sleep is better than not enough

2011-02-11T21:31:00.008+02:00

Timing is an important aspect of automated testing. Tests should run in minimum time so that they give rapid feedback and are not bottle necks. But when testing complex systems some operations unfortunately just take time and this has to be handled in tests. This post is about handling those situation in tests.

Usually the first solution is to use sleeps. A sleep (in Robot Framework BuiltIn.Sleep) does nothing for a specified time period and continues the test after this. A Sleep isn't usually the optimal solution as it doesn't guarantee anything else than that the give time has passed. In other words: It guarantees that the test will take longer and doesn't guarantee that the waited event has happened.

Sleeps don't always work. This results in flickering tests.

To make Sleep work always it has to be very long so that it will be enough for every environment where the system under test should work. This is most likely longer time than what it would take in average.

Another solution for waiting is polling (or busy wait). In polling (in Robot Framework BuiltIn.Wait Until Keyword Succeeds) the waited condition is checked repeatedly until the condition passes or a timeout period expires. Polling guarantees that the waited event has happened.

Polling will usually result in a less total time of test execution than using sleeps. Sometimes this doesn't happen. For example the polling check could exhaust the system under test. Repeating the keyword will also produce more test logs.

If the repeated keyword takes enough time wait until could take more time than needed.
For example keyword that would take 9 seconds, in a situation where there has to be at least 10 seconds of waiting before the keyword can succeed, it will take 27 seconds to execute the Wait Until Keyword Succeeds but it would take 19 seconds to sleep 10 seconds and then execute the keyword.

Other methods to handle time:
* If possible simulate the passing of time
* Hollywood principle
* First sleep and then use wait until keyword succeeds to minimize polling

Functional Robot

2011-01-28T21:54:00.008+02:00

Robot has some interesting keywords that allow users to make keywords that use function delegation or are higher order functions. Here are some examples that I think could be useful.

Keyword that opens and closes a resource

Basic idea is to have keyword that does something (opens a resource) before executing a delegated keyword and does something (closes a resource) after it has executed the delegated keyword.

Here is an example keyword that executes the argument keyword without producing log output.


With No Logging
    [Arguments]  @{keyword with args}
    ${original log level}=  Set Log Level  NONE
    ${result}=  Run Keyword  @{keyword with args}
    Set Log Level  ${original log level}
    [Return]  ${result}

Example use:


With No Logging  Generated Huge Report  Is Valid

Keyword for consuming unknown number of items


Consume All
    [Arguments]  ${producer function}  ${consumer function}
    ${product}=  Run Keyword  ${producer function}
    Run Keyword If  ${product}  ${consumer function}  ${product}
    Run Keyword If  ${product}  Consume All  ${producer function}  ${consumer function}

Consume All keyword takes a function that produces items and another function that consumes those items. Item could be anything for example a web page url or a custom xml message from a server. Consume All keyword stops when producer returns an empty value.
This keyword is useful in situations where there isn't total control of the number of produced items or when that number isn't really interesting from the testing point of view or it takes too much time to first find the number of items and then go through all of them. For cases where the number of items or the list of items is easy to get robotframework has BuiltIn.Repeat Keyword and :FOR.

Example uses:


All Server Messages Have Valid Hash
  Consume All  Server.Pull Message  Validate Message Hash

All Pages Contain Company Logo
  Consume All  PageCrawler.Next Page  Page Contains Company Logo

Acceptance Test Driven Development with an internal customer

2010-12-06T10:20:00.000+02:00

In my experience and what I hear about problems with Acceptance Driven Development (ATDD) practice, is that it's not easy to have the customer in the same room to create highest abstraction level Test Cases. This is of course a problem to business people and it might be a little bit hard to sell, if you don't have any experience of your own about the topic.

One of the easier way how a corporation can start to practice this is to try it out with internal customers. Usually there should be a situation where the whole product is not created in one place. These kind of relationships are eg. platform vs. application, where platform is created in one part of organization and application(s) in another. It's easier to create Test Cases and get people in the same meeting room when they are close to each others and there are no company barriers.

These activities aim to raise competence and good knowledge about practice inside the company. ATDD of course aims for communication with real customers, but this is a good way to get hands-on.

Robot framework training

2010-10-25T14:04:00.003+03:00

I joined with Pekka Klärck and Ismo Aro to do my first Robot training and coaching sessions this October in Athens, Greece. It was a very nice way start coaching Robot usage with such an enthusiastic and eager to learn crowd. Many thanks to everybody who participated!

In the (generally very positive) feedback that we gathered from the sessions two main points of criticism stand out:

Two days is a very short time to learn test automation with Robot (I don't think this is in any way Robot specific issue.)
People prefer learning by doing exercises over listening to me going on about the theory. Also it was pointed out that it would be nice to have more time for the exercises that we have.

Planning the next training

The first issue is really no surprise. Although the basic syntax and usage of Robot is very easy to learn, learning how to do test automation well in general takes time. Also for learning the more exotic features of Robot Framework, a course of two days is just a too short of a time. When giving Robot training we do usually stay for a few days afterwards to coach and help clients with their real tests. This is a very important part of the training and should be considered a part of the course really. (Maybe next time, I will ask for the feedback after the coaching is done also...)

I do understand that the two day crash course into test automation with robot can be a bit like drinking from a fire hose. We have to think whether it would be better to give Robot training for example first one day, then coaching and some time for the people to try out what they learned and then after a week or so, to give a second day of training with more advanced concepts.

The second point about more hands on training forces us to rethink how to present the more advanced concepts and also what concepts are really necessary in a basic training. It is a lot easier to cover many advanced concepts in one day when you just read them through from slides. To actually let the people try them out, we have to give them some time and therefore also to skip some subjects from the training. But in the end, I guess it is better to teach few things well, than a lot of things too quickly.

This has been an important learning experience and we are looking forward to making the next training even better than this one.

Creating Robot Framework jar distribution

2010-10-18T06:56:00.003+03:00

As of Robot Framework 2.5.2, it is also distributed as a single stand alone jar file.

The goal was to create a jar distribution with following properties:

Standalone executable jar file, i.e. java -jar robotframework.jar mystest.txt would suffice to run tests.
The exit code of the above command must be the same as if mytests.txt was executed with pybot.
There would also be a programmatic API to start Robot Framework test execution from any Java program

While there was a wealth of information about Jython-Java interaction available in the net, I could not find a single set of instructions describing all the necessary steps to create the required jar. Since I was able to do it and since the process is general, I decided to write it down.

First requirement is the Jython standalone jar, which can be created with the Jython installer. Following the instructions in the Jython Book, I created three Java classes and one Python for the interaction. These classes implement the one-to-one Jython object factory described in the Jython book. Source code of these classes is available in Robot Framework source repository.

The java classes had to (of course) be compiled before inserting them into the jar file. Additionally, a MANIFEST.MF file with correct main-class had to be created. Once these were done, the files could be inserted into the Jython jar:

Robot Framework source code must be placed in Lib directory inside the jar
The java code must be inserted starting from the root of the and according to package structure
The MANIFEST.MF has be to in META-INF directory

That's pretty standard stuff expect for one gotcha: Zip file format (and jar files are Zip files) supports multiple entries with same path. Since the Jython jar already contains MANIFEST.MF and since Python's zipfile module does not support overriding or removing entries, the first released version of the jar distribution contained two manifest files. For reasons unknown to me, the one we inserted was effective when the jar file was executed with java, but adding files to the jar file with jar failed because of the duplicate manifest.

This was resolved by unpacking the Jython jar and replacing the MANIFEST.MF with our own on the file system before repackaging. .

All the Python source files were compiled with Jython's compileall module before they were inserted into the jar file. This presumably increases performance slightly.

I will probably do another post about how we finally got the jar distribution uploaded to Maven central, since that's an interesting story too.

Implementing asynchronous Robot Framework keywords

2010-10-07T13:05:00.000+03:00

Before tests can be run I have to start all the necessary systems (processes, servers, stuff and things). Let's imagine that there are multiple systems that need to be started. I would like to start them all at the same time and wait until they are all ready to rock 'n' roll before starting my tests (assume that it would take a lot longer if I started all the systems in sequence).

So how to do this with Robot? Basic idea is to have re-usable keywords for starting each system and a keyword for waiting until the systems are ready so that testing can begin (and we don't have to use ugly and unreliable sleeps). Doing this on Robot keyword level makes it possible to have different combinations of systems in different tests.

*** Settings ***
Documentation   Example of using parallel things
Suite Setup     StartSystems
Library         SystemStarterLibrary.py

*** Test Cases ***
... Here should be my tests

*** Keywords ***
StartSystems
   ${SYSTEM1_STARTED}=  Asynchronously Start System 1
   ${SYSTEM2_STARTED}=  Asynchronously Start System 2
   Wait until  ${SYSTEM1_STARTED}  ${SYSTEM2_STARTED}

The way I'm going to implement this is by using python decorator that executes the function that it decorates in a separate thread. The decorated function will return the thread object so that it can be used to implement the waiting functionality.

I'm using this little code for the decorator.

After I've imported that to my SystemStarterLibrary.py I can implement system starter functions as normal functions.

@run_async
def asynchronously_start_system_1():
  .. do stuff to start system 1

@run_async
def asynchronously_start_system_2():
  .. do stuff to start system 2

Now all I need to do is to implement Wait until.

def wait_until(*stuff):
  for something in stuff :
     something.join()

This is kind of OK but it will wait forever if starting of some system will take forever. So it is better to have some timeout that will tricker setup failure after the timeout.

*** Keywords ***
StartSystems
    [Timeout]  5 minutes
    ${SYSTEM1_STARTED}=  Asynch Start System 1
    ${SYSTEM2_STARTED}=  Asynch Start System 2
    Wait until  ${SYSTEM1_STARTED}  ${SYSTEM2_STARTED}

And that should do it.

Hello World Robot Framework library

2010-10-07T08:40:00.000+03:00

This is the way I did my first Robot Framework keyword library. It should show the basic steps to add your own python keywords.

Let's do it in a test-driven way!

Failing test case

Add a file called HelloWorld.txt. This is our robot test suite file.

Add following text to the file:

*** Test Cases ***
HelloWorld
   Hello World

After this run pybot HelloWorld.txt - this will execute your Hello World test case.
Output should be something like:

==============================================================================
HelloWorld                                                                   
==============================================================================
HelloWorld                                                            | FAIL |
No keyword with name 'Hello World' found.
------------------------------------------------------------------------------
HelloWorld                                                            | FAIL |
1 critical test, 0 passed, 1 failed
1 test total, 0 passed, 1 failed
==============================================================================

Now we have a failing test case! So we can begin to implement our super cool Hello World keyword.

Keyword file

Add a file called HelloWorld.py to the same directory as our HelloWorld.txt test suite.

Add following text to the file:

def hello_world():
   print "HELLO WORLD!"

Now we have implemented our fine keyword that prints "HELLO WORLD!". Although our test still fails..

Passing test case

We have to import our super cool library to our test suite. Add following lines to the HelloWorld.txt (before test cases):

*** Settings ***
Library         HelloWorld.py

After this run pybot HelloWorld.txt - and watch it PASS:

==============================================================================
HelloWorld                                                                    
==============================================================================
HelloWorld.HelloWorld                                                         
==============================================================================
HelloWorld                                                            | PASS |
------------------------------------------------------------------------------
HelloWorld.HelloWorld                                                 | PASS |
1 critical test, 1 passed, 0 failed
1 test total, 1 passed, 0 failed
==============================================================================
HelloWorld                                                            | PASS |
1 critical test, 1 passed, 0 failed
1 test total, 1 passed, 0 failed
==============================================================================

Thats it.

Installing Robot Framework on Ubuntu

2010-10-07T08:09:00.000+03:00

Lately I've been learning to use the robot framework. These are my notes on how to install it.

Installing pybot - normal robot thing

First install python if it's not already installed.


sudo apt-get install python

Then install easy_install and robotframework.


sudo apt-get install python-setuptools
sudo easy_install robotframework

After this you should have pybot (normal robot thing) installed.


pybot --version
== Should output something like ==>
Robot Framework 2.5.4 (Python 2.6.5 on linux2)

Installing jybot - jython version of robot

I'll assume you have done all the things to install pybot so far (it also installed jybot and all you have to do now is to install correct version of jython).

First a word of warning: Ubuntu is still (you should check is this still valid point if your reading this in the future) using old version of Jython that doesn't work with current jybot.

So we have to first download Jython 2.5 (or later) from jython webpage. Follow jythons installation instructions.

Add JYTHON_HOME to PATH so that you can use it.

After this jybot should work.


jybot --version
== Should output something like ==>
Robot Framework 2.5.4 (Jython 2.5.1 on java1.6.0_20)

More conferences featuring Robot Framework

2009-09-24T01:13:00.003+03:00

Before the summer I wrote about conferences I was going to talk about Robot Framework or Acceptance Test-Driven Development (ATDD) using it. Most of those conferences are already in the past and the feedback has been pretty positive. It seems that the number of users is growing steadily and a lot of that is probably due to these presentations.

Here's an updated list of the conferences somehow featuring Robot Framework that are still ahead this year.

PyCon India 2009

- 26th-27th September, Bengaluru, India
- http://in.pycon.org/2009

Sai Venkatakrishnan gives a talk titled Driving Development using Examples - A Robot Framework Approach. The talk looks very interesting, and because the conferences looks also good otherwise and is pretty cheap, I highly recommend this to all Robot Framework users in Bengaluru.

Agile Testing Days

- 12th-14th October, Berlin, Germany
- http://www.agiletestingdays.com

As I already wrote earlier, I will give Introduction to Robot Framework talk with pretty self explaining content. A bigger news is that Elisabeth Hendrickson uses Robot Framework in her Acceptance Test Driven Development (ATDD) in Practice tutorial.

Agile Testing Days has a really good speaker lineup and I'm really looking forward to it myself. As a speaker I got these discount codes to share with you: SPEAKERSTUT010 gives 10% discount for the tutorials and SPEAKERCON020 20% for the conference.

Scandinavian Agile Conference 2009

- 15th-16th October, Helsinki, Finland
- http://www.scan-agile.org/

Juha Rantanen and Janne Härkönen and I organize similar Executable Requirements in Practice workshop/presentation we did at XP 2009 conference. For more information see http://code.google.com/p/robotframework/wiki/XP2009Workshop.

Scan-Agile is a local conference for me but you cannot really tell that from the excellent international speakers list.

Comments to Robot Framework review by Gojko Adzic

2009-06-08T02:23:00.014+03:00

Gojko Adzic organized Agile acceptance testing tools round-up on 27 May 2009 at Skills Matter in London. One of the tools he presented was Robot Framework and he published Robot Framework review on his blog afterwards.

Gojko's review is very well written, especially thinking that it is mainly based on the documentation on the project pages. The review is also pretty positive which obviously makes me happy. I had a change to see his text already before the presentation and some of my comments are included in the blog post. After re-reading the text I decided to add few more comments here. The comments below are organized into same sections as in the original review.

Keywords

It was great that Gojko spent some time to explain the keyword-driven concept behind Robot Framework and included my comment about how it makes the library API simple. I hope he had also shown that although this approach is simple, it's also very powerful and it is possible to create different kind of test cases using it. Robot Framework Quick Start Guide has very good examples of "normal" keyword driven tests, data-driven tests, and tests using the behavior-driven development (BDD) style.

Organising and editing tests

This is a good section about creating tests. Unfortunately the example, taken from the examples we use for explaining complex features, is pretty technical. For example the earlier mentioned Quick Start Guide has examples that are much better as tests. Normally when I talk how tests are organized into files and directories, I also mention how trivial it is to put them into a version control system. This is probably obvious to everyone reading the review, though.

Filtering

This section discusses tags and how they can be used for filtering tests and marking tests critical. Gojko's examples are very good, I especially like that they are from his own domain, and they illustrate these important features very well. The only thing I wish was mentioned too is how tags can be used to get statistics about passed and failed tests.

Compared to other tools

Gojko writes that Robot Framework is best suited for scripting tasks. Although I totally agree that RF suits very well for traditional scripted test automation, I'd like to highlight that creating also other kind of tests is possible. This is illustrated by the examples in the Quick Start Guide, and the executable requirements we created when developing RFDoc tool act as more realistic examples. In my opinion it is also a huge benefit that the same tool can be used for "normal" scripted testing, for creating BDD style executable specifications, and for constructing tabular examples with only input and output values.

One of the many points where I fully agree with Gojko is that Robot Framework can be used for creating domain specific languages. This can actually be done on many levels. For example SeleniumLibrary has its own web specific language with keywords like Click Link and Title Should Be, application specific language could have keywords like Input Password or Welcome Page Should Be Open, and finally on the specification level the language often sounds something like Given a user has logged in.

Reports and logs

The only bigger subject missing from the review was a section with this title. Reports and logs are something where Robot Framework is, at least in my opinion, ahead of other acceptance testing tools by a wide margin. The high-level report can be used for visualizing the overall status, detailed logs give accurate information about the test execution, and XML outputs can be easily integrated e.g. with CI systems and even combined to generate higher level reports.

Conference appearances

2009-05-25T03:00:00.006+03:00

I have been fortunate to be accepted to speak at several conferences in 2009. All these talks and workshops are at least related to Robot Framework, and they are listed below in chronological order.

UPDATE: See also a latter post about more conferences featuring Robot Framework.

XP2009

- 25th-28th May, Sardinia, Italy
- http://xp2009.org

I organize a workshop titled Executable Requirements in Practice with fellow Robot Framework developers Juha Rantanen and Janne Härkönen. The workshop is scheduled for Thursday, 28th May 14:00-17:30 and more information about it can be found from http://code.google.com/p/robotframework/wiki/XP2009Workshop.

EuroPython

- 28th June-4th July, Birmingham, UK
- http://europython.eu

My talk Acceptance Testing with Robot Framework will introduce the audience to Robot Framework and explain how to extend it with Python. For more information see http://www.europython.eu/talks/talk_abstracts/#talk33.

Agile 2009

- 24th-28th August, Chicago, USA
- http://agile2009.com

Elisabeth Hendrickson and I organize a demonstration titled Acceptance Test Driven Development (ATDD) in Practice. It is scheduled for Tuesday, 25th August 16:00-17:30, and we'll be presenting on the Main Stage. Full details are available at http://agile2009.com/node/641.

Agile Testing Days

- 12th-14th October, Berlin, Germany
- http://www.agiletestingdays.com

I give a talk Introduction to Robot Framework with pretty self explaining content.

The (in)famous first post

2009-05-05T01:18:00.002+03:00

Hello, world! My name is Pekka Klärck and I'm a tester-developer and software contractor from Finland. I am also the lead developer of Robot Framework, a highly flexible open source acceptance testing framework. I work through my one-man company Eliga Oy and you can find some more information about me and my services from those pages.

This blog is likely to contain a lot of posts somehow related to Robot Framework. Other topics that are likely to show up include test automation in general, Agile testing and Python programming. I don't plan to use this blog too much for personal posts, but that doesn't mean that I wouldn't ever write about my favorite hobbies like telemark skiing.

I've been thinking about starting a blog nearly a year and I'm happy to finally get this first post off my back. I hope it doesn't take that long time until the first post with some real content!