Friday, February 20, 2015

Speeding up test execution with Pabot

Many projects end up in to situations where the time spend running automated test cases becomes a major problem. There are many methods that can help in this kind of situations. One of them is to parallelise the test execution. In best cases test parallelisation can significantly drop the actual time spend when running the automated test cases.

Pabot is a tool for parallelising Robot Framework test execution with multiple executors. One test suite file is executed by one executor. In this post I'll show how to make pabot work with your test cases.

Lets start with a non-parallelised test suite structure and work our way from there to parallelised version. Note that this is not a tutorial to Robot Framework. I expect that you are familiar with Robot Framework syntax.

*** Settings *** 
Suite Setup  Setup Systems 

*** Keywords *** 
Setup Systems 
  Clear items 
  Add users 
  Add items 

*** Test Cases *** 
Test 1 
  Connect to system  ${url} 
  Login  ${username}  ${password} 
  Modify item  ${itemID} 
  Verify modifications for item  ${itemID} 
  [Teardown]   Logout 

*** Test Cases *** 
Test 2
  Connect to system  ${url} 
  Login  ${username}  ${password}
  Delete item   ${itemID}
  Verify that item does not exists  ${itemID} 
  [Teardown]  Logout 

This imaginary suite structure has two test cases. Both login to the system and make modifications to some imaginary data items.

After we have this test material, to first thing that we need to do to start using pabot is to install it.
Pabot can be installed and updated with pip "pip install -U robotframework-pabot"
Pabot can be started from the command line with command "pabot suite". This will execute test cases in suite directory. Pabot supports all the command line options of pybot so it should be rather easy to replace a test execution script pybot calls with pabot calls.

Unfortunately parallelising can be tricky as it is with our example material. Sometimes the tests will interfere with each other.

First thing you might notice is that both tests in different suites by default modify the same item. Actually the second test case removes it. So might be that when running those suites at the same time suite1.robot will fail as the item has already been removed. We must ensure that the tests aren't using the same test data.

Both tests log in to the system with same username and password. This might also be a problem.

Setup phase is executed twice when using two parallel executors. Could result in double number of test material (or zero if timings are bad).

There are several ways that we could fix these problems. Pabot offers a shared remote library called PabotLib that can help. This library has implementations for locks, value set sharing and ensuring that some setup keywords are only executed once.

For the case of using different test data one might try using PabotLib with --resourcefile option.

To run the setup only once keyword PabotLib.Run Only Once will work.


Anonymous said...

This is a pretty clean example ,
But then How about an approach where different test suites can run on different test machines against different SUTS?
Our infrastructure currently works without a pabot? would like to adapated to it.

Mikko Korpela said...

This is something that I would loved to implement next :D

binith said...

Has anybody used pabot with selenium tests successfully ?

Anonymous said...

"""This is something that I would loved to implement next :D"""

Very good example. Have you been able to implement the above scenario where diff suites running on diff test machines?

Thank you.

Mikko Korpela said...

Yes it can be done with current pabot: