Monkey Testing: a Technique for Detecting Nasty Bugs

I'm going to tell about a technique that could help you find those bugs that no one else can find or are classified as "non-reproducible".

Some real life bugs

First let me show you that the technique also works with a real and recent 0.37 version of RIDE by demonstrating some hard to detect bugs that I detected with the tool I made. NOTE: These bugs have been in RIDE for very long time so they should be repeatable also in older versions.

First bug

1. Open RIDE


2. Open a keyword or a suite editor


3. Insert some text to the first row


4. Select the first row and select move row up from row context menu -- Nothing happens


5. Undo the last command (Ctrl+Z) -- produces a stack trace either to RIDE log or to command prompt

Second bug

This is a bit trickier but it has an issue in RIDE issue tracker -- unlike the previous one. Most likely because the underlying problem could have more ways to reveal it self than just this one.

1. Open RIDE


2. Open a keyword editor


3. Insert some text to the fifth line


4. Delete first row


5. Delete sixth row


6. Save file


7. Delete third row -- produces a stack trace either to RIDE log or to command prompt

Monkey testing

So how did I detect those bugs and how did I find a way to reproduce them?

I used a technique called monkey testing. It's name comes from the saying: "a thousand monkeys at a thousand typewriters will eventually type out the entire works of Shakespeare".
The basic idea is to make an automated system that randomly executes actions in the system under test. If the actions are selected in a way that will keep the system under test in a running state the test can just go on forever (or until it finds a bug).
To make the detected bugs reproducible the randomness in the test system must be controlled. Basic method is to control and log the seed value of the used (pseudo) random number generator.
Usually a failing test run will result in a very long trace (a list of executed actions) that needs to be pruned to find out the actions that resulted in the detected error. This pruning can of course be automatic.
You can find the related code from http://code.google.com/p/robotframework-ride/source/browse/#hg%2Frtest.

How do I know (well almost) that there is no more of these bugs around?

Because I have now let those monkeys run for several hours without catching anything. This gives me confidence to say that it is very unlikely that there are bugs that the monkey testing tool could produce and detect. In RIDE I mean that there most likely are no more non-GUI code bugs that will throw exceptions while editing test cases or keywords. And I can still put the monkeys to work to make me even more confident that the bugs that I have detected so far are all there is (I have corrected the bugs so the monkeys will not stumble on them again).
It is very important that a monkey testing tool can produce enough different kinds of actions to make it possible for it to express different types of defecting traces. The tool should also have good enough detection mechanisms so it will catch the defects -- but remember that more complexity means more complexity (the error could be in the tool if the tool is too complex).
In my case with the RIDE the detection mechanism has so far been only to catch exceptions but I've been thinking of taking some basic action related assertions in to use.
If you find this technique useful you could also check out model based testing to make monkeys that handle more complex action sequences and situations.

Super Sized JavaScript

The most technically challenging improvements in Robot Framework 2.6 are the new logs. They are about 1/10 of the size of the old logs and they are completely generated from very large JSON objects. One of the challenges in the new format is that the log is a single file that includes all the generated JavaScript, HTML and CSS code.

The largest new style log-files so far have been about 100 MB (that is a lot of JavaScript) and I believe that because we are working with such a big JavaScript objects we've encountered many difficulties that others have not yet have to deal with.

Although these files take time to download when loaded from servers, once loaded they work very well. Actually after we had first figured out how to make these almost 100 MB log-files work the reality came in and we had to figure out a way to split those large files to reasonable sized pieces (the method that we used is also explained in this post)..

Here's a list of tricks that we have learned during the process of super sized JavaScript development.

Doing large computations in JavaScript and how to prevent browser from freezing.

One of the first problems we had was that the extra large html/JavaScript files would freeze almost all browsers while expanding all the log elements in logs tree view. The final solution to this problem required some thinking and experimentation as the second point (not putting everything to the event queue) wasn't obvious.

JavaScript engines are one threaded and event based. This means that a big task will freeze everything. You can split your big task with setTimeout function to smaller parts. But if you split your task in too many parts and queue them all to the task queue the browser will again freeze as there is no space for users tasks.

The solution is to have a separate queue for the tasks that are generated during execution and to have only one task in the real event queue in any given time.

function timerTasker() {
    var currentTask = tasksQueue.nextTask();
    currentTask.do();
    tasksQueue.appendAll(currentTask.tasks()); //add tasks generated during current task execution
    if (!tasksQueue.isEmpty()) {
       setTimeout(timerTasker, 0);
    }
}

Lazy domain objects.

Everything should be as lazy as possible when dealing with a huge serialized data. It would be very sad to run out of memory or CPU when generating thousands of useless objects that no one will ever use.

IE9 JavaScript parsing out of memory.

Everything seemed to work ok even with very large files (Over 20 MB) but then we tried them in the IE9 and some of the logs just didn't work. After hard debugging we found that IE9 had a very odd problem that none of the other browsers had.

For some reason IE9 JavaScript parsing with reasonable small sized lists containing nested lists and integers and strings (the total size in our case was "only" 1.5 MB) will run in to out of memory error. This can be prevented by not mixing integers and strings.. In my opinion this could be a bug in IE9 as IE8 doesn't have these problems.

Too Large JavaScript.

After IE9 problem was fixed everything was again OK. Until we finally tried to generate extremely big JavaScript log files. This time the problems were with Firefox. Luckily these were simple problems with easy fixes -- just had to invent a clever way to split our data.

Firefox 4 (and 5) will at some point between 40MB - 80MB start to say that your JavaScript block is too large.. To prevent this use multiple JavaScript blocks instead of just one. The memory errors seem to occur during parsing, thus you can handle extremely large objects but you just have to keep the parser happy.

<script type="text/javascript">
    window.data = [.. [big data subelement] ..];
</script>

-- transform this to: --

<script type="text/javascript">
    window.dataSubElement = [big data subelement];
</script>
<script type="text/javascript">
    window.data = [.. window.dataSubElement ..];
</script>

Loading more JavaScript on the fly + Chrome safety.

100 MB log files were not something that all of our users wanted to work with. So we had to find a way to split the logs. The method had to work also locally. After brainstorming with ideas and spiking with techniqus we finally found a solution that works.

There are many ways to load more JavaScript while a page is open. I think that AJAX request to a server is the most common way but in our case there is no server.

There is a convenient function in jQuery for downloading new scripts ($.getScript) but it doesn't work for local files when using Chrome (@see Chrome issue 40787).

Our solution was to insert new script blocks to dom-tree on the fly (How to Dynamically Insert JavaScript). This works at least in our case.

Robot Framework 2.6 with new logs and reports has been finally released and everything seems to run smoothly (knock on wood). It was very interesting experience (with a lot of unexpected problems) to develop them. Respect to all my teammates!