Test Automation Tools for Mobile Applications: A Brief Survey

Overview

Mobile Test Automation Challenges

Test automation is the use of software to automate and control the setting up of test preconditions, execution of tests, test control and test reporting functions, with minimal user intervention. It is not practical to try to automate everything, especially for mobile devices. However, leveraging Common Off The Shelf (COTS) tools can greatly benefit the automation process.

The following are some of the reasons that make effective testing automation of mobile applications challenging:

• A mobile device is much more restricted compared to desktops – the underlying OS typically sandboxes each application and allows very limited inter process access, unless a phone is ‘rooted’.
• Controlling the UI navigation of a mobile device is harder – in addition to lack of control mentioned in the previous point, mobile UIX response times are harder to predict than desktop equivalents and hence makes screen grab based predictions of pass/fail harder.
• Mobile devices, by definition are not statically located entities – depending on where the device is, the network may introduce many challenges that completely break down a prior tested use-case
• The proliferation of different screen sizes and form factors make UI based testing more challenging as well

However, mobile test automation provides many advantages, including:

• Improves testing efficiency
• Consistent and repeatable testing process
• Improved regression tests
• More tests can be run in less time – Improved coverage in shorter time
• 24/7 operation – better resources utilization
• Human resources are free to perform advanced manual tests
• Simple reproduction of found defects

Types of Test Automation Tools

Figure 1: Mobile Application Development

Figure 1 describes the typical phases in mobile application testing and feature enhancement. Most mobile applications today are developed in tandem with an Agile software development methodology which involves very frequent interim releases. In such cases, implementing test automation can greatly help reduce iterative time to market. To achieve test automation it is worthwhile to discuss various tools available and the nature of testing they enable that could be aligned to the an Agile software delivery process. This is discussed next.

GUI Test Automation Tools for iOS and Android

There are quite a few GUI test automation tools available in the market. We have broadly categorized these tools as below:

• Platform Specific Tools -. These are often provided by the Phone OS vendor (example, bundled by Google as part of the SDK, or by Apple as part of Xcode) and have integrated support with the SDK IDE. Platform specific tools are capable of testing the application both on the target device and on the emulator. The more popular platform specific tools are:
  • Instrument – Instrument is a Mac application provided by Apple as part of Xcode (IDE for IOS Application Development). A new feature called “Automation” was added in Xcode 4(onwards) to automate GUI testing of iOS Apps. The Automation instrument requires a basic understanding of JavaScript for test scripts development. Apple provides a set of JavaScript libraries that can be used to drive tests and simulate user interaction.
  • Monkeyrunner – Monkeyrunner provides an API for writing programs that can control an Android device or emulator remotely. With monkeyrunner, you can write a Python program that installs an Android application or test package, run it, send keystrokes to it, take screenshots of its user interface, and store screenshots on the workstation. This provides a powerful hook to integrate monkey runner with other Desktop based test tools to control the device and compare the resulting screenshots with reference screenshots for automation purposes. The monkeyrunner tool is primarily designed to test applications and devices at the functional/framework level and for running unit test suites.
• Generic Script Based Tools– These tools are typically developed by third parties and usually control the execution of the application using scripts. These scripts can be written in a text editor or it can be generated automatically by recording events from your application. Below are the list of few tools in this category. In general, generic script based tools often need to integrate with some type of platform specific tool to complete automation, where the platform specific tool is responsible for communicating with the device, while the generic script based tool interfaces with it via an adaptor (that either is provided by the third party, or needs to be developed). Some of the popular tools here are:
  • Sikuli- Sikuli is a visual technology to automate and test graphical user interfaces (GUI) using images (screenshots). Sikuli includes Sikuli Script, a visual scripting API for Jython, and Sikuli IDE, an integrated development environment for writing visual scripts with screenshots easily. Sikuli Script automates anything you see on the screen without internal API support. You can programmatically control a web page, a Windows/Linux/Mac OS X desktop application, or even an iPhone or android application running in a simulator or via VNC.
  • Robot Framework- Robot Framework is a generic test automation framework for acceptance testing and acceptance test-driven development (ATDD). It has a easy-to-use tabular test data syntax and utilizes a keyword-driven testing approach. Its testing capabilities can be extended by test libraries implemented either with Python or Java, and users can create new keywords from existing ones using the same syntax that is used for creating test cases
• Random Event Generator – There are many testing tools available, which test the stability of the application by sending random events to it. These tools are not intended to run any particular test case but quite often use to test stability of the application.
  • UI/Application Exerciser Monkey – Monkey (not to be confused with Monkeyrunner above) is a program that runs on an Android emulator on Android device and generates pseudo-random streams of user events such as clicks, touches, or gestures, as well as a number of system-level events. You can use Monkey to stress-test applications that you are developing, in a random yet repeatable manner.
  • Automation – By default iOS Automation template doesn’t have any random event generator but one can easily develop it utilizing the JavaScript and iOS automation API’s. Using automation JavaScript one can create custom modules which can provide all the capabilities like touches, gestures, system-level events and add the capabilities to test the application for custom designed tests suits.
• Whitebox testing tools –Whitebox testing tools can examine the internal structure of the application via control flow testing, data flow testing, and code branch testing. For white box testing, the test developer needs to have access to the source code for application under test, or at the least be able to link their test library to the Application Under Test (AUT) for a special build. Following are some of the tools that can be used for whitebox testing of mobile applications:
  • Android Instrumentation: Android instrumentation test framework is based on JUnit. It allows JUnit to be used for testing of Classes/Method that does not use Android specific APIs. For testing of Android specific Classes/Methods Android extensions of JUnit are used.
  • Instruments: Instruments tool is part of the iOS application development IDE and allows Whitebox testing of iOS applications. Frank and TestStudio tools also allow Whitebox testing.
• Blackbox testing tools –Blackbox testing tools check the functionality of the application and not how that functionality is achieved. Blackbox test tools do not require the test developer to have access to application code or to know how the code is organized or how it internally works. Following are some of the tools that can be used for Blackbox testing of mobile applications:
  • Sikuli: As explained earlier, Sikuli uses a visual technology for test automation and does not requires test developer to have any knowledge of the application code.
  • Robotium : Robotium is a test framework for Android that allows test developers to create blackbox as well as white box test cases for Android applications.

Subsequent sections will describe the applicability of some of these tools in more detail.

Sikuli

Sikuli is a visual technology tool to automate testing of graphical user interfaces (GUI) using image recognition. Sikuli scripts can be used to take a snapshot of a GUI element and compare it with some reference snapshot. Since Sikuli uses a visually driven approach (i.e.,drag/drop images to test against), a test script writer can write test cases without any knowledge of the target application code. Sikuli provides an IDE that allows for rapid development of test scripts with application screen shots. It allows automation of anything that is visible on a screen. Sikuli can be used to control a web page, a desktop application, an android application or even an iPhone application running in a Simulator.

An application that satisfies the following criteria can be tested with Sikuli:

• Application screen can be transferred to a desktop (Windows/Mac/Linux) where Sikuli is being run.
• User input to application can be transferred from desktop to application

Sikuli scripts are Jython scripts and can be extended to use other available Jython modules.

Sikuli uses images patterns to deliver mouse/keyboard events to appropriate GUI elements. Sikuli consists of two main parts

• Java.awt.Rotot: This component of Sikuli delivers user events to the desired location on screen.
• C++ Engine: Sikuli’s core image processing engine is based on OpenCV (Open Source Computer Vision). This component is responsible for searching image patterns on screen.

Sikuli for Mobile Platforms

Sikuli runs on Windows/Linux/Mac platforms. To be able to test mobile applications with Sikuli, the mobile phone/tablet screen should be transferred to Desktops where Sikuli is installed. Following are some of the ways to test mobile applications with Sikuli:

iOS application testing with Sikuli

One of the possible way to test an iOS application with Sikuli is to run the iOS application in an iPhone/iPad simulator within Xcode. The other possibility is to run a VNC server on the iOS device and run a VNC client on the desktop to get the iPhone/iPad screen on the Desktop. This approach has a disadvantage that it can only be used with a jail broken iOS device because VNC servers are not available for non-jail broken devices.

Android application testing with Sikuli

For testing Android applications there are two possibilities to transfer the mobile phone/tablet screen to the desktop.

• Screen Cast application which uses the Android Debug Bridge
• VNC server

The Screen cast application can transfer an Android phone/tablet screen to the desktop. However, it requires the phone to be rooted to be able to pass keyboard/touch events to the phone. Most of the VNC servers also require rooted Android phones. One VNC server, “VNCLite” available at Google Play does not require a rooted phone and works well with Sikuli.

Note: For ICS and later releases, Screen cast application cannot pass type/touch event to phone even for rooted phones.

Sample Script

Below is a sample Sikuli test script. This test script automates the test case to add a news category and RSS news feed for Hughes Systique Pace application. Following is the list of activities performed by the test script

• Click on Home button so that any running application is put in background
• Click on application tray button and then launch “HSC Pace” Application
• Wait for splash screen to finish
• Click on button to add news category
• Type the category name
• Check for successful addition of category
• Click on button to add RSS feed and then type RSS URL
• Check for successful addition of URL

Pros

• Does not require any knowledge of the application code/design
• Easy and quick to automate
• Scripts are not dependent on underlying platform
• Can be used for testing any GUI
• Sikuli provides a ‘fuzzy logic’ image comparison that can be adjusted during the testing process to determine degree of match.

Cons

• Its not easy to match images if their dimensions/locations change – Sikuli offers various primitives but depending on nature of change, matching logic may or may not work.
• Not suitable for performance testing of mobile application because mobile phone screen transfer to Desktop using VNC/Screencast is not very smooth. This is actually a VNC/Screencast limitation rather than Sikuli limitation.

Monkeyrunner (For Android)

Monkeyrunner tool is part of AOSP (Android Open Source Project) which provides APIs for writing programs that can control an Android device remotely. With MonkeyRunner, we can write python programs that install an Android application, send keystrokes to it, take screenshots of its user interface, and store screenshots on the Test Driver. The screen shots can then be compared with sample baseline screenshots and checked for accuracy.

MonkeyRunner, in general, provides the facilities for functional testing, regression testing as well as ways to extend the tool and write frameworks on top of it. MonkeyRunner provides access to three basic classes MonkeyRunner, MonkeyDevice and MonkeyImage. These classes provide a set of APIs which could be invoked from custom Python programs to create and execute test cases.

The Plug-in architecture of MonkeyRunner also allows users to add their own classes to the API.

It does not need to be installed separately as it is part of Android SDK.

Sample Script

The sample script below demonstrates a simple test case for the following actions:

• Install an Android Application
• Open a UI screen
• Click on a button to open another screen
• Takethe UI snapshot
• Compare it with the reference snapshot

Note: Please note that com.android. monkeyrunner.easy class of monkeyrunner used in the code snippet above is still (Android Jellybean) under development and may change without notice.

Pros

• MonkeyRunner can be used for testing on an actual device without any need to export screens to the desktop
• Good documentation
• MonkeyRunner can take screenshots for offline analysis. Screenshots can also be used for doing automated image comparisons against reference images using an inbuilt functionality or using external tools.
• Monkeyrunner can be extended using a plug-in architecture
• Monkeyrunner can be used to control multiple devices at the same time

Cons

• Requires some knowledge of the application code; for example the user need to know the name of the activities and views.
• MonkeyRunner is a low level API based tool – there is no GUI/IDE that is provided along with it by default.

UITestAutomation (FOR IOS)

‘Instruments’ is a performance, analysis, and testing tool for dynamically tracing and profiling OS X and iOS code.
Automation is a type of template in Instruments, which allows performing testing of user interfaces on iOS device/simulator. Testers can write test cases in JavaScript utilizing the UI Automation APIs (provided by Apple) and automate them using the Automation template. It enables testers to quickly track regression and performance issues. Automation comes with an in-line editor and it supports recording of events/gestures in an easy manner. These scripts/tests can be used to not only automate a user’s interaction with the application, but can also contain “asserts” which can test if the application is behaving according to the developer’s expectations. UITestAutomation is a part of the Automation template that Instruments invoke for the purpose of UI test automation.

Various Instruments tools

Instruments not only provides GUI Test automation tool but it has various other automation tools, which help a developers perform various kind of tests related to memory leaks, graphics, animation, time profiling and other areas as shown below:

It is part of the Xcode developer tool suite and it come as a pre-bundled tool with Xcode 4.0+.

Automated GUI Testing

UI Automation is an instrument that uses JavaScript to drive interaction with an application’s User Interface. These scripts can be structured as a series of tests to be run against the application. Based on the presence of certain UI elements or their value (object id), these tests can be made to pass or fail. Additionally, one can script repetitive tasks to run over a long duration, and then log performance data using other instruments, such as memory usage or CPU load.

There are however some limitations to the UI Automation instrument. If running against a device, that device must be running iOS 4.0+ and it must have hardware capable of multitasking (this won’t work on iPhone 3G or the second generation iPod touch).

Writing GUI Test Scripts

To get started with UI Automation testing, one will need to launch Instruments and select the Automation instrument. One can start with a blank document and based on the application need can write test scripts in simple JavaScript.

Below is a very simple GUI test script targeted for an iPad application. This script will generate random touch events for the application.

Interacting with user interface elements

One can simulate user interaction with the application interface in a number of ways. If we have buttons or other touch-responding elements, we can simulate a variety of touches and gestures. First, we need to find the correct element. To do that, one can use manual logging and traversal of the UI elements as described in the code fragment above, or one can do a more generic reference based on element type and name.
For example, the following will tap the button named “Equation library”, which is accessible via the root view in the application:

What’s happening here is that we’re accessing an array containing only the buttons within this view, then finding the one with the name Equation library. Once we have that item, we use the tap() function to simulate a user tapping on the button.
Supported Events/Gestures

Instruments supports the following events/gestures:.

• tap()
• tapAndHold(Number duration)
• doubleTap()
• twoFingerTap()
• touchWithOptions(Object options)
• dragInsideWithOptions(Object options)
• flickInsideWithOptions(Object options)
• scrollToVisible()

Pros

• JavaScript and UI Automation JavaScript API are fairly easy to learn.
• One can combine the various templates of Instruments with Automation.For example, start the test script and add Leaks / Allocations / Core Data / Time Profiler / System Trace / etc. to test other aspects of the application (memory usage, threads created, heap allocations, battery usage, and many more) while automating them. No other testing tool provides such facilities integrated together so conveniently.
• More closely linked to device and works with device and simulator.
• Supports all gestures (pinch, zoom, swipe, flick, long press, scroll, etc.)
• Apple community support (apart from forums like stackoverflow).
• Able to run native commands like ImageMagic shell command or other scripts to compare images along with advanced image comparison algorithms.

Cons

• Requires linking with application to produce a special build
• Tester needs to write custom test script to get consolidated report of all test cases .

Froglogic Squish

Froglogic provides wide range of cross-platform GUI test automation tools, which specifically support the automation of tests based on QT. Froglogic has three main products:

• Squish GUI Testing
• Squish Central
• Squish COCO

Squish runs a small server (squishserver) that handles the communication between the Application Under Test (AUT) and the test script. The “squishserver” starts the AUT and injects the Squish hook into it. The hook is a small library that makes the AUT’s lives running objects accessible and that can communicate with the squish server. With the hook in place, the squish server can query AUT objects regarding their state and can execute commands—all on behalf of the squish runner. The following diagram illustrates how the individual Squish tools work together.

Squish supports QT based mobile application development. It also has support for native iOS application testing.

iOS native Apps testing

Testing iOS native Apps on an actual iPhone or iPad device is less convenient as compared to simulator testing. To do this one must add a Squish-specific wrapper library to Xcode, make a small modification to the application’s main function as written below:

Once the main function is setup correctly for Squish, then the developer needs to add a static library “libsquishioswrapper.a” into the Xcode project. This library is shipped with the Squish package and can be found in the package’s lib directory.

Pros

• Provides script less test creation interface to tester similar to TestStudio.
• It supports separation of test data from test cases, which allows reusability of test cases.
• It supports composite logging in XML format and has a log viewer module,
• Supports five scripting languages – JavaScript, Perl, Python, Tcl & Ruby.
• Source code shipped with the tool so that one can enhance the tool for any extra support.

Cons

• •Commercial license
• iOS module doesn’t support memory leak info, performance or load testing.

Frank

Pros

• Comes with Symbiote, a free application inspector, which you can use to get detailed information on your running app.
• Record video of test runs to show the application in action.
• Works for both device and simulator
• Clean CSS-like selector syntax, increases readability.

Cons

• Very limited support for gestures.
• HTML app support is missing.
• As compared to other testing tools, it requires modifications in configuration files to run on real device.

Feature Comparision: Summary

References

• http://sikuli.org/
• http://groups.csail.mit.edu/uid/projects/sikuli/sikuli-uist2009.pdf
• http://developer.android.com/tools/index.html
• http://developer.android.com/tools/help/monkeyrunner_concepts.html
• http://developer.apple.com
• http://code.google.com/p/robotframework/
• http://docs.blackberry.com/en/developers/deliverables/11958/Event_injection_808536_11.jsp
• http://www.telerik.com/automated-testing-tools/support/documentation/mobile-testing/deployment/uikit-testing.aspx
• http://www.telerik.com/automated-testing-tools
• http://doc.froglogic.com/squish/4.2/testing..iphone..apps.on.an.iphone.html
• http://www.sunsetlakesoftware.com/sites/default/files/Fall2010CourseNotes/ui%20automation.html
• http://forums.pragprog.com/forums/248/topics/10999