While UI test automation is well understood for the desktop market, with a plethora of good tools available, the mobile market is different. Due to platform fragmentation and restrictions imposed by mobile device OSs, UI automation is a harder problem to solve. Over time, several tools have evolved that offer different mechanisms for UI automation for mobile devices, each with certain advantages and disadvantages.
The goal of this document is to compare and contrast some of the popular tools against various dimensions to help developers assess suitability of such tools for their need. The document limits itself only to provide information about the features / capabilities available in different tools and does not describe in detail on how to use that particular tool. Further, this document will only focus on Android and iOS when comparing such tools.
Mobile Test Automation Challenges
Test automation is the use of software to automate and control the setting up of test preconditions, execution of tests, test control and test reporting functions, with minimal user intervention. It is not practical to try to automate everything, especially for mobile devices. However, leveraging Common Off The Shelf (COTS) tools can greatly benefit the automation process.
The following are some of the reasons that make effective testing automation of mobile applications challenging:
A mobile device is much more restricted compared to desktops – the underlying OS typically sandboxes each application and allows very limited inter process access, unless a phone is ‘rooted’.
Controlling the UI navigation of a mobile device is harder – in addition to lack of control mentioned in the previous point, mobile UIX response times are harder to predict than desktop equivalents and hence makes screen grab based predictions of pass/fail harder.
Mobile devices, by definition are not statically located entities – depending on where the device is, the network may introduce many challenges that completely break down a prior tested use-case
The proliferation of different screen sizes and form factors make UI based testing more challenging as well
However, mobile test automation provides many advantages, including:
Improves testing efficiency
Consistent and repeatable testing process
Improved regression tests
More tests can be run in less time – Improved coverage in shorter time
24/7 operation – better resources utilization
Human resources are free to perform advanced manual tests
Simple reproduction of found defects
Types of Test Automation Tools
Figure 1: Mobile Application Development
Figure 1 describes the typical phases in mobile application testing and feature enhancement. Most mobile applications today are developed in tandem with an Agile software development methodology which involves very frequent interim releases. In such cases, implementing test automation can greatly help reduce iterative time to market. To achieve test automation it is worthwhile to discuss various tools available and the nature of testing they enable that could be aligned to the an Agile software delivery process. This is discussed next.
GUI Test Automation Tools for iOS and Android
There are quite a few GUI test automation tools available in the market. We have broadly categorized these tools as below:
Platform Specific Tools -. These are often provided by the Phone OS vendor (example, bundled by Google as part of the SDK, or by Apple as part of Xcode) and have integrated support with the SDK IDE. Platform specific tools are capable of testing the application both on the target device and on the emulator. The more popular platform specific tools are:
Monkeyrunner – Monkeyrunner provides an API for writing programs that can control an Android device or emulator remotely. With monkeyrunner, you can write a Python program that installs an Android application or test package, run it, send keystrokes to it, take screenshots of its user interface, and store screenshots on the workstation. This provides a powerful hook to integrate monkey runner with other Desktop based test tools to control the device and compare the resulting screenshots with reference screenshots for automation purposes. The monkeyrunner tool is primarily designed to test applications and devices at the functional/framework level and for running unit test suites.
Generic Script Based Tools– These tools are typically developed by third parties and usually control the execution of the application using scripts. These scripts can be written in a text editor or it can be generated automatically by recording events from your application. Below are the list of few tools in this category. In general, generic script based tools often need to integrate with some type of platform specific tool to complete automation, where the platform specific tool is responsible for communicating with the device, while the generic script based tool interfaces with it via an adaptor (that either is provided by the third party, or needs to be developed). Some of the popular tools here are:
Sikuli- Sikuli is a visual technology to automate and test graphical user interfaces (GUI) using images (screenshots). Sikuli includes Sikuli Script, a visual scripting API for Jython, and Sikuli IDE, an integrated development environment for writing visual scripts with screenshots easily. Sikuli Script automates anything you see on the screen without internal API support. You can programmatically control a web page, a Windows/Linux/Mac OS X desktop application, or even an iPhone or android application running in a simulator or via VNC.
Robot Framework- Robot Framework is a generic test automation framework for acceptance testing and acceptance test-driven development (ATDD). It has a easy-to-use tabular test data syntax and utilizes a keyword-driven testing approach. Its testing capabilities can be extended by test libraries implemented either with Python or Java, and users can create new keywords from existing ones using the same syntax that is used for creating test cases
Random Event Generator – There are many testing tools available, which test the stability of the application by sending random events to it. These tools are not intended to run any particular test case but quite often use to test stability of the application.
UI/Application Exerciser Monkey – Monkey (not to be confused with Monkeyrunner above) is a program that runs on an Android emulator on Android device and generates pseudo-random streams of user events such as clicks, touches, or gestures, as well as a number of system-level events. You can use Monkey to stress-test applications that you are developing, in a random yet repeatable manner.
Whitebox testing tools –Whitebox testing tools can examine the internal structure of the application via control flow testing, data flow testing, and code branch testing. For white box testing, the test developer needs to have access to the source code for application under test, or at the least be able to link their test library to the Application Under Test (AUT) for a special build. Following are some of the tools that can be used for whitebox testing of mobile applications:
Android Instrumentation: Android instrumentation test framework is based on JUnit. It allows JUnit to be used for testing of Classes/Method that does not use Android specific APIs. For testing of Android specific Classes/Methods Android extensions of JUnit are used.
Instruments: Instruments tool is part of the iOS application development IDE and allows Whitebox testing of iOS applications. Frank and TestStudio tools also allow Whitebox testing.
Blackbox testing tools –Blackbox testing tools check the functionality of the application and not how that functionality is achieved. Blackbox test tools do not require the test developer to have access to application code or to know how the code is organized or how it internally works. Following are some of the tools that can be used for Blackbox testing of mobile applications:
Sikuli: As explained earlier, Sikuli uses a visual technology for test automation and does not requires test developer to have any knowledge of the application code.
Robotium : Robotium is a test framework for Android that allows test developers to create blackbox as well as white box test cases for Android applications.
Subsequent sections will describe the applicability of some of these tools in more detail.
Sikuli is a visual technology tool to automate testing of graphical user interfaces (GUI) using image recognition. Sikuli scripts can be used to take a snapshot of a GUI element and compare it with some reference snapshot. Since Sikuli uses a visually driven approach (i.e.,drag/drop images to test against), a test script writer can write test cases without any knowledge of the target application code. Sikuli provides an IDE that allows for rapid development of test scripts with application screen shots. It allows automation of anything that is visible on a screen. Sikuli can be used to control a web page, a desktop application, an android application or even an iPhone application running in a Simulator.
An application that satisfies the following criteria can be tested with Sikuli:
Application screen can be transferred to a desktop (Windows/Mac/Linux) where Sikuli is being run.
User input to application can be transferred from desktop to application
Sikuli scripts are Jython scripts and can be extended to use other available Jython modules.
Sikuli uses images patterns to deliver mouse/keyboard events to appropriate GUI elements. Sikuli consists of two main parts
Java.awt.Rotot: This component of Sikuli delivers user events to the desired location on screen.
C++ Engine: Sikuli’s core image processing engine is based on OpenCV (Open Source Computer Vision). This component is responsible for searching image patterns on screen.
Sikuli for Mobile Platforms
Sikuli runs on Windows/Linux/Mac platforms. To be able to test mobile applications with Sikuli, the mobile phone/tablet screen should be transferred to Desktops where Sikuli is installed. Following are some of the ways to test mobile applications with Sikuli:
iOS application testing with Sikuli
One of the possible way to test an iOS application with Sikuli is to run the iOS application in an iPhone/iPad simulator within Xcode. The other possibility is to run a VNC server on the iOS device and run a VNC client on the desktop to get the iPhone/iPad screen on the Desktop. This approach has a disadvantage that it can only be used with a jail broken iOS device because VNC servers are not available for non-jail broken devices.
Android application testing with Sikuli
For testing Android applications there are two possibilities to transfer the mobile phone/tablet screen to the desktop.
Screen Cast application which uses the Android Debug Bridge
The Screen cast application can transfer an Android phone/tablet screen to the desktop. However, it requires the phone to be rooted to be able to pass keyboard/touch events to the phone. Most of the VNC servers also require rooted Android phones. One VNC server, “VNCLite” available at Google Play does not require a rooted phone and works well with Sikuli.
Note: For ICS and later releases, Screen cast application cannot pass type/touch event to phone even for rooted phones.
Below is a sample Sikuli test script. This test script automates the test case to add a news category and RSS news feed for Hughes Systique Pace application. Following is the list of activities performed by the test script
Click on Home button so that any running application is put in background
Click on application tray button and then launch “HSC Pace” Application
Wait for splash screen to finish
Click on button to add news category
Type the category name
Check for successful addition of category
Click on button to add RSS feed and then type RSS URL
Check for successful addition of URL
Does not require any knowledge of the application code/design
Easy and quick to automate
Scripts are not dependent on underlying platform
Can be used for testing any GUI
Sikuli provides a ‘fuzzy logic’ image comparison that can be adjusted during the testing process to determine degree of match.
Its not easy to match images if their dimensions/locations change – Sikuli offers various primitives but depending on nature of change, matching logic may or may not work.
Not suitable for performance testing of mobile application because mobile phone screen transfer to Desktop using VNC/Screencast is not very smooth. This is actually a VNC/Screencast limitation rather than Sikuli limitation.
Monkeyrunner (For Android)
Monkeyrunner tool is part of AOSP (Android Open Source Project) which provides APIs for writing programs that can control an Android device remotely. With MonkeyRunner, we can write python programs that install an Android application, send keystrokes to it, take screenshots of its user interface, and store screenshots on the Test Driver. The screen shots can then be compared with sample baseline screenshots and checked for accuracy.
MonkeyRunner, in general, provides the facilities for functional testing, regression testing as well as ways to extend the tool and write frameworks on top of it. MonkeyRunner provides access to three basic classes MonkeyRunner, MonkeyDevice and MonkeyImage. These classes provide a set of APIs which could be invoked from custom Python programs to create and execute test cases.
The Plug-in architecture of MonkeyRunner also allows users to add their own classes to the API.
It does not need to be installed separately as it is part of Android SDK.
The sample script below demonstrates a simple test case for the following actions:
Install an Android Application
Open a UI screen
Click on a button to open another screen
Takethe UI snapshot
Compare it with the reference snapshot
Note: Please note that com.android. monkeyrunner.easy class of monkeyrunner used in the code snippet above is still (Android Jellybean) under development and may change without notice.
MonkeyRunner can be used for testing on an actual device without any need to export screens to the desktop
MonkeyRunner can take screenshots for offline analysis. Screenshots can also be used for doing automated image comparisons against reference images using an inbuilt functionality or using external tools.
Monkeyrunner can be extended using a plug-in architecture
Monkeyrunner can be used to control multiple devices at the same time
Requires some knowledge of the application code; for example the user need to know the name of the activities and views.
MonkeyRunner is a low level API based tool – there is no GUI/IDE that is provided along with it by default.
UITestAutomation (FOR IOS)
‘Instruments’ is a performance, analysis, and testing tool for dynamically tracing and profiling OS X and iOS code.
Various Instruments tools
Instruments not only provides GUI Test automation tool but it has various other automation tools, which help a developers perform various kind of tests related to memory leaks, graphics, animation, time profiling and other areas as shown below:
It is part of the Xcode developer tool suite and it come as a pre-bundled tool with Xcode 4.0+.
Automated GUI Testing
There are however some limitations to the UI Automation instrument. If running against a device, that device must be running iOS 4.0+ and it must have hardware capable of multitasking (this won’t work on iPhone 3G or the second generation iPod touch).
Writing GUI Test Scripts
Below is a very simple GUI test script targeted for an iPad application. This script will generate random touch events for the application.
Interacting with user interface elements
One can simulate user interaction with the application interface in a number of ways. If we have buttons or other touch-responding elements, we can simulate a variety of touches and gestures. First, we need to find the correct element. To do that, one can use manual logging and traversal of the UI elements as described in the code fragment above, or one can do a more generic reference based on element type and name.
For example, the following will tap the button named “Equation library”, which is accessible via the root view in the application:
What’s happening here is that we’re accessing an array containing only the buttons within this view, then finding the one with the name Equation library. Once we have that item, we use the tap() function to simulate a user tapping on the button.
Instruments supports the following events/gestures:.
One can combine the various templates of Instruments with Automation.For example, start the test script and add Leaks / Allocations / Core Data / Time Profiler / System Trace / etc. to test other aspects of the application (memory usage, threads created, heap allocations, battery usage, and many more) while automating them. No other testing tool provides such facilities integrated together so conveniently.
More closely linked to device and works with device and simulator.
Supports all gestures (pinch, zoom, swipe, flick, long press, scroll, etc.)
Apple community support (apart from forums like stackoverflow).
Able to run native commands like ImageMagic shell command or other scripts to compare images along with advanced image comparison algorithms.
Requires linking with application to produce a special build
Tester needs to write custom test script to get consolidated report of all test cases .
Froglogic provides wide range of cross-platform GUI test automation tools, which specifically support the automation of tests based on QT. Froglogic has three main products:
Squish GUI Testing
Squish runs a small server (squishserver) that handles the communication between the Application Under Test (AUT) and the test script. The “squishserver” starts the AUT and injects the Squish hook into it. The hook is a small library that makes the AUT’s lives running objects accessible and that can communicate with the squish server. With the hook in place, the squish server can query AUT objects regarding their state and can execute commands—all on behalf of the squish runner. The following diagram illustrates how the individual Squish tools work together.
Squish supports QT based mobile application development. It also has support for native iOS application testing.
iOS native Apps testing
Testing iOS native Apps on an actual iPhone or iPad device is less convenient as compared to simulator testing. To do this one must add a Squish-specific wrapper library to Xcode, make a small modification to the application’s main function as written below:
Once the main function is setup correctly for Squish, then the developer needs to add a static library “libsquishioswrapper.a” into the Xcode project. This library is shipped with the Squish package and can be found in the package’s lib directory.
Provides script less test creation interface to tester similar to TestStudio.
It supports separation of test data from test cases, which allows reusability of test cases.
It supports composite logging in XML format and has a log viewer module,
Source code shipped with the tool so that one can enhance the tool for any extra support.
iOS module doesn’t support memory leak info, performance or load testing.
Frank is an open source UI testing tool for native iOS apps. Frank uses Cucumber and JSON through which once can write structured test/acceptance tests and have them execute against the iOS application. Frank also includes a powerful “app inspector” (called Symbiote) that one can use to get detailed information on the running application.
Create a file named navigation.feature inside Frank/features/ and write the test case in the file:
Create a step definition file called ‘navigation_steps.rb’ inside Frank/features/step_definitions/ write step definitions as follows: