Covid Safety Solution Empowering businesses in a post-COVID world 
Next Generation Hotspot Experience seamless & secure Wifi connectivity 
Intelligent Monetization Platform Get meaningful insights from multiple customer attributes 
Unified Test Automation Framework OS & platform agnostic test automation framework 
August 24, 2016
Today’s world of software is all about delivering large number of quality product with fewer resources and limited time. With this paradigm shift, Test cycle needs to repeat numerous times to achieve the following:
Choosing a right tool can help an Organization in the following ways:
Test tool selection largely depends on the technology that the AUT is built on. A specific test tool might be pretty popular in its own space but might fail to address the testing needs for certain types of applications. For ex. QTP is a famous test automation tool but it cannot support Informatica so it cannot be used to test applications built on Informatica. Every tool has its own set of pros and cons. Before starting to invest in a particular test tool, it is important to do a fine print reading to find out ìWhat a tool canít doî. Generally, it is a common trend for all tools, open source or commercial, to highlight how mighty a certain tool is. What is often hidden under the hood are the things it cannot do. Before selecting an automation tool, it is highly advised to understand the technology that the application is built on and the test requirements. For instance, one may want to determine whether the application testing requires only functional or both functional and performance testing. Some applications also require Image testing/verification etc. Answers to these questions will help to gauge whether those requirements are fulfilled by a particular test tool. Few common points to ponder upon are:
This requirement is necessitated by the growing number of platforms day by day. Application developers have the inherent, implicit requirement to support the application on all possible platforms, which thereby imposes the requirement to test such application deployments on all supported platforms. Even in one variant of the platform, various versions need to be supported. For example, if a desktop application claims to support Windows, then it has to run on Windows 7 (32-bit/64-bit), Windows XP etc., or on Linux, then various flavours of Linux like RHEL, Ubuntu etc. Similarly, a mobile application could be supported on Android, iOS, Windows Mobile etc. For an automation tool this is an implicit requirement that once a test script or test case is written, it should be able to test the application on all supported platforms with a minimal change, say in a configuration file. While setting the goal for an automation tool, therefore, it becomes imperative to carry out an extensive study to ensure that it has support for doing cross-platform testing for all required platforms with maximum reuse of the test scripts.
This factor becomes very important once the tool has been selected and deployed or implemented. Before a tool is finalized, it should be considered how the end users will perceive it. This can be achieved by creating variations, gauging the impact of the tool and learning about its end users. These steps have to be considered before arriving at an automation tool that is targeted to address the needs of the organization and a more generic testing group. For instance, based on the complexity of the tool, the usage stage could either be painful or an appreciative phase. Few questions that should be asked from usage perspective are as follows:
As a starting step it is important to clear these questions. With the passage of time, as the experience and comfort level of teams increases, one can deploy a tool of varying complexities to service any type of test. But while starting out, it is important to achieve a basic threshold of test velocity. This helps to streamline the learning process.
Popularity of the tool directly relates to the chances of finding the relevant testing engineers for maintaining and using the testing tool. Less popular tools will have fewer engineers who would have worked on it earlier and thus new skills have to be acquired to understand and implement the tool. On the other hand, if the testing tool is popular enough there are high chances that several engineers have already worked on it or at least have some exposure to the tool during their experience. For instance, let us consider Selenium vs Sahi for web applications automation domain. While Selenium is more popular and widely used in the industry, Sahi is not so common. Popularity of a tool is directly proportional to the availability of support, technical forums and documentation quality.
Support and documentation have been an integral part of any technology product. This factor adds up exponentially if the tool is costly or complex. For commercial tools, there are various packages available under different license umbrellas, which come along with different levels and types of support. In-house trainings, developer services, consultation of course at some additional costs are some of the support activities associated with the above. For organizations that are low on IT or in-house development support, it is always wise to acquire extensive, hands-on support.
On the other hand, few open source programs also have these training sessions for an extra cost, but more often than not, you are on your own when choosing to go for an open source tool. In such a scenario, it becomes important that you carefully examine the tool support. Following is a list of few key items though not limited to, to consider before choosing an open-source:
An important concern while choosing any product or service is the cost. It is also the case when choosing a testing tool. The important thing to take care on this aspect is to balance between the needs and the budget. It might not be as simple as comparing the posted rates on the website. Every license cost is different in terms of the pricing of their products. Some offer monthly rates while some others offer annual. Some charge by the number of simultaneous users while others by the number of tests run. What fits your budget depends on the targets and situation as much it does on the amount of money to spend. Some tools offer a bare minimum service for a nominal license cost. However, it becomes important to read the fine print and find out the extent of the feature set provided. Each upgrade to the next set of features could be a costly affair which may also end up in becoming much more than the initial license cost. Other than the cost of the add-ons, there could be additional price like the support fee, training fee and upgrade fee.
Open source license is another lucrative option for many organizations; while commercial licenses guarantee better support open source license, if admissible for its no license fee weighs out lack of support in many cases. However, while choosing an open-source tool, the license agreement associated with the tool needs to be carefully assessed since there are too many open-source licenses, each with its own caveat. For example. GPL ensure open source, but any changes done to the software have to be returned to the community, and hence it can become an IPR issue for many organizations. Generally, before choosing an open-source testing tool and making any changes to the base software, it is advisable to carefully examine the use of the tool, whether it will be totally for internal use or needs to be redistributed.
There is a good chance that the organizations are already using a test case management or bug management tool. It is an obvious need that any new automation testing tool should be integrated with their existing test management tool so that the whole testing cycle can be managed alongside other available tools. This aspect is also worth noting before a new test tool selection is made. For example, QTP supports QLM, TestComplete support QAComplete.
Reporting is another critical factor while choosing an automation test tool. One important aspect is whether the tool allows customized reporting and what form the data can be exported/imported from the tool. If the organizations already have reporting system, it is important to check whether the same reporting can be derived from the new testing tool. In many tools, these options are built-in while in others, there are ways to make the report more comprehensive. If a tool gives a comprehensive report on failure, then it is the best for the organization.
While this factor is closely linked with the above point, it nevertheless becomes an important consideration to know while making a test tool selection as to whether a testing tool can work with already existing platforms. Some testing tools have ready plugins for common bug reporting and project management tools like Jira, Fogbugz, RedMine etc. It is important to carefully analyze the support of the integration with these tools available in the new automation tool. While in some cases, some API implementation may be required but that requires development skills to be integrated. It could be a simple integration with the e-mail service in an organization or an SMS-based reporting in case of failure, but these requirements could be essential for a testing program in an organization so should be carefully considered.
This helps to define the category of Test Automation Requirements like Test Cases that are to be automated and segregating them as per the relevant category along with the target audience from the usage part
Preliminary study shall be carried out in order to shortlist a tool and at this stage one should evaluate around 5-6 tools any one of which could qualify to be the final tool by satisfying ultimate choice.
As a next step, download an evaluation or trial version of the tool for refining the decision. At this stage this should clearly explain all your requirements of testing along with set of constraints to the tool supplier
This is a process followed by organizations to define a formal method of evaluating with respect to factors considered for selection
As an outcome of the above DAR Sheet, one can select 2-3 tools for Proof of concept.
The scope of the POC is defined by identifying the subset of test scenarios and the needs that have to be evaluated for the tool to be considered suitable. Once the scope is decided, POC can be commenced.
Results of the POC phase are compared and the most fitting tool can be selected.
