automation robot

Choosing the best test automation tool requires you to understand that all test automation tools are not alike and that all software applications are different. Therefore, the best test automation tools FOR YOU may differ from those listed in the following post. That’s because they must have capabilities that match the software under test, and the software under test must have clear objectives for using test automation. But since you could be one of our potential clients, we will advise on the best test automation tools as experienced through the lenses of one of our clients, where we determined which test automation tool fitted their needs the best.

Key Criteria for choosing a test automation tool

If you are a software and trying to choose the best tool for software test automation, here are some of the things you should consider:

selection criteria

❒ Reporting and scheduling capabilities

❒ Integration with other tools such as test case and test planning tools

❒ Supports functionally rich applications and test scenarios

❒ Supports flexible parameterization and architecture

❒ User-friendly definition and execution of tests

❒ Support for various applications specific to platforms installed

❒ Supports user control of test execution

Whether you are considering QuickTest Professional, SilkTest, Selenium, TestComplete, or any of the many others, you need to consider which one of these criteria is important to you before choosing. This will depend on your software, your capabilities, and what your test automation goals are.

Evaluating test automation tools – Getting started

When doing an automation tool evaluation, you soon realize there are dozens of test automation tools out there and that your evaluation could be overwhelming. But after you consider the particular needs of your business and your software, it will help you narrow down the list.

It’s most important to understand the software under test and its critical business functions and hence the contextual requirements of the test automation tool. Special considerations include:

The need to support concurrent and integrated platforms; Windows, WebApp, and Mobile

The ability of the tool’s scripts to be integrated into your CI/CD pipeline, which could be Azure DevOps or various others such as Jenkins or CircleCI. 

Other criteria that could become more or less paramount as you work through your list of considerations given the context of the business and its software.


5 Of The Biggest Challenges In Automation Testing

By  | January 3rd, 2023

5 Of The Most Common Challenges In Automation Testing

Testing environment

We analyzed different test automation tools and determine automation feasibility for a desktop and a web-based application. After initial investigations of those tools that could work in both of these environments in an integrated manner, we are able to select TestComplete and WinAppDriver + Selenium to carry out a detailed comparison, including tool characteristics and implementation.

Candidate 1 = TestCompleteTestCompleteTestComplete
Candidate 2 = WinAppDriver + SeleniumWinAppDriverSelenium

Automation tools comparison

We used the most critical selection criteria to narrow the choices of tools to a handful.

test automation tool boxInitial tool selection criteria:

❒ Support both desktop and web applications with integrated solution

❒ Multi-language support

❒ Commercial license versus open-source

❒ Maintenance and support (if open source, then popularity and availability of community solutions)

Short-list of the 7 best test automation tools 

Using these criteria and our own experience and expertise, we shortened the list to the following tools:

testarchitect - a test automation toolTelerik Test Studio - a test automation toolAPPIUM - a test automation tool

 Selenium-based tools

test automation tool test complete

WINAPPDRIVER - a test automation tool

Selenium-based tools

Ranorex - a test automation tool

microfocus - a test automation tool

 (formerly HP QTP)

After an initial examination of the tools above, we further narrowed the choice down to two tools: TestComplete Versus WinAppDriver + Selenium for the following reasons.

The ultimate 2 best test automation tools

1. TestComplete           5 stars

test automation tool test completePositiveAmong the most popular test automation tools, TestComplete is a leading automation tool for Windows applications. 
PositiveOur team has accumulated more knowledge and experience with TestComplete than the other tools for Windows app automation testing.
PositiveUsers have the option of creating scripts by record and play, or manually creating keyword-driven scripts.
PositiveThe initial trial test scenario can be automated, so it passes the feasibility criteria that it can support test automation for both desktop applications and web applications in one project.
NegativeRequires licensing and maintenance costs. 
NegativeAdditional costs for additional modules and accessories.


TestComplete allows testers to create automated tests for Microsoft Windows, iOS, Web, and Android applications. It comes with a smart object repository and supports more than 500 controls.

2. WinAppDriver + Selenium           4.5 stars

winappdriver and selenium test automation toolPositiveIt is Selenium based; the most popular web based test automation tool.
PositiveSelenium has a larger community of users, supporters, plugins, and online forums to assist in troubleshooting any problems encountered.
PositiveThe initial trial test scenario can be automated with WinAppDriver, so it passes the feasibility criteria that it can carry test automation for both desktop applications and web applications with Selenium WebDriver in one project.
PositiveIt’s a free tool.
PositiveWinAppDriver is developed and backed by Microsoft, a strong support.
PositiveCompared to the other free tools on Windows App test automation, there are multiple element locating tools available, for instance, Inspect.exe, Spy++, etc.
NegativeThe test team needs to have good programming skills and experience in order to integrate WinAppDriver with other tools and frameworks.
NegativeNo built-in object spy tool and object repository, have to manually maintain object properties.
NegativeCompared with TestComplete, it takes longer to set up and maintain the framework, 
NegativeMore time required to get the tool itself updated or issues fixed or not, release info.

WinAppDriver + Selenium

WinAppDriver supports testing multiple platforms including Classic Windows (Win32) apps, Universal Windows Platform (UWP), Windows Presentation Foundation (WPF), and Windows Forms (WinForms) on Windows 10 PCs. The Windows Application Driver complies with the JSON Wire Protocol standard and some application management functionalities defined by Appium. 


The Ultimate Guide To Manual Testing vs Automation Testing

By |September 14th, 2022

thumbnail manual testing vs automation testing

Comparing these 2 test automation tools side-by-side

We did a more detailed analysis with our semi-finalists, TestComplete and WinAppDriver + Selenium. We exercised the tools for critical functions such as object identification and ease of and degree of platform integration. This section presents the characteristics of both tools.

CRITERIAtest automation tool test completewinappdriver and selenium test automation tool
Software Type❒ Windows-based application and thus cannot run on Linux/Unix systems.Selenium, 

❒ A set of APIs which can be used on at least three major operating systems: Windows, Linux, and macOS.


❒ Windows-based tool and thus cannot run on Linux/Unix systems.

Application Layer❒ Front end and API❒ Front end
Integration of Desktop App and Web App*❒ Yes❒ Yes
Object Identification*❒ It has an Object Recognition Engine with Artificial Intelligence. Selenium

❒ No such feature provided by Selenium.


❒ No built-in object spy. It requires a plug-in for WinAppDriver, such as inspect.exe (Windows SDK) and mentally maps inspect.exe properties to WinAppDriver properties.

Object storage and maintenance❒ Built-in object storage library to detect common objects❒ User Defined
Operating System support❒ Only supports Windows 10.Selenium,

❒ Windows, Linux and macOS.


❒ Only runs on Windows 10, not Linux, or macOS.

Language SupportJavaScript, Python, VBScript, Jscript, DelphiScript, C#, and C+ mentioned in the following doc❒ C#, Java, Python…
Record and Play*❒ Yes, supported.❒ Yes, supported. With UIRecorder in WinAppDriver, it can be used for recording, not able to playback.
Browser Support❒ Major browsers ❒ Major browsers
IDE Support❒ Supports only inbuilt TestComplete IDE❒ Eclipse, IntelliJ and any other IDE which supports Java
Test Report❒ Built-in Reporting❒ Need to integrate reporting tools
Test Script Maintenance❒ Easier❒ Maintenance becomes challenging with growing test scripts.
Learning Curve❒ Medium❒ High
Programming Skills❒ Suitable for non-technical to start, but advanced skills are required to build up a well-structured and  maintainable test automation.❒ Advanced skills are required to start and integrate various tools.
Support❒ Good documentation and support services.Many free online tutorials and help communities are available
Cost❒ License and paid.❒ Open Source/Free tool.
Application Type❒ Windows desktop app, web application, mobile application❒ Selenium, used for web app.

❒ WinAppDriver, Windows desktop app.

❒ Appium, mobile app.

General Integration❒ Can be integrated with many third-party tools.❒ Need to integrate with many other tools to make a powerful framework.
CI/CD Integration*❒ Yes❒ Yes
* Indicates the aspects that will be elaborated upon in the latter sections. Namely:

  • Integration of Desktop App and Web App*
  • Object Identification*
  • CI/CD Integration*
  • Record and Play*

Real-case Integration of Desktop App and Web App in Test Automation

Point of sales software Based on our experience with a cloud-based POS system client we know that TestComplete and WinAppDriver + Selenium can both implement test automation with integration of desktop and web applications. Both test automation tools apply WebDriver API for interacting with web applications, so the major difference between these two lies in the Windows POS application in automation tests. 


The following issues occurred during our research and exercises with WinAppDriver.

It’s not stable when launching POS. 

    • The issue encountered is not being able to pull the POS application on UI even while the process is running. We were unable to uncover the root cause of it, then moved on to a workaround solution to launch it from the terminal.

The object recognition takes more time compared to TestComplete.

    • Need to use multiple tools together to identify the objects.


This test automation implements the trial test scenario integrated with POS and Hub launching. It successfully launches both Windows app, POS and Web app, Hub under one TestComplete project. 

A TestComplete project goes in the following structure, containing

❒ Execution plan

❒ Object NameMapping

❒ Script

❒ TestedApp (Windows app specific)

The web application doesn’t get declared or listed under TestedApp even though it gets tested in our TestComplete project. This is because TestComplete delegates it to a browser driver.

Object Identification

Point of sales softwareOur initial exercise covered the following types/categories of objects in Windows application POS,

  • Button
  • Text 
  • Grid
  • Image
  • Frame

The common mechanism of object recognition leverages the following identification aspects of an object. Both TestComplete and WinAppDriver + Selenium use these mechanisms.

  • Properties or custom properties including multiple property combinations.
  • CSS selectors
  • XPath expression

All the objects covered in the trial test case for our POS and HUB client are identifiable within TestComplete and WinAppDriver + Selenium. The major difference between these tools in terms of object recognition lies in the object storing and maintenance. By default, TestComplete stores all the objects with which your tests interact in the Name Mapping repository. WinAppDriver does not have this capability, thus it requires automation test engineers to manually identify, store and maintain the objects.

In a nutshell, TestComplete has a better object identification and management mechanism. 

📌 Object Identification with TestComplete

Object Management

The Object browser within TestComplete helps save effort in object maintenance.

object management test complete

Complete Object Recognition Engine

Based on the built-in object management, TestComplete has AI-powered self-healing capabilities that will automatically identify unexpected errors and recommend an alternative. This prevents the test from failing and gives the opportunity to accept the fix after test execution.

Object Recognition Flexibility

This capability allows test engineers to use advanced methods to identify objects in an efficient manner including:

❒ Property combination

❒ Logical operation

❒ Regex

❒ Property priority for recognition

📌 Object Identification with WinAppDriver + Selenium

The mechanism of object identification in WinAppDriver.

1. With other tools, WinAppDriver detects objects. However, by itself, it doesn’t include an object identification feature. It is supported by some other tools, listed below:

  • inspect.exe
  • WinAppDriver UIRecorder
  • Accessibility Insights For Windows

2. Manually define the locators of an object.

Automation test engineers use their own judgement to select which properties to use as criteria.

Object clarification in WinAppDriver.

It is self-building from scratch for test automation in WinAppDriver. Object identification begins with a project file with class declaration.

Object clarification in WinAppDriver

The screenshot below presents the code on object definition with WinAppDriver, including attributes or properties to locate the objects and the method attached to the object.

code on object definition with WinAppDriver

CI/CD Integration Capability

In theory with the following official documents, TestComplete and WinAppDriver + Selenium can be integrated with Azure DevOps. And our team has experience of the test automation tool integration.

WinAppDriver with Azure DevOps

TestComplete with Jenkins

TestComplete, integrated with Azure DevOps

WinAppDriver + Selenium, integrated with Azure DevOps

Record and Playback 

Although Record and Playback is a good option when learning test automation tools and conducting simple proof of concept exercises, it is not a heavily weighted desired feature for test automation tools due to the following reasons.

High maintenance cost

These record and playback tools create the procedural code, but these procedural codes are a big problem. For every change in the code, it needs to be recorded. So doing this repetitively increases the maintenance cost.

Limited test coverage

The main drawback of this tool is that it does what you have previously recorded (nothing more or less). It means these tools typically do little more than the basic navigation testing. And these tools won’t do more complex verifications, like verify a page actually loaded.

Poor integration

The record and playback tools by and large don’t integrate well with your SDLC process.  They typically have their own interface and test runners (triggering tests). This often means you need to manually kick off tests.  This can also mean tests need exclusive control over the machine they are running on, which means the tester running the tests doesn’t do much while the tests are running.  Once you do have test results they are often in a flat file or “fancy” report format that follows a custom format for that specific tool. These custom formats make uploading the results to your test management or build tool challenging.

Issues with recording

With TestComplete, the POS objects which were not captured during recording, can be recognized during scripting.

For instance, 

The clock on POS, seconds in time are moving items.

The non-editable, non-selected control.

The object identifier is able to catch the text through XPath expressions or CSS selectors

record and playback


📌 Tool Costs

 1. TestComplete – Per License/Year Cost

Tool Combination CandidatesWhat’s IncludedNode-Locked – Perpetual term

Floating – Perpetual term

TestComplete Platform & Desktop + Web Modules❒ Object recognition engine

❒ Scriptless Record and Replay or keyword-driven tests

❒ Automated reporting and analysis

❒ Desktop and web testing

TestComplete Platform & Desktop + Web + Mobile ModulesObject recognition engine

❒ Scriptless Record and Replay or keyword-driven tests

❒ Automated reporting and analysis

❒ Desktop, mobile and web testing

TestComplete Pro BundleObject recognition engine

❒ Scriptless Record and Replay or keyword-driven tests

❒ Automated reporting and analysis

❒ Desktop, mobile and web testing

❒ Access the parallel testing engine, TestExecute


The cost of purchasing the TestComplete license will be impacted by the following aspects:

  1. Task arrangement
  2. How the team is structured or distributed
  3. Which stage your test automation is at
  4. Tool license option selected

#1 to #3 should be raised in the test strategy and depend on client’s input here. 

testcomplete pricing range


The recommended license purchase method is floating TestComplete Pro Bundle based on the following:

    • ✅ Desktop, Mobile and Web testing are all included in a single license.
    • TestExecute is included, no need to purchase separately.
    • ✅ Same service, better price.


2. WinAppDriver 

No cost, it’s a free tool.

📌 Indirect Costs

Indirect cost is the expense beyond the tool itself. The cost consists of the following areas at different levels as estimated.

Cost ItemCost Level of TestCompleteCost Level of WinAppDriver + Selenium
1Automation test project setupMediumHigh. Start from Scratch.
2Automation test maintenanceMediumHigh
3Automation Test Development Collaboration (knowledge share, transfer, etc)MediumHigh. Steep learning curve.
4Automation test execution setupMedium.High

Our ultimate tool recommendation

We recommend TestComplete as the best test automation tool based on the Pros and Cons listed above, key considerations, and cost. The weaknesses of WinAppDriver may cause critical blockers on some key scenarios testing or result in more time spent. 

Contact us today for your Software Automation Testing.

automation testing contact button

We Recommend

5 Of The Most Common Challenges In Automation Testing

5 Of The Biggest Challenges In Automation Testing

By  | January 3rd, 2023

Discover the biggest challenges in automation testing, experienced even by seasoned practitioners.

mobile testing in the cloud

The 4 Best Platforms For Mobile Testing In The Cloud

By  | December 28th, 2022

In this blog post, we will explore how mobile testing in the cloud works, the benefits they offer, and some of the limitations to be aware of. We will also provide examples of some commercially available cloud-based testing platforms and discuss their unique features and capabilities.

banking domain testing - a complete guide

Banking Domain Testing: A Complete Guide

By  | November 28th, 2022

The banking, financial services, and insurance (BFSI) industry works with sensitive information — and this makes it a prime target for cyberattacks. That’s why any piece of technology that the BFSI sector uses — banking applications, banking systems, and other software — calls for thorough testing.