Test Automation Framework

A test automation framework is a set of assumptions, concepts, and practices that provide support for automated software testing.

The main advantage of such a framework is the low cost for maintenance. If there is change to any test case then only the test case file needs to be updated and the Driver Script and Startup Script will remain the same. There’s no need to update the scripts in case of changes to the application.

This section gives the introduction about test automation framework, various types of the framework and the analysis of the best suitable framework for the application under test (the application under test is referred as AUT). This also includes the detailed description of the format of the input (the input to the framework is referred as the test tables) that is given to our test automation framework.

1.1 RECORD/ PLAYBACK MYTH

The test automation tool’s vendor market their product as the main feature of the tool is the ability to capture the user actions and later to playback them. Here is the basic paradigm(prototype) for GUI-based automated regression testing – the so called Record/Playback method (also called as Capture/Replay approach)

1. Design a test case in the test management tool.

2. Using the capture feature of the automation testing tool record the user actions. The result is a macro-like script where each user action is presented.

3. Enhance the recorded script with verification points, where some property or data is verified against an existing baseline. Add delay and wait states points where the different actions are synchronized.

4. Playback the scripts and observe the results in the log of the test management tool.

The basic drawback in this method is the scripts resulting from this method contain (1)hard-coded values which must change if anything at all changes in our AUT. The (2)costs associated with maintaining such scripts are astronomical, and unacceptable. These (3)scripts are not reliable, even if the application has not changed, and often fail on replay (pop-up

windows, messages, and other things can happen that did not happen when the test was recorded).

If the tester makes an error entering data, etc., the test must be (4)re-recorded. If the application changes the test must be re-recorded. All that is being tested are things that already work. Areas that have errors are encountered in the recording process (which is manual testing, after all). These bugs are reported, but a script cannot be recorded until the software is corrected. So logically nothing is tested by this approach.

So, avoid using “Record/Playback” as a method of automating testing. This method is fraught with problems, and is the most costly (time consuming) of all methods over the long term. The record/playback feature of the test tool is useful for determining how the tool is trying to process or interact with the application under test, and can give us some ideas about how to develop your test scripts, but beyond that, its usefulness ends quickly.

1.2 TYPES OF TEST AUTOMATION FRAMEWORKS

As we have eliminated Record/Playback method, let us explore about the existing automation methodologies. There are several test automation frameworks available, among these the selection is made based on the factors such as re-usability of both the scripts and the test assets. The different five basic test automation frameworks available are as follows,

Test Script Modularity

Test Library Architecture

Data-Driven Testing

Keyword-Driven or Table-Driven Testing

Hybrid Test Automation

1.2.1 Test Script Modularity

The test script modularity framework is the most basic of the frameworks. It creations small, independent scripts that represent modules, sections, and functions of the application-under-test. These small scripts are then used in a hierarchical fashion to construct larger tests, realizing a particular test case.

It’s a well-known programming strategy to build an abstraction layer in front of a component to hide the component from the rest of the application. This insulates the application from modifications in the component and provides modularity in the application design. The use of this framework will yield a higher degree of modularization and add to the overall maintainability of the test scripts.

1.2.2 Test Library Architecture

The test library architecture framework is very similar to the test script modularity framework and offers the same advantages, but it divides the application-under-test into procedures and functions (or objects and methods depending on the implementation language) instead of scripts. This framework requires the creation of library files (SQABasic libraries, APIs, DLLs, and such) that represent modules, sections, and functions of the application-under-test. These library files are then called directly from the test case script. Much like script modularization this framework also yields a high degree of modularization and adds to the overall maintainability of the tests.

1.2.3 Data-Driven Testing Framework

A data-driven framework is where test input and output values are read from data files (ODBC sources, CVS files, Excel files, DAO objects, ADO objects, and such) and are loaded into variables in captured or manually coded scripts. In this framework, variables are used for both input values and output verification values. Navigation through the program, reading of the data files, and logging of test status and information are all coded in the test script. This is similar to table-driven testing (which is discussed shortly) in that the test case is contained in the data file and not in the script; the script is just a “driver,” or delivery mechanism, for the data. In data-driven testing, only test data is contained in the data files.

1.2.3.1 Merits of data-driven testing

The merits of the Data-Driven test automation framework are as follows,

Scripts may be developed while application development is still in progress

Utilizing a modular design, and using files or records to both input and verify data, reduces redundancy and duplication of effort in creating automated test scripts

If functionality changes, only the specific “Business Function” script needs to be updated

Data input/output and expected results are stored as easily maintainable text records.

Functions return “TRUE” or “FALSE” values to the calling script, rather than aborting, allowing for more effective error handling, and increasing the robustness of the test scripts. This, along with a well-designed “recovery” routine, enables “unattended” execution of test scripts.

1.2.3.2 Demerits of data-driven testing

The demerits of the Data-Driven test automation framework are as follows,

Requires proficiency in the Scripting language used by the tool (technical personnel)

Multiple data-files are required for each Test Case. There may be any number of data-inputs and verifications required, depending on how many different screens are accessed. This usually requires data-files to be kept in separate directories by Test Case

Tester must not only maintain the Detail Test Plan with specific data, but must also re-enter this data in the various required data-files

If a simple “text editor” such as Notepad is used to create and maintain the data-files, careful attention must be paid to the format required by the scripts/functions that process the files, or script-processing errors will occur due to data-file format and/or content being incorrect

1.2.4 Keyword-Driven Testing

This requires the development of data tables and keywords, independent of the test automation tool used to execute them and the test script code that “drives” the application-under-test and the data. Keyword-driven tests look very similar to manual test cases. In a keyword-driven test, the functionality of the application-under-test is documented in a table as well as in step-by-step instructions for each test. In this method, the entire process is data-driven, including functionality.

1.2.4.1 Example

In order to open a window, the following table is devised, and it can be used for any other application, just it requires just changing the window name.

Test Table for Opening a Window

Window

Control

Action

Arguments

Window Name

Menu

Click

File, Open

Window Name

Menu

Click

Close

Window Name

Pushbutton

Click

Folder Name

Window Name

Verify

Results

Once creating the test tables, a driver script or a set of scripts is written that reads in each step executes the step based on the keyword contained the Action field, performs error checking, and logs any relevant information.

1.2.4.2 Merits of keyword driven testing

The merits of the Keyword Driven Testing are as follows,

The Detail Test Plan can be written in Spreadsheet format containing all input and verification data.

If “utility” scripts can be created by someone proficient in the automated tool’s Scripting language prior to the Detail Test Plan being written, then the tester can use the Automated Test Tool immediately via the “spreadsheet-input” method, without needing to learn the Scripting language.

The tester need only learn the “Key Words” required, and the specific format to use within the Test Plan. This allows the tester to be productive with the test tool very quickly, and allows more extensive training in the test tool to be scheduled at a more convenient time.

1.2.4.3 Demerits of keyword driven testing

The demerits of the Keyword Driven Testing are as follows,

Development of “customized” (Application-Specific) Functions and Utilities requires proficiency in the tool’s Scripting language. (Note that this is also true for any method)

If application requires more than a few “customized” Utilities, this will require the tester to learn a number of “Key Words” and special formats. This can be time-consuming, and may have an initial impact on Test Plan Development. Once the testers get used to this, however, the time required to produce a test case is greatly improved.

1.2.5 Hybrid Test Automation Framework

The most commonly implemented framework is a combination of all of the above techniques, pulling from their strengths and trying to mitigate their weaknesses. This hybrid test automation framework is what most frameworks evolve into over time and multiple projects. The most successful automation frameworks generally accommodate both Keyword-Driven testing as well as Data-Driven scripts.

This allows data driven scripts to take advantage of the powerful libraries and utilities that usually accompany a keyword driven architecture. The framework utilities can make the data driven scripts more compact and less prone to failure than they otherwise would have been.

The utilities can also facilitate the gradual and manageable conversion of existing scripts to keyword driven equivalents when and where that appears desirable. On the other hand, the framework can use scripts to perform some tasks that might be too difficult to re-implement in a pure keyword driven approach, or where the keyword driven capabilities are not yet in place. The following sections describe its architecture, merits and demerits.

1.2.5.1 Hybrid Test Automation Framework Architecture

The framework is defined by the Core Data Driven Engine, the Component Functions, and the Support Libraries. While the Support Libraries provide generic routines useful even outside the context of a keyword driven framework, the core engine and component functions are highly dependent on the existence of all three elements.

The test execution starts with the LAUNCH TEST(1) script. This script invokes the Core Data Driven Engine by providing one or more High-Level Test Tables to CycleDriver(2). CycleDriver processes these test tables invoking the SuiteDriver(3) for each Intermediate-Level Test Table it encounters. SuiteDriver processes these intermediate-level tables invoking StepDriver(4) for each Low-Level Test Table it encounters. As StepDriver processes these low-level tables it attempts to keep the application in synch with the test. When StepDriver encounters a low-level command for a specific component, it determines what Type of component is involved and invokes the corresponding Component Function(5) module to handle the task.

All of these elements of the framework rely on the information provided in the App Map to interface or bridge the automation framework with the application being tested. The App Map is the only means by which the framework could identify the objects in the application under test. Each of these elements is described in more detail in the following sections. The following figure shows the diagrammatic representation of the Hybrid Test Automation Framework.