DATA INDEPENDENT TEST SCIPTS

 Abstract

Following the introduction of the Trusted Test System (TTS) in 1995, Bankers Trust Australia Limited (BTAL) reduced the number of bugs per function point in its software by 50%.  Consequently the bank improved time to market and slashed its software development cost by 25%.

Comprising of a concept, a method and a set of tools, TTS can be used to test a wide variety of system interfaces including GUI and character cell.

This paper looks at what has been learned in BTAL about a concept central to automated testing; the detachment of test data from test scripts.  It begins by describing the benefits of separating test data from test Scripts.  It then looks at a way to use the concept to improve time to market and prevent bugs.  The paper concludes with a description of TTS, a system that detaches test data from SQA Robot and MS Visual Test Scripts.

Foreword

It's no secret that in today's world, responsiveness is at the heart of competitive advantage.  Time-to-market is critical to success.  While markets change rapidly and new products may be conceived almost overnight, system development takes much longer.  In the search for competitive advantage, it is therefore natural to centre our attention on how to speed up the development process.

At Bankers Trust Australia Limited (BTAL), software testing is the most time-consuming part of that process, accounting for about 50% of elapsed time.  So it is here that we found the biggest single opportunity to reduce time-to-market for new or revised products.

TTS, the Trusted Testing System, provides automated assistance to testers.  Not only does it dramatically reduce testing time, it reduces cost and improves quality as well. TTS is a concept, a method and a set of tools.  TTS may be used to test systems having a wide range of user interfaces including GUI and character-cell.

TTS was first used with the TRUST system, a fund manager’s unit holding registry.  On the first project, it halved test times and costs, while sharply increasing software quality.  In acceptance testing for this project, TTS halved incidents per function point compared with a similar project - clear evidence that TTS uncovers and prevents more bugs than manual methods.

Approaches to Automated Testing

TTS adopts a unique approach to automated testing that is best described by comparing TTS with conventional automated test tools such as Microsoft Visual Test and vanilla SQA Robot.

Conventional Automated Testing

With conventional test tools, automated system testing proceeds as follows.  The test tool watches while a QA Engineer runs the system under test and performs a system function:

·       Inputs that the QA Engineer applied to the system (such as keyboard input and mouse operations), and

·       Outputs that the system produced as a result (such as screen contents).

·       The QA Engineer saves the recording if the system under test ('the system') behaves as expected. This process is repeated with many combinations of functions and data.

·       At a later time, when a new version of the system has been developed, the QA Engineer can use these recordings for regression testing. 

In regression testing, the test tool plays back the earlier recordings to simulate a user, automatically applying the same inputs to the system, and automatically comparing the system outputs to those originally obtained.  For each system function, the test tool generates a report showing whether the outputs remain the same.  If the outputs do remain the same, the system passes the test.  The aim of regression testing is to develop confidence that changes incorporated in the new version have not unintentionally altered functions which were already present in the original version.

Disadvantages of Conventional Automated Tools

Automated test tools of this type remove some of the work associated with manual testing.  However the process can still be labour intensive:

·       The database must be set up before each test.

·       To make each recording, the QA Engineer must run the system and actually perform the system function.

·       Usually the number of combinations of functions and data is large, so 'nose-to-screen' time is very high.  Much of this work is repetitive and therefore error-prone.

·       Reports from the test runs are bulky. Reconciling test results is complex and takes a long time. 

·       Even trivial changes to functionality or screen layout will force complete re-recording.  Note that the test tool records inputs to the system quite blindly. Since the program has no knowledge of the business functions involved, or the structure of the input, it cannot ease the repetitive burden on the QA Engineer.

TTS

The remarkable difference between TTS and the test tools described above is that TTS can be given knowledge of the business functions.  By applying this knowledge, TTS is able to substantially reduce testing effort.  With TTS you first create a repository of simple reusable

modules, called TTS Scripts, covering the business functionality of the system.

Each TTS Script:

·       Has intimate knowledge of a single business transaction.

·       Can apply any desired input to the business transaction.

·       Can compare the actual output of the business transaction with the required output .

·       Can send a message to other TTS Scripts, to share information.

Drawing on this repository, you can quickly specify a whole battery of tests involving any combination of business transactions, input and required output.  To create this specification, or TTS Plan as it is called, you use a normal text editor.  The language is simple, highly compact, and mirrors the business functions in a natural way.

You do not have to run the system to write a TTS Plan, so 'nose-to-screen' time is minimised.  TTS runs your TTS Plan automatically and unattended.  During the run, TTS gets the system to process the business transactions referenced in the TTS Plan, and produces an extremely condensed report (Test Log) on the outcome of each.  Before running a TTS Plan, TTS verifies that the TTS Plan has no syntax errors.

 Advantages of TTS

TTS offers a number of advantages over other automated test tools:

·       TTS completely eliminates the most time-consuming aspect of test preparation, namely attended operation of the system during test development - the 'nose-to-screen' time described above.

·       Since the Test Logs are small and easy to read, interpretation of test results is fast and less prone to error.

·       Tests can be free-standing, without dependence on earlier tests or initial database content, because TTS Scripts provide a convenient mechanism for tests to initialise their own test beds.

·       Your investment in test suites is preserved over many versions of the system.  With minor changes, TTS Scripts and TTS Plans may be used with new versions of the system.

·       TTS improves time to market, because automated testing begins much earlier.  Test suites may be prepared from the Functional Specification while development is in progress.

·       Automated testing begins the day system development is complete. 

·       Because a TTS Script can handle any number of data combinations, the number of scripts is dramatically reduced.

·       TTS improves developer productivity by automating complex business logic during debugging.

·       For a given system, the repository of TTS Scripts forms a common resource which may be shared and re-used by all project teams, leading to productivity improvements.

·       Audibility is greatly enhanced.  Because TTS Plans mirror the business, it is easier to ensure adequacy of coverage. And the highly compact nature of TTS Test Logs makes for quicker and more thorough audit.

Disadvantages of TTS

TTS requires resources to write and maintain TTS Scripts, but this does not seriously erode the overall work-force savings.

Roles

With TTS, the following roles are defined:

Project Manager.

Works with the business line to determine testing requirements. Allocates testing resources, schedules testing phases and monitors progress.  Found in development and testing groups.

Needs a general knowledge of TTS and the ability to structure TTS efforts in support of project goals.

Test Administrator.

Runs TTS Plans and examines Test Logs to check whether the system passed all tests. Found in testing groups.  Must be familiar with the system under test and with the theory of software testing.  Needs in-depth knowledge of all aspects of TTS.  For larger projects, this role may be split between a Test Manager, who oversees the testing process, and a Test Administrator, who performs the actual tests.

Plan Writer.

Writes TTS Plans, working from the Functional Specification and other available specifications.  Found in development and testing Groups.  Must be deeply familiar with the system under test, and knowledgeable about all aspects of TTS except Script Programming.  Should have an appreciation of the theory of software testing.

Script Programmer.

Develops TTS Scripts for business transactions in the system under test.  Needs a sound knowledge of all aspects of TTS, knowledge of the system under test, familiarity with the elements of Visual Basic, a programming background and an appreciation of the theory of software testing.

Shell Programmer.

Maintains the TTS software and develops it further as needed.  Should be an experienced developer of system software (for example compilers) with a sound knowledge of Visual Basic.

An individual may assume different roles at different times, at the discretion of the Project Manager.  Roles relate simply to what an individual is doing at a given time.  Resource allocation is driven by project needs, not role definitions.  If Test Administrators are also Plan Writers, they should not test TTS Plans that they have written themselves.

Testing Life Cycle

With TTS, automated testing begins at an early stage.  Figure 2 shows the testing life cycle in the TTS environment.

Development

As the Development box (Figure 2) shows, the process begins with Project Initiation.  Project Definition, Analysis and Design stages follow, with the Functional Specification being written in the Analysis stage.  Construction then begins.  Finally the system enters the Deployment State.

Deployment stage.

These stages tend to be iterative, as indicated by the loops, and reversions to earlier stages can occur.  During the Construction stage, a number of Builds are released for testing, corresponding for example to development milestones.  Builds are also released for testing during the Deployment stage.

Testing

In the TTS box (Figure 2), we see that Test Planning begins as soon as Project Initiation is complete, and proceeds in parallel with Project Definition. A business decision may be made to begin Plan Writing and Script Programming while the Functional Specification is still in a draft stage. This gives a substantial time to market benefit, which must be traded off against the possibility of rework costs if the draft undergoes major change.

During Construction each new Build undergoes preliminary testing using the TTS Plans and TTS Scripts written to date.  Stubs may be used when needed.  In this iterative process, many defects in Builds, TTS Plans and TTS Scripts are uncovered and corrected before formal testing begins.  By the time the Construction stage is complete, the TTS Plans and TTS Scripts required will have been developed and proven.  Final overall testing can be started immediately. Of course, by this time any stubs in the system have been replaced with the required user interfaces.

Sometimes we need to refine (or develop) TTS Plans and TTS Scripts for functionality that is already present in the existing system.  In that case the existing system may be used to check out the TTS Plans and TTS Scripts involved.  This can be done at an early stage, for example while Specifications are still under development.

Functional and System Specifications seldom go down to the intricate level of detail that a Script Programmer requires.  The Script Programmer then is obliged to refer to a previous version of the system itself, for guidance.  In this situation there is a need to guard against unwarranted assumptions that the previous version works correctly. To minimise exposure in this area, the Script Programmer should isolate screen handling from other functions.  During unit testing, stubs may be present in the system under test.  However during user testing, only supported user interfaces are permitted. 

Monitoring

In the Monitoring box (Figure 2) we have Evaluation and Defect Tracking, which are complementary to TTS.  Evaluation begins as soon as tests are run on the first Build released by Programming.  All observed defects are entered into a repository which is structured to reflect the testing strategy.  SQA Manager, a component of SQA, Suite provides such a repository.

Defect Tracking begins as soon as Requirements Definition is complete, and continues until all Test Runs are finished.  Defect Tracking provides a means of assessing overall progress in the testing effort.

A question of fundamental importance is when to stop testing.  SQA Manager does provide some qualitative basis for this decision.  It can tell us, for example, that after three months we have tested 70% of the system and found five severe errors and twelve cosmetic errors.  There are limitations to such interpretations, because any test regime can cover only a tiny fraction of the system's functionality.  Project Managers need to draw on their experience of testing other systems.

TTS Plan

When a new release of a system is developed, there is a need to confirm it operates as expected.  To this end, the system will be subjected to a collection of automated and manual tests.  Automated tests are formally specified in text files known as TTS Plans. 

A TTS Plan details:

·       A sequence of business functions which the system under test ('the system') is to perform.

·       The outputs which the Plan Writer expects from the system when it performs these business functions.

TTS reads the TTS Plan and:

·       Converts the business functions into inputs for the system.

·       Applies these inputs to the system (which generates output as a result).

·       Compares the system's output to the expected output.

·       Creates a Test Log summarising the outcome.

·       Passes intermediate values to other TTS Plans.

A TTS Plan should be divided into Test Cases.  A Test Case groups related business functions. A TTS Plan invokes a sequence of TTS Scripts.  TTS Scripts are Visual Basic programs.  When planning a test, you select suitable TTS Scripts from an existing repository.  Each TTS Script corresponds to a single business transaction.  A sequence of one or more TTS Scripts corresponds to a Test Case.

TTS Script

A TTS Script is a Visual Basic program, corresponding to a single business transaction such as adding a new client. When you run a TTS Plan, the TTS User Interface (TTS Shell) invokes a sequence of TTS Scripts, as specified in the TTS Plan. The TTS Shell does this by asking SQA Robot or MS Visual Test to run the Visual Basic code in the TTS Scripts. SQA Robot is a component of SQA Suite, a commercially available PC application supporting automated testing.  When a TTS Script is invoked, it asks the TTS Shell to supply any parameters specified in the TTS Plan, such as:

·       Data to be entered into screen input fields and

·       Comparison values for screen output fields.

Using this information, the TTS Script directs appropriate inputs to the system under test ('the system'), and monitors what the system does in response.  Before exiting, the TTS Script supplies the TTS Shell with an execution status (success, failure or fatal error).  The criteria for success or failure are at the discretion of the Script Programmer.  Typically, a success status indicates that an action completed without error or that system outputs matched expectations.  The TTS Shell will terminate the test run if an error threshold has been exceeded, or if a fatal error status was returned.

The TTS Script may also supply the TTS Shell with a sequence of one or more return values (for example the ID of a new client).  Other TTS Scripts, invoked later in the test run, may use these return values.  Finally the TTS Script appends an entry to the Test Log, consisting of a success or failure indicator together with an explanatory text message.  The TTS Script may append an entry to the Test Log at any time during the test run.

For those familiar with SQA Team Test, it may be helpful to know that SQA Robot and MS Visual Test believe (wrongly) that the TTS Script is a recording of an actual test run.  To this end the TTS Script makes calls to SQA Robot and MS Visual Test, for example InitPlay, SetProcID, Window SetContext, WriteLogMessage and EndPlay.  In reality, TTS *.REC files are synthetic.  In the case of character cell systems like TRUST, TTS Scripts may call routines which sense fields on the screen, a fact of which SQA Robot or MS Visual Test are unaware.  From SQA Robot's perspective, the TTS Shell also appears to be a recording.  In reality the TTS Shell is a specially written program which causes a sequence of TTS Scripts to be executed.