GENERATING TEST CASES
Background of the Invention Field of the Invention The present invention relates to testing of applications, and more specifically to a method and apparatus for generating test cases used to test an application.
Related Art
Testing generally refers to evaluation of an application (designed to perform a specific task) to determine conformance to requirement, the performance and/or reliability of the application. As used in the present application, an application may be implemented in the form of a combination of one or more of hardware, software and firmware.
A typical testing transaction entails providing a set of input values to an application and determining whether the application behaves in a desired fashion. Often, in applications such as graphical user interfaces implemented using software, the input values of a set are provided in sequence, and the corresponding output/ behavior is verified against the expected output/ behavior. The set of input/output values for a single transaction is referred to as a test case.
One challenge often encountered in application development is generation of test cases, generally to test the application exhaustively. In oiie approach, a test designer examines the specifications and/or uses the application to manually and intuitively identify the test cases. The identified test cases are then used to test the application (or various versions thereof as the application evolves with fewer problems/bugs and/or enhanced features).
One problem with such an approach is that a test designer may unintentionally miss identifying (or including) test cases, which could identify possible problems/bugs with the application implementation. Often such problems are uncovered later when the application is deployed on a wide-scale, with users using portions of the application not tested exhaustively. At least to avoid such unintentional omission of useful test cases, an improved approach to generate test cases is desirable. Generating exhaustive test cases systematically, to achieve
required test coverage remains to be problem faced by test designers.
Brief Description of the Drawings
The present invention will be described with reference to the following accompanying drawings:
Figure (Fig.)l is a block diagram illustrating an example system in which the present invention can be implemented.
Figure 2 is flowchart illustrating an example method using which test cases may be generated according to an aspect of present invention. Figure 3 is a display screen illustrating an example application using use cases for which test cases can be generated according to various aspects of the present invention.
Figure 4A illustrates use case diagram of 'Edit Proj ect' use case implemented according to an aspect of present invention.
Figure 4B is a display screen illustrating the manner in which 'Edit Project1 use case is represented using a flow-graph in an embodiment of present invention.
Figure 5 is a display screen which, depicts the test paths generated for 'Edit Project' in an embodiment of present invention.
Figure 6A is a display screen illustrating the manner in which variables (used later for defining test cases) of all the nodes in 'Edit Proj ect' use case flow-graph may be specified in an embodiment of present invention.
Figure 6B is a pop-up screen illustrating the manner in which variables and type may be specified for each node in an embodiment of present invention.
Figure 7 A is a display screen illustrating the manner in which input/output (I/O)' variables for each node may be specified in an embodiment of present invention. Figure 7B is a window 780 illustrating the manner in which input/output variables, type and values for each node may be specified in an embodiment of present invention.
Figure 8A is a display screen illustrating the manner in which test cases may be generated in an embodiment of the present invention.
Figure 8B is a display screen illustrating the manner in which a user may be provided a convenient interface for specifying various input variables corresponding to a test case according to an aspect of present invention.
Figure 9 is a network centric architecture illustrating implementation of several aspects of the present invention.
In the drawings, like reference numbers generally indicate identical, functionally similar, and or structurally similar elements. The drawing in which an element first appears is indicated by the leftmost digit(s) in the corresponding reference number.
Detailed Description of the Preferred Embodiments
1. Overview An aspect of the present invention examines a flow-graph representing the specification of an application, and generates test cases by traversing the flow graph. As the flow-graph permits flexibility such as representation of loops in an application flow/use, application flow/use can be effectively specified by flow-graphs. By traversing the flow-graph, test paths covering potentially all the paths (and all input combinations) of the applications can be systematically generated.
Several aspects of the invention are described below with reference to examples for illustration. It should be understood that numerous specific details, relationships, and methods are set forth to provide a full understanding of the invention. One skilled in the relevant art, however, will readily recognize that the invention can be practiced without one or more of the specific details, or with other methods, etc. In other instances, well-known structures or operations are not shown in detail to avoid obscuring the invention.
2. Example System Figure 1 is a block diagram of computer system 100 illustrating an example system in which various aspects of the present invention can be implemented. Computer system 100 may contain one or more processors such as central processing unit (CPU) 110, random access memory (RAM) 120, secondary memory 130, graphics controller 160, display unit 170, network interface 180, and input interface 190. All the components except display unit 170 may communicate with each other over communication path 150, which may contain several buses as is well known in the relevant arts. The components of Figure 1 are described below in
further detail.
CPU 110 may execute instructions stored in RAM 120 to provide several features of the present invention. CPU 110 may contain multiple processing units, with each processing unit potentially being designed for a specific task. Alternatively, CPU 110 may contain only a single processing unit. RAM 120 may receive instructions from secondary memory 130 using communication path 150. Data representing test cases of various paths of an application may be stored and retrieved from secondary memory 130 (and/or RAM 120) during the execution of the instructions.
Graphics controller 160 generates display signals (e.g., in RGB format) to display unit 170 based on data/instructions received from CPU 110. Display unit 170 contains a display screen to display the images defined by the display signals. Input interface 190 may correspond to a key-board and/or mouse, and generally enables a user to provide inputs. Network interface 180 enables some of the inputs (and outputs) to be provided on a network. Display unit 170, input interface 190 and network interface 180 may be used to enable a user to generate test cases for applications. Display unit 170, input interface 190 and network interface 180 may be implemented in a known way.
Secondary memory 130 may contain hard drive 135, flashmemory 136 andremovable storage drive 137. Secondary memory 130 may store the data (e.g., use-cases, flow-graphs, test cases, described in detailed in sections below) and software instructions (e.g., to generate test cases), which enable computer system 100 to provide several features in accordance with the present invention. Some or all of the data and instructions may be provided on removable storage unit 140, and the data and instructions may be read and provided by removable storage drive 137 to CPU 110. Floppy drive, magnetic tape drive, CD-ROM drive, DVD Drive, Flash memory, removable memory chip (PCMCIA Card, EPROM) are examples of such removable storage drive 137.
Removable storage unit 140 may be implemented using medium and storage format compatible with removable storage drive 137 such that removable storage drive 137 can read
the data and instructions. Thus, removable storage unit 140 includes a computer readable storage medium having stordd therein computer software and/or data. An embodiment of the present invention is implemented using software running (that is, executing) in computer system 100.
In this document, the term "computer program product" is used to generally refer to removable storage unit 140 or hard disk installed in hard drive 135. These computer program products are means for providing software to computer system 100. CPU 110 may retrieve the software instructions, and execute the instructions to provide various features of the present invention as described below in detail.
3. Method
Figure 2 is flowchart illustrating an example method using which test cases may be generated according to an aspect of present invention. The description is provided with reference to Figure 1 merely for illustration. Various aspects of the present invention can be implemented in other systems as well. The flowchart starts in step 201, in which control immediately passes to step 210.
In step 210, computer system 100 may enable a user to specify use cases associated with high level functionalities of an application sought to be implemented. In step 220, computer system enables the user to expand higher level use cases into one or more lower level use cases. Use cases are described in further detail in a book entitled, "Applying Use Cases", ISBN Number: 0-201-30981-5, by Gei Schneider and Jason P. Winters and "Object -Oriented Software Engineering A Use Case Driven Approach", ISBN Number: 0-201 -54435-0, by Ivar Jacobson et al. Example high level functionalities and lower level use cases are illustrated in further detail with an example in sections below.
In step 230, computer system 100 may enable the user to generate a flow graph representing each use case. A flow-graph generally contains several nodes connected by edges. A transition from one node to another generally occurs on an event (including providing an appropriate input(s) of a test case, generating a corresponding output(s), etc.). Each node may
be viewed as realizing a behavior of functionality as in specification for a corresponding portion of the application,, and then control is transferred, to a next.ngde asspecified in the flow graph. An example approach to represent a flow-graph is described below in further detail.
Each node may represent an atomic operation, a use case, a branch or a junction of multiple paths. A use case at a node can in turn be defined in further detail with lower level nodes and edges. The recursive definition of nodes representing use cases can continue until none of the nodes in a flow-graph at that level represents a.use case. In addition, loops in a flow graph enable repeated invocation of use cases and atomic operations. In conjunction with branches, loops generally provide a mechanism to implement complex functionalities.
In step 250, computer system 100 may identify various paths in the flow graphs by traversing the flow-graphs. The paths can potentially traverse a single use case or multiple use cases, potentially at different levels/granularity of definition. In an embodiment, a set of test paths are generated to thoroughly test all paths of a use case, and the tested use case is then used as a single node ('black box') while testing the application at a higher level of granularity. Such an approach enables the overall number of test cases to be minimized while still potentially exhaustively testing all paths of an application in structured and bottom-up manner.
In step 270, computer system 100 may enable a user to provide inputs and outputs to define the test cases for each path. Various type of inputs and outputs may be provided according to their data types, and various approaches may be employed in generating test cases based on the user inputs. In general, the test cases need to be designed to ensure an application operates as desired for correct input, and yet handles unexpected inputs gracefully.
Thus, in an embodiment, a user specifies the type of input(s)/output(s) (e.g., integer, text, character) and the range of permissible/correct input(s)/output(s), and computer system 100 may automatically generate test cases according to the formal testing techniques (boundary-value analysis, equivalence portioning etc.). Designing such test cases will be apparent to one skilled in the relevant arts by reading the disclosure provided herein.
In step 290, the test cases thus generated may be used to test the application. Such testing may be performed manually or using an appropriate automation technique (such as generating test scripts which automatically feed the correct sequence of inputs to an application and verifying the actual outputs against the expected outputs while testing). Such testing techniques are not described in the present application, as not being necessary for an understanding of the present invention. Control then passes to step 299, in which the method ends.
Thus, using the approaches of above, test cases can be generated for various applications. It may be appreciated that the flow-graphs are described as being described associated with use cases merely to integrate the approach into some common deign methodology, however various aspects of the present invention can be implemented without use of such use cases. The manner in which the approach(es) of above can be used is described in further detail with reference to an example application below.
4. Example Application
Figure 3 is a display screen illustrating an example application using use cases for which test cases can be generated according to various aspects of the present invention. The display screen is shown containing two windows 305 and 315. Window 305 is shown containing a root directory 'C-TestTool' which in turn is shown containing sub classes-'class diagram', 'use case diagram', Tester(user) and use cases which are shown in window 315.
Window 315 is shown containing user 310 specifying four use cases -Authenticate user320, Open Proj ect330, Edit Proj ect340, and Close Proj ect350. Each use case is assumed to represent a high level functionality. Each use case may be defined in further detail as described with reference to Edit Project340.
Figure 4A depicts the further detailed definition of Edit Prqject340. As may be appreciated the user is shown specifying that Edit Project340 uses the use cases - Save410 and Editwindow415. Editwindow415 is in turn shown using the use cases - Project Settings420, Set compile options425 and Environment settings430.
Figure 4B is a display screen containing two windows 440 and 445. Window 440 is shown containing a directory C-TestToolD which in turn is shown containing two use case flow graphs uc 1 EdifProj ectFlowGraph (representing Edit Proj ect) and 'uc2OpenProj ectFlowGraph' . User 310 may use ViewD option in the pop-up menu 495 to view the flow-graph in window 445. The user may continue to edit the flow-graph to represent the specification of the application sought to be tested.
Window 445 depicts the flow-graph for Edit Project340. The flow graph is shown containing starting node 435, decision boxes 450 and 480, junctions 460 and 470, Edit window415, Project Settings420, Set compile options425, Environment settings430, Save 410, and end junction 490. The nodes are shown connected by various edges. The manner in which the nodes and edges together specify application flow is described below in further detail.
Transition to decision box 450 from starting node 435 is shown to occur on event e2.
From decision box 450, control transfers to Edit window415 on event e3, and to end junction 490 on event e4. From Edit window415, control is shown transferring to junction 460 on event e5. From junction 460, control is transferred to Project Settings420, Set compile options425, Environment settings430 respectively upon events e7, e6, and e8.
Control transfers to junction 470 from Project Settings420, Set compile options425, Environment settings430 upon the occurrence of respective events el 4, e9 and el 5. From junction 470, control transfers to Save410 upon occurrence of event elO. From Save410, control transfers to decision box 480 on event el l. From decision box 480, control transfers to end junction 490 on event el3, and to Edit window415 on event el2. Thus, the flow graph of Figure 4B is shown containing a loop between nodes 415, 460, 470, 410 and 480.
The description is continued with reference to the manner in which test paths may be generated from flow-graphs. While a simple example is described for illustration, it should be understood that the approaches are applicable to complex applications as well.
5. Test Paths from Flow Graphs
Continuing with reference to Figure 4B, user 310 may use Create PathsD option in popup menu 495 to create test paths from the flow-graph in window 445. Figure 5 depicts the test paths generated for Edit Project340. As may be appreciated, the nodes in the flow-graph (of Figure 4B) are traversed to determine the various possible paths, which form the test paths.
Window 505 is shown containing seven paths (pi - p7), with each path indicating a test path. Some of the paths are described below for illustration. As may be appreciated, each path is conveniently identified by the event symbols associated with transitions. However, each path may be. identified by the corresponding nodes.
Path 510 is shown depicting test path P 1 = e2, e4. The path corresponds to a situation in which e2 event causes a transition to decision node 450, and e4 event cause control to be transferred to end node 490. As may be appreciated, transfer of control from edit project 340 to end node 490 indicates that user 310 opted to quit editing.
Path 520 is shown depicting P2 = e2, e3, e5, e6, e9, elO, el 1, el3, corresponding to a situation in which user 310 opts to set compile opt, save changes and quit editing. Test path P3 in path 530 is shown equaling (e2, e3, e5, e6, e9, elO, ell, el2), which corresponds to a situation in which a user returns to editing after saving changes related to the environment. The remaining test paths P4 - P7 are described similarly. The description is continued with reference to generating test cases from the test paths.
6. Generating Test Cases from Test Paths It may be appreciated that a user may need to provide different combinations of input/output values to test each identified path exhaustively. Various interfaces may be provided to simplify the task of providing such input/output values, as will be apparent to one skilled in the relevant arts such as functional testing techniques (such as boundary value analysis, equivalence class portioning etc.). Such interfaces are intended to be covered by various aspects of the present invention.
According to one approach, a convenient interface is provided enabling a user to specify the input/output variables associated with each test path. A number of test case can then be generated by providing a combination of input/output values for the specified input/output variables. Various combinations of input/output values can be provided to test the path exhaustively. The selection of the combination of input/output values can be provided based on the formal functional testing techiniques. Thus, the description is continued with reference to an example interface, which facilitates the user to specify various input/output variables that may be relevant to testing of each test path.
7. Identifying Input/Output Variables for Each Test Path
Figure 6 A is a display screen illustrating the manner in which variables of all the nodes in Edit Project use case flow-graph may be specified in an embodiment of present invention.
When a flow-graph represents the detail of a use case represented at higher levels granularity, the information on variables may be received from displays/interfaces corresponding to. such higher levels.
In window 610, the user is shown highlighting the use case 'EditProj ectFlowGraph', and causing pop-up menu 495 to be activated/generated by appropriate action (e.g., by clicking the appropriate button of a mouse). The user is then shown selecting Define VariablesD option to cause pop-up window 650 of Figure 6B to be displayed. The user is shown specifying the input/output variables (File Name, Path Name, Drive Name, Compiler, Java Version, Count) in column 654 and variable type in column 658. Similar to type, other attributes for variables can be added. These attribute information can be used to ensure that application can handle these variables gracefully.
The user indicates completion of specifying the input/output variables by clicking on button 651. The input/output data may be caused to be ignored by clicking on button 652. The information thus provided can be used in identifying the input/output variables for each path as described below in further detail with reference to Figure 7A.
Figure 7A is a display screen illustrating the manner in which input/output (I/O)
variables for each node (in the context of a test path) may be specified in an embodiment of present invention. The input/output variables thus specified may be consolidated to generate the input/output variables associated with each test path.
In window 760, the flow-graph corresponding to Edit Project340 is shown. The user is shown selecting (by underlining) test path P2 in window 710, and the corresponding path in window 760 is shown high-lighted (bold lines). Pop-up menu 775 is shown when the user clicks on Edit window415. Window 780 may be displayed when the user selects the 'Edit I/O' option.
Window 780 of Figure 7B is shown containing tables 785 and 789 respectively for indicating input variables and output variables associated with the selected path for node 415. Table 785 is shown containing three columns 781, 782, and 783 respectively representing Variables, Type, and default value (Vail). The input and output variables in columns 781 and 786 need to be contained in variables column 654 of Figure 6B, and appropriate checks may be performed before accepting the input variables specified in columns 781 and 786.
The variables specified in columns 654 and 658 may be displayed, and a user may be permitted to dragD each displayed variable as an input variable or output variable in columns 781 and 786. The implementation of several such interfaces will be apparent to one skilled in the relevant arts.
It should be understood that the input/output (I/O) variables for a node may be different in each test path. For example, when a node represents a use case, different paths internal to the use case may be tested in different test paths, and accordingly the I/O variables for a node may be different in different test paths. Thus, an interface such as that described in Figures 7A and
7B may be used to define the I/O variables and values for (all the nodes) in each test path.
The set of input output variables for a test path may be determined by combining the input/output variables specified for various nodes in the context of the test path. The determined set may be used to provide a convenient user interface to define test cases as
described below.
8. Defining Input/Output Values for Input/Output Variables
Broadly, several test cases may be generated from each test path by using corresponding number of combinations of input/output values for input/output variables identified for each path. In general, it is desirable to test a test path using different values based on formal testing techniques, some that are contemplated as expected values and some others which may be due to user errors. It is typically desirable to ensure that the application conforms to a desired behavior in case of expected inputs, in addition to gracefully handling (e.g., ignoring, with an appropriate comment) any unexpected input values (e.g., value of a number as an input when the expectation is a character).
Keeping such considerations in mind, various test cases can be generated from the test paths identified above, as described below with reference to Figures 8 A and 8B with examples. It should be further understood that not all input/output variables of a test path may need to be provided for a test case. A user (defining test cases) may determine such sub-set based on an understanding of the application-flow. The description is continued with reference to an example user interface enabling a user to define test cases for test paths.
Figure 8A is a display screen illustrating the manner in which test cases may be generated in an embodiment of the present invention. Window 810 is shown displaying various test paths, and the user is shown selecting one of the test paths (Underlined). Pop-up menu 830 is shown generated upon an appropriate user action (such as right-clicking on the selected test path). The user may generate test cases by selecting the option entitled/Define TC as described below with reference to Figure 8B.
Figure 8B is a display screen (generated in response to selection of 'Define TC option in
Figure 8A) illustrating the manner in which a user may be provided a convenient interface for specifying various input/output variables corresponding to a test case. As may be readily observed, columns 861, 862, and 863 respectively note the node name in a test path, the input/output variable name and variable type. Based on the information, the input/output values
provided are shown in column 864 and 865.
Assuming only three of the nodes in the path need to provided input/output variables and that the three nodes respectively have 3 (File Name, Path Name and Drive Name, as defined above with reference to Figure 6B), 3, and 1 input variables, the corresponding 7 input variables may be provided. Once the user clicks on button 871 , the test case may be accepted and saved. The saving of the test case may be reflected by an additional entry below the already present test case 812. A user may edit a test case displayed in area 810 by clicking on the test case.
Using the interface of Figures 8A and 8B, a user may specify as many test cases as desired for each test path. The input variables of a test case may be provided to an application by any convenient user interface (including just manually). Thus, the approach(es) and user interfaces described above can be used to generate test cases for an application.
For a test path, using formal testing techniques on the nodes, test cases can be generated automatically. In such automatically generated test cases, user may edit or modify the input/output values of variables by selecting the test case in area 810 and with user interaction 830
In general, the approach(es) described above may be implemented in various environments as will be apparent to one skilled in the relevant arts by reading the disclosure provided herein. Some general considerations in an example embodiment are noted below briefly.
8. Test Case Optimization and Test Completion Criteria
Using the test path generation from flow graph (Section 5) and generation of test cases from Test paths (Section 6), appropriate level of test case optimization and test stop criteria can be defined. The optimization of test cases can be derived based on the hierarchy of the flow graph, The loop counts, traversal of all nodes, traversal of all decision blocks in the flow graph, traversal of unique paths in the flow graphs, usage of formal techniques (equivalence, boundary
etc .) . Similarly the test coverage can be computed based on the paths actually traversed and test completion criteria can be achieved. The implementation of generation of optimal test cases, computation of such test coverage and test completion criteria will be apparent to one skilled in the relevant arts by reading the disclosure provided herein.
9. Implementation Considerations
In one embodiment, software instructions are implemented in computer system 100 to provide an editor (or editors) using which a user generates use cases (e.g., as in Figure 3) and flow-graphs (Figure 4B). The software instructions may keep track of the various use cases specified and provide a suitable interface by which the user can specify flow-graphs. For example, the use cases for which test cases are already generated may be shown in one color
(e.g., green), and the remaining use cases in another color (e.g., red) in area 440 of Figure 4B.
As the user continues to edit define a flow-graph for a use case, the software instructions need to store appropriate data representing the nodes and connections, which together define the topology of the flow-graph for the use case. Data structures such as linked lists and/or arrays (with appropriate pointers) may be used to represent the topology. The data structures may need to further enable storing of various I/O variable names associated with each node for each path.
The software instructions need to be designed to traverse the flow-graph to determine various test paths. The implementation of storing of topology, traversal and determination of test paths will be apparent to one skilled in the relevant arts by reading the disclosure provided herein. The test paths also may be stored in appropriate data structures.
The software instructions may further be designed to enable a user to specify various input variables associated with each test path to generate a test case. The test cases thus generated may be used for testing the application. While the above description is provided with reference to computer system 100 operating as a stand-alone system, it should be understood that test generation tools may be implemented using multiple systems. For example, a network- centric architecture depicted in Figure 9 may be used.
In such an environment, server system 950 (which may be implemented using a web server andor an application server, widely available in the market place) provides the various screens of Figures 3, 4A, 4B, 5, 6A, 6B, 7A, 7B, 8A and 8B in the form of web pages on network 940 (e.g., implemented using TCP/IP) to user system 910 (or 920), and the user provides the appropriate inputs, as described above, to generate test cases. In general, data is transferred between server system 950 and user system 910 representing the various screens and data exchanged. Thus, various features of the present invention can be implemented on standalone systems as well as networked systems.
10. Conclusion
While various embodiments of the present invention have been described above, it should be understood that they have been presented by way of example only, and not limitation.
Thus, the breadth and scope of the present invention should not be limited by any of the above described exemplary embodiments, but should be defined only in accordance with the following claims and their equivalents.