Automatically generating software tests based on metadata

• US 7,272,822 B1
• Filed: 09/17/2002
• Issued: 09/18/2007
• Est. Priority Date: 09/17/2002
• Status: Expired due to Fees

• Current Assignee: Cisco Technology, Inc. (Cisco Systems, Inc.)
• Original Assignee: Cisco Technology, Inc. (Cisco Systems, Inc.)
• Inventors: Feldstein, Jeffrey, Hewitt, Jeffrey, Riggins, Michael
• Primary Examiner(s): Zhen; Wei
• Assistant Examiner(s): CHOW, CHIH CHING
• Application Number:
  US10/246,340
• Time in Patent Office: 1,827 Days
• Field of Search: 717/124, 717/127, 717/130, 714/38
• US Class Current: 717/124
• CPC Class Codes: G06F 11/3414 Workload generation, e.g. scripts, playback
• CPC Class Codes: G06F 11/3688 for test execution, e.g. scheduling of test suites
• CPC Class Codes: G06F 2201/865 Monitoring of software
• CPC Class Codes: G06F 9/451 Execution arrangements for user interfaces

Abstract

A method of automatically generating tests for a software program based on metadata that is used to control the operational behavior of a software product is disclosed. A definition is received of a page hierarchy and security characteristics associated with the software program and representing output pages that are generated by the program in operation. The definition is parsed to identify information associated with one or more pages of the program. Automatically, instructions in a test language are generated. The instructions, when executed by a testing system, cause the testing system to perform the steps of: navigating through the page hierarchy and displaying each of the pages; determining whether each of the pages is properly displayed by the program; and generating output indicating whether each of the pages is properly displayed.

Claims

1. A method of automatically generating tests, the method comprising:

receiving a definition for a set of application pages, wherein the definition comprises an ordering of how the application pages appear in a hierarchical menu system and is written in a first language, and wherein each page in the set of application pages is written in a second language and is one visual display among a plurality of visual displays that form a page hierarchy;

wherein the definition is a file;

parsing the definition to determine one or more page attributes associated with each page in the set of application pages, wherein the one or more page attributes comprise a screen identifier, hierarchy location, or a security level, and wherein the screen identifier, hierarchy location, or security level are associated with the each page in the set of application pages; and

automatically generating a plurality of test scripts in a third language, wherein the plurality of test scripts, when executed by a testing system, cause the testing system to perform the steps of;

placing the set of application pages under test;

using the one or more page attributes to navigate through the set of application pages;

displaying an output for each application page in the set of application pages;

determining whether the output for each application page is properly displayed; and

generating an indication of whether the output for each application page is properly displayed.

2. A method as recited in claim 1, wherein the test scripts, when executed by the testing system, further cause the testing system to perform the steps of:

for each user security level of a plurality of user security levels allowed by the set of application pages;

logging into the set of application pages under the each user security level;

navigating through the page hierarchy and displaying each page;

determining whether the each page is properly displayed based on the each user security level; and

generating an indication of whether the each page is properly accessed based on the each user security level.

3. A method as recited in claim 2, wherein the test scripts, when executed by the testing system, further cause the testing system to perform the steps of:

determining whether the each user security level is valid for the each page;

determining whether an invalid user screen is displayed when the each user security level is invalid for the each page.

4. A method as recited in claim 2, wherein the test scripts, when executed by the testing system, further cause the testing system to perform the step of determining whether all screen objects of the each page are displayed.

5. A method as recited in claim 1, wherein the test scripts, when executed by the testing system, further cause the testing system to perform the steps of:

navigating through the page hierarchy and displaying each page using a plurality of different navigation mechanisms;

determining whether the each page is properly displayed after navigation using the plurality of different navigation mechanisms.

6. A method as recited in claim 1, wherein the test scripts, when executed by the testing system, further cause the testing system to perform the steps of:

navigating through the page hierarchy and displaying each page using a plurality of different navigation mechanisms, wherein the navigation mechanisms are selected from among;

navigating and displaying using mouse clicks;

navigating and displaying using menu acceleration keys; and

navigating and displaying using a direct screen display call;

determining whether the each page is properly displayed after navigation using the plurality of different navigation mechanisms.

7. A method as recited in claim 1, wherein the parsing step further includes identifying a screen identifier value that is uniquely associated with each page, and wherein the step of navigating and displaying using a direct screen display call comprises the step of navigating and displaying a page by requesting a URL associated with the set of application pages and including the screen identifier value.

8. A method as recited in claim 1, wherein the parsing step further includes identifying a screen identifier value that is uniquely associated with each page, and wherein the step of navigating and displaying using a direct screen display call comprises the step of navigating and displaying a page by requesting a URL associated with the set of application pages and including the screen identifier value, using a bookmarking function of a runtime engine that controls execution of the program.

9. A method as recited in claim 1, wherein the testing system is Mercury Interactive WinRunner, and wherein the third language is Test Script Language.

10. A method as recited in claim 1, wherein the testing system is RobotJ, and wherein the third language is JAVA.

11. A method as recited in claim 1, wherein the definition comprises an Extensible Markup Language (XML) document that conforms to a Document Type Definition (DTD) comprising tags that define menu levels, menu level entries, screen identifiers, and security levels of the set of application pages.

12. A computer-readable storage medium carrying one or more sequences of instructions for automatically generating tests, which instructions, when executed by one or more processors, cause the one or more processors to carry out the steps of:

receiving a definition for a set of application pages, wherein the definition comprises an ordering of how the application pages appear in a hierarchical menu system and is written in a first language, and wherein each page in the set of application pages is written in a second language and is one visual display among a plurality of visual displays that form a page hierarchy;

wherein the definition is a file;

parsing the definition to determine one or more page attributes associated with each page in the set of application pages, wherein the one or more page attributes comprise screen identifier, hierarchy location, or a security level and wherein the screen identifier, hierarchy location, or security level are associated with the each page in the set of application pages; and

automatically generating a plurality of test scripts in a third language, wherein the plurality of test scripts, when executed by a testing system, cause the testing system to perform the steps of;

placing the set of application pages under test;

using the one or more page attributes to navigate through the set of application pages;

displaying an output for each application page in the set of application pages;

determining whether the output for each application page is properly displayed; and

generating an indication of whether the output for each application page is properly displayed.

13. A computer-readable storage medium as recited in claim 12, wherein the test scripts, when executed by the testing system, further cause the testing system to perform the steps of:

for each user security level of a plurality of user security levels allowed by the set of application pages;

logging into the set of application pages under the each user security level;

navigating through the page hierarchy and displaying each page;

determining whether the each page is properly displayed based on the each user security level; and

generating an indication of whether the each page is properly accessed based on the each user security level.

14. A computer-readable storage medium as recited in claim 13, wherein the test scripts, when executed by the testing system, further cause the testing system to perform the steps of determining whether all screen objects of the then-current page are displayed.

15. A computer-readable storage medium as recited in claim 12, wherein the test scripts, when executed by the testing system, further cause the testing system to perform the steps of:

navigating through the page hierarchy and displaying each page using a plurality of different navigation mechanisms;

determining whether the each page is properly displayed after navigation using the plurality of different navigation mechanisms.

16. A computer-readable storage medium as recited in claim 12, wherein the test scripts, when executed by the testing system, further cause the testing system to perform the steps of:

navigating through the page hierarchy and displaying each page using a plurality of different navigation mechanisms, wherein the navigation mechanisms are selected from among;

navigating and displaying using mouse clicks;

navigating and displaying using menu acceleration keys; and

navigating and displaying using a direct screen display call;

determining whether the each page is properly displayed after navigation using the plurality of different navigation mechanisms.

17. A computer-readable storage medium as recited in claim 12, the parsing step further includes identifying a screen identifier value that is uniquely associated with each page, and wherein the step of navigating and displaying using a direct screen display call comprises the step of navigating and displaying a page by requesting a URL associated with the set of application pages and including the screen identifier value.

18. A computer-readable storage medium as recited in claim 12, wherein the parsing step further includes identifying a screen identifier value that is uniquely associated with each page, and wherein the step of navigating and displaying using a direct screen display call comprises the step of navigating and displaying a page by requesting a URL associated with the set of application pages and including the screen identifier value, using a bookmarking function of a runtime engine that controls execution of the program.

19. A computer-readable storage medium as recited in claim 12, wherein the test scripts, when executed by the testing system, further cause the testing system to perform the steps of:

determining whether the each user security level is valid for the each page;

determining whether an invalid user screen is displayed when the each user security level is invalid for the then each page.

20. A computer-readable storage medium as recited in claim 12, wherein the testing system is Mercury Interactive WinRunner, and wherein the third language is Test Script Language.

21. A computer-readable storage medium as recited in claim 12, wherein the testing system is RobotJ, and wherein the third language is JAVA.

22. A computer-readable storage medium as recited in claim 12, wherein the definition comprises an Extensible Markup Language (XML) document that conforms to a Document Type Definition (DTD) comprising tags that define menu levels, menu level entries, screen identifiers, and security levels of the set of application pages.

23. An apparatus for generating user interface tests for a software program based on metadata describing the program, comprising:

a network interface that is coupled to the data network for receiving one or more packet flows therefrom;

a processor;

one or more stored sequences of instructions which, when executed by the processor, cause the processor to carry out the steps of;

receiving a definition for a set of application pages, wherein the definition comprises an ordering of how the application pages appear in a hierarchical menu system and is written in a first language, and wherein each page in the set of application pages is written in a second language and is one visual display among a plurality of visual displays that form a page hierarchy;

wherein the definition is a file;

parsing the definition to determine one or more page attributes associated with each page in the set of application pages, wherein the one or more page attributes comprise screen identifier, hierarchy location, or a security level and wherein the screen identifier, hierarchy location, or security level are associated with the each page in the set of application pages; and

automatically generating a plurality of test scripts in a third language, wherein the plurality of test scripts, when executed by a testing system, cause the testing system to perform the steps of;

placing the set of application pages under test;

using the one or more page attributes to navigate through the set of application pages;

displaying an output for each application page in the set of application pages;

determining whether the output for each application page is properly displayed; and

generating an indication of whether the output for each application page is properly displayed.

24. An apparatus as recited in claim 23, wherein the test scripts, when executed by the testing system, further cause the testing system to perform the steps of:

for each user security level of a plurality of user security levels allowed by the set of application pages;

logging into the set of application pages under the each user security level;

navigating through the page hierarchy and displaying each page;

determining whether the each page is properly displayed based on the each user security level; and

generating an indication of whether the each page is properly accessed based on the reach user security level.

25. An apparatus as recited in claim 24, wherein the test scripts, when executed by the testing system, further cause the testing system to perform the step of determining whether all screen objects of the each page are displayed.

26. An apparatus as recited in claim 23, wherein the test scripts, when executed by the testing system, further cause the testing system to perform the steps of:

navigating through the page hierarchy and displaying each page using an plurality of different navigation mechanisms;

determining whether the each page is properly displayed after navigation using the plurality of different navigation mechanisms.

27. An apparatus as recited in claim 23, wherein the parsing step further includes means for identifying a screen identifier value that is uniquely associated with each page, and wherein the step of navigating and displaying using a direct screen display call comprises the step of navigating and displaying a page by requesting a URL associated with the set of application pages and including the screen identifier value.

28. An apparatus as recited in claim 23, wherein the definition comprises an Extensible Markup Language (XML) document that conforms to a Document Type Definition (DTD) comprising tags that define menu levels, menu level entries, screen identifiers, and security levels of the set of application pages.

29. An apparatus for generating user interface tests, comprising:

means for receiving a definition for a set of application pages, wherein the definition comprises an ordering of how the application pages appear in a hierarchical menu system and is written in a first language, and wherein each page in the set of application pages is written in a second language and is one visual display among a plurality of visual displays that form a page hierarchy;

wherein the definition is a file;

means for parsing the definition to determine one or more page attributes associated with each page in the set of application pages, wherein the one or more page attributes comprise screen identifier, hierarchy location, or a security level, and wherein the screen identifier, hierarchy location, or security level are associated with the each page in the set of application pages; and

means for automatically generating a plurality of test scripts in a third language, wherein the plurality of test scripts, when executed by a testing system, cause the testing system to perform the steps of;

placing the set of application pages under test;

using the one or more page attributes to navigate through the set of application pages;

displaying an output for each application page in the set of application pages;

determining whether the output for each application page is properly displayed; and

generating an indication of whether the output for each application page is properly displayed.

30. An apparatus as recited in claim 29, wherein the test scripts, when executed by the testing system, further cause the testing system to perform the steps of:

for each user security level of a plurality of user security levels allowed by the set of application pages;

logging into the set of application pages under the each user security level;

navigating through the page hierarchy and displaying each page;

determining whether the each page is properly displayed-based on the each user security level; and

generating an indication of whether the each page is properly accessed based on the each user security level.

31. An apparatus as recited in claim 29, wherein the test scripts, when executed by the testing system, further cause the testing system to perform the steps of:

navigating through the page hierarchy and displaying each page using a plurality of different navigation mechanisms;

determining whether the each page is properly displayed after navigation using the plurality of different navigation mechanisms.

32. An apparatus as recited in claim 29, wherein the means for parsing further includes means for identifying a screen identifier value that is uniquely associated with each page, and wherein the means for navigating and displaying using a direct screen display call comprises the means for navigating and displaying a page by requesting a URL associated with the set of application pages and including the screen identifier value.

33. An apparatus as recited in claim 29, wherein the test scripts, when executed by the testing system, further cause the testing system to perform the step of determining whether all screen objects of the then-current page are displayed.

34. An apparatus as recited in claim 29, wherein the definition comprises an Extensible Markup Language (XML) document that conforms to a Document Type Definition (DTD) comprising tags that define menu levels, menu level entries, screen identifiers, and security levels of the set of application pages.