Methods and systems for testing tool with comparative testing

• US 8,132,156 B2
• Filed: 06/14/2007
• Issued: 03/06/2012
• Est. Priority Date: 06/14/2007
• Status: Active Grant

• Current Assignee: Red Hat, Inc. (International Business Machines Corporation)
• Original Assignee: Red Hat, Inc. (International Business Machines Corporation)
• Inventors: Malcolm, David H.
• Primary Examiner(s): NGUYEN, PHILLIP H
• Application Number:
  US11/763,416
• Publication Number:
  US 20080313616A1
• Time in Patent Office: 1,727 Days
• Field of Search: 717/143, 717/144, 717/122, 717/126, 717/130, 717/157, 714/27
• US Class Current: 717/124
• CPC Class Codes: G06F 11/3688 for test execution, e.g. scheduling of test suites

Abstract

Embodiments of the present invention provide a multi-input comparative testing tool. The present invention looks at the internal structure of the software being tested and compares the two pairs of internal structures. In the present invention, the testing tool is capable of receiving multiple program code files or data files as its input. For each program code file, the testing tool generates a collection of problem objects and compares pairs of the problem collections with each other. The present invention may also run on data files having program code. For example, the present invention is capable of testing extensible markup language (XML) files to determine if they validate against a schema. Problem objects that are in both collections are identified and set aside. The remaining problem objects are then marked either as being resolved or new depending in which collection they exist.

Claims

1. A method of performing comparative testing on contents of a program code file, said method comprising:

parsing, by a computing device executing a testing tool, the contents of a new version of the program code file into a tree data structure;

generating, by the computing device, problem objects for errors in the contents of the new version of the program code file, wherein each problem object comprises a class and a pointer to a location in the tree data structure, wherein the class is used to classify an error in the new version of the program code file;

comparing the problem objects of the contents of the new version of the program code file to problem objects of a previous version of the program code file; and

marking at least one of the problem objects of the contents of the new version of the program code file or the problem objects of the previous version of the program code file with a level of significance based on the comparison, the level of significance reflecting a tolerability of the problem objects in the program code file, wherein problem objects that are present in only the new version of the program code file are marked as new problems and problem objects that are present in only the previous version of the program code file are marked as resolved problems.

2. The method of claim 1, wherein marking the problem objects comprises marking the problem objects that are present in multiple versions of the contents of the program code file as a low level of significance.

3. The method of claim 1, wherein marking the problem objects comprises marking the problem objects that are present in all versions of the contents of the program code file as a false positive error.

4. The method of claim 1, wherein marking the problem objects comprises marking the problem objects that are present in only a new version of contents of the program code file as a high level of significance.

5. The method of claim 1, wherein marking the problem objects comprises marking the problem objects that are present in only an old version of contents of the program code file as a medium level of significance.

6. A method of identifying errors in program code based on comparative testing, wherein the comparative testing is performed on at least a first program code file having a first collection of problem objects and a second program code file having a second collection of problem objects, said method comprising:

receiving, by a computing device executing a testing tool, the first program code file and the second program code file;

determining, by the computing device, a first structure of the first program code file and a second structure of the second program code file;

determining, by the computing device, the first collection of problem objects based on the first structure and the second collection of problem objects based on the second structure, wherein at least one problem object in the first collection of problem objects comprises a class and a pointer to a location in the first structure and at least one problem object in the second collection of problem objects comprises a class and a pointer to a location in the second structure, wherein the class is used to classify an error in at least one of the first program code file or the second program code file;

deleting problem objects that are present in both the first and second collections;

marking remaining problem objects in the first collection as being resolved in the second program code file; and

marking remaining problem objects in the second collection as being new in the second program code file.

7. The method of claim 6, wherein receiving the first program code file comprises receiving a program code file having a known set of errors.

8. A non-transitory computer readable medium including instructions for a comparative testing tool that, when executed by a processing device cause the processing device to perform a method, comprising:

receiving multiple program code files;

identifying, by the processing device using a problem collection generator executed by the processing device, collections of problem objects present in each of the multiple program code files based on an object oriented model that classifies errors in the multiple program code files based on a class and behavior, wherein the problem collection generator is configured to determine pointers for the problem objects that indicate a location of the errors in the multiple program code files, wherein each problem object in the collections of problem objects comprises a class and a pointer to a location of an error in one of the multiple program code files, the class classifying the error in one of the multiple program code files;

comparing, by the processing device using a set of comparison engines executed by the processing device, respective internal structures of the multiple program code files, pairs of the problem objects from the collections; and

determining which of the problem objects have been resolved and which of the problem objects are new.

9. The non-transitory computer readable medium of claim 8, the method further comprising parsing the multiple program code files into data structures that model contents of the multiple program code files.

10. The non-transitory computer readable medium of claim 8, wherein the set of comparison engines are configured to compare a specific class of the problem objects.

11. A non-transitory computer readable storage medium including instructions that, when executed by a processing device, cause the processing device to perform a method comprising:

receiving, by the processing device, a first program code file and a second program code file;

determining, by the processing device, first structures of the first program code file and second structures of the second program code file;

determining, by the processing device, a first collection of problem objects of the first program code file based on the first structures and a second collection of problem objects of the second program code file based on the second structures, wherein at least one problem object in the first collection of problem objects comprises a class and a pointer to a location in the first structures and at least one problem object in the second collection of problem objects comprises a class and a pointer to a location in the second structures, wherein the class is used to classify an error in at least one of the first program code file or the second program code file;

deleting problem objects that are present in both the first collection of problem objects and the second collection of problem objects;

marking remaining problem objects in the first collection as being resolved in the second program code file; and

marking remaining problem objects in the second collection as being new in the second program code file.

12. The non-transitory computer readable storage medium of claim 11, wherein receiving the first program code file comprises receiving a program code file having a known set of errors.

13. A computing apparatus comprising:

a memory to store instructions for a testing tool; and

a processing device coupled to the memory, wherein the processing device is configured to;

parse contents of a new version of a program code file into a tree data structure;

generate problem objects for errors in the contents of the new version of the program code file, wherein each problem object comprises a class and a pointer to a location in the tree data structure, wherein the class is used to classify an error in the new version of the program code file;

compare the problem objects of the contents of the new version of the program code file to problem objects of a previous version of the program code file; and

mark at least one of the problem objects of the contents of the new version of the program code file or the problem objects of the previous version of the program code file with a level of significance based on the comparison, the level of significance reflecting a tolerability of the problem objects in the program code file, wherein problem objects that are present in only the new version of the program code file are marked as new problems and problem objects that are presented in only the previous versions of the program code file are marked as resolved problems.

14. The computing apparatus of claim 13, wherein marking the problem objects comprises marking the problem objects that are present in multiple versions of the contents of the program code file as a low level of significance, marking the problem objects that are present in only a new version of contents of the program code file as a high level of significance and marking the problem objects that are present in only an old version of contents of the program code file as a medium level of significance.