Systems and methods for automated classification and analysis of large volumes of test result data

• US 20050257086A1
• Filed: 04/21/2004
• Published: 11/17/2005
• Est. Priority Date: 04/21/2004
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Dar, Affan Arshad, Milbradt, Andre, Triou, Edward Jr., Agbonile, Osarumwemse U.
• Granted Patent: US 7,509,538 B2
• Time in Patent Office: Days
• US Class Current: 714/25
• CPC Class Codes: G06F 11/3688 for test execution, e.g. scheduling of test suites

Abstract

Test result data can be classified across multiple tests, test scenarios, and lab runs. Test result files can be parsed by extracting information from them. Extracted information can be compared to failure information in a database. If a match is found, the extracted information can be linked to the failure, creating a history for each failure. New failures can be identified when no match is found. Failure data can be cross-referenced to further aid in results analysis. For each failure, lists of useful information can be accessed. Analysis information can be associated with failures, for example whether the failure is new, occurred previously, is for a different reason than expected, or has been added to a baseline of expected failures. A Graphic User Interface (“GUI”) is also provided to expose the analyzed results to the result analyzers.

Claims

1. A method for analyzing test results, comprising:

comparing test result data corresponding to at least two test failures; and

determining at least one representative test failure corresponding to at least one related test failure; and

linking said at least one related test failure to said at least one representative test failure in a database.

2. A method according to claim 1, wherein said method is accomplished by a stored procedure in a database.

3. A method according to claim 1, further comprising cross-referencing said test result data corresponding to at least two test failures such that at least one property of said at least one related test failure is accessible from said at least one related test failure without accessing all properties of said at least one related test failure.

4. A method according to claim 1, further comprising exposing said at least one representative test failure through a Graphic User Interface (“

GUI”

).

5. A method according to claim 1, further comprising marking said at least one representative test failure as an expected failure.

6. A method according to claim 5, further comprising deemphasizing said at least one representative test failure in a GUI with respect to any unexpected failures.

7. A computer readable medium bearing instructions automated test result analysis, comprising:

instructions for comparing test result data corresponding to at least two test failures; and

instructions for determining at least one representative test failure corresponding to at least one related test failure; and

instructions for linking said at least one related test failure to said at least one representative test failure in a database.

8. A method for classifying test results, comprising:

extracting data from a test result file; and

comparing said data to failure characteristics stored in a database; and

linking said data from a test result file to said failure characteristics if said data from a test result file matches said failure characteristics.

9. A method according to claim 8, further comprising marking properties of test result files to be ignored during said comparing.

10. A method according to claim 8, wherein at least one of said failure characteristics is an abstract characteristic that can be matched by a variety to data from a test result file.

11. A method according to claim 8, further comprising adding new failure characteristics to said database if said data from a test result file does not match said failure characteristics, wherein said new failure characteristics correspond to said data from a test result file.

12. A method according to claim 11, further comprising emphasizing said new failure characteristics in a GUI with respect to failure characteristics that are not new.

13. A method according to claim 8, wherein said data from a test result file is describes results from a tested software operation.

14. A method according to claim 8 further comprising marking failure characteristics to indicate that the failure they represent is expected.

15. A system for analyzing test results, comprising:

a first process for extracting test result data from test result files; and

a second process for comparing said test result data and classifying said test result data into groups; and

a database for storing said groups; and

a Graphic User Interface (“

GUI”

) for exposing said groups stored in said database.

16. The system of claim 15, further comprising a test operation that produces an output test result file with properties that are marked to be ignored by said first process or by said second process.

17. The system of claim 15, wherein said test result files are the output of automated software tests.

18. The system of claim 15, wherein classifying said test result data into groups comprises cross-referencing at least one property of said test result data so all variations of the at least one property are grouped together.

19. The system of claim 15, wherein said groups are identified by representative test failures.

20. The system of claim 19, further comprising marking at least one representative test failure as an expected failure.

21. A method according to claim 20, further comprising deemphasizing said at least one representative test failure in the GUI.

22. A method for classifying test results, comprising:

extracting data from a test result file; and

comparing said data from a test result file to failure characteristics stored in a database, wherein first data that identifies a test operation is used in said comparison and second data that identifies a test scenario is not used in said comparison; and

if a match is discovered from said comparing, identifying said data from a test result file and said failure characteristics as a single failure.

23. A method according to claim 22, wherein said comparing is accomplished by a stored procedure in a database.

24. A method according to claim 22, further comprising cross-referencing said data from a test result file such that at least one property of said data from a test result file accessible from said single failure.

25. A method according to claim 22, further comprising exposing said single failure through a Graphic User Interface (“

GUI”

).

26. A method according to claim 22, wherein said data from a test result file is automated software testing data.

27. A method according to claim 26, wherein said data from a test result file is in Extensible Markup Language (“

XML”

) format.