Software reliability analysis using alerts, asserts and user interface controls

• US 7,681,085 B2
• Filed: 06/15/2007
• Issued: 03/16/2010
• Est. Priority Date: 06/15/2007
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Mahmud, Adnan Azfar, Li, Yantao, Wang, Qiang, Zhang, Haidong, Dang, Yingnong, Wang, Jian, Zhu, Wenli
• Primary Examiner(s): Lohn; Joshua A
• Application Number:
  US11/818,612
• Publication Number:
  US 20080313507A1
• Time in Patent Office: 1,005 Days
• Field of Search: 714/26, 714/37, 714/38, 714/45, 714/46, 714/47, 714/57, 709/224
• US Class Current: 714/45
• CPC Class Codes: G06F 11/008 Reliability or availability analysis
• CPC Class Codes: G06F 11/3616 using software metrics
• CPC Class Codes: G06F 11/366 using diagnostics G06F11/0703 takes precedence

Abstract

Described is a technology by which software instrumentation data collected during software program usage sessions is analyzed to identify potential problems with software program usage, such as based on frequency of problem occurrence during the usage sessions. Reliability metrics may be calculated from the information. Failure data additionally collected during the usage sessions may be accessed to derive details that correspond to the potential problems. In one example, the information may be analyzed to determine which alerts and/or asserts occurred most often, and/or to determine a relationship between user interface control operations (e.g., clicks and usage of commands) and alerts or asserts.

Claims

1. A method performed by a computer, the method comprising:

analyzing a set of software instrumentation data collected and stored by software instrumentation during software program usage sessions to identify potential problems with at least one software program corresponding to the software instrumentation data, wherein a usage session comprises a session of use of the software program by a user, wherein the potential problems are identified based on frequency of occurrence of asserts during the usage sessions, wherein the software instrumentation data is collected at least in part by executing the asserts in the program, where each assert comprises a statement including a logical condition that is evaluated when the assert is executed, wherein when the logical condition evaluates to true the assert statement has no effect on the program and when the assert statement evaluates to false corresponding indicia is recorded in the software instrumentation data; and

accessing failure data additionally collected during the usage sessions, the failure data having been collected by a crash handling system separate from the software instrumentation, the failure data comprising one or more crash dumps captured by the crash handling system when the software program crashed, and locating, within the failure data, portions thereof that correspond to the identified potential problems, and obtaining details of the potential problems from the portions of failure data that correspond to the potential problems.

2. The method of claim 1 wherein analyzing the software instrumentation data includes filtering the software instrumentation data to locate a subset of session data corresponding to only program sessions that terminated abnormally.

3. The method of claim 2 wherein analyzing the information further comprises processing the subset to determine which alerts or asserts, or both alerts and asserts, occurred most often within that subset of sessions.

4. The method of claim 2 wherein the analyzing further comprises calculating reliability metrics from the software instrumentation data.

5. The method of claim 4 wherein calculating the reliability metrics comprises computing at least one of:

crash information corresponding to a number of sessions of a user that crashed, failure information corresponding to a number of sessions that failed of a user, mean time to crash information corresponding to session length and sessions of a user that crashed, mean time to failure information corresponding to session length and number of sessions of a user that failed, mean commands to crash information, or mean commands to failure information, or any combination of crash information, failure information, mean time to crash information, mean commands to crash information, or mean commands to failure information.

6. The method of claim 5 further comprising, outputting a distribution mapping for at least one calculated reliability metric.

7. The method of claim 1 further comprising, determining the frequency of consecutive failures for a software product.

8. The method of claim 1 further comprising, determining which computer programs including at least one application of a selected software product were running concurrently with the software program when a problem occurred with the software program.

9. The method of claim 1 wherein analyzing the information comprises determining a relationship between user interactions with a interface control and indicia of alerts or asserts in the software instrumentation data.

10. The method of claim 1 wherein the analyzing comprises performing an analysis on a selected group of users, or a selected program version, or a combination of a selected group of users and a selected program version.

11. A computer-readable medium having computer executable instructions, which when executed by a computer cause the computer to perform steps comprising:

using recorded session data stored by the computer to form a set of users determined to be having problems with a software product the session data comprising execution traces generated and recorded by the software product while the software product was being operated by the users, and determining from the recorded session data one or more reliability metrics comprising at least one of;

crash information corresponding to a number of sessions of a user that crashed, failure information corresponding to a number of sessions that failed of a user, mean time to crash information corresponding to session length and sessions of a user that crashed, mean time to failure information corresponding to session length and number of sessions of a user that failed, mean commands to crash information, or mean commands to failure information, or any combination of crash information, failure information, mean time to crash information, mean commands to crash information, or mean commands to failure information, and

For the set of users, determining, using a crash handling system separate from the trace generating system, whether each user belongs to a class of users for which failure data was recorded, and if so, accessing the failure data to derive details from the failure data that correspond to the recorded session data for that user, the failure data comprising crash dumps of the software product.

12. The computer-readable medium of claim 11 having further computer-executable instructions comprising, outputting information corresponding to a subset of users based on their problems with the software product.

13. The computer-readable medium of claim 11 having further computer executable instructions comprising, filtering the recorded session data to locate the set of users based on filtering criterion or criteria corresponding to at least one parameter in the recorded session data.

14. The computer-readable medium of claim 11 having further computer executable instructions comprising, outputting a distribution mapping for each determined reliability metric.

15. The computer-readable medium of claim 11 having further computer-executable instructions comprising, determining the frequency of consecutive failures for the software product.

16. The computer-readable medium of claim 11 having further computer executable instructions comprising, determining which computer programs including at least one application of the software product were running concurrently when a problem occurred.

17. A computer comprising:

an analyzer that processes software instrumentation trace data recorded from user software program usage sessions to determine which alerts or asserts, or both alerts and asserts, occurred during sessions of a user using the software program, the asserts and alerts comprising statements of the software program, the software instrumentation trace data having been captured by software instrumentation facilities on the computer, the software instrumentation facilities being available for any arbitrary software program to use to trace execution thereof;

a locator component that accesses failure data for at least one alert or assert determined by the analyzer, the failure data comprising or derived from one or more crash dumps of the software program, the locator component obtaining from the failure data failure information that corresponds to a trace or alert determined by the analyzer; and

a storage component storing results corresponding to alert or assert, or alert and assert occurrences, and any located failure data for each alert or assert.

18. The computer of claim 17 wherein the analyzer processes the software instrumentation trace data to determine a relationship between user interface operations and alerts.

19. The computer of claim 17 wherein the analyzer processes the information to determine a relationship between user interface operations and asserts.

20. The computer of claim 17 wherein the computer filters the software instrumentation trace data based on one or more session criteria or user criteria, or both session criteria and user criteria, to determine which alerts or asserts, or both alerts and asserts occurred during only those sessions that meet the criteria.