Analyzing software users with instrumentation data and user group modeling and analysis

• US 20080313617A1
• Filed: 06/15/2007
• Published: 12/18/2008
• Est. Priority Date: 06/15/2007
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Li, Yantao, Sun, Bing, Wang, Qiang, Zhang, Haidong, Wang, Min, Wang, Jian, Ye, Shuguang, Zhu, Wenli
• Granted Patent: US 7,739,666 B2
• Time in Patent Office: Days
• US Class Current: 717/127
• CPC Class Codes: G06F 11/321 Display for diagnostics, e.g. diagnostic result display, self-test user interface
• CPC Class Codes: G06F 11/328 Computer systems status display G06F11/327 takes precedence
• CPC Class Codes: G06F 11/3414 Workload generation, e.g. scripts, playback
• CPC Class Codes: G06F 11/3466 Performance evaluation by tracing or monitoring
• CPC Class Codes: G06F 11/3604 Software analysis for verifying properties of programs byte-code verification G06F9/44589
• CPC Class Codes: G06F 2201/81 Threshold
• CPC Class Codes: G06F 2201/865 Monitoring of software
• CPC Class Codes: G06F 2201/88 Monitoring involving counting

Abstract

Described is a technology by which software instrumentation data collected from user program sessions are analyzed, including by determining program usage metrics and/or command usage metrics. Information representative of the program usage metrics and/or the command usage metrics is output, such as in the form of a report. The software instrumentation data may be further analyzed, such as to determine at least one usage trend over time, and to determine user groups. For example, a usage subset of sessions that meet specified session usage criteria based on a set of session data may be located, along with a subset of users based on users whose sessions meet specified user criteria. The usage and user subsets may be combined via Boolean logic to produce a result set.

Claims

1. In a computing environment, a method comprising:

analyzing information corresponding to software instrumentation data collected from user sessions corresponding to one or more programs, including determining program usage metrics or determining command usage metrics, or determining program usage metrics and command usage metrics; and

outputting information representative of the program usage metrics, the command usage metrics, or both the program usage metrics and the command usage metrics.

2. The method of claim 1 further comprising, collecting the instrumentation data during actual user sessions.

3. The method of claim 1 wherein determining the program usage metrics comprises determining for a set of users at least one of:

session count information based on a number of application sessions, session frequency information based on a time measurement between sessions, running time information based on session time, session length information based on session time and session count, or depth of usage information based on a percentage of commands used, or any combination of session count information, session frequency information, running time information, session length information or depth of usage information.

4. The method of claim 1 wherein determining the command usage metrics comprises determining for a set of users and a selected command at least one of:

user count information based on a number of users of the set who use the selected command, percentage of users information corresponding to a percentage of users of the set who use the selected command, session count information based on a number of sessions in which the selected command occurred, percentage of session information corresponding to a percentage of application sessions in which the selected command was used, click count information corresponding to a number of clicks corresponding to the selected command, percentage of click count information corresponding to a percentage of program clicks corresponding to the selected command, click count per user information based on click count and user count of the selected command, or click count per session information corresponding to a click count per session, or any combination of user count information, percentage of users information, session count information, percentage of session information, click count information, percentage of click count information, click count per user information, or click count per session information.

5. The method of claim 1 further comprising, analyzing the information corresponding to the software instrumentation data to determine at least one usage trend over time.

6. The method of claim 5 wherein analyzing the information corresponding to the software instrumentation data to determine at least one usage trend over time comprises determining at least one of:

user count information corresponding to a number of users that used a program during a reporting interval, session count information corresponding to a number of sessions of a program during a reporting interval, session count per user information based on a number of sessions and a total number of users that used a program during a reporting interval, running time per user information corresponding to a session length of users and a number of users that used a program during a reporting interval, cumulative session count per user information corresponding to a total number of sessions and a number of users from a start time of a period to analyze to an end of each reporting interval, or cumulative running time per user information corresponding to a session length of users that used a program and a number of users from a start time of a period to analyze to an end of each reporting interval, or any combination of user count information, session count information, session count per user information, running time per user information, cumulative session count per user information, or cumulative running time per user information.

7. The method of claim 1 further comprising, analyzing the information corresponding to the software instrumentation data to determine at least one type of user.

8. The method of claim 7 wherein analyzing the information corresponding to the software instrumentation data to determine at least one type of user comprises categorizing users by depth of usage, or categorizing users by types of activities in which they engage, or categorizing users by depth of usage and categorizing users by the types of activities in which they engage.

9. The method of claim 1 further comprising, analyzing the information corresponding to the software instrumentation data to determine at least potential outlier corresponding to command usage that appears different from command usage of other users.

10. The method of claim 1 wherein analyzing the information corresponding to the software instrumentation data to determine at least potential outlier comprises computing an entropy value corresponding to an occurrence distribution of a command, computing an average occurrence per session of the command usage by a potential outlier, and using the entropy value or the average occurrence per session, or both, as outlier criterion or criteria.

11. The method of claim 1 further comprising, locating a subset of the user sessions based on at least one session criterion, wherein each session criteria comprises a dimension and a value for that dimension, and each dimension comprises a variable recorded in a session, a feature, or a set of one or more variables computed from a plurality of variables recorded in a session, and wherein analyzing the information corresponding to the software instrumentation data comprises analyzing the subset.

12. The method of claim 11 wherein locating the subset comprises mathematically combining each of at least two session criteria.

13. The method of claim 1 further comprising, providing a mechanism for modeling a user group, including providing an interface for receiving one or more measures that specify one or more conditions that any user needs to meet in order to belong to the user group, and locating the users from the information corresponding to the software instrumentation data based on the one or more measures.

14. The method of claim 13 wherein locating the users comprises determining at least one of:

session length information corresponding to a session length of the sessions of a user, crash information corresponding to a number of sessions of a user that crashed, depth of usage information corresponding to which commands were used by a user, enrollment length information corresponding to a time between an earliest session and a latest session of a user, failure information corresponding to a number of sessions that failed of a user, mean time to crash information corresponding to session length and sessions of a user that crashed, mean time to failure information corresponding to session length and number of sessions of a user that failed, session count information corresponding to a total number of sessions of a user, session frequency information corresponding to time between consecutive sessions of a user, or total running time information corresponding to total session time of sessions of a user, or any combination of session length information, crash information, depth of usage information, enrollment length information, failure information, mean time to crash information, mean time to failure information, session count information, session frequency information, or total running time information.

15. A computer-readable medium having computer executable instructions, which when executed perform steps comprising, locating a subset of sessions that meet specified session criteria based on a set of session data, locating a subset of users based on users whose sessions meet specified user criteria based on the set of session data, and combining the subset of sessions with the subset of users via Boolean logic to produce a result set.

16. The computer-readable medium of claim 15 wherein the session criteria comprise at least one session criterion and another session criterion combined with each other via Boolean logic, and wherein the user criteria comprise at least one user criterion and another user criterion combined with each other via Boolean logic.

17. In a computing environment, a system comprising:

an analyzer that processes information corresponding to software instrumentation data recorded from user software program usage sessions to produce a first subset comprising software usage data;

a group modeling mechanism that processes the information corresponding to the software instrumentation data a second subset comprising user data; and

means for combining the first subset with the second subset to provide an output that corresponds to a selected group of users and their software program usage.

18. The system of claim 17 further comprising a user interface for facilitating selection of one or more criteria by which the first subset and second subset are located from the information.

19. The system of claim 17 further comprising means for recording the software instrumentation data.

20. The system of claim 17 wherein the analyzer comprises means for performing at least one of an application usage analysis, a command usage analysis, or a trend analysis, or any combination of an application usage analysis, a command usage analysis, or a trend analysis.