Indexing and annotating a usability test recording

• US 10,649,634 B2
• Filed: 06/06/2014
• Issued: 05/12/2020
• Est. Priority Date: 06/06/2014
• Status: Active Grant

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: De Paula, Rogerio Abreu, Pinhanez, Claudio Santos, De Santana, Vagner Figueredo
• Primary Examiner(s): Vu, Kieu D
• Assistant Examiner(s): Chung, Andrew
• Application Number:
  US14/298,137
• Publication Number:
  US 20150356062A1
• Time in Patent Office: 2,167 Days
• CPC Class Codes: G06F 11/3438 monitoring of user actions tracking the activity of the user H04L67/535
• CPC Class Codes: G06F 17/40 Data acquisition and logging for input to computer G06F3/00
• CPC Class Codes: G06F 2201/835 Timestamp
• CPC Class Codes: G06F 2201/86 Event-based monitoring
• CPC Class Codes: G06F 3/048 Interaction techniques based on graphical user interfaces [GUI]
• CPC Class Codes: G06F 3/04817 using icons graphical or visual programming using iconic symbols G06F8/34
• CPC Class Codes: G06F 3/0484 for the control of specific functions or operations, e.g. selecting or manipulating an object, an image or a displayed text element, setting a parameter value or selecting a range
• CPC Class Codes: G06F 3/04842 Selection of displayed objects or displayed text elements G06F3/0482 takes precedence

Abstract

A method for indexing a user interface test recording includes capturing, during a session, a recording of a plurality of interactions with the user interface, capturing, during the session, an event stream including a plurality of user interface events, synchronizing, in time, the plurality of interactions and the plurality of user interface events, identifying a point of interest in the event stream, wherein the point of interest is correlated to a time in the recording by the synchronization, and annotating the recording at a time correlated to when the point of interest occurred.

Claims

1. A method for indexing a user interface test recording comprising:

capturing, during a usability test session, a recording of a plurality of interactions of a user with the user interface;

capturing, during the usability test session, an event stream including a plurality of user interface events performed by the user, each user interface event being defined by one or more types of user interface input submitted via the user interface;

synchronizing, in time, the plurality of interactions and the plurality of user interface events;

identifying a point of interest in the event stream, wherein the point of interest is correlated to a time in the recording of the plurality of interactions synchronized with the plurality of user interface events; and

annotating the recording at a time correlated to when the point of interest occurred,

wherein identifying the point of interest in the event stream further comprises building an event digraph comprising a plurality of nodes corresponding to respective ones of the plurality of user interface events performed by the user, the respective ones of the plurality of user interface events corresponding to respective ones of the plurality of interactions of the user with the user interface,

wherein the point of interest is identified as corresponding to a certain node in the event digraph when;

a number of nodes to reach the certain node is greater than a threshold number of nodes;

an amount of time to reach the certain node is greater than a threshold time;

or

an elapsed time corresponding to the certain node is above a threshold elapsed time.

2. The method of claim 1, wherein the point of interest corresponds to an activity of a user.

3. The method of claim 1, wherein the point of interest corresponds to a period of inactivity of a user.

4. The method of claim 1, the event digraph further comprising a plurality of edges representing a sequence of the plurality of user interface events over time, wherein the event digraph models the plurality of user interface events over time.

5. The method of claim 1, wherein building the event digraph comprises labeling each of the plurality of nodes based on a concatenation of a corresponding user interface event type and a user interface element identification.

6. The method of claim 1, wherein the recording comprises a video of the plurality of user interactions with the user interface under test.

7. The method of claim 6, wherein the recording further comprises audio corresponding to at least one of the plurality of interactions with the user interface.

8. The method of claim 1, wherein the event digraph further comprises a root node representing a beginning of the usability test session, the method further comprising:

associating each node corresponding to a respective one of the plurality of user interface events with a mean distance to the root node for all occurrences of the respective one of the plurality of user interface events and a mean timestamp representing time from the beginning of the usability test session for all occurrences of the respective one of the plurality of user interface events.

9. The method of claim 8, further comprising updating the mean distance and the mean timestamp for the node corresponding to said respective one of the plurality of user interface events upon each repeated occurrence of said respective one of the plurality of user interface events.

10. The method of claim 8, wherein identifying the point of interest comprises determining that the certain node in the event digraph is part of a cyclic pattern based on at least one of:

a difference between the mean distance from the root node to the certain node and the mean distance from the root node to a set of neighbor nodes of the certain node; and

a difference between the mean timestamp of the certain node and the mean timestamp of the set of neighbor nodes of the certain node.

11. The method of claim 10, wherein identifying the point of interest comprises determining that the certain node in the event digraph is part of a cyclic pattern based further on at least one of:

a standard deviation of the mean distance from the root node to the set of neighbor nodes of the certain node; and

a standard deviation of the mean timestamp of the set of neighbor nodes of the certain node.

12. The method of claim 1, wherein the point of interest corresponds to a usability problem candidate.

13. The method of claim 1, wherein identifying the point of interest comprises determining that the certain node in the event digraph is part of a cyclic pattern.

14. A computer program product for indexing a user interface test recording, the computer program product comprising a computer readable storage medium having program instructions embodied therewith, the program instructions executable by a processor to cause the processor to perform a method comprising:

capturing, during a usability test session, a recording of a plurality of interactions of a user with the user interface;

capturing, during the usability test session, an event stream including a plurality of user interface events performed by the user, each user interface event being defined by one or more types of user interface input submitted via the user interface;

synchronizing, in time, the plurality of interactions and the plurality of user interface events;

identifying a point of interest in the event stream, wherein the point of interest is correlated to a time in the recording of the plurality of interactions synchronized with the plurality of user interface events; and

annotating the recording at a time correlated to when the point of interest occurred, wherein identifying the point of interest in the event stream further comprises building an event digraph comprising a plurality of nodes corresponding to respective ones of the plurality of user interface events performed by the user, the respective ones of the plurality of user interface events corresponding to respective ones of the plurality of interactions of the user with the user interface, wherein the point of interest is identified as corresponding to a certain node in the event digraph when;

a number of nodes to reach the certain node is greater than a threshold number of nodes;

an amount of time to reach the certain node is greater than a threshold time;

or

an elapsed time corresponding to the certain node is above a threshold elapsed time.

15. An apparatus configured to index a user interface test recording comprising:

a recording device configured to capture, during a user interface test session, a video of a plurality of user interactions with the user interface; and

a computer program product comprising a computer readable storage medium having program instructions embodied therewith, the program instructions executable by a processor to;

capture, during the user interface test session, an event stream including a plurality of user interface events performed by a user,

synchronize, in time, the plurality of interactions and the plurality of user interface events performed by the user, each user interface event being defined by one or more types of user interface input submitted via the user interface,

generate a model of the user interface events,

identify a point of interest in the event stream, wherein the point of interest is correlated to a time in the recording of the plurality of interactions synchronized with the plurality of user interface events, and

annotate the video at a time correlated to when the point of interest occurred,

wherein identifying the point of interest in the event stream further comprises building an event digraph comprising a plurality of nodes corresponding to respective ones of the plurality of user interface events performed by the user, the respective ones of the plurality of user interface events corresponding to respective ones of the plurality of interactions of the user with the user interface,

wherein the point of interest is identified as corresponding to a certain node in the event digraph when;

a number of nodes to reach the certain node is greater than a threshold number of nodes;

an amount of time to reach the certain node is greater than a threshold time;

or

an elapsed time corresponding to the certain node is above a threshold elapsed time.

16. The apparatus of claim 15, further comprising a display displaying the video including at least one annotation correlated to the time in the video the point of interest occurred.

17. The apparatus of claim 15, wherein the recording device records audio corresponding to at least one of the plurality of interactions with the user interface.

18. The apparatus of claim 15, the event digraph further comprising a plurality of edges representing a sequence of the plurality of user interface events over time, wherein the event digraph models the plurality of user interface events over time, and the digraph is stored in a memory accessible to the processor.

19. The apparatus of claim 15, wherein building the event digraph comprises labeling each of the plurality of nodes based on a concatenation of a corresponding user interface event type and a user interface element identification.

20. The apparatus of claim 15, wherein a signal is generated by the processor, wherein the signal is used to synchronize, in time, the plurality of interactions and the plurality of user interface events.