INTERACTION SENSING AND RECORDING OF A PROCESS TO CONTROL A COMPUTER SYSTEM

• US 20160162168A1
• Filed: 04/24/2015
• Published: 06/09/2016
• Est. Priority Date: 12/05/2014
• Status: Abandoned Application

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Shabazz, Sharrief, Bondarenko, Maksym, Prodan, Alexandru, Ghosh, Dipanjan
• Application Number:
  US14/695,658
• Publication Number:
  US 20160162168A1
• Time in Patent Office: Days
• CPC Class Codes: G06F 11/323 Visualisation of programs or trace data
• CPC Class Codes: G06F 11/3414 Workload generation, e.g. scripts, playback
• CPC Class Codes: G06F 11/3438 monitoring of user actions tracking the activity of the user H04L67/535
• CPC Class Codes: G06F 2201/865 Monitoring of software
• CPC Class Codes: G06F 3/04842 Selection of displayed objects or displayed text elements G06F3/0482 takes precedence
• CPC Class Codes: G06F 3/04847 Interaction techniques to control parameter settings, e.g. interaction with sliders or dials

Abstract

User interactions with a computing system are sensed and recorded, during runtime operation of a process that controls a computer system. The responsive computer system actions that are taken based upon the sensed user interactions are also sensed and recorded. A causation between the sensed interactions and the sensed computer system actions is identified and recorded, for later playback and automatic control of the computer system.

Claims

1. A computing system, comprising:

a user interface component that controls a display device to display a user interface display for an application;

a user interaction detector component that detects a user interaction indicative of a user performing an action in the application;

an action recording component that generates a recording by opening a task recording structure in response to the detection of the user interaction and recording, in the task recording structure, the detected user interaction; and

a scope generator system that records, as scope information that is part of the task recording structure, any detected actions that happen within the application based on the detected user interaction.

2. The computing system of claim 1 wherein the scope generator system comprises:

a scope open component that opens a scope record based on the detected user interaction; and

a scope action detector that detects the actions, within the application, that are performed by the application in response to the detected user interaction.

3. The computing system of claim 2 wherein the action recording component records the application actions, in a sequence in which they occurred, in the task recording structure in relation to the detected user interaction.

4. The computing system of claim 3 wherein the scope generator system comprises:

a scope close component that detects scope close criteria and closes the scope record for the detected user interaction.

5. The computing system of claim 4 wherein the scope generator system comprises:

a scope trimming component that removes predefined actions from the scope record in response to the scope close component closing the scope.

6. The computing system of claim 3 wherein the user interface component controls the display device to display a recording pane with recording control user input mechanisms and detects a start recording user input through a given recording control user input mechanism, the user interaction detector detecting user interactions based on the detected start recording user input.

7. The computing system of claim 6 wherein the user interface display for the application includes user actuatable input mechanisms that are actuatable to perform different actions in the application, and further comprising:

a record preview component that detects a user selecting a given user actuatable input mechanism, after the start recording user input is detected, and visually identifies whether the action recording component will open the task recording structure in response to user actuation of the given user actuatable input mechanism.

8. The computing system of claim 7 wherein the record preview component detects the user selecting the given user actuatable input mechanism by detecting that the user is hovering a cursor over the given user actuatable input mechanism.

9. The computing system of claim 7 and further comprising:

a rollback component that detects a rollback user input and rolls back any changes made by the application during the recording.

10. The computing system of claim 3 and further comprising:

a recording edit system that generates a user edit display with edit user input mechanisms that are actuated to edit the recording.

11. The computing system of claim 10 wherein the recording edit system comprises:

a grouping component that displays a grouping user input mechanism that is actuated to group selected user interactions and corresponding scope information into groups of steps within the task recording structure.

12. The computing system of claim 10 wherein the recording edit system comprises:

a task identifier component that displays a task identifier user input mechanism that is actuated to identify a task corresponding to the task recording structure.

13. The computing system of claim 12 wherein the recording edit system comprises:

a required/example component that displays a required/example user input mechanism that is actuated to indicate whether a detected value corresponding to a detected user interaction is an example value or a required value for the identified task.

14. The computing system of claim 13 wherein the recording edit system comprises:

an added material component that displays an added material user input mechanism that is actuated to add notes or an attachment to a step in the task recording structure.

15. A computer implemented method, comprising:

controlling a display device to display a user interface display for an application;

detecting a user interaction indicative of a user performing an action in the application;

generating a recording by opening a task recording structure in response to the detection of the user interaction and recording, in the task recording structure, the detected user interaction;

detecting actions, within the application, that are performed by the application in response to the detected user interaction; and

recording scope information that, is part of the task recording structure, the scope information being indicative of the detected actions that performed by the application in response to the detected user interaction.

16. The computer implemented method of claim 15 wherein recording scope information comprises:

opening a scope record based on the detected user interaction; and

recording the scope information as the actions performed by the application, in a sequence in which they occurred, in the scope record.

17. The computer implemented method of claim 16 wherein controlling the display device includes controlling the display device to display a recording pane with recording control user input mechanisms and detecting a start recording user input through a given recording control user input mechanism, and wherein detecting user interactions comprises detecting the user interactions based on the detected start recording user input.

18. The computer implemented method of claim 17 wherein the user interface display for the application includes user actuatable input mechanisms that are actuatable to perform different actions in the application, and further comprising:

detecting a user selecting a given user actuatable input mechanism, after the start recording user input is detected; and

visually identifying whether the action recording component will open the task recording structure in response to user actuation of the given user actuatable input mechanism

19. A computing system, comprising:

a user interface component that controls a display device to display a user interface display for an application;

a user interaction detector component that detects a user interaction indicative of a user performing an action in the application;

an action recording component that generates a recording by opening a task recording structure in response to the detection of the user interaction and recording, in the task recording structure, the detected user interaction;

a scope open component that opens a scope record based on the detected user interaction; and

a scope action detector that detects application actions, within the application, that are performed by the application in response to the detected user interaction, wherein the action recording component records the application actions, in a sequence in which they occurred, in the task recording structure in relation to the detected user interaction.

20. The computing system of claim 19 and further comprising:

a rollback component that detects a rollback user input and rolls back any changes made by the application during the recording; and

a recording edit system that generates a user edit display with edit user input mechanisms that are actuated to edit the recording.