AUTOMATIC PROCESS GUIDANCE

• US 20160162475A1
• Filed: 12/03/2015
• Published: 06/09/2016
• Est. Priority Date: 12/05/2014
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Bondarenko, Maksym, Dongre, Manali, Khabazan, Maryam, Shabazz, Sharrief, Have, Hans, Ghosh, Dipanjan
• Granted Patent: US 9,965,464 B2
• Time in Patent Office: Days
• CPC Class Codes: G06F 11/263 Generation of test inputs, e.g. test vectors, patterns or sequences ; with adaptation of the tested hardware for testability with external testers
• CPC Class Codes: G06F 11/3414 Workload generation, e.g. scripts, playback
• CPC Class Codes: G06F 11/3438 monitoring of user actions tracking the activity of the user H04L67/535
• CPC Class Codes: G06F 11/3636 by tracing the execution of the program
• CPC Class Codes: G06F 11/3664 Environments for testing or debugging software
• CPC Class Codes: G06F 11/3684 for test design, e.g. generating new test cases
• CPC Class Codes: G06F 2201/865 Monitoring of software
• CPC Class Codes: G06F 40/40 Processing or translation of natural language natural language analysis G06F40/20; semantic analysis G06F40/30
• CPC Class Codes: G10L 13/00 Speech synthesis; Text to speech systems
• CPC Class Codes: G10L 15/22 Procedures used during a speech recognition process, e.g. man-machine dialogue

Abstract

User interactions with a computing system are sensed and recorded. The recording represents a process for controlling a computer system. Voice input are received and the computer system actions that are taken based upon the voice inputs and a task recording.

Claims

1. A computing system, comprising:

a linguistic processing system that receives a task initiating speech input and generates a recognition result indicative of a task identified in the task initiating speech input; and

a script engine that identifies a script corresponding to the task, based on the recognition result, the script including step elements that generate step outputs indicative of steps to be performed in completing the task, the script engine running the script by generating the step outputs, in an order identified by the script and based on responsive user inputs, the linguistic processing system generating a speech output indicative of each of the step outputs generated by the script engine.

2. The computing system of claim 1 and further comprising:

an interaction system that generates an interaction with a computing system component, based on user performance of the task.

3. The computing system of claim 2 wherein the interaction system generates the interactions with the computing system component in a separate computing system.

4. The computing system of claim 2 wherein the linguistic processing system comprises:

a speech recognition component that receives the task initiating speech input and the responsive user inputs and generates corresponding textual outputs.

5. The computing system of claim 4 wherein the linguistic processing system comprises:

a natural language understanding component that receives the textual outputs from the speech recognition system and generates an interpretation corresponding to each of the textual outputs and provides the interpretations to the script engine, as the recognition result.

6. The computing system of claim 5 wherein the linguistic processing system comprises:

a text-to-speech component that generates the speech outputs based on the step outputs generated by the script engine.

7. The computing system of claim 1 wherein the speech outputs, corresponding to the script outputs, comprise audible prompts to perform a next step in the task.

8. The computing system of claim 1 wherein the script engine generates the step outputs based on responsive user speech inputs.

9. The computing system of claim 1 and further comprising:

a touch sensitive user interface device and wherein the script engine generates the step outputs based on responsive user touch inputs.

10. A computer implemented, comprising:

receiving a task initiating speech input;

generating a recognition result indicative of a task identified in the task initiating speech input;

identifying a script corresponding to the task, based on the recognition result, the script including step elements that generate step outputs indicative of steps to be performed in completing the task;

running the script by generating the step outputs, in an order identified by the script and based on responsive user inputs and by generating a speech output indicative of each of the step outputs.

11. The computer implemented method of claim 10 and further comprising:

generating an interaction with a computing system component, based on user performance of the task.

12. The computer implemented method of claim 11 wherein generating the interactions with the computing system component comprises:

generating the interactions with the computing system component in a separate computing system.

13. The computer implemented method of claim 11 wherein generating the recognition result comprises:

receiving the task initiating speech input and the responsive user inputs; and

generating corresponding textual outputs.

14. The computer implemented method of claim 13 wherein generating the recognition result comprises:

generating an interpretation corresponding to each of the textual outputs; and

outputting the interpretations as the recognition result.

15. The computer implemented method of claim 14 wherein generating the recognition result comprises:

generating the speech outputs based on the step outputs generated by the script engine.

16. The computer implemented method of claim 10 wherein generating the speech outputs, corresponding to the script outputs, comprises:

generating audible prompts to perform a next step in the task.

17. The computer implemented method of claim 10 wherein generating the step outputs comprises:

generating the step outputs based on responsive user speech inputs.

18. The computer implemented method of claim 10 wherein generating the step outputs comprises:

generating the step outputs based on responsive user touch inputs.

19. A computing system, comprising:

a linguistic processing system that receives a task initiating speech input and generates a recognition result indicative of a task identified in the task initiating speech input;

a script engine that identifies a script corresponding to the task, based on the recognition result, the script including step elements that generate step outputs indicative of steps to be performed in completing the task, the script engine running the script by generating the step outputs, in an order identified by the script, the linguistic processing system generating speech outputs based on the step outputs, the speech outputs audible instructing the user to perform the steps, the script engine receiving responsive user inputs that are responsive to the step outputs, and generating any additional step outputs based on responsive user inputs, the linguistic processing system generating additional speech outputs indicative of each of the additional step outputs generated by the script engine; and

an interaction system that generates an interaction with a backend computing system component, based on user performance of the steps in the task.

20. The computing system of claim 19 wherein the linguistic processing system comprises:

a speech recognition component that receives the task initiating speech input and the responsive user inputs and generates corresponding textual outputs;

a natural language understanding component that receives the textual outputs from the speech recognition system and generates an interpretation corresponding to each of the textual outputs and provides the interpretations to the script engine, as the recognition result; and

a text-to-speech component that generates the speech outputs based on the step outputs generated by the script engine.