Automatic process guidance

• US 9,965,464 B2
• Filed: 12/03/2015
• Issued: 05/08/2018
• Est. Priority Date: 12/05/2014
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Bondarenko, Maksym, Dongre, Manali, Khabazan, Maryam, Shabazz, Sharrief, Have, Hans, Ghosh, Dipanjan
• Primary Examiner(s): GUERRA-ERAZO, EDGAR X
• Application Number:
  US14/958,489
• Publication Number:
  US 20160162475A1
• Time in Patent Office: 887 Days
• CPC Class Codes: G06F 11/263 Generation of test inputs, e.g. test vectors, patterns or sequences ; with adaptation of the tested hardware for testability with external testers
• CPC Class Codes: G06F 11/3414 Workload generation, e.g. scripts, playback
• CPC Class Codes: G06F 11/3438 monitoring of user actions tracking the activity of the user H04L67/535
• CPC Class Codes: G06F 11/3636 by tracing the execution of the program
• CPC Class Codes: G06F 11/3664 Environments for testing or debugging software
• CPC Class Codes: G06F 11/3684 for test design, e.g. generating new test cases
• CPC Class Codes: G06F 2201/865 Monitoring of software
• CPC Class Codes: G06F 40/40 Processing or translation of natural language natural language analysis G06F40/20; semantic analysis G06F40/30
• CPC Class Codes: G10L 13/00 Speech synthesis; Text to speech systems
• CPC Class Codes: G10L 15/22 Procedures used during a speech recognition process, e.g. man-machine dialogue

Abstract

User interactions with a computing system are sensed and recorded. The recording represents a process for controlling a computer system. Voice input are received and the computer system actions that are taken based upon the voice inputs and a task recording.

Claims

1. A computing system, comprising:

a processor; and

memory storing instructions executable by the processor, wherein the instructions, when executed, configure the computing system to provide;

a linguistic processing system configured to;

receive an indication of a task initiating speech input; and

generate a recognition result indicative of a task identified in the task initiating speech input; and

a script engine configured to;

based on the recognition result, identify a script that correspond to the task and includes step elements configured to generate step outputs indicative of computing steps to be performed in completing the task; and

run the script to;

generate the step outputs, in an order identified by the script and based on indications of responsive user inputs, wherein the linguistic processing system is configured to generate a speech output indicative of each of the step outputs generated by the script engine; and

generate machine instructions that execute the computing steps in the computing system based on the indications of the responsive user inputs.

2. The computing system of claim 1, wherein the machine instructions comprise an interaction with a computing system component.

3. The computing system of claim 2, wherein the computing system comprises a first computing system, and the computing system component comprises a component in a second computing system, that is separate from the first computing system.

4. The computing system of claim 1, wherein the linguistic processing system comprises:

a speech recognition component configured to;

receive the indication of the task initiating speech input,

receive the indications of the responsive user inputs, and

generate a textual output that corresponds to one or more of the task initiating speech input and the responsive user inputs.

5. The computing system of claim 4, wherein the linguistic processing system comprises:

a natural language understanding component configured to;

receive the textual output from the speech recognition system, and

generate an interpretation corresponding to the textual output, and

provide the interpretation to the script engine, as the recognition result.

6. The computing system of claim 5, wherein the linguistic processing system comprises:

a text-to-speech component configured to;

generate the speech outputs based on the step outputs generated by the script engine.

7. The computing system of claim 1, wherein the speech outputs, corresponding to the script outputs, comprise audible prompts indicative of a next step in the task.

8. The computing system of claim 1, wherein the indications of responsive user inputs comprise indications of responsive user speech inputs, and the script engine is configured to generate the step outputs based on the indications of the responsive user speech inputs.

9. The computing system of claim 1, wherein the indications of responsive user inputs comprise indications of responsive user touch inputs received from a touch sensitive user interface device, and the script engine is configured to generate the step outputs based on the indications of the responsive user touch inputs.

10. A computer implemented method, comprising:

receiving an indication of a task initiating speech input;

generating a recognition result indicative of a task identified in the task initiating speech input;

based on the recognition result, identifying a script that corresponds to the task and includes step elements configured to generate step outputs indicative of computing steps to be performed in completing the task; and

running the script by;

generating the step outputs, in an order identified by the script and based on indications of responsive user inputs;

generating a speech output indicative of each of the step outputs; and

generating machine instructions that execute the computing steps in a computing system based on the indications of the responsive user inputs.

11. The computer implemented method of claim 10, wherein the machine instructions comprise an interaction with a computing system component.

12. The computer implemented method of claim 11, wherein the machine instructions are generated by a first computing system, and the computing system component comprises a component in a second computing system that is separate from the first computing system.

13. The computer implemented method of claim 11, wherein generating the recognition result comprises:

receiving the indications of the task initiating speech input and the responsive user inputs; and

generating a textual output that corresponds to one or more of the task initiating speech input and the responsive user inputs.

14. The computer implemented method of claim 13, wherein generating the recognition result comprises:

generating an interpretation corresponding to the textual output; and

outputting the interpretation as the recognition result.

15. The computer implemented method of claim 10, wherein generating the speech outputs, corresponding to the script outputs, comprises:

generating audible prompts indicative of a next step in the task.

16. The computer implemented method of claim 10, wherein the indications of responsive user inputs comprise indications of responsive user speech inputs, and wherein generating the step outputs comprises:

generating the step outputs based on the indications of the responsive user speech inputs.

17. The computer implemented method of claim 10, wherein the indications of responsive user inputs comprise indications of responsive user touch inputs received from a touch sensitive user interface device, and wherein generating the step outputs comprises:

generating the step outputs based on the indications of the responsive user touch inputs.

18. A computing system, comprising:

a processor; and

memory storing instructions executable by the processor, wherein the instructions, when executed, configure the computing system to provide;

a linguistic processing system configured to;

receive an indication of a task initiating speech input; and

generate a recognition result indicative of a task identified in the task initiating speech input;

a script engine configured to;

based on the recognition result, identify a script that corresponds to the task and includes step elements configured to generate step outputs indicative of computing steps to be performed in completing the task;

run the script to;

generate the step outputs, in an order identified by the script, wherein

the linguistic processing system is configured to generate speech outputs that are based on the step outputs represent audible instructions indicative of the steps,

the script engine is configured to receive indications of responsive user inputs that are responsive to the step outputs, and generate an additional step output based on indications of the responsive user inputs, and

the linguistic processing system is configured to generate an additional speech output indicative of the additional step output generated by the script engine; and

an interaction system configured to generate machine instructions that execute the computing steps in the computing system based on the indications of the responsive user inputs.

19. The computing system of claim 18, wherein the linguistic processing system comprises:

a speech recognition component configured to receive an indications of the task initiating speech input and the responsive user inputs, and to generate a textual output that corresponds to one or more of the task initiating speech input and the responsive user inputs;

a natural language understanding component configured to receive the textual output from the speech recognition system, generate an interpretation corresponding to the textual output, and provide the interpretation to the script engine, as the recognition result; and

a text-to-speech component configured to generate the speech outputs based on the step outputs generated by the script engine.