DETECTING THE END OF A USER QUESTION

• US 20140249811A1
• Filed: 03/01/2013
• Published: 09/04/2014
• Est. Priority Date: 03/01/2013
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google Inc. (Alphabet Inc.)
• Inventors: Doherty, Ryan P., Johnston, Nicholas
• Granted Patent: US 9,123,340 B2
• Time in Patent Office: Days
• US Class Current: 704/231
• CPC Class Codes: G06V 40/20 Movements or behaviour, e.g. gesture recognition recognition of facial expressions G06V40/16
• CPC Class Codes: G10L 15/22 Procedures used during a speech recognition process, e.g. man-machine dialogue
• CPC Class Codes: G10L 2015/226 using non-speech characteristics

Abstract

Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for classifying voice inputs. The methods, systems, and apparatus include actions of providing an answer to a first voice input from a user and receiving visual or audio data corresponding to a second voice input. Further actions include classifying the second voice input as a follow on request to the first voice input or as deliberation on the answer, based on the visual data or the audio data. Additionally, the actions include determining whether to provide a response to the second voice input based on the classification of the second voice input.

Claims

1. A computer-implemented method comprising:

providing an answer to a first voice input from a user;

receiving visual or audio data corresponding to a second voice input;

classifying the second voice input as a follow on request to the first voice input or as deliberation on the answer, based on the visual data or the audio data; and

determining whether to provide a response to the second voice input based on the classification of the second voice input.

2. The method of claim 1, further comprising:

receiving visual or audio data corresponding to the first voice input; and

entering an active listen mode based on the visual or audio data corresponding to the first voice input,

wherein determining whether to provide the response to the second voice input comprises;

exiting the entered active listen mode if the second voice input is classified as deliberation.

3. The method of claim 1, further comprising:

determining an angle of the user'"'"'s head,

wherein classifying the second voice input as a follow on request to the first voice input or deliberation on the answer is based on the determined angle of the user'"'"'s head.

4. The method of claim 1, further comprising:

determining a number of people in an area that includes the user based on the visual data,

wherein classifying the second voice input as a follow on request to the first voice input or deliberation on the answer is based on the determined number of people.

5. The method of claim 1, further comprising:

determining whether the lips of the user are moving based on the visual data,

wherein classifying the second voice input as a follow on request to the first voice input or deliberation on the answer is based on the determination if lips of the user are moving.

6. The method of claim 1, further comprising:

determining words corresponding to the second voice input based on the audio data,

wherein classifying the second voice input as a follow on request to the first voice input or deliberation on the answer is based on the determined words of the second voice input.

7. The method of claim 1, wherein the response comprises an answer to the second voice input or a reaction to the second voice input perceivable by the user.

8. A system comprising:

one or more computers and one or more storage devices storing instructions that are operable, when executed by the one or more computers, to cause the one or more computers to perform operations comprising;

providing an answer to a first voice input from a user;

receiving visual or audio data corresponding to a second voice input;

classifying the second voice input as a follow on request to the first voice input or as deliberation on the answer, based on the visual data or the audio data; and

determining whether to provide a response to the second voice input based on the classification of the second voice input.

9. The system of claim 8, the operations further comprising:

receiving visual or audio data corresponding to the first voice input; and

entering an active listen mode based on the visual or audio data corresponding to the first voice input,

wherein determining whether to provide the response to the second voice input comprises;

exiting the entered active listen mode if the second voice input is classified as deliberation.

10. The system of claim 8, the operations further comprising:

determining an angle of the user'"'"'s head,

wherein classifying the second voice input as a follow on request to the first voice input or deliberation on the answer is based on the determined angle of the user'"'"'s head.

11. The system of claim 8, the operations further comprising:

determining a number of people in an area that includes the user based on the visual data,

wherein classifying the second voice input as a follow on request to the first voice input or deliberation on the answer is based on the determined number of people.

12. The system of claim 8, the operations further comprising:

determining whether the lips of the user are moving based on the visual data,

wherein classifying the second voice input as a follow on request to the first voice input or deliberation on the answer is based on the determination if lips of the user are moving.

13. The system of claim 8, the operations further comprising:

determining words corresponding to the second voice input based on the audio data,

wherein classifying the second voice input as a follow on request to the first voice input or deliberation on the answer is based on the determined words of the second voice input.

14. The system of claim 8, wherein the response comprises an answer to the second voice input or a reaction to the second voice input perceivable by the user.

15. A non-transitory computer-readable medium storing software comprising instructions executable by one or more computers which, upon such execution, cause the one or more computers to perform operations comprising:

providing an answer to a first voice input from a user;

receiving visual or audio data corresponding to a second voice input;

classifying the second voice input as a follow on request to the first voice input or as deliberation on the answer, based on the visual data or the audio data; and

determining whether to provide a response to the second voice input based on the classification of the second voice input.

16. The medium of claim 15, the operations further comprising:

receiving visual or audio data corresponding to the first voice input; and

entering an active listen mode based on the visual or audio data corresponding to the first voice input,

wherein determining whether to provide the response to the second voice input comprises;

exiting the entered active listen mode if the second voice input is classified as deliberation.

17. The medium of claim 15, the operations further comprising:

determining an angle of the user'"'"'s head,

wherein classifying the second voice input as a follow on request to the first voice input or deliberation on the answer is based on the determined angle of the user'"'"'s head.

18. The medium of claim 15, the operations further comprising:

determining a number of people in an area that includes the user based on the visual data,

wherein classifying the second voice input as a follow on request to the first voice input or deliberation on the answer is based on the determined number of people.

19. The medium of claim 15, the operations further comprising:

determining whether the lips of the user are moving based on the visual data,

wherein classifying the second voice input as a follow on request to the first voice input or deliberation on the answer is based on the determination if lips of the user are moving.

20. The medium of claim 15, the operations further comprising:

determining words corresponding to the second voice input based on the audio data,

wherein classifying the second voice input as a follow on request to the first voice input or deliberation on the answer is based on the determined words of the second voice input.