Ambient assistant device

• US 10,210,866 B2
• Filed: 05/18/2017
• Issued: 02/19/2019
• Est. Priority Date: 01/20/2017
• Status: Active Grant

• Current Assignee: Essential Products, Inc.
• Original Assignee: Essential Products, Inc.
• Inventors: Segal, Mara Clair, Roman, Manuel, Desai, Dwipal, Rubin, Andrew E.
• Primary Examiner(s): Jackson, Jakieda R
• Application Number:
  US15/599,402
• Publication Number:
  US 20180211658A1
• Time in Patent Office: 642 Days
• Field of Search: 704235, 704275
• CPC Class Codes: G06F 3/167 Audio in a user interface, e.g. using voice commands for navigating, audio feedback
• CPC Class Codes: G10L 15/08 Speech classification or search
• CPC Class Codes: G10L 15/22 Procedures used during a speech recognition process, e.g. man-machine dialogue
• CPC Class Codes: G10L 15/24 Speech recognition using non-acoustical features
• CPC Class Codes: G10L 15/30 Distributed recognition, e.g. in client-server systems, for mobile phones or network applications
• CPC Class Codes: G10L 2015/088 Word spotting
• CPC Class Codes: G10L 2015/223 Execution procedure of a spoken command
• CPC Class Codes: G10L 2015/226 using non-speech characteristics
• CPC Class Codes: G10L 25/78 Detection of presence or absence of voice signals switching of direction of transmission by voice frequency in two-way loud-speaking telephone systems H04M9/10

Abstract

Ambient assistance is described. An assistant device can detect speech in its environment and determine that the speech includes words or phrases of a local dictionary of the assistant device. The assistant device can then generate an interaction opportunity based on the words or phrases of the speech.

Claims

1. A home assistant device, comprising:

a speaker;

a microphone;

a display screen;

one or more processors; and

memory storing instructions, wherein the processor is configured to execute the instructions such that the processor and memory are configured to;

detect first speech spoken within an environment of the home assistant device using the microphone;

determine that the first speech includes content having one or more words or phrases included in a local dictionary of the home assistant device;

provide a first interaction opportunity with the home assistant device based on the one or more words or phrases of the first speech corresponding to the local dictionary, the first interaction opportunity providing a speech response using the speaker based on the content of the first speech and based on the first speech including the content having the one or more words or phrases included in the local dictionary of the home assistant device;

detect second speech spoken within the environment of the home assistant device, the first speech being different than the second speech;

determine that the content of the second speech does not include the one or more words or phrases corresponding to the local dictionary;

provide the second speech to a cloud server to determine content related to the second speech;

receive response data from the cloud server based on the second speech; and

provide a second interaction opportunity with the home assistant device based on the response data received from the cloud server, the second interaction opportunity different than the first interaction opportunity, the second interaction opportunity providing a visual response on the display screen based on the content of the second speech and based on the second speech being provided to the cloud server.

2. A method, comprising:

detecting first speech spoken within an environment of an assistant device;

determining, by a processor, that the first speech includes content having one or more words or phrases corresponding to a local dictionary of the assistant device;

generating a first interaction opportunity with the assistant device based on the one or more words or phrases of the first speech corresponding to the local dictionary, the first interaction opportunity being a first type of interaction opportunity based on the first speech including content having the one or more words or phrases corresponding to the local dictionary of the assistant device;

detecting second speech spoken within the environment of the assistant device, the first speech being different than the second speech;

determining that the second speech does not include the one or more words or phrases corresponding to the local dictionary;

providing the second speech to a cloud server to determine content or interactions related to the second speech;

receiving response data from the cloud server based on the content of the second speech; and

generating a second interaction opportunity with the assistant device based on the response data received from the cloud server, the second interaction opportunity being a second type of interaction opportunity that is different than the first type of interaction opportunity, the second interaction opportunity being the second type based on providing the second speech to the cloud server.

3. The method of claim 2, wherein the cloud server is selected from among a first cloud server and a second cloud server based on the second speech, the first cloud server and the second cloud server corresponding to different services.

4. The method of claim 3, wherein the cloud server is selected based on characteristics of the second speech, the characteristics including one or more of time, content, complexity, or time duration.

5. The method of claim 2, wherein the first interaction opportunity includes providing additional information related to content of the first speech.

6. The method of claim 2, wherein the local dictionary includes information related to translating portions of the first speech into text.

7. The method of claim 2, wherein the local dictionary includes information related to commands capable of being performed by the assistant device.

8. The method of claim 2, wherein the first interaction opportunity is a speech response responsive to the first speech, and the second interaction opportunity is a visual response responsive to the second speech.

9. An assistant device, comprising:

one or more processors; and

memory storing instructions, wherein the processor is configured to execute the instructions such that the processor and memory are configured to;

detect first speech spoken within an environment of the assistant device;

determine that the first speech includes content having one or more words or phrases corresponding to a local dictionary of the assistant device;

generate a first interaction opportunity with the assistant device based on the one or more words or phrases of the first speech corresponding to the local dictionary, the first interaction opportunity being a first type of interaction opportunity based on the first speech including content having the one or more words or phrases corresponding to the local dictionary of the assistant device;

detect second speech spoken within the environment of the assistant device, the first speech being different than the second speech;

determine that the second speech does not include the one or more words or phrases corresponding to the local dictionary;

provide the second speech to a cloud server to determine content or interactions related to the second speech;

receive response data from the cloud server based on the content of the second speech; and

generate a second interaction opportunity with the assistant device based on the response data received from the cloud server, the second interaction opportunity being a second type of interaction opportunity that is different than the first type of interaction opportunity, the second interaction opportunity being the second type based on providing the second speech to the cloud server.

10. The assistant device of claim 9, wherein the cloud server is selected from among a first cloud server and a second cloud server based on the second speech, the first cloud server and the second cloud server corresponding to different services.

11. The assistant device of claim 10, wherein the cloud server is selected based on characteristics of the second speech, the characteristics including one or more of time, content, complexity, or time duration.

12. The assistant device of claim 9, wherein the first interaction opportunity includes providing additional information related to content of the first speech.

13. The assistant device of claim 9, wherein the local dictionary includes information related to translating portions of the first speech into text.

14. The assistant device of claim 9, wherein the local dictionary includes information related to commands capable of being performed by the assistant device.

15. The assistant device of claim 9, wherein the first interaction opportunity is a speech response responsive to the first speech, and the second interaction opportunity is a visual response responsive to the second speech.

16. A computer program product, comprising one or more non-transitory computer-readable media having computer program instructions stored therein, the computer program instructions being configured such that, when executed by one or more computing devices, the computer program instructions cause the one or more computing devices to:

detect first speech spoken within an environment of an assistant device;

determine that the first speech includes content having one or more words or phrases corresponding to a local dictionary of the assistant device;

generate a first interaction opportunity with the assistant device based on the one or more words or phrases of the first speech corresponding to the local dictionary, the first interaction opportunity being a first type of interaction opportunity based on the first speech including content having the one or more words or phrases corresponding to the local dictionary of the assistant device;

detect second speech spoken within the environment of the assistant device, the first speech being different than the second speech;

determine that the second speech does not include the one or more words or phrases corresponding to the local dictionary;

provide the second speech to a cloud server to determine content or interactions related to the second speech;

receive response data from the cloud server based on the content of the second speech; and

generate a second interaction opportunity with the assistant device based on the response data received from the cloud server, the second interaction opportunity being a second type of interaction opportunity that is different than the first type of interaction opportunity, the second interaction opportunity being the second type based on providing the second speech to the cloud server.

17. The computer program product of claim 16, wherein the cloud server is selected from among a first cloud server and a second cloud server based on the second speech, the first cloud server and the second cloud server corresponding to different services.

18. The computer program product of claim 17, wherein the cloud server is selected based on characteristics of the second speech, the characteristics including one or more of time, content, complexity, or time duration.

19. The computer program product of claim 16, wherein the first interaction opportunity includes providing additional information related to content of the first speech.

20. The computer program product of claim 16, wherein the local dictionary includes information related to translating portions of the first speech into text.

21. The assistant device of claim 16, wherein the local dictionary includes information related to commands capable of being performed by the assistant device.

22. The computer program product of claim 16, wherein the first interaction opportunity is a speech response responsive to the first speech, and the second interaction opportunity is a visual response responsive to the second speech.