Privacy control in a connected environment

• US 10,204,623 B2
• Filed: 05/04/2017
• Issued: 02/12/2019
• Est. Priority Date: 01/20/2017
• Status: Active Grant

• Current Assignee: Essential Products, Inc.
• Original Assignee: Essential Products, Inc.
• Inventors: Segal, Mara Clair, Roman, Manuel, Desai, Dwipal, Rubin, Andrew E.
• Primary Examiner(s): Abebe, Daniel
• Application Number:
  US15/587,230
• Publication Number:
  US 20180211657A1
• Time in Patent Office: 649 Days
• Field of Search: 704275
• CPC Class Codes: G10L 15/02 Feature extraction for speech recognition; Selection of recognition unit
• CPC Class Codes: G10L 15/08 Speech classification or search
• CPC Class Codes: G10L 15/1822 Parsing for meaning understanding
• CPC Class Codes: G10L 15/22 Procedures used during a speech recognition process, e.g. man-machine dialogue
• CPC Class Codes: G10L 15/30 Distributed recognition, e.g. in client-server systems, for mobile phones or network applications
• CPC Class Codes: G10L 2015/088 Word spotting
• CPC Class Codes: G10L 2015/223 Execution procedure of a spoken command
• CPC Class Codes: G10L 2015/228 of application context
• CPC Class Codes: G10L 25/51 for comparison or discrimination
• CPC Class Codes: G10L 25/78 Detection of presence or absence of voice signals switching of direction of transmission by voice frequency in two-way loud-speaking telephone systems H04M9/10
• CPC Class Codes: H04L 63/00 Network architectures or network communication protocols for network security cryptographic mechanisms or cryptographic arrangements for secret or secure communication H04L9/00; network architectures or network communication protocols for wireless network security H04W12/00; security arrangements for protecting computers or computer systems against unauthorised activity G06F21/00
• CPC Class Codes: H04L 63/04 for providing a confidential data exchange among entities communicating through data packet networks
• CPC Class Codes: H04L 67/10 in which an application is distributed across nodes in the network software deployment G06F8/60; multiprogramming arrangements G06F9/46
• CPC Class Codes: H04W 12/02 Protecting privacy or anonymity, e.g. protecting personally identifiable information [PII]
• CPC Class Codes: H04W 12/03 Protecting confidentiality, e.g. by encryption
• CPC Class Codes: H04W 4/70 Services for machine-to-machine communication [M2M] or machine type communication [MTC]

Abstract

Privacy control in a connected environment is described. An assistant device can detect speech spoken by a user. The speech can include a hardware activation phrase that indicates whether the user intends for at least a portion of the speech to be provided to local resources or cloud resources. The speech can then be provided to the appropriate resource based on the hardware activation phrase.

Claims

1. A home assistant device, comprising:

a microphone;

a speaker;

one or more processors; and

memory storing instructions, wherein the processor is configured to execute the instructions such that the processor and memory are configured to;

detect first speech including a first question spoken by a user of the home assistant device using the microphone;

determine that the first speech includes a first hardware activation phrase preceding the first question, wherein the first hardware activation phrase indicates an intention of the user to provide the first question of the first speech to a cloud server;

provide the first question to the cloud server;

receive a first response from the cloud server providing a first answer to the first question;

play back the first answer using the speaker, the playback of the first answer having first characteristics;

detect second speech including a second question spoken by the user of the home assistant device using the microphone;

determine that the second speech includes a second hardware activation phrase preceding the second question, the first hardware activation phrase and the second hardware activation phrase being different phrases, wherein the second hardware activation phrase indicates an intention of the user to provide the second question of the second speech to local resources of the home assistant device rather than the cloud server due to increased privacy expectations regarding content of the second question in comparison with privacy expectations regarding content of the first question;

provide the second question to the local resources of the home assistant device;

receive a second response from the local resources providing a second answer to the second question; and

play back the second answer using the speaker, the playback of the second answer having second characteristics that are different than the first characteristics of the playback of the first answer due to the first response being received from the cloud server rather than the local resources.

2. The home assistant device of claim 1, wherein the local resources include one or both of hardware resources of the home assistant device or resources of other devices communicatively coupled with the home assistant device on a wireless network.

3. A method for privacy control in a connected environment, comprising:

detecting speech spoken by a user of an assistant device;

determining, by a processor, that the speech includes a hardware activation phrase corresponding to an intention of the user to provide a first portion of the speech to one or both of local resources of the assistant device or a cloud server based on the hardware activation phrase;

providing the first portion of the speech to the local resources or the cloud server based on the hardware activation phrase;

receiving a first response data providing an answer to the first portion of the speech from the local resources;

playing back the first response data, characteristics of the playback of the first response data based on the first response data being received from the local resources;

providing a second portion of the speech to the cloud server;

receiving a second response data providing an answer to the second portion of the speech from the cloud server; and

playing back the second response data, characteristics of the playback of the second response data being different than the characteristics of the playback of the first response data due to the second portion of the speech being received from the cloud server rather than the local resources.

4. The method of claim 3, wherein the local resources include one or both of hardware resources of the assistant device or resources of other devices connected with the assistant device on a wireless network.

5. The method of claim 3, further comprising:

determining that the speech includes the first portion and the second portion, the first portion provided to the local resources, the second portion provided to the cloud server.

6. The method of claim 5, further comprising:

providing a response to the speech based on the first response data received from the local resources and the second response data received from the cloud resources.

7. The method of claim 5, wherein the first portion and the second portion are determined based on content of the speech.

8. The method of claim 3, wherein the first hardware activation phrase corresponds to providing the first portion of the speech to the local resources, and a second hardware activation phrase corresponds to providing the second portion of the speech to the cloud server, the first hardware activation phrase and the second hardware activation phrase being different.

9. The method of claim 3, wherein the first portion and the second portion are determined based on characteristics of the the speech.

10. An electronic device, comprising:

one or more processors; and

memory storing instructions, wherein the processor is configured to execute the instructions such that the processor and memory are configured to;

detect speech being spoken by a user of an assistant device;

determine that the speech includes a hardware activation phrase corresponding to an intention of the user to provide a first portion of the speech to one or both of local resources of the assistant device or a cloud server based on the hardware activation phrase;

provide the first portion of the speech to the local resources or the cloud server based on the hardware activation phrase;

receive a first response data providing an answer to the first portion of the speech from the local resources;

play back the first response data, characteristics of the playback of the first response data based on the first response data being received from the local resources;

provide a second portion of the speech to the cloud server;

receive a second response data providing an answer to the second portion of the speech from the cloud server; and

play back the second response data, characteristics of the playback of the second response data being different than the characteristics of the playback of the first response data due to the second portion of the speech being received from the cloud server rather than the local resources.

11. The electronic device of claim 10, wherein the local resources include one or both of hardware resources of the assistant device or resources of other devices connected with the assistant device on a wireless network.

12. The electronic device of claim 10, wherein the processor is configured to execute the instructions such that the processor and memory are configured to:

determine that the speech includes the first portion and the second portion, the first portion provided to the local resources, the second portion provided to the cloud server.

13. The electronic device of claim 12, wherein the processor is configured to execute the instructions such that the processor and memory are configured to:

provide a response to the portion of the speech based on the first response data received from the local resources and the second response data received from the cloud resources.

14. The electronic device of claim 12, wherein the first portion and the second portion are determined based on content of the speech.

15. The electronic device of claim 10, wherein the first hardware activation phrase corresponds to providing the first portion of the speech to the local resources, and a second hardware activation phrase corresponds to providing the second portion of the speech to the cloud server, the first hardware activation phrase and the second hardware activation phrase being different.

16. The electronic device of claim 10, wherein the first portion and the second portion are determined based on characteristics of the speech.

17. A computer program product, comprising one or more non-transitory computer-readable media having computer program instructions stored therein, the computer program instructions being configured such that, when executed by one or more computing devices, the computer program instructions cause the one or more computing devices to:

detect speech being spoken by a user of an assistant device;

determine that the speech includes a hardware activation phrase corresponding to an intention of the user to provide a first portion of the speech to one or both of local resources of the assistant device or a cloud server based on the hardware activation phrase;

provide the first portion of the speech to the local resources or the cloud server based on the hardware activation phrase;

receive a first response data providing an answer to the first portion of the speech from the local resources;

play back the first response data, characteristics of the playback of the first response data based on the first response data being received from the local resources;

provide a second portion of the speech to the cloud server;

receive a second response data providing an answer to the second portion of the speech from the cloud server; and

play back the second response data, characteristics of the playback of the second response data being different than the characteristics of the playback of the first response data due to the second portion of the speech being received from the cloud server rather than the local resources.

18. The computer program product of claim 17, wherein the local resources include one or both of hardware resources of the assistant device or resources of other devices connected with the assistant device on a wireless network.

19. The computer program product of claim 17, wherein the computer program instructions cause the one or more computing devices to:

determine that the portion of the speech includes the first portion and the second portion, the first portion provided to the local resources, the second portion provided to the cloud server.

20. The computer program product of claim 19, wherein the computer program instructions cause the one or more computing devices to:

provide a response to the portion of the speech based on the first response data received from the local resources and the second response data received from the cloud resources.

21. The computer program product of claim 19, wherein the first portion and the second portion are determined based on content of the speech.

22. The computer program product of claim 17, wherein the first hardware activation phrase corresponds to providing the first portion of the speech to the local resources, and a second hardware activation phrase corresponds to providing the second portion of the speech to the cloud server, the first hardware activation phrase and the second hardware activation phrase being different.