Smart home automation systems and methods

• US 10,367,652 B2
• Filed: 02/21/2017
• Issued: 07/30/2019
• Est. Priority Date: 10/08/2012
• Status: Active Grant

• Current Assignee: Nant Holdings IP, LLC (NantWorks, LLC)
• Original Assignee: Nant Holdings IP, LLC (NantWorks, LLC)
• Inventors: Ehsani, Farzad, Witt-Ehsani, Silke Maren, Rolandi, Walter
• Primary Examiner(s): Han, Qi
• Application Number:
  US15/438,428
• Publication Number:
  US 20170163435A1
• Time in Patent Office: 889 Days
• Field of Search: 704 10, 704 1, 704 2, 704 3, 704 7, 704 9, 704270, 7042701, 704275, 704257
• CPC Class Codes: G05B 15/02 electric
• CPC Class Codes: G05B 2219/2642 Domotique, domestic, home control, automation, smart house
• CPC Class Codes: G06F 3/167 Audio in a user interface, e.g. using voice commands for navigating, audio feedback
• CPC Class Codes: G06F 40/242 Dictionaries
• CPC Class Codes: G06F 40/35 Discourse or dialogue representation
• CPC Class Codes: G10L 15/1815 Semantic context, e.g. disambiguation of the recognition hypotheses based on word meaning
• CPC Class Codes: G10L 15/183 using context dependencies, e.g. language models
• CPC Class Codes: G10L 15/22 Procedures used during a speech recognition process, e.g. man-machine dialogue
• CPC Class Codes: G10L 2015/223 Execution procedure of a spoken command
• CPC Class Codes: G10L 2015/226 using non-speech characteristics
• CPC Class Codes: G10L 21/06 Transformation of speech into a non-audible representation, e.g. speech visualisation or speech processing for tactile aids G10L15/26 takes precedence
• CPC Class Codes: H04L 12/281 indicating a format for calling an appliance service function in a home automation network for remote control or remote monitoring of applications H04L67/025
• CPC Class Codes: H04L 12/282 based on user interaction within the home receiver circuitry for displaying additional information being controlled by a remote control apparatus H04N21/42204
• CPC Class Codes: H04L 2012/285 Generic home appliances, e.g. refrigerators
• CPC Class Codes: H04L 67/12 specially adapted for proprietary or special-purpose networking environments, e.g. medical networks, sensor networks, networks in vehicles or remote metering networks
• CPC Class Codes: H04M 1/72415 for remote control of appliances

Abstract

A smart home interaction system is presented. It is built on a multi-modal, multithreaded conversational dialog engine. The system provides a natural language user interface for the control of household devices, appliances or household functionality. The smart home automation agent can receive input from users through sensing devices such as a smart phone, a tablet computer or a laptop computer. Users interact with the system from within the household or from remote locations. The smart home system can receive input from sensors or any other machines with which it is interfaced. The system employs interaction guide rules for processing reaction to both user and sensor input and driving the conversational interactions that result from such input. The system adaptively learns based on both user and sensor input and can learn the preferences and practices of its users.

Claims

1. An interaction apparatus comprising:

a user interface for receiving an active user input signal representing a voice utterance that includes a system name associated with the interaction apparatus, converting the active user input signal to a processed active input signal having a machine-readable format, receiving a passive user input signal representing a voice utterance that does not include the system name prior to receiving the active user input signal, and converting the passive user input signal to a processed passive input signal having a machine-readable format;

a processor configured to execute software instructions stored on a tangible,

non-transitory computer readable storage medium, said processor for;

retrieving, from a location-specific vocabulary library, location-specific vocabulary associated with the location of the interaction apparatus and specific to a set of one or more local devices available at the location and external to the interaction apparatus;

converting the processed passive input signal into a digital representation of a semantic meaning of the passive user input signal by parsing the processed passive input signal using the location-specific vocabulary;

storing the digital representation of the semantic meaning of the passive user input signal as context data associated with an interaction rule;

converting the processed active input signal into a digital representation of a semantic meaning of the active user input signal by parsing the processed active input signal using the location-specific vocabulary; and

,

in response to the digital representation of the semantic meaning of the active user input signal matching the interaction rule, generating a device command based on the digital representation of the semantic meaning of the active user input signal and further based on the context data associated with the interaction rule; and

a device interface for providing the device command to a local device from among the one or more local devices.

2. The interaction apparatus of claim 1, wherein the user interface is configured to communicate with a user input device comprising a smart phone, a wall panel, a microphone, a game console, a tablet, a vehicle, a computer, or a server.

3. The interaction apparatus of claim 1, further comprising a sensor interface for receiving a sensor input signal, wherein the processor generates the device command further based on the sensor input signal.

4. The interaction apparatus of claim 3, wherein the sensor interface is configured to communicate with a sensor comprising a camera, a thermostat, a precipitation sensor, a medical sensor, a microphone, a proximity sensor, an accelerometer, or an optical sensor.

5. The interaction apparatus of claim 1, wherein the device interface comprises a serial interface, an Ethernet interface, a LonWorks protocol interface, an X10 protocol interface, a wireless interface, an HTTP server, or a line of sight interface.

6. The interaction apparatus of claim 1, wherein the location of the interaction apparatus comprises a home, an office, a store, a public location, a building, or a classroom.

7. The interaction apparatus of claim 1, wherein the location of the interaction apparatus comprises a home, and the location-specific vocabulary is specific to the home'"'"'s inhabitants.

8. The interaction apparatus of claim 1, wherein the digital representation of the semantic meaning of the active user input signal comprises a query, and the processor submits the query to an indexed interaction guide library to retrieve the interaction rule and generates the device command based on the retrieved interaction rule.

9. The interaction apparatus of claim 8, further comprising a sensor interface for receiving a sensor input signal, wherein the processor generates the device command further based on the sensor input signal.

10. The interaction apparatus of claim 9, wherein the sensor interface is configured to communicate with a sensor comprising a camera, a thermostat, a precipitation sensor, a medical sensor, a microphone, a proximity sensor, an accelerometer, or an optical sensor.

11. The interaction apparatus of claim 1, wherein the local device comprises a household appliance, device, or system, an entertainment device, a communication device, or a security device.

12. An interaction apparatus comprising:

a sensor interface, configured to communicate with one or more microphones, for receiving, from the one or more microphones, an active input signal representing a voice utterance that includes a system name associated with the interaction apparatus and, prior to receiving the active input signal, receiving a passive input signal representing a voice utterance that does not include the system name;

a processor configured to execute software instructions stored on a tangible,

non-transitory computer readable storage medium, said processor for;

retrieving, from a location-specific vocabulary library, location-specific vocabulary associated with the location of the interaction apparatus and specific to a set of one or more local devices available at the location and external to the interaction apparatus;

converting the passive input signal into a digital representation of a semantic meaning of the passive input signal by parsing the passive input signal using the location-specific vocabulary;

storing the digital representation of the semantic meaning of the passive input signal as context data associated with an interaction rule;

converting the active input signal into a digital representation of a semantic meaning of the active input signal by parsing the active input signal using the location-specific vocabulary; and

,

in response to the digital representation of the semantic meaning of the active input signal matching the interaction rule, generating a device command based on the digital representation of the semantic meaning of the active input signal and further based on the context data associated with the interaction rule; and

a device interface for providing the device command to a local device from among the one or more local devices.

13. The interaction apparatus of claim 12, wherein the sensor interface is configured to further communicate with a sensor comprising a camera, a thermostat, a precipitation sensor, a medical sensor, a proximity sensor, an accelerometer, or an optical sensor.

14. The interaction apparatus of claim 12, wherein the device interface comprises a serial interface, an Ethernet interface, a LonWorks protocol interface, an X10 protocol interface, a wireless interface, an HTTP server, or a line of sight interface.

15. The interaction apparatus of claim 12, wherein the location of the interaction apparatus comprises a home, an office, a store, a public location, a building, or a classroom.

16. The interaction apparatus of claim 12, wherein the location of the interaction apparatus comprises a home, and the location-specific vocabulary is specific to the home'"'"'s inhabitants.

17. The interaction apparatus of claim 12, wherein the digital representation of the semantic meaning of the active input signal comprises a query, and the processor submits the query to an indexed interaction guide library to retrieve the interaction rule and generates the device command based on the retrieved interaction rule.

18. The interaction apparatus of claim 12, wherein the local device comprises a household appliance, device, or system, an entertainment device, a communication device, or a security device.