Smart home automation systems and methods

• US 9,230,560 B2
• Filed: 10/08/2013
• Issued: 01/05/2016
• Est. Priority Date: 10/08/2012
• Status: Active Grant

• Current Assignee: Nant Holdings IP, LLC (NantWorks, LLC)
• Original Assignee: Nant Holdings IP, LLC (NantWorks, LLC)
• Inventors: Ehsani, Farzad, Witt-Ehsani, Silke Maren, Rolandi, Walter
• Primary Examiner(s): Han, Qi
• Application Number:
  US14/049,158
• Publication Number:
  US 20140108019A1
• Time in Patent Office: 819 Days
• Field of Search: 704/10, 704/1, 704/2, 704/3, 704/7, 704/9, 704/270, 704/270.1, 704/275
• US Class Current: 1/1
• CPC Class Codes: G05B 15/02 electric
• CPC Class Codes: G05B 2219/2642 Domotique, domestic, home control, automation, smart house
• CPC Class Codes: G06F 3/167 Audio in a user interface, e.g. using voice commands for navigating, audio feedback
• CPC Class Codes: G06F 40/242 Dictionaries
• CPC Class Codes: G06F 40/35 Discourse or dialogue representation
• CPC Class Codes: G10L 15/1815 Semantic context, e.g. disambiguation of the recognition hypotheses based on word meaning
• CPC Class Codes: G10L 15/183 using context dependencies, e.g. language models
• CPC Class Codes: G10L 15/22 Procedures used during a speech recognition process, e.g. man-machine dialogue
• CPC Class Codes: G10L 2015/223 Execution procedure of a spoken command
• CPC Class Codes: G10L 2015/226 using non-speech characteristics
• CPC Class Codes: G10L 21/06 Transformation of speech into a non-audible representation, e.g. speech visualisation or speech processing for tactile aids G10L15/26 takes precedence
• CPC Class Codes: H04L 12/281 indicating a format for calling an appliance service function in a home automation network for remote control or remote monitoring of applications H04L67/025
• CPC Class Codes: H04L 12/282 based on user interaction within the home receiver circuitry for displaying additional information being controlled by a remote control apparatus H04N21/42204
• CPC Class Codes: H04L 2012/285 Generic home appliances, e.g. refrigerators
• CPC Class Codes: H04L 67/12 specially adapted for proprietary or special-purpose networking environments, e.g. medical networks, sensor networks, networks in vehicles or remote metering networks
• CPC Class Codes: H04M 1/72415 for remote control of appliances

Abstract

A smart home interaction system is presented. It is built on a multi-modal, multithreaded conversational dialog engine. The system provides a natural language user interface for the control of household devices, appliances or household functionality. The smart home automation agent can receive input from users through sensing devices such as a smart phone, a tablet computer or a laptop computer. Users interact with the system from within the household or from remote locations. The smart home system can receive input from sensors or any other machines with which it is interfaced. The system employs interaction guide rules for processing reaction to both user and sensor input and driving the conversational interactions that result from such input. The system adaptively learns based on both user and sensor input and can learn the preferences and practices of its users.

Claims

1. An interaction system comprising:

a device interface configured to couple with at least one local device;

a multi-modal interface configured to receive input signals representative of multi-modal input;

a location specific vocabulary library comprising a corpus associated with a location;

a multi-modal dialog interaction engine coupled with the multi-modal user interface and the location specific vocabulary library, and configured to;

obtain the input signals from the multi-modal interface;

convert the input signals into a query by mapping each of the input signals to an interaction object using the location-specific vocabulary library, the query comprising a digital representation of a semantic meaning of the input signal;

retrieve at least one interaction rule object from an interaction guide library in response to submitting the query to the interaction guide library;

generate at least one device command from the at least one interaction rule object;

configure the at least one device to execute the at least one device command via the device interface; and

configure an output device to present a response related to the device command and responsive to the input signals.

2. The system of claim 1, wherein the multi-modal interface comprises a user interface.

3. The system of claim 2, wherein the multi-modal interface comprises a remote user interface.

4. The system of claim 2, wherein the multi-modal user interface comprises at least one of the following:

a smart phone, a wall panel, a hidden microphone, a game console, a tablet, a vehicle, a computer, and a server.

5. The system of claim 1, wherein the multi-modal interface comprises a sensor interface.

6. The system of claim 5, wherein the sensor interface is configured to interact with at least one of the following sensors:

a camera, a thermostat, a precipitation sensor, a medical sensor, a microphone, a proximity sensor, an accelerometer, and an optical sensor.

7. The system of claim 5, wherein the input signals comprise sensor data from at least one sensor.

8. The system of claim 1, wherein the device interface comprises at least one of the following:

a serial interface, an Ethernet interface, a LonWorks protocol interface, an X10 protocol interface, a wireless interface, an HTTP server, and a line of sight interface.

9. The system of claim 1, wherein the input signals represent a text modality.

10. The system of claim 1, wherein the input signals represent at least one of the following:

a speech modality and an audio modality.

11. The system of claim 1, wherein the input signals represent a reaction modality.

12. The system of claim 1, wherein the input signals represent an emotion modality.

13. The system of claim 1, wherein the input signals represent a gesture modality.

14. The system of claim 1, wherein the input signals represent a tactile modality.

15. The system of claim 1, wherein the input signals represent at least one of the following:

an image modality, and a video modality.

16. The system of claim 1, wherein the location includes at least one of the following:

a home, an office, a store, a public location, a building, and a classroom.

17. The system of claim 1, further comprising the interaction guide library.

18. The system of claim 17, wherein the interaction guide library stores a plurality of interaction rule objects according to a location specific indexing scheme.

19. The system of claim 17, wherein interaction rule objects comprises device interaction rules.

20. The system of claim 19, wherein the device interaction rules depend on observed data.

21. The system of claim 1, further comprising a domain specific vocabulary library.

22. The system of claim 21, wherein the multi-modal dialog interaction engine is communicatively coupled with the domain specific vocabulary library.

23. The system of claim 22, wherein the multi-modal dialog interaction engine is configured to convert the input signals into a query by mapping each of the input signals to an interaction object using the location-specific normalized vocabulary library and the domain specific vocabulary library.