Smart home automation systems and methods

• US 9,614,690 B2
• Filed: 11/23/2015
• Issued: 04/04/2017
• Est. Priority Date: 10/08/2012
• Status: Active Grant

• Current Assignee: Nant Holdings IP, LLC (NantWorks, LLC)
• Original Assignee: Nant Holdings IP, LLC (NantWorks, LLC)
• Inventors: Ehsani, Farzad, Witt-Ehsani, Silke Maren, Rolandi, Walter
• Primary Examiner(s): Han, Qi
• Application Number:
  US14/948,934
• Publication Number:
  US 20160080165A1
• Time in Patent Office: 498 Days
• Field of Search: 704 10, 704 1, 704 2, 704 3, 704 7, 704 9, 704270, 7042701, 704275
• CPC Class Codes: G05B 15/02 electric
• CPC Class Codes: G05B 2219/2642 Domotique, domestic, home control, automation, smart house
• CPC Class Codes: G06F 3/167 Audio in a user interface, e.g. using voice commands for navigating, audio feedback
• CPC Class Codes: G06F 40/242 Dictionaries
• CPC Class Codes: G06F 40/35 Discourse or dialogue representation
• CPC Class Codes: G10L 15/1815 Semantic context, e.g. disambiguation of the recognition hypotheses based on word meaning
• CPC Class Codes: G10L 15/183 using context dependencies, e.g. language models
• CPC Class Codes: G10L 15/22 Procedures used during a speech recognition process, e.g. man-machine dialogue
• CPC Class Codes: G10L 2015/223 Execution procedure of a spoken command
• CPC Class Codes: G10L 2015/226 using non-speech characteristics
• CPC Class Codes: G10L 21/06 Transformation of speech into a non-audible representation, e.g. speech visualisation or speech processing for tactile aids G10L15/26 takes precedence
• CPC Class Codes: H04L 12/281 indicating a format for calling an appliance service function in a home automation network for remote control or remote monitoring of applications H04L67/025
• CPC Class Codes: H04L 12/282 based on user interaction within the home receiver circuitry for displaying additional information being controlled by a remote control apparatus H04N21/42204
• CPC Class Codes: H04L 2012/285 Generic home appliances, e.g. refrigerators
• CPC Class Codes: H04L 67/12 specially adapted for proprietary or special-purpose networking environments, e.g. medical networks, sensor networks, networks in vehicles or remote metering networks
• CPC Class Codes: H04M 1/72415 for remote control of appliances

Abstract

A smart home interaction system is presented. It is built on a multi-modal, multithreaded conversational dialog engine. The system provides a natural language user interface for the control of household devices, appliances or household functionality. The smart home automation agent can receive input from users through sensing devices such as a smart phone, a tablet computer or a laptop computer. Users interact with the system from within the household or from remote locations. The smart home system can receive input from sensors or any other machines with which it is interfaced. The system employs interaction guide rules for processing reaction to both user and sensor input and driving the conversational interactions that result from such input. The system adaptively learns based on both user and sensor input and can learn the preferences and practices of its users.

Claims

1. A building-specific device interaction method comprising:

obtaining, by a dialog interaction engine, input signals from a multi-modal interface;

retrieving, by the dialog interaction engine, a building-specific location vocabulary associated with a building;

identifying, by the dialog interaction engine, based on the input signals at least one domain specific vocabulary library having a vocabulary relevant to a specific domain;

converting, by the dialog interaction engine, the input signals into a query by mapping the input signals to an interaction object using at least one of the building-specific location vocabulary library and the domain specific vocabulary, the query comprising a digital representation of a semantic meaning of the input signal;

retrieving, by the dialog interaction engine, at least one interaction rule object from an interaction guide library in response to submitting the query to an interaction guide library;

generating at least one device command from the at least one interaction rule object and from the input signals; and

configuring, by the dialog interaction engine and via a device interface, at least one device to execute the at least one device command.

2. The method of claim 1, further comprising configuring an output device to present a response related to the device command and responsive to the input signals.

3. The method of claim 1, wherein the domain specific vocabulary library comprises a vocabulary relevant to wellness.

4. The method of claim 1, wherein the domain specific vocabulary library includes a vocabulary relevant to a group of people.

5. The method of claim 1, wherein the domain specific vocabulary library includes a vocabulary relevant to an identity of a person.

6. The method of claim 1, wherein the building-specific location vocabulary library includes a room-specific vocabulary.

7. The method of claim 1, wherein the domain-specific vocabulary includes a room-specific vocabulary.

8. The method of claim 1, wherein the dialog interaction engine comprises a desktop computer.

9. The method of claim 1, wherein the dialog interaction engine comprises a portable device.

10. The method of claim 9, wherein the portable device comprises a smart phone.

11. The method of claim 1, further comprising, receiving, by the dialog interaction engine, the input signals from at least one of the following sensors:

a camera, a thermometer, a precipitation sensor, a medical sensor, a microphone, a proximity sensor, an accelerometer, and an optical sensor.

12. The method of claim 1, further comprising detecting spoken words in the input signal.

13. The method of claim 1, wherein the step of retrieving the at least one interaction objects depend on detected changes in the environment based on the input signals.

14. The method of claim 1, wherein interaction rule objects comprises device interaction rules.

15. The method of claim 14, further comprising populating a device command template within the device interaction rules based on parameters derived from the input signals.

16. The method of claim 14, further comprising iterating through the device interaction rules by evaluating the device interaction rules.

17. The method of claim 16, wherein the step of generating the at least one device command includes generate the at least one device command from the device interaction rules that evaluates to true.

18. The method of claim 17, further comprising generating the at least one device command from a first device interaction rules that evaluates to true.

19. The method of claim 1, wherein the at least one device is local to the building.