History-based key phrase suggestions for voice control of a home automation system

• US 10,018,977 B2
• Filed: 10/05/2015
• Issued: 07/10/2018
• Est. Priority Date: 10/05/2015
• Status: Active Grant

• Current Assignee: Savant Systems, LLC
• Original Assignee: Savant Systems, LLC
• Inventors: Cipollo, Nicholas J., Kitson, Ryan E., Katsiris, George T.
• Primary Examiner(s): Lee, Adam
• Application Number:
  US14/875,210
• Publication Number:
  US 20170097618A1
• Time in Patent Office: 1,009 Days
• Field of Search: None
• CPC Class Codes: G05B 15/02 electric
• CPC Class Codes: G08C 2201/31 Voice input
• CPC Class Codes: G10L 15/22 Procedures used during a speech recognition process, e.g. man-machine dialogue
• CPC Class Codes: G10L 17/22 Interactive procedures; Man-machine interfaces
• CPC Class Codes: G10L 2015/223 Execution procedure of a spoken command
• CPC Class Codes: G10L 2015/225 Feedback of the input speech
• CPC Class Codes: G10L 2015/228 of application context
• CPC Class Codes: H04L 12/2803 Home automation networks
• CPC Class Codes: H04L 12/2816 Controlling appliance services of a home automation network by calling their functionalities arrangements in telecontrol or telemetry systems for selectively calling a substation from a main station; in which substation desired apparatus is selected for applying a control signal thereto or for obtaining measured values therefrom H04Q9/00
• CPC Class Codes: H04L 12/282 based on user interaction within the home receiver circuitry for displaying additional information being controlled by a remote control apparatus H04N21/42204
• CPC Class Codes: H04Q 2213/13103 Memory
• CPC Class Codes: H04Q 9/00 Arrangements in telecontrol or telemetry systems for selectively calling a substation from a main station, in which substation desired apparatus is selected for applying a control signal thereto or for obtaining measured values therefrom

Abstract

In one embodiment, history-based key phrase suggestions are provided for voice control of a home automation system. A host controller builds a history database that maintains a history of events, each event indicating one or more services activated, one or more scenes activated or one or more types of media content presented by the home automation system. A predictions process on the host controller selects a subset of the events of the history database based on a present condition of the home automation system to produce a predication dataset. A grammar process on a remote control or mobile device selects one or more intentions from a list used by a voice recognition engine, based on the prediction dataset. Each intention includes a key phrase paired with an action. A voice user interface (UI) process suggests key phrases from the one or more selected intentions in a voice control UI on the remote control or mobile device.

Claims

1. A method comprising:

building a history database on a storage device that maintains events that have occurred in a home automation system, each event associated with a time and indicating one or more services activated, one or more scenes activated or one or more types of media content presented by the home automation system;

selecting a subset of the events of the history database based on a comparison of each event and associated time with a present condition of the home automation system and a present time, to produce a scored prediction dataset, wherein event scores in the scored prediction dataset are determined at least in part based on a frequency of each event within a time window, with a shorter time window causing a higher score than a longer time window;

selecting one or more intentions from a list of intentions based on the event scores from the scored prediction dataset, wherein each intention from the list is recognized by a voice recognition engine and includes a key phrase paired with an action to be performed by the home automation system;

displaying, to a current user of a remote control or mobile device of the home automation system in a user interface on the remote control or mobile device, one or more key phrases from the one or more selected intentions as one or more key phrase suggestions;

receiving, at the voice recognition engine, speech input of the current user from a microphone of the remote control or mobile device;

comparing, by the voice recognition engine, the speech input against the key phrases of the list of intentions to find a matching intention; and

triggering the action associated with the matching intention, wherein the action controls one or more devices within a room of the home automation system.

2. The method of claim 1, wherein the present time includes a present time of day and a present day of week and the comparison determines events in the history database that occurred on a same day of the week as the present day of week, in a time window that encompasses the present time of day on any day of the week, or in a time window that encompasses the present time of day on a same day of the week as the present day of the week.

3. The method of claim 1, wherein the scoring is adjusted by a factor based on the type of event, with types of events indicating more specific user action receiving higher scores than events indicating less specific user action.

4. The method of claim 1, further comprising:

dynamically generating the list of intentions based upon at least one of configuration information for the home automation system, user content defined in the home automation system or a system state of the home automation system.

5. The method of claim 1, further comprising:

dynamically generating the list of intentions based upon the current user or a room being controlled.

6. The method of claim 1, further comprising:

storing the history database on a host controller that is in communication with the remote control or mobile device over a network.

7. The method of claim 1, wherein the list of intentions includes one or more service intentions whose action when performed activates one or more services within a room.

8. The method of claim 1, wherein the list of intentions includes one or more scenes intentions whose action when performed activates one or more scenes.

9. The method of claim 1, wherein the action is a system action that activates one or more services or activates one or more scenes, in the home automation system.

10. The method of claim 1, wherein the action is a user interface (UI) action that controls one or more aspects of a home automation system control UI.

11. A home automation system comprising:

a host controller that is coupled to and configured to control one or more audio/video (A/V) devices, lighting devices, heating ventilation and cooling (HVAC) devices or security devices within a structure to provide home automation, the host controller including;

host activity recording software stored on the host controller that is configured to build a history database that maintains events that have occurred in the structure, each event indicating one or more services activated, one or more scenes activated or one or more types of media content presented, and

a predictions software process stored on the host controller that is configured to select a subset of the events of the history database based on a present condition of the home automation system and a frequency of each event within a time window to produce a scored prediction dataset that scores each event based at least in part on the time window, with a shorter time window causing a higher score than a longer time window; and

a remote control or mobile device in communication with the host controller over a network, the remote control or mobile device including;

a voice recognition engine stored on the remote control or mobile device, and

voice suggestion software stored on the remote control or mobile device and configured to

select one or more intentions from a list of intentions based on the event scores of the scored prediction dataset wherein each intention from the list is recognized by the voice recognition engine and includes a key phrase paired with an action to be performed by the home automation system, and

display on a screen of the remote control or mobile device to a current user of the home automation system one or more key phrases from the one or more selected intentions as one or more key phrase suggestions, wherein the voice recognition engine includes a recognizer process that is configured to receive speech input of the current user from a microphone of the remote control or mobile device, compare the speech input against the key phrases of the list of intentions to find a matching intention, and trigger the action associated with the matching intention and wherein the action controls one or more devices within a room of the home automation system.

12. The home automation system of claim 11, wherein each event in the history of events includes at least an indication of a time associated with the event, the present condition of the home automation system includes a present time, and the selection is based on a comparison of the time of each event with the present time.

13. The home automation system of claim 11, wherein the voice suggestion software includes a grammar process that is configured to dynamically generate the list of intentions based upon at least one of configuration information for the home automation system, user content defined in the home automation system or a system state of the home automation system.

14. The home automation system of claim 11, wherein the voice suggestion software includes a grammar process that is configured to dynamically generate the list of intentions based upon the current user or a room being controlled.

15. A non-transitory electronic-device readable medium storing software that when executed on one or more processors is operable to:

receive a scored prediction dataset that includes a frequency of events that occurred in a home automation system within a time window that prior to a present time, each event indicating one or more services activated, one or more scenes activated or one or more types of media content presented by the home automation system, wherein each event is scored, and a shorter time window causes a higher score than a longer time window;

select one or more key phrases from a list of key phrases used by a voice recognition engine of the home automation system based on the event scores;

display the selected one or more key phases on a remote control or mobile device of the home automation system to a user as one or more key phrase suggestions;

receive speech input by the remote control or mobile device from the user corresponding to one of the one or more key phrase selections;

compare the speech input against the list of key phrases used by the voice recognition engine to find a match; and

trigger an action associated with the matching key phrase, wherein the action controls one or more devices within a room of the home automation system.

16. The non-transitory electronic-device readable medium of claim 15, wherein the list of key phrases is part of a list of intentions, with each intention pairing a key phrase and an action performed in response to the key phrase, and the selection of one or more key phrases includes the selection of one or more intentions.

17. The non-transitory electronic-device readable medium of claim 15, wherein the software when executed on the one or more processors is further operable to:

dynamically generate the list of intentions based upon at least one of configuration information for the home automation system, user content defined in the home automation system or a system state of the home automation system.

18. The non-transitory electronic-device readable medium of claim 15, wherein the software when executed on the one or more processors is further operable to:

dynamically generate the list of intentions based upon the current user or a room being controlled.