Intelligent assistant for home automation

• US 10,699,717 B2
• Filed: 10/30/2018
• Issued: 06/30/2020
• Est. Priority Date: 05/30/2014
• Status: Active Grant

• Current Assignee: Apple Inc.
• Original Assignee: Apple Inc.
• Inventors: Orr, Ryan M., Nell, Garett R., Brumbaugh, Benjamin L.
• Primary Examiner(s): Yang, Qian
• Application Number:
  US16/175,208
• Publication Number:
  US 20190074016A1
• Time in Patent Office: 609 Days
• CPC Class Codes: G05B 15/02 electric
• CPC Class Codes: G05B 2219/23373 Interactive guidance by voice message
• CPC Class Codes: G05B 2219/23386 Voice, vocal command or message
• CPC Class Codes: G05B 2219/2642 Domotique, domestic, home control, automation, smart house
• CPC Class Codes: G06F 3/167 Audio in a user interface, e.g. using voice commands for navigating, audio feedback
• CPC Class Codes: G10L 17/22 Interactive procedures; Man-machine interfaces
• CPC Class Codes: H04L 12/2816 Controlling appliance services of a home automation network by calling their functionalities arrangements in telecontrol or telemetry systems for selectively calling a substation from a main station; in which substation desired apparatus is selected for applying a control signal thereto or for obtaining measured values therefrom H04Q9/00
• CPC Class Codes: H04L 12/282 based on user interaction within the home receiver circuitry for displaying additional information being controlled by a remote control apparatus H04N21/42204

Abstract

This relates to systems and processes for using a virtual assistant to control electronic devices. In one example process, a user can speak an input in natural language form to a user device to control one or more electronic devices. The user device can transmit the user speech to a server to be converted into a textual representation. The server can identify the one or more electronic devices and appropriate commands to be performed by the one or more electronic devices based on the textual representation. The identified one or more devices and commands to be performed can be transmitted back to the user device, which can forward the commands to the appropriate one or more electronic devices for execution. In response to receiving the commands, the one or more electronic devices can perform the commands and transmit their current states to the user device.

Claims

1. A user device comprising:

one or more processors;

memory; and

one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, the one or more programs including instructions for;

receiving an audio input representing a user request for an electronic device to perform an action,

wherein the audio input is in natural language form, and

wherein the audio input is ambiguous with respect to defining the electronic device for performing the action;

determining one or more possible electronic device characteristics corresponding to the audio input;

retrieving a data structure representing a set of electronic devices of a predetermined location;

determining, based on the data structure, one or more candidate electronic devices from the set of electronic devices, the one or more candidate electronic devices corresponding to the audio input;

determining, based on the one or more possible electronic device characteristics and one or more actual electronic device characteristics of the one or more candidate electronic devices, a user intent corresponding to the audio input; and

providing a command that causes an electronic device of the one or more candidate electronic devices to perform the action corresponding to the user intent.

2. The user device of claim 1, wherein the audio input is in natural language form, and wherein the audio input is ambiguous with respect to defining the action.

3. The user device of claim 1, wherein the one or more possible electronic device characteristics comprise one or more words associated with electronic devices that are controllable by the user device, or associated with one or more states corresponding to the electronic devices that are controllable by the user device.

4. The user device of claim 1, wherein the data structure defines one or more actual device characteristics associated with each electronic device in the set of electronic devices.

5. The user device of claim 4, wherein the one or more actual electronic device characteristics comprise one or more words associated with each electronic device in the set of electronic devices, or associated with one or more states corresponding to each electronic device in the set of electronic devices.

6. The user device of claim 1, wherein the data structure defines a current state of each electronic device in the set of electronic devices.

7. The user device of claim 6, wherein the one or more programs comprise further instructions for:

determining the user intent further based on the current state of each of the one or more candidate electronic devices.

8. The user device of claim 6 wherein the one or more programs comprise further instructions for:

after providing the command, receiving an updated state from the electronic device; and

updating the current state of the electronic device based on the updated state received from the electronic device.

9. The user device of claim 8, wherein the one or more programs comprise further instructions for:

outputting an indication of a result of the command based on the updated state of the electronic device received from the electronic device.

10. The user device of claim 9, wherein the indication of the result of the command is an audio indication, a visual indication, or both an audio and a visual indication.

11. The user device of claim 1, wherein the one or more programs comprise further instructions for:

determining a location corresponding to the audio input; and

determining, based on the determined location, the predetermined location from a plurality of predetermined locations.

12. The user device of claim 1, wherein determining the one or more candidate electronic devices from the set of electronic devices comprises:

determining one or more alternative terms associated with a term in the audio input, wherein the one or more candidate electronic devices is determined based on the one or more alternative terms.

13. The user device of claim 1, wherein determining the user intent further comprises:

determining the electronic device from the one or more candidate electronic devices based on the one or more possible electronic device characteristics and the one or more actual electronic device characteristics of the one or more candidate electronic devices.

14. The user device of claim 1, wherein the one or more programs comprise further instructions for:

determining a relative location of the user device with respect to an area within the predetermined location, wherein the user intent is determined based on the relative location.

15. A method for controlling an electronic device using a virtual assistant on a user device having one or more processors and a memory, the method comprising:

at the user device;

receiving an audio input representing a user request for an electronic device to perform an action,

wherein the audio input is in natural language form, and

wherein the audio input is ambiguous with respect to defining the electronic device for performing the action;

determining one or more possible electronic device characteristics corresponding to the audio input;

retrieving a data structure representing a set of electronic devices of a predetermined location;

determining, based on the data structure, one or more candidate electronic devices from the set of electronic devices, the one or more candidate electronic devices corresponding to the audio input;

determining, based on the one or more possible electronic device characteristics and one or more actual electronic device characteristics of the one or more candidate electronic devices, a user intent corresponding to the audio input; and

providing a command that causes an electronic device of the one or more candidate electronic devices to perform the action corresponding to the user intent.

16. The method of claim 15, wherein the audio input is in natural language form, and wherein the audio input is ambiguous with respect to defining the action.

17. The method of claim 15, wherein the data structure defines one or more actual device characteristics associated with each electronic device in the set of electronic devices.

18. The method of claim 17, wherein the one or more actual electronic device characteristics comprise one or more words associated with each electronic device in the set of electronic devices, or associated with one or more states corresponding to each electronic device in the set of electronic devices.

19. The method of claim 15, further comprising:

determining a location corresponding to the audio input; and

determining, based on the determined location, the predetermined location from a plurality of predetermined locations.

20. The method of claim 15, wherein determining the one or more candidate electronic devices from the set of electronic devices comprises:

determining one or more alternative terms associated with a term in the audio input, wherein the one or more candidate electronic devices is determined based on the one or more alternative terms.

21. The method of claim 15, further comprising:

determining a relative location of the user device with respect to an area within the predetermined location, wherein the user intent is determined based on the relative location.

22. A non-transitory computer-readable storage medium comprising instructions for one or more programs for execution by one or more processors of a user device, the one or more programs including instructions which, when executed by the one or more processors, cause the user device to perform the method of:

receiving an audio input representing a user request for an electronic device to perform an action,

wherein the audio input is in natural language form, and

wherein the audio input is ambiguous with respect to defining the electronic device for performing the action;

determining one or more possible electronic device characteristics corresponding to the audio input;

retrieving a data structure representing a set of electronic devices of a predetermined location;

determining, based on the data structure, one or more candidate electronic devices from the set of electronic devices, the one or more candidate electronic devices corresponding to the audio input;

determining, based on the one or more possible electronic device characteristics and one or more actual electronic device characteristics of the one or more candidate electronic devices, a user intent corresponding to the audio input; and

providing a command that causes an electronic device of the one or more candidate electronic devices to perform the action corresponding to the user intent.

23. The non-transitory computer-readable storage medium of claim 22, wherein the audio input is in natural language form, and wherein the audio input is ambiguous with respect to defining the action.

24. The non-transitory computer-readable storage medium of claim 22, wherein the data structure defines one or more actual device characteristics associated with each electronic device in the set of electronic devices.

25. The non-transitory computer-readable storage medium of claim 24, wherein the one or more actual electronic device characteristics comprise one or more words associated with each electronic device in the set of electronic devices, or associated with one or more states corresponding to each electronic device in the set of electronic devices.

26. The non-transitory computer-readable storage medium of claim 22, wherein the one or more programs comprise further instructions for:

determining a location corresponding to the audio input; and

determining, based on the determined location, the predetermined location from a plurality of predetermined locations.

27. The non-transitory computer-readable storage medium of claim 22, wherein determining the one or more candidate electronic devices from the set of electronic devices comprises:

determining one or more alternative terms associated with a term in the audio input, wherein the one or more candidate electronic devices is determined based on the one or more alternative terms.

28. The non-transitory computer-readable storage medium of claim 22, wherein the one or more programs comprise further instructions for:

determining a relative location of the user device with respect to an area within the predetermined location, wherein the user intent is determined based on the relative location.