INTELLIGENT ASSISTANT FOR HOME AUTOMATION

• US 20150348554A1
• Filed: 09/30/2014
• Published: 12/03/2015
• Est. Priority Date: 05/30/2014
• Status: Active Grant

• Current Assignee: Apple Inc.
• Original Assignee: Apple Inc.
• Inventors: Orr, Ryan M., Nell, Garett R., Brumbaugh, Benjamin L.
• Granted Patent: US 10,170,123 B2
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: G05B 15/02 electric
• CPC Class Codes: G05B 2219/23373 Interactive guidance by voice message
• CPC Class Codes: G05B 2219/23386 Voice, vocal command or message
• CPC Class Codes: G05B 2219/2642 Domotique, domestic, home control, automation, smart house
• CPC Class Codes: G06F 3/167 Audio in a user interface, e.g. using voice commands for navigating, audio feedback
• CPC Class Codes: G10L 17/22 Interactive procedures; Man-machine interfaces
• CPC Class Codes: H04L 12/2816 Controlling appliance services of a home automation network by calling their functionalities arrangements in telecontrol or telemetry systems for selectively calling a substation from a main station; in which substation desired apparatus is selected for applying a control signal thereto or for obtaining measured values therefrom H04Q9/00
• CPC Class Codes: H04L 12/282 based on user interaction within the home receiver circuitry for displaying additional information being controlled by a remote control apparatus H04N21/42204

Abstract

This relates to systems and processes for using a virtual assistant to control electronic devices. In one example process, a user can speak an input in natural language form to a user device to control one or more electronic devices. The user device can transmit the user speech to a server to be converted into a textual representation. The server can identify the one or more electronic devices and appropriate commands to be performed by the one or more electronic devices based on the textual representation. The identified one or more devices and commands to be performed can be transmitted back to the user device, which can forward the commands to the appropriate one or more electronic devices for execution. In response to receiving the commands, the one or more electronic devices can perform the commands and transmit their current states to the user device.

Claims

1. A method for controlling electronic devices using a virtual assistant on a user device, the method comprising:

receiving, by the user device, an audio input comprising user speech;

transmitting data corresponding to the audio input to one or more servers;

receiving, from the one or more servers;

an identification of a first electronic device determined by the one or more servers based on the data corresponding to the audio input; and

a first command to be performed by the first electronic device determined by the one or more servers based on the data corresponding to the audio input; and

transmitting the first command to the first electronic device.

2. The method of claim 1, further comprising:

receiving, after transmitting the first command to the first electronic device, a current state of the first electronic device from the first electronic device; and

transmitting the current state of the first electronic device to the one or more servers.

3. The method of claim 2, wherein the first electronic device comprises a light bulb.

4. The method of claim 3, wherein the first command comprises a command to set an ON/OFF state, dimmable state, or color state of the light bulb.

5. The method of claim 4, wherein the current state of the first electronic device comprises the ON/OFF state, dimmable state, or color state of the light bulb after transmitting the first command to set the ON/OFF state, dimmable state, or color state of the light bulb.

6. The method of claim 2, wherein the first electronic device comprises an electrical outlet.

7. The method of claim 6, wherein the first command comprises a command to set an ACTIVE/INACTIVE state of the electrical outlet.

8. The method of claim 7, wherein the current state of the first electronic device comprises the ACTIVE/INACTIVE state of the electrical outlet after transmitting the command to set the ACTIVE/INACTIVE state of the electrical outlet.

9. The method of claim 2, wherein the first electronic device comprises a switch.

10. The method of claim 9, wherein the first command comprises a command to set an ON/OFF state of the switch.

11. The method of claim 10, wherein the current state of the first electronic device comprises the ON/OFF state of the electrical outlet after transmitting the command to set the ON/OFF state of the switch.

12. The method of claim 2, wherein the first electronic device comprises a door lock.

13. The method of claim 12, wherein the first command comprises a command to set a LOCKED/UNLOCKED state of the door lock.

14. The method of claim 13, wherein the current state of the first electronic device comprises the LOCKED/UNLOCKED state of the door lock after transmitting the command to set the LOCKED/UNLOCKED state of the door lock.

15. The method of claim 2, wherein the first electronic device comprises a garage door.

16. The method of claim 15, wherein the first command comprises a command to set an OPEN/CLOSED state of the garage door.

17. The method of claim 16, wherein the current state of the first electronic device comprises the OPEN/CLOSED state of the garage door after transmitting the command to set the OPEN/CLOSED state of the garage door.

18. The method of claim 2, wherein the first electronic device comprises a thermostat.

19. The method of claim 18, wherein the first command comprises a command to set a numerical value of a temperature setting of the thermostat.

20. The method of claim 19, wherein the current state of the first electronic device comprises the numerical value of the temperature setting of the thermostat after transmitting the first command to set the numerical value of the temperature setting of the thermostat.

21. The method of claim 2, further comprising outputting a notification associated with the current state of the first electronic device in response to determining that a predetermined condition has been satisfied.

22. The method of claim 2, wherein the user device comprises a database for storing a state of each of a plurality of electronic devices, the plurality of electronic devices comprising the first electronic device, and wherein the method further comprises updating a state of the first electronic device stored in the database based at least in part on the current state of the first electronic device received from the first electronic device.

23. The method of claim 2, wherein the method further comprises outputting an indication of a result of the first command based on the first command and the current state of the first electronic device received from the first electronic device, wherein the indication of the result is an audio, visual, or both an audio and visual indication.

24. The method of claim 1, wherein the user device comprises a mobile phone, desktop computer, laptop computer, tablet computer, portable media player, television, television set-top box, or wearable electronic device.

25. The method of claim 1, further comprising:

receiving, from the one or more servers;

an identification of a second electronic device determined by the one or more servers based on the data corresponding to the audio input; and

a second command to be performed by the second electronic device determined by the one or more servers based on the data corresponding to the audio input;

transmitting the second command to the second electronic device;

receiving, after transmitting the second command to the second electronic device, a current state of the second electronic device from the second electronic device; and

transmitting the current state of the second electronic device to the one or more servers.

26. A method for controlling electronic devices using a virtual assistant on a user device, the method comprising:

receiving, by the user device, an audio input comprising a user speech;

transmitting data corresponding to the audio input to one or more servers;

receiving, from the one or more servers, an instruction to query a state of each of a plurality of electronic devices determined by the one or more servers based on the data corresponding to the audio input; and

transmitting a state query to each of the plurality of electronic devices.

27. The method of claim 26, further comprising:

receiving a current state of each of the plurality of electronic devices from the plurality of electronic devices; and

transmitting the current state of each of the plurality of electronic devices to the one or more servers to be stored as a configuration.

28. A non-transitory computer-readable storage medium comprising instructions for:

receiving, by the user device, an audio input comprising user speech;

transmitting data corresponding to the audio input to one or more servers;

receiving, from the one or more servers;

an identification of a first electronic device determined by the one or more servers based on the data corresponding to the audio input; and

a first command to be performed by the first electronic device determined by the one or more servers based on the data corresponding to the audio input; and

transmitting the first command to the first electronic device.

29. The non-transitory computer-readable storage medium of claim 28, further comprising instructions for:

receiving, after transmitting the first command to the first electronic device, a current state of the first electronic device from the first electronic device; and

transmitting the current state of the first electronic device to the one or more servers.

30. The non-transitory computer-readable storage medium of claim 29, further comprising instructions for:

outputting a notification associated with the current state of the first electronic device in response to determining that a predetermined condition has been satisfied.

31. A system comprising:

one or more processors;

memory;

one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, the one or more programs including instructions for;

receiving an audio input comprising user speech;

transmitting data corresponding to the audio input to one or more servers;

means for receiving, from the one or more servers;

an identification of an electronic device determined by the one or more servers based on the data corresponding to the audio input; and

a command to be performed by the electronic device determined by the one or more servers based on the data corresponding to the audio input;

transmitting the command to the electronic device;

receiving, after transmitting the command to the electronic device, a current state of the electronic device from the electronic device; and

transmitting the current state of the electronic device to the one or more servers.

32. The system of claim 31, further comprising instructions for:

receiving, after transmitting the first command to the first electronic device, a current state of the first electronic device from the first electronic device; and

transmitting the current state of the first electronic device to the one or more servers.

33. The system of claim 32, further comprising instructions for:

outputting a notification associated with the current state of the first electronic device in response to determining that a predetermined condition has been satisfied.

34. The system of claim 32, wherein the user device comprises a database for storing a state of each of a plurality of electronic devices, the plurality of electronic devices comprising the first electronic device, and wherein the method further comprises updating a state of the first electronic device stored in the database based at least in part on the current state of the first electronic device received from the first electronic device.