Intelligent assistant for home automation

• US 10,714,095 B2
• Filed: 10/29/2018
• Issued: 07/14/2020
• Est. Priority Date: 05/30/2014
• Status: Active Grant

• Current Assignee: Apple Inc.
• Original Assignee: Apple Inc.
• Inventors: Orr, Ryan M., Nell, Garett R., Brumbaugh, Benjamin L.
• Primary Examiner(s): Yang, Qian
• Application Number:
  US16/174,046
• Publication Number:
  US 20190074015A1
• Time in Patent Office: 624 Days
• CPC Class Codes: G05B 15/02 electric
• CPC Class Codes: G05B 2219/23373 Interactive guidance by voice message
• CPC Class Codes: G05B 2219/23386 Voice, vocal command or message
• CPC Class Codes: G05B 2219/2642 Domotique, domestic, home control, automation, smart house
• CPC Class Codes: G06F 3/167 Audio in a user interface, e.g. using voice commands for navigating, audio feedback
• CPC Class Codes: G10L 17/22 Interactive procedures; Man-machine interfaces
• CPC Class Codes: H04L 12/2816 Controlling appliance services of a home automation network by calling their functionalities arrangements in telecontrol or telemetry systems for selectively calling a substation from a main station; in which substation desired apparatus is selected for applying a control signal thereto or for obtaining measured values therefrom H04Q9/00
• CPC Class Codes: H04L 12/282 based on user interaction within the home receiver circuitry for displaying additional information being controlled by a remote control apparatus H04N21/42204

Abstract

This relates to systems and processes for using a virtual assistant to control electronic devices. In one example process, a user can speak an input in natural language form to a user device to control one or more electronic devices. The user device can transmit the user speech to a server to be converted into a textual representation. The server can identify the one or more electronic devices and appropriate commands to be performed by the one or more electronic devices based on the textual representation. The identified one or more devices and commands to be performed can be transmitted back to the user device, which can forward the commands to the appropriate one or more electronic devices for execution. In response to receiving the commands, the one or more electronic devices can perform the commands and transmit their current states to the user device.

Claims

1. A user device comprising:

one or more processors;

memory; and

one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, the one or more programs including instructions for;

receiving, by the user device, an audio input comprising a user speech;

transmitting data corresponding to the audio input to one or more servers;

receiving, from the one or more servers;

a plurality of commands to set a state of each of a plurality of electronic devices determined by the one or more servers based on a stored configuration, wherein the stored configuration defines a state of each of the plurality of electronic devices to use in response to a command that references the configuration; and

identifications associated with each of the plurality of commands, wherein the identifications identify each of the plurality of electronic devices for performing each of the plurality of commands; and

transmitting the plurality of commands to the plurality of electronic devices based on the identifications associated with each of the plurality of commands.

2. The user device of claim 1, wherein the one or more programs comprise further instructions for:

before transmitting the plurality of commands, querying each of the plurality of electronic devices to determine a current state of each of the plurality of electronic devices.

3. The user device of claim 1, wherein the one or more programs comprise further instructions for:

after transmitting the plurality of commands, receiving an updated state of each of the plurality of electronic devices from the plurality of electronic devices; and

transmitting the updated state of each of the plurality of electronic devices to the one or more servers.

4. The user device of claim 3, wherein the one or more programs comprise further instructions for:

outputting an indication of a result of each of the plurality of commands based on the updated state of each of the plurality of electronic devices received from the plurality of electronic devices.

5. The user device of claim 4, wherein the indication of the result of each of the plurality of commands is an audio indication, a visual indication, or both an audio and a visual indication.

6. The user device of claim 3, wherein the plurality of electronic devices comprises a light bulb.

7. The user device of claim 6, wherein the plurality of commands comprises a command to set an ON/OFF state, dimmable state, or color state of the light bulb.

8. The user device of claim 7, wherein the updated state of the light bulb comprises the ON/OFF state, dimmable state, or color state of the light bulb after transmitting the command to set the ON/OFF state, dimmable state, or color state of the light bulb.

9. The user device of claim 3, wherein the plurality of electronic devices comprises an electrical outlet.

10. The user device of claim 9, wherein the plurality of commands comprises a command to set an ACTIVE/INACTIVE state of the electrical outlet.

11. The user device of claim 10, wherein the updated state of the electrical outlet comprises the ACTIVE/INACTIVE state of the electrical outlet after transmitting the command to set the ACTIVE/INACTIVE state of the electrical outlet.

12. The user device of claim 3, wherein the plurality of electronic devices comprises a switch.

13. The user device of claim 12, wherein the plurality of commands comprises a command to set an ON/OFF state of the switch.

14. The user device of claim 13, wherein the updated state of the electrical outlet comprises the ON/OFF state of the electrical outlet after transmitting the command to set the ON/OFF state of the switch.

15. The user device of claim 3, wherein the plurality of electronic devices comprises a door lock.

16. The user device of claim 15, wherein the plurality of commands comprises a command to set a LOCKED/UNLOCKED state of the door lock.

17. The user device of claim 16, wherein the updated state of the door lock comprises the LOCKED/UNLOCKED state of the door lock after transmitting the command to set the LOCKED/UNLOCKED state of the door lock.

18. The user device of claim 3, wherein the plurality of electronic devices comprises a garage door.

19. The user device of claim 18, wherein the plurality of commands comprises a command to set an OPEN/CLOSED state of the garage door.

20. The user device of claim 19, wherein the updated state of the garage door comprises the OPEN/CLOSED state of the garage door after transmitting the command to set the OPEN/CLOSED state of the garage door.

21. The user device of claim 3, wherein the plurality of electronic devices comprises a thermostat.

22. The user device of claim 21, wherein the plurality of commands comprises a command to set a numerical value of a temperature setting of the thermostat.

23. The user device of claim 22, wherein the updated state of the thermostat comprises the numerical value of the temperature setting of the thermostat after transmitting the command to set the numerical value of the temperature setting of the thermostat.

24. The user device of claim 3, wherein the user device comprises a database for storing a current state of each of the plurality of electronic devices, and

wherein the current state of each of the plurality of electronic devices is updated based at least in part on the updated state of each of the plurality of electronic devices received from the plurality of electronic devices.

25. The user device of claim 1, wherein the plurality of commands are transmitted to the plurality of electronic devices in response to a determination that a predetermined condition is satisfied.

26. The user device of claim 25, wherein the determination that a predetermined condition is satisfied comprises determining that the user device enters a predetermined area.

27. The user device of claim 25, wherein the determination that a predetermined condition is satisfied comprises determining that the user device exits a predetermined area during a predetermined window of time.

28. A method for controlling a plurality of electronic devices using a virtual assistant on a user device having one or more processors and a memory, the method comprising:

at the user device;

receiving an audio input comprising a user speech;

transmitting data corresponding to the audio input to one or more servers;

receiving, from the one or more servers;

a plurality of commands to set a state of each of a plurality of electronic devices determined by the one or more servers based on a stored configuration, wherein the stored configuration defines a state of each of the plurality of electronic devices to use in response to a command that references the configuration; and

identifications associated with each of the plurality of commands, wherein the identifications identify each of the plurality of electronic devices for performing each of the plurality of commands; and

transmitting the plurality of commands to the plurality of electronic devices based on the identifications associated with each of the plurality of commands.

29. The method of claim 28, further comprising:

before transmitting the plurality of commands, querying each of the plurality of electronic devices to determine a current state of each of the plurality of electronic devices.

30. The method of claim 28, further comprising:

after transmitting the plurality of commands, receiving an updated state of each of the plurality of electronic devices from the plurality of electronic devices; and

transmitting the updated state of each of the plurality of electronic devices to the one or more servers.

31. The method of claim 28, wherein the plurality of commands are transmitted to the plurality of electronic devices in response to a determination that a predetermined condition is satisfied.

32. The method of claim 31, wherein the determination that a predetermined condition is satisfied comprises determining that the user device enters a predetermined area.

33. The method of claim 31, wherein the determination that a predetermined condition is satisfied comprises determining that the user device exits a predetermined area during a predetermined window of time.

34. A non-transitory computer-readable storage medium comprising instructions for one or more programs for execution by one or more processors of a user device, the one or more programs including instructions which, when executed by the one or more processors, cause the user device to perform the method of:

receiving an audio input comprising a user speech;

transmitting data corresponding to the audio input to one or more servers;

receiving, from the one or more servers;

a plurality of commands to set a state of each of a plurality of electronic devices determined by the one or more servers based on a stored configuration, wherein the stored configuration defines a state of each of the plurality of electronic devices to use in response to a command that references the configuration; and

identifications associated with each of the plurality of commands, wherein the identifications identify each of the plurality of electronic devices for performing each of the plurality of commands; and

transmitting the plurality of commands to the plurality of electronic devices based on the identifications associated with each of the plurality of commands.

35. The non-transitory computer-readable storage medium of claim 34, wherein the one or more programs comprise further instructions for:

before transmitting the plurality of commands, querying each of the plurality of electronic devices to determine a current state of each of the plurality of electronic devices.

36. The non-transitory computer-readable storage medium of claim 34, wherein the one or more programs comprise further instructions for:

after transmitting the plurality of commands, receiving an updated state of each of the plurality of electronic devices from the plurality of electronic devices; and

transmitting the updated state of each of the plurality of electronic devices to the one or more servers.

37. The non-transitory computer-readable storage medium of claim 34, wherein the plurality of commands are transmitted to the plurality of electronic devices in response to a determination that a predetermined condition is satisfied.

38. The non-transitory computer-readable storage medium of claim 37, wherein the determination that a predetermined condition is satisfied comprises determining that the user device enters a predetermined area.

39. The non-transitory computer-readable storage medium of claim 37, wherein the determination that a predetermined condition is satisfied comprises determining that the user device exits a predetermined area during a predetermined window of time.