Controlling connected devices using a relationship graph

• US 10,157,613 B2
• Filed: 11/17/2016
• Issued: 12/18/2018
• Est. Priority Date: 11/17/2016
• Status: Active Grant

• Current Assignee: BrainofT, Inc.
• Original Assignee: BrainofT, Inc.
• Inventors: Saxena, Ashutosh, Berman, Brendan, Deng, Deng, Kroc, Lukas
• Primary Examiner(s): SHIN, SEONG-AH A
• Application Number:
  US15/354,115
• Publication Number:
  US 20180137858A1
• Time in Patent Office: 761 Days
• Field of Search: 704234, 704235, 704255, 704275
• CPC Class Codes: G06F 3/167 Audio in a user interface, e.g. using voice commands for navigating, audio feedback
• CPC Class Codes: G10L 15/22 Procedures used during a speech recognition process, e.g. man-machine dialogue
• CPC Class Codes: G10L 15/30 Distributed recognition, e.g. in client-server systems, for mobile phones or network applications
• CPC Class Codes: G10L 2015/223 Execution procedure of a spoken command
• CPC Class Codes: H04L 12/2816 Controlling appliance services of a home automation network by calling their functionalities arrangements in telecontrol or telemetry systems for selectively calling a substation from a main station; in which substation desired apparatus is selected for applying a control signal thereto or for obtaining measured values therefrom H04Q9/00
• CPC Class Codes: H04L 12/282 based on user interaction within the home receiver circuitry for displaying additional information being controlled by a remote control apparatus H04N21/42204

Abstract

Network connected devices are controlled. A command is received. A graph model is applied to identify a subset of nodes related to the command. The graph model includes a plurality of nodes that each correspond to a device and the graph model includes a plurality of edges that specify relationships between the plurality of nodes. The subset of nodes is activated in response to the command.

Claims

1. A method for controlling network connected devices, comprising:

receiving a command;

using a processor to apply a graph model to identify a subset of nodes related to the command, wherein the graph model includes a plurality of nodes that each correspond to a device and the graph model includes a plurality of edges that specify relationships between the plurality of nodes, at least one of the plurality of nodes of the graph model corresponds to a physical area within a larger physical deployment environment, and at least one of the plurality of edges specifies a physical spatial relationship between corresponding nodes connected by the at least one corresponding edge; and

activating the subset of nodes in response to the command, wherein activating the subset of nodes includes automatically controlling at least a portion of the network connected devices,

using a result of the activation of the subset of nodes in evaluating a plurality of output candidates according to an optimization function, and selecting an output that optimizes the optimization function among the output candidates; and

receiving a feedback associated with the output and updating one or more reference optimization evaluation values based at least in part on the received feedback; and

wherein the subset of nodes related to the command is identified including by identifying a specific node of the subset of nodes as being associated with an associated physical area of the command within the larger physical deployment environment and identifying one or more of the network connected devices associated with the associated physical area including by identifying one or more other nodes of the subset of nodes connected to the identified specific node by one or more physical spatial relationship identifying edges.

2. The method of claim 1, wherein the command is not an explicit user command and the command was detected based on a detection of an environmental property.

3. The method of claim 1, wherein one or more of the plurality of edges are associated with a dynamic weight value.

4. The method of claim 1, wherein the graph model is implemented as a sparse table or a sparse matrix.

5. The method of claim 1, wherein the command is associated with an automation rule.

6. The method of claim 1, wherein the graph model was generated using one or more graph model templates.

7. The method of claim 1, wherein the graph model is hierarchical and includes a plurality of levels.

8. The method of claim 1, wherein one or more of the plurality of edges identify a physical relationship between one or more of the plurality of nodes.

9. The method of claim 1, wherein one or more of the plurality of edges identify a functional relationship between one or more of the plurality of nodes.

10. The method of claim 1, wherein for a particular behavioral activity, one or more of the plurality of edges identify a relationship between one or more of the plurality of nodes.

11. The method of claim 1, wherein a plurality of different types of edges connect two same nodes of the plurality of nodes.

12. The method of claim 1, wherein the command is associated with performing a test to verify one or more nodes and edges of the graph model.

13. The method of claim 1, wherein the command includes a voice command.

14. The method of claim 1, wherein the command is associated with an automatically discovered automation rule and the automation rule was discovered using the graph model.

15. The method of claim 1, wherein applying the graph model includes identifying a location of a pet or a human user and a corresponding node of the graph model to the location of the pet or the human user.

16. The method of claim 1, wherein applying the graph model includes using an automation rule discovered using a different graph model.

17. The method of claim 1, wherein applying the graph model includes determining nodes associated with a location identified by the command.

18. The method of claim 1, wherein the subset of nodes related to the command has been identified based on properties associated with one or more edges of the subset of nodes.

19. The method of claim 1, wherein at least one of the subset of nodes related to the command has been identified based on a relationship of the at least one node with respect to an outdoor environmental property.

20. A system for controlling network connected devices, including:

a processor configured to;

receive a command;

apply a graph model to identify a subset of nodes related to the command, wherein the graph model includes a plurality of nodes that each correspond to a device and the graph model includes a plurality of edges that specify relationships between the plurality of nodes, at least one of the plurality of nodes of the graph model corresponds to a physical area within a larger physical deployment environment, and at least one of the plurality of edges specifies a physical spatial relationship between corresponding nodes connected by the at least one corresponding edge; and

activate the subset of nodes in response to the command, wherein activating the subset of nodes includes automatically controlling at least a portion of the network connected devices, using a result of the activation of the subset of nodes in evaluating a plurality of output candidates according to an optimization function, and selecting an output that optimizes the optimization function among the output candidates; and

receive a feedback associated with the output and update one or more reference optimization evaluation values based at least in part on the received feedback; and

wherein the subset of nodes related to the command is identified including by identifying a specific node of the subset of nodes as being associated with an associated physical area of the command within the larger physical deployment environment and identifying one or more of the network connected devices associated with the associated physical area including by identifying one or more other nodes of the subset of nodes connected to the identified specific node by one or more physical spatial relationship identifying edges; and

a memory coupled with the processor, wherein the memory is configured to provide the processor with instructions.

21. The system of claim 20, wherein the command is associated with an automation rule.

22. The system of claim 20, wherein the command includes a voice command.

23. A computer program product for controlling network connected devices, the computer program product being embodied in a non-transitory computer readable storage medium and comprising computer instructions for:

receiving a command;

applying a graph model to identify a subset of nodes related to the command, wherein the graph model includes a plurality of nodes that each correspond to a device and the graph model includes a plurality of edges that specify relationships between the plurality of nodes, at least one of the plurality of nodes of the graph model corresponds to a physical area within a larger physical deployment environment, and at least one of the plurality of edges specifies a physical spatial relationship between corresponding nodes connected by the at least one corresponding edge; and

activating the subset of nodes in response to the command, wherein activating the subset of nodes includes automatically controlling at least a portion of the network connected devices, using a result of the activation of the subset of nodes in evaluating a plurality of output candidates according to an optimization function, and selecting an output that optimizes the optimization function among the output candidates; and

receiving a feedback associated with the output and updating one or more reference optimization evaluation values based at least in part on the received feedback; and

wherein the subset of nodes related to the command is identified including by identifying a specific node of the subset of nodes as being associated with an associated physical area of the command within the larger physical deployment environment and identifying one or more of the network connected devices associated with the associated physical area including by identifying one or more other nodes of the subset of nodes connected to the identified specific node by one or more physical spatial relationship identifying edges.