SYSTEM AND METHOD FOR GENERATING MACHINE-CURATED SCENES

• US 20180300645A1
• Filed: 05/24/2017
• Published: 10/18/2018
• Est. Priority Date: 04/17/2017
• Status: Active Grant

• Current Assignee: Essential Products, Inc.
• Original Assignee: Essential Products, Inc.
• Inventors: Segal, Mara Clair, Roman, Manuel, Desai, Dwipal, Rubin, Andrew E.
• Granted Patent: US 10,380,493 B2
• Time in Patent Office: Days
• CPC Class Codes: G06N 20/00 Machine learning
• CPC Class Codes: G06N 5/048 Fuzzy inferencing
• CPC Class Codes: G06N 7/02 using fuzzy logic computing arrangements based on biological models G06N3/00; computing arrangements using knowledge-based models G06N5/00
• CPC Class Codes: G06N 7/023 Learning or tuning the parameters of a fuzzy system
• CPC Class Codes: H04L 67/12 specially adapted for proprietary or special-purpose networking environments, e.g. medical networks, sensor networks, networks in vehicles or remote metering networks
• CPC Class Codes: H04L 67/535 Tracking the activity of the user network monitoring arrangements H04L43/00; recording of computer activity G06F11/34
• CPC Class Codes: H04W 84/12 WLAN [Wireless Local Area Networks]

Abstract

The present disclosure contemplates a variety of improved methods and systems for creating a unique user experience using an ambient operating system connected to a variety of disparate IoT devices. The described solution includes a curated scene or set of actions initiated by an assistant device. For example, the assistant device detects it is a weekday morning and initiates actions associated with the wakeup routine such as opening the blinds, making coffee and notifying the user of the current traffic report.

Claims

1. A method for automatically generating a scene for an assistant device to cause one or more devices to perform functionality automatically within an environment, comprising:

receiving, by a processor, an image frame depicting a user interaction with a first device of one or more devices within the environment, and receiving audio information indicating the user interaction with the first device of the one or more devices within the environment, the one or more devices communicatively coupled with the assistant device via a wireless local area network (WLAN);

using the image frame and the audio information to identify the first device of the one or more devices with which a user is interacting;

using the image frame or the audio information identifying the functionality the user interaction causes the first device to perform;

identifying a previous user interaction of the user causing a second device of the one or more devices to perform a second functionality which is related to the user interaction with the first device;

determining using fuzzy matching that the previous user interaction with the second device and the user interaction with the first device matches a scene template representing a standardized set of user interactions with the one or more devices in the environment;

generating the scene associated with the scene template, and comprising a trigger representing an instruction for initiating the scene, and causing the first device, to perform the functionality associated with the user interaction, and the second device to perform the second functionality associated with the previous user interaction; and

receiving a second image frame or a second audio information indicating the user providing the trigger associated with the scene;

using the second image frame or the second audio information to identify the trigger associated with the scene;

performing automatically the functionality of the first device associated with the user interaction and the second functionality of the second device associated with the previous user interaction associated with the scene responsive to receiving the trigger.

2. A method, comprising:

receiving, by a processor, an image frame depicting a user interaction with a first device of one or more devices within the environment, and receiving audio information indicating the user interaction with the first device of the one or more devices within an environment, the one or more devices communicatively coupled with an assistant device via a wireless local area network (WLAN);

using the image frame and the audio information to identify the first device of the one or more devices with which a user is interacting;

using the image frame or the audio information to identify a functionality the user interaction causes the first device to perform;

identifying a previous user interaction of the user causing a second device of the one or more devices to perform a second functionality which is related to the user interaction with the first device;

determining a scene template by matching the scene template to the previous user interaction with the second device and the user interaction with the first device to a template associated with the scene, the first device, the functionality associated with the user interaction, the second device, and the second functionality associated with the previous user interaction;

generating the scene associated with the scene template, and comprising a trigger representing an instruction for initiating the scene causing the first device, to perform the functionality associated with the user interaction, and the second device to perform the second functionality associated with the previous user interaction;

receiving the trigger representing the instruction for initiating the scene;

and

performing the functionality of the first device associated with the user interaction and the second functionality of the second device associated with the previous user interaction responsive to receiving the trigger associated with the scene.

5. The method of claim 2, wherein the scene template is determined using a fuzzy matching algorithm.

6. The method of claim 5, wherein initiating the scene causes a third device to perform a third functionality.

7. The method of claim 6, comprises receiving configuration information from a user;

and wherein generating the scene includes the configuration information.

8. The method of claim 7, wherein the one or more devices are connected on a wireless network with the assistant device.

9. The method of claim 8, wherein one or more users are identified in the image frame.

10. The method of claim 9, wherein generating the scene includes associating the trigger with the one or more users who have authority to cause the scene to be performed.

20. The method of claim 10, wherein a location of the first device is determined using the image frame, and the location of the first device is used to determine the scene template.

3-4. -4. (canceled)

11. An electronic device, comprising:

one or more processors;

memory storing instructions, wherein the processor is configured to execute the instructions such that the processor and memory are configured to;

receive an image frame depicting a user interaction with a first device of one or more devices within the environment, and receiving audio information indicating the user interaction with the first device of the one or more devices within an environment, the one or more devices communicatively coupled with an assistant device via a wireless local area network (WLAN);

use the image frame and the audio information to identify the first device of the one or more devices with which a user is interacting;

identify using the image frame or the audio information a functionality which the user interaction causes the first device to perform;

identify a previous user interaction of the user causing a second device of the one or more devices to perform a second functionality which is related to the user interaction with the first device;

determine a scene template by matching the scene template to the previous user interaction with the second device and the user interaction with the first device to a template associated with the scene, the first device, the functionality associated with the user interaction, the second device, and the second functionality associated with the previous user interaction;

generate the scene associated with the scene template, and comprising a trigger representing an instruction for initiating the scene causing the first device, to perform the functionality associated with the user interaction, and the second device to perform the second functionality associated with the previous user interaction;

receiving the trigger representing the instruction for initiating the scene; and

performing the functionality of the first device associated with the user interaction and the second functionality of the second device associated with the previous user interaction responsive to receiving the trigger associated with the scene.

14. The electronic device of claim 11, wherein the scene template is determining using a fuzzy matching algorithm.

15. The electronic device of claim 14, wherein initiating the scene includes causing a third device to perform a third functionality.

16. The electronic device of claim 15, further comprising the processor and memory are configured to receive configuration information from a user and generate the scene including the configuration information.

17. The electronic device of claim 16, wherein the one or more devices are connected on a wireless network with the assistant device.

18. The electronic device of claim 17, wherein one or more users are identified in the image frame.

19. The electronic device of claim 18, wherein generation of the scene includes associating the trigger with the one or more users who have authority to cause the scene to be performed.

21. The electronic device of claim 19, wherein a location of the first device is determined using the image frame, and the location of the first device is used to determine the scene template.

12-13. -13. (canceled)

22. A computer program product including non-transitory computer-readable media storing computer program instructions, execution of which by a processor causes a processing system to perform operations comprising:

receiving an image frame depicting a user interaction with a first device of one or more devices within the environment, and receiving audio information indicating the user interaction with the first device of the one or more devices within an environment, the one or more devices communicatively coupled with an assistant device via a wireless local area network (WLAN);

using the image frame and the audio information to identify the first device of the one or more devices with which a user is interacting;

using the image frame or the audio information to identify a functionality the user interaction causes the first device to perform;

identifying a previous user interaction of the user causing a second device of the one or more devices to perform a second functionality which is related to the user interaction with the first device;

determining a scene template by matching the scene template to the previous user interaction with the second device and the user interaction with the first device to a template associated with the scene, the first device, the functionality associated with the user interaction, the second device, and the second functionality associated with the previous user interaction;

generating the scene associated with the scene template, and comprising a trigger representing an instruction for initiating the scene, and thereby causing the first device, to perform the functionality associated with the user interaction, and the second device to perform the second functionality associated with the previous user interaction;

receiving the trigger representing the instruction for initiating the scene;

and

performing the functionality of the first device associated with the user interaction and the second functionality of the second device associated with the previous user interaction responsive to receiving the trigger associated with the scene.