Home device application programming interface

• US 10,073,681 B2
• Filed: 08/02/2017
• Issued: 09/11/2018
• Est. Priority Date: 01/23/2017
• Status: Active Grant

• Current Assignee: Essential Products, Inc.
• Original Assignee: Essential Products, Inc.
• Inventors: Roman, Manuel, Segal, Mara Clair, Desai, Dwipal, Rubin, Andrew E.
• Primary Examiner(s): Chen, Qing
• Application Number:
  US15/667,351
• Publication Number:
  US 20180210710A1
• Time in Patent Office: 405 Days
• Field of Search: 717106-109
• CPC Class Codes: G06F 8/20 Software design
• CPC Class Codes: G06F 8/30 Creation or generation of source code
• CPC Class Codes: G06F 8/34 Graphical or visual programming
• CPC Class Codes: G06F 8/38 for implementing user interfaces
• CPC Class Codes: G06N 20/00 Machine learning
• CPC Class Codes: H04L 12/2816 Controlling appliance services of a home automation network by calling their functionalities arrangements in telecontrol or telemetry systems for selectively calling a substation from a main station; in which substation desired apparatus is selected for applying a control signal thereto or for obtaining measured values therefrom H04Q9/00

Abstract

The present disclosure contemplates a variety of improved methods and systems for enabling developers to program and control a variety of disparate devices coupled to the framework and/or ambient operating system. The Home API can expose operation of the devices in a natural and intuitive manner, facilitating control of unknown devices by high-level commands.

Claims

1. A method for seamlessly and automatically building an application using an application programming interface (API) to interface with one or more devices within an environment with an assistant device having artificial intelligence (AI), comprising:

receiving, via a processor, a graphic representing a layout of one or more graphical user interface (GUI) elements for the application to instruct the assistant device to control the one or more devices within the environment;

identifying, by the processor, a section of the graphic corresponding to the one or more GUI elements for the application that is to be a selectable item on the application to implement a functionality to be performed within the environment using the assistant device;

identifying, by the processor, a graphical content within the section of the graphic;

determining, by the processor, object characteristics of the graphical content within the section of the graphic corresponding to the one or more GUI elements for the application that is to be the selectable item on the application to implement the functionality to be performed within the environment using the assistant device;

identifying, by the processor, a textual content within the section of the graphic;

determining, by the processor, text characteristics of the textual content within the section of the graphic corresponding to the one or more GUI elements for the application that is to be the selectable item on the application to implement the functionality to be performed within the environment using the assistant device;

determining, by the processor using a database, the functionality to be performed within the environment using the assistant device based on the text characteristics of the textual content within the section of the graphic and the object characteristics of the graphical content within the section of the graphic;

identifying, by the processor, the one or more devices capable of performing the functionality within the environment; and

generating, by the processor, the application having the section of the graphic, wherein the section of the graphic, when selected, instructs the assistant device to cause the identified one or more devices to perform the functionality within the environment.

2. A method, comprising:

receiving, via a processor, a graphic representing a layout of one or more graphical user interface (GUI) elements for an application to instruct an assistant device to control one or more devices within an environment;

identifying, by the processor, a section of the graphic corresponding to the one or more GUI elements for the application corresponding to a functionality to be performed within the environment using the assistant device;

identifying, by the processor, a graphical content within the section of the graphic;

determining, by the processor, characteristics of the graphical content within the section of the graphic;

determining, by the processor using a database, the functionality to be performed within the environment using the assistant device based on the characteristics of the graphical content within the section of the graphic;

identifying, by the processor, the one or more devices and associated adapters capable of performing the functionality within the environment; and

generating, by the processor, the application having the one or more GUI elements for the application which, when selected, instruct the assistant device to initialize one or more adapters of the associated adapters causing the identified one or more devices to perform the functionality within the environment.

3. The method of claim 2, wherein the characteristics of the graphical content within the section of the graphic include a position of the graphical content within the section of the graphic.

4. The method of claim 3, wherein the characteristics of the graphical content within the section of the graphic include a size of the graphical content within the section of the graphic in relation to a size of the section of the graphic.

5. The method of claim 2, wherein the characteristics of the graphical content within the section of the graphic include meaning of a depicted text.

6. The method of claim 2, wherein the characteristics of the graphical content within the section of the graphic include a color of a depicted object found on the graphic outside the section of the graphic.

7. The method of claim 2, wherein the characteristics of the graphical content within the section of the graphic include a color of the section of the graphic.

8. The method of claim 2, wherein the characteristics of the graphical content within the section of the graphic include a color of a depicted object found in the section of the graphic.

9. The method of claim 2, wherein determining the functionality to be performed within the environment using the assistant device includes a first machine learning algorithm.

10. The method of claim 9, wherein the graphical content within the section of the graphic includes a depicted object or a depicted text.

11. The method of claim 9, wherein identifying the one or more devices capable of performing the functionality within the environment includes a second machine learning algorithm, and wherein the first machine learning algorithm and the second machine learning algorithm are different machine learning algorithms.

12. The method of claim 2, wherein the one or more adapters of the associated adapters are stored on the assistant device.

13. The method of claim 2, wherein the one or more adapters of the associated adapters are stored on remote resources.

14. A system, comprising:

a processor; and

a memory storing instructions, wherein the processor is configured to execute the instructions such that the processor and the memory are configured to;

receive a graphic representing a layout of one or more graphical user interface (GUI) elements for an application to instruct an assistant device to control one or more devices within an environment;

identify a section of the graphic corresponding to the one or more GUI elements for the application corresponding to a functionality to be performed within the environment using the assistant device;

identify a graphical content within the section of the graphic;

determine characteristics of the graphical content within the section of the graphic;

determine, using a database, the functionality to be performed within the environment using the assistant device based on the characteristics of the graphical content within the section of the graphic;

identify the one or more devices and associated adapters capable of performing the functionality within the environment; and

generate the application having the one or more GUI elements for the application which, when selected, instruct the assistant device to initialize one or more adapters of the associated adapters causing the identified one or more devices to perform the functionality within the environment.

15. The system of claim 14, wherein the characteristics of the graphical content within the section of the graphic include a position of the graphical content within the section of the graphic.

16. The system of claim 15, wherein the characteristics of the graphical content within the section of the graphic include a size of the graphical content within the section of the graphic in relation to a size of the section of the graphic.

17. The system of claim 14, wherein the characteristics of the graphical content within the section of the graphic include meaning of a depicted text.

18. The system of claim 14, wherein the characteristics of the graphical content within the section of the graphic include a color of a depicted object found on the graphic outside the section of the graphic.

19. The system of claim 14, wherein the characteristics of the graphical content within the section of the graphic include a color of the section of the graphic.

20. The system of claim 14, wherein the characteristics of the graphical content within the section of the graphic include a color of a depicted object found in the section of the graphic.

21. The system of claim 14, wherein determining the functionality to be performed within the environment using the assistant device includes a first machine learning algorithm.

22. The system of claim 21, wherein the graphical content within the section of the graphic includes a depicted object or a depicted text.

23. The system of claim 21, wherein identifying the one or more devices capable of performing the functionality within the environment includes a second machine learning algorithm, and wherein the first machine learning algorithm and the second machine learning algorithm are different machine learning algorithms.

24. The system of claim 14, wherein the one or more adapters of the associated adapters are stored on the assistant device.

25. The system of claim 14, wherein the one or more adapters of the associated adapters are stored on remote resources.

26. A computer program product, comprising one or more non-transitory computer-readable media having computer program instructions stored therein, the computer program instructions being configured such that, when executed by one or more computing devices, the computer program instructions cause the one or more computing devices to:

receive a graphic representing a layout of one or more graphical user interface (GUI) elements for an application to instruct an assistant device to control one or more devices within an environment;

identify a section of the graphic corresponding to the one or more GUI elements for the application corresponding to a functionality to be performed within the environment using the assistant device;

identify a graphical content within the section of the graphic;

determine characteristics of the graphical content within the section of the graphic;

determine, using a database, the functionality to be performed within the environment using the assistant device based on the characteristics of the graphical content within the section of the graphic;

identify the one or more devices and associated adapters capable of performing the functionality within the environment; and

generate the application having the one or more GUI elements for the application which, when selected, instruct the assistant device to initialize one or more adapters of the associated adapters causing the identified one or more devices to perform the functionality within the environment.

27. The computer program product of claim 26, wherein the characteristics of the graphical content within the section of the graphic include a position of the graphical content within the section of the graphic.

28. The computer program product of claim 27, wherein the characteristics of the graphical content within the section of the graphic include a size of the graphical content within the section of the graphic in relation to a size of the section of the graphic.

29. The computer program product of claim 26, wherein the characteristics of the graphical content within the section of the graphic include meaning of a depicted text.

30. The computer program product of claim 26, wherein the characteristics of the graphical content within the section of the graphic include a color of a depicted object found on the graphic outside the section of the graphic.

31. The computer program product of claim 26, wherein the characteristics of the graphical content within the section of the graphic include a color of the section of the graphic.

32. The computer program product of claim 26, wherein the characteristics of the graphical content within the section of the graphic include a color of a depicted object found in the section of the graphic.

33. The computer program product of claim 26, wherein determining the functionality to be performed within the environment using the assistant device includes a first machine learning algorithm.

34. The computer program product of claim 33, wherein the graphical content within the section of the graphic includes a depicted object or a depicted text.

35. The computer program product of claim 33, wherein identifying the one or more devices capable of performing the functionality within the environment includes a second machine learning algorithm, and wherein the first machine learning algorithm and the second machine learning algorithm are different machine learning algorithms.

36. The computer program product of claim 26, wherein the one or more adapters of the associated adapters are stored on the assistant device.

37. The computer program product of claim 26, wherein the one or more adapters of the associated adapters are stored on remote resources.