Predictive insertion of graphical objects in a development environment

• US 10,466,863 B1
• Filed: 06/01/2016
• Issued: 11/05/2019
• Est. Priority Date: 06/01/2016
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google LLC (Alphabet Inc.)
• Inventors: Guy, Romain P.
• Primary Examiner(s): Pan, Yongjia
• Application Number:
  US15/170,260
• Time in Patent Office: 1,252 Days
• Field of Search: None
• CPC Class Codes: G06F 3/0482 Interaction with lists of selectable items, e.g. menus
• CPC Class Codes: G06F 3/04845 for image manipulation, e.g. dragging, rotation, expansion or change of colour
• CPC Class Codes: G06F 3/04883 for inputting data by handwriting, e.g. gesture or text
• CPC Class Codes: G06F 8/33 Intelligent editors
• CPC Class Codes: G06F 8/34 Graphical or visual programming
• CPC Class Codes: G06F 8/38 for implementing user interfaces

Abstract

An example method includes outputting, by a computing device and for display in a graphical development environment used during development of a software application, a graphical user interface associated with the software application, receiving, by the computing device, a development specification that associates a region of the graphical user interface with one or more candidate graphical objects for insertion into the graphical user interface, and receiving, by the computing device, an indication of a selection of the region of the graphical user interface. The example method further includes determining, by the computing device, based at least in part on the development specification and the indication of the selection of the region of the graphical user interface, a graphical object from the one or more candidate graphical objects, and outputting, by the computing device and for display at the region of the graphical user interface, an indication of the graphical object.

Claims

1. A method comprising:

outputting, by a computing device and for display in a graphical development environment that is used during development of a software application, a graphical user interface that is associated with the software application;

receiving, by the computing device, a development specification that associates a region of the graphical user interface with one or more candidate graphical objects for insertion into the graphical user interface;

receiving, by the computing device, an indication of a selection of the region of the graphical user interface;

determining, by the computing device, one or more adjacent graphical objects that are located within a threshold distance of the region of the graphical user interface;

determining, by the computing device, a graphical object from the one or more candidate graphical objects, wherein determining the graphical object is based at least in part on (i) the development specification, (ii) the indication of the selection of the region of the graphical user interface, and (iii) the one or more adjacent graphical objects that are located within the threshold distance of the region; and

outputting, by the computing device and for display at the region of the graphical user interface, an indication of the graphical object.

2. The method of claim 1,

wherein outputting the indication of the graphical object comprises outputting, by the computing device and for display at the region of the graphical user interface, a preview of the graphical object as a suggested graphical object for insertion at the region of the graphical user interface, and

wherein the method further comprises:

receiving, by the computing device, an indication of a selection of the preview of the graphical object; and

responsive to receiving the indication of the selection of the preview of the graphical object, outputting, by the computing device and for display at the region of the graphical user interface, the graphical object.

3. The method of claim 1,

wherein the one or more candidate graphical objects comprises a plurality of candidate graphical objects,

wherein the development specification associates the region of the graphical user interface with the plurality of candidate graphical objects, and

wherein determining the graphical object from the one or more candidate graphical objects comprises:

outputting, by the computing device and for display, a plurality of options, wherein each option of the plurality of options represents a respective candidate graphical object of the plurality of candidate graphical objects; and

receiving, by the computing device, an indication of a selection an option from the plurality of options, wherein the selected option represents the graphical object.

4. The method of claim 3, wherein each option of the plurality of options comprises a textual representation of the respective candidate graphical object of the plurality of candidate graphical objects.

5. The method of claim 3, wherein determining the graphical object from the one or more candidate graphical objects further comprises:

determining, by the computing device, a plurality of confidence values, wherein each confidence value of the plurality of confidence values is associated with a respective candidate graphical object of the plurality of candidate graphical objects; and

sorting, by the computing device and based on the plurality of confidence values, the plurality of options to create a sorted order of the plurality of options,

wherein outputting the plurality of options comprises outputting, by the computing device and for display, the plurality of options in the sorted order.

6. The method of claim 1,

wherein determining the graphical object from the one or more candidate graphical objects comprises determining, by the computing device, one or more of a type of the graphical object, a size of the graphical object, an insertion location of the graphical object, or an alignment of the graphical object relative to one or more other graphical objects in the graphical user interface, and

wherein outputting the indication of the graphical object comprises outputting, by the computing device and for display at the region of the graphical user interface, the indication of the graphical object based at least in part on the one or more of the type of the graphical object, the size of the graphical object, the insertion location of the graphical object, or the alignment of the graphical object with respect to the one or more other graphical objects in the graphical user interface.

7. The method of claim 1,

wherein the one or more adjacent graphical objects comprise an adjacent graphical object, and

wherein determining the graphical object from the one or more candidate graphical objects comprises:

determining, by the computing device, a type of the adjacent graphical object;

determining, by the computing device, a location of the region of the graphical user interface relative to the adjacent graphical object; and

determining, by the computing device, based at least in part on the type of the adjacent graphical object and on the location of the region of the graphical user interface relative to the adjacent graphical object, a type of the graphical object.

8. The method of claim 1, wherein determining the graphical object from the one or more candidate graphical objects comprising determining, by the computing device, one or more of an insertion location of the graphical object relative to the one or more adjacent graphical objects or an alignment of the graphical object relative to the one or more adjacent graphical objects.

9. The method of claim 1, wherein the development specification identifies the graphical object based at least in part on the region of the graphical user interface and the one or more adjacent graphical objects.

10. The method of claim 1, wherein outputting the indication of the graphical object comprises outputting, by the computing device and for display at the region of the graphical user interface, the graphical object.

11. The method of claim 1, wherein the one or more candidate graphical objects each comprise one of a graphical button, a graphical checkbox, a label, an input field, or a date selector.

12. A computing device, comprising:

at least one processor; and

a computer-readable storage device storing instructions that, when executed by the at least one processor, cause the at least one processor to;

output, for display in a graphical development environment that is used during development of a software application, a graphical user interface that is associated with the software application;

receive a development specification that associates a region of the graphical user interface with one or more candidate graphical objects for insertion into the graphical user interface;

receive an indication of a selection of the region of the graphical user interface;

determine one or more adjacent graphical objects that are located within a threshold distance of the region of the graphical user interface;

determine a graphical object from the one or more candidate graphical objects, wherein determining the graphical object is based at least in part on (i) the development specification, (ii) the indication of the selection of the region of the graphical user interface, and (iii) the one or more adjacent graphical objects that are located within the threshold distance of the region; and

output, for display at the region of the graphical user interface, an indication of the graphical object.

13. The computing device of claim 12,

wherein the one or more candidate graphical objects comprises a plurality of candidate graphical objects,

wherein the development specification associates the region of the graphical user interface with the plurality of candidate graphical objects, and

wherein the instructions that cause the at least one processor to determine the graphical object from the one or more candidate graphical objects comprise instructions that cause the at least one processor to:

output, for display, a plurality of options, wherein each option of the plurality of options represents a respective candidate graphical object of the plurality of candidate graphical objects; and

receive an indication of a selection an option from the plurality of options, wherein the selected option represents the graphical object.

14. The computing device of claim 13, wherein the instructions that cause the at least one processor to determine the graphical object from the one or more candidate graphical objects further comprise instructions that cause the at least one processor to:

determine a plurality of confidence values, wherein each confidence value of the plurality of confidence values is associated with a respective candidate graphical object of the plurality of candidate graphical objects; and

sort, based on the plurality of confidence values, the plurality of options to create a sorted order of the plurality of options,

wherein the instructions that cause the at least one processor to output the plurality of options comprise instructions that cause the at least one processor to output, for display, the plurality of options in the sorted order.

15. The computing device of claim 12,

wherein the one or more adjacent graphical objects comprise an adjacent graphical object, and

wherein the instructions that cause the at least one processor to determine the graphical object from the one or more candidate graphical objects comprise instructions that cause the at least one processor to:

determine a type of the adjacent graphical object;

determine a location of the region of the graphical user interface relative to the adjacent graphical object; and

determine, based at least in part on the type of the adjacent graphical object and on the location of the region of the graphical user interface relative to the adjacent graphical object, a type of the graphical object.

16. A computer-readable storage device storing instructions that, when executed by at least one processor of a computing device, cause the at least one processor to perform operations comprising:

outputting, for display in a graphical development environment that is used during development of a software application, a graphical user interface that is associated with the software application;

receiving a development specification that associates a region of the graphical user interface with one or more candidate graphical objects for insertion into the graphical user interface;

receiving an indication of a selection of the region of the graphical user interface;

determining one or more adjacent graphical objects that are located within a threshold distance of the region of the graphical user interface;

determining a graphical object from the one or more candidate graphical objects, wherein determining the graphical object is based at least in part on (i) the development specification, (ii) the indication of the selection of the region of the graphical user interface, and (iii) the one or more adjacent graphical objects that are located within the threshold distance of the region; and

outputting, for display at the region of the graphical user interface, an indication of the graphical object.

17. The computer-readable storage device of claim 16,

wherein the one or more candidate graphical objects comprises a plurality of candidate graphical objects,

wherein the development specification associates the region of the graphical user interface with the plurality of candidate graphical objects, and

wherein determining the graphical object from the one or more candidate graphical objects comprises:

outputting, for display, a plurality of options, wherein each option of the plurality of options represents a respective candidate graphical object of the plurality of candidate graphical objects; and

receiving an indication of a selection an option from the plurality of options, wherein the selected option represents the graphical object.

18. The computer-readable storage device of claim 17, wherein determining the graphical object from the one or more candidate graphical objects further comprises:

determining a plurality of confidence values, wherein each confidence value of the plurality of confidence values is associated with a respective candidate graphical object of the plurality of candidate graphical objects; and

sorting, based on the plurality of confidence values, the plurality of options to create a sorted order of the plurality of options,

wherein outputting the plurality of options comprises outputting, for display, the plurality of options in the sorted order.

19. The computer-readable storage device of claim 16,

wherein the one or more adjacent graphical objects comprise an adjacent graphical object, and

wherein determining the graphical object from the one or more candidate graphical objects comprises:

determining a type of the adjacent graphical object;

determining a location of the region of the graphical user interface relative to the adjacent graphical object; and

determining, based at least in part on the type of the adjacent graphical object and on the location of the region of the graphical user interface relative to the adjacent graphical object, a type of the graphical object.