Automatic identification of types of user interface components

• US 10,761,818 B2
• Filed: 10/31/2018
• Issued: 09/01/2020
• Est. Priority Date: 10/31/2018
• Status: Active Grant

• Current Assignee: Salesforce.com, Inc.
• Original Assignee: Salesforce.com, Inc.
• Inventors: Hou, Kevin C.
• Primary Examiner(s): Lyons, Andrew M.
• Application Number:
  US16/177,050
• Publication Number:
  US 20200133643A1
• Time in Patent Office: 671 Days
• Field of Search: 717107
• CPC Class Codes: G06F 16/2264 Multidimensional index structures
• CPC Class Codes: G06F 18/22 Matching criteria, e.g. proximity measures
• CPC Class Codes: G06F 18/23211 with adaptive number of clusters
• CPC Class Codes: G06F 8/38 for implementing user interfaces
• CPC Class Codes: G06F 9/451 Execution arrangements for user interfaces

Abstract

Techniques are disclosed relating to identifying types of user interface components based on one or more existing user interfaces. The disclosed techniques may include, for example, determining a plurality of visible elements of a graphical user interface, based on user interface code. Additionally, techniques include determining coordinates for bounding boxes for ones of the plurality of visible elements, based on the user interface code. Disclosed techniques may also include grouping the visible elements into at least first and second groups and determining types of elements within the first and second groups. The techniques also include, in response to detecting a match between the types of elements within the first and second groups, determining a similarity metric for the first and second groups based on the coordinates of determined bounding boxes within the first and second groups. In response to the similarity metric meeting a threshold value, the techniques include storing information defining a component type that corresponds to the first and second groups.

Claims

1. A method, comprising:

determining, based on user interface code, a plurality of visible elements of a graphical user interface;

determining, based on the user interface code, coordinates for bounding boxes for ones of the plurality of visible elements;

grouping the visible elements into at least first and second groups;

determining types of elements within the first and second groups;

in response to detecting a match between the types of elements within the first and second groups, determining a similarity metric for the first and second groups based on the coordinates of determined bounding boxes within the first and second groups, wherein detecting the match includes;

determining whether a number of elements of each element type matches to within a threshold similarity for the first and second groups of elements; and

determining whether an element displayed at a position within the first group of elements is the same element type as an element displayed at a similar position within the second group of elements; and

storing information defining a component type that corresponds to the first and second groups in response to the similarity metric meeting a threshold value.

2. The method of claim 1, wherein the similarity metric is determined based on respective pixel representations for the first and second groups.

3. The method of claim 2, wherein the pixel representations are based on the determined bounding boxes for ones of the plurality of visible elements.

4. The method of claim 2, wherein the pixel representations are based on ones of the plurality of visible elements of the graphical user interface.

5. The method of claim 2, wherein the pixel representations for the first and second groups are represented using respective matrix data structures;

and

further comprising;

flattening the matrix data structures for the first and second groups.

6. The method of claim 5, further comprising:

transposing the flattened matrix data structures for the first and second groups; and

wherein the determining the similarity metric includes;

determining a first dot product of the flattened matrix data structure for the first group and the transposed matrix data structure for the second group;

determining a second dot product of the flattened matrix data structure for the first group and the transposed matrix data structure for the first group; and

calculating a result for the similarity metric based on a ratio between the first dot product and the second dot product.

7. The method of claim 6, wherein the first dot product corresponds to a similarity between the first and second groups of elements, and wherein the second dot product corresponds to a value associated with an exact match.

8. A non-transitory computer-readable medium having instructions stored thereon that are capable of execution by a computing device to perform operations comprising:

determining, based on user interface code, a plurality of visible elements of a graphical user interface;

determining, based on the user interface code, coordinates for bounding boxes for ones of the plurality of visible elements;

grouping the visible elements into at least first and second groups;

determining types of elements within the first and second groups;

in response to detecting a match between the types of elements within the first and second groups, determining a similarity metric for the first and second groups based on the coordinates of determined bounding boxes within the first and second groups, wherein detecting the match includes;

determining whether a number of elements of each element type matches to within a threshold similarity for the first and second groups of elements; and

determining whether an element displayed at a position within the first group of elements is the same element type as an element displayed at a similar position within the second group of elements; and

storing information defining a component type that corresponds to the first and second groups in response to the similarity metric meeting a threshold value.

9. The non-transitory computer-readable medium of claim 8, wherein the similarity metric is determined based on respective pixel representations for the first and second groups.

10. The non-transitory computer-readable medium of claim 9, wherein the pixel representations are based on the determined bounding boxes for ones of the plurality of visible elements.

11. The non-transitory computer-readable medium of claim 9, wherein the pixel representations are based on ones of the plurality of visible elements of the graphical user interface.

12. The non-transitory computer-readable medium of claim 9, wherein the pixel representations for the first and second groups are represented using respective matrix data structures;

wherein the operations further comprise;

flattening the matrix data structures for the first and second groups.

13. The non-transitory computer-readable medium of claim 12, wherein the operations further comprise:

transposing the flattened matrix data structures for the first and second groups; and

wherein the determining the similarity metric includes;

determining a first dot product of the flattened matrix data structure for the first group and the transposed matrix data structure for the second group;

determining a second dot product of the flattened matrix data structure for the first group and the transposed matrix data structure for the first group; and

calculating a result for the similarity metric based on a ratio between the first dot product and the second dot product.

14. An apparatus comprising:

one or more processing elements configured to;

determine, based on user interface code, a plurality of visible elements of a graphical user interface;

determine, based on the user interface code, coordinates for bounding boxes for ones of the plurality of visible elements;

group the visible elements into at least first and second groups;

determine types of elements within the first and second groups;

in response to detecting a match between the types of elements within the first and second groups, determine a similarity metric for the first and second groups based on the coordinates of determined bounding boxes within the first and second groups, wherein detecting the match includes;

determining whether a number of elements of each element type matches to within a threshold similarity for the first and second groups of elements; and

determining whether an element displayed at a position within the first group of elements is the same element type as an element displayed at a similar position within the second group of elements; and

store information defining a component type that corresponds to the first and second groups in response to the similarity metric meeting a threshold value.

15. The apparatus of claim 14, wherein the similarity metric is determined based on respective pixel representations for the first and second groups.

16. The apparatus of claim 15, wherein the pixel representations are based on the determined bounding boxes for ones of the plurality of visible elements.

17. The apparatus of claim 15, wherein the pixel representations are based on ones of the plurality of visible elements of the graphical user interface.

18. The apparatus of claim 15, wherein the pixel representations for the first and second groups are represented using respective matrix data structures;

wherein the apparatus is further configured to;

flatten the matrix data structures for the first and second groups.

19. The apparatus of claim 18, wherein the apparatus is further configured to:

transpose the flattened matrix data structures for the first and second groups; and

wherein, to determine the similarity metric, the apparatus is configured to;

determine a first dot product of the flattened matrix data structure for the first group and the transposed matrix data structure for the second group;

determine a second dot product of the flattened matrix data structure for the first group and the transposed matrix data structure for the first group; and

calculate a result for the similarity metric based on a ratio between the first dot product and the second dot product.