Automatic Classification of User Interface Elements

• US 20200133644A1
• Filed: 10/31/2018
• Published: 04/30/2020
• Est. Priority Date: 10/31/2018
• Status: Active Grant

• Current Assignee: Salesforce.com, Inc.
• Original Assignee: Salesforce.com, Inc.
• Inventors: Hou, Kevin C.
• Granted Patent: US 10,949,174 B2
• Time in Patent Office: Days
• CPC Class Codes: G06F 40/14 Tree-structured documents parsing G06F40/205; validation G06F40/226
• CPC Class Codes: G06F 8/38 for implementing user interfaces

Abstract

Techniques are disclosed relating to classifying user interface elements of existing user interfaces. This may include for example, storing information specifying known metadata values for a plurality of metadata fields and indications of relationships between ones of the known metadata values and a plurality of types of visible user interface elements. The techniques also include determining respective metadata values for a plurality of visible elements of a graphical user interface, where the metadata values are included in user interface code that specifies the plurality of visible elements. The disclosed techniques also include, based on the stored indications of relationships and the determined metadata values, scoring ones of the plurality of visible elements to generate score values for each of the plurality of types of visible elements. Finally, the disclosed techniques include, based on the scoring, classifying the plurality of visible elements according to the plurality of types of visible elements and storing information specifying the classified elements.

Claims

1. A method comprising:

storing information specifying known metadata values for a plurality of metadata fields and predetermined indications of relationships between ones of the known metadata values and a plurality of types of visible user interface elements, wherein the relationships include a relationship between a first known metadata value and a text type, and a relationship between a second known metadata value and an image type;

determining respective metadata values for a plurality of visible elements of a graphical user interface, wherein the metadata values are included in user interface code that specifies the plurality of visible elements;

scoring ones of the plurality of visible elements, based on the stored predetermined indications of relationships and the determined metadata values, to generate score values for each of the plurality of types of visible elements;

classifying, based on the score values for each of the plurality of types of visible elements, the plurality of visible elements according to the plurality of types of visible elements, including classifying at least one element as a text type, and at least one element as an image type; and

storing information specifying the classified elements.

2. The method of claim 1, further comprising:

determining, based on the stored information specifying the classified elements, whether a set of element types for a first group of elements is the same as a set of element types for a second group of elements.

3. The method of claim 2, further comprising:

determining, based on the stored information specifying the classified elements, whether a number of elements of each element type is the same for the first group of elements and the second group of elements.

4. The method of claim 2, further comprising:

determining, based on the stored information specifying the classified elements, whether an element displayed at a first position within the first group of elements is the same element type as an element displayed within a threshold distance from the first position within the second group of elements.

5. The method of claim 4, further comprising:

determining whether the first group of elements and the second group of elements have at least a threshold similarity, based on the determining whether a set of element types for the first group of elements is the same as a set of element types for the second group of elements and the determining whether a number of elements of each element type is the same for the first group of elements and the second group of elements.

6. The method of claim 1, wherein the scoring is performed separately for different ones of the metadata fields.

7. The method of claim 1, wherein the scoring is performed separately for different ones of the plurality of types of user interface elements.

8. The method of claim 1, further comprising:

determining and storing a confidence value associated with classification of one of the classified elements, wherein the determining is based on scores for multiple different types of elements generated for the one of the classified elements.

9. The method of claim 8, wherein the confidence value is determined based on a ratio of the score for a classified type for the classified element to a sum of scores for all scored types for the classified element.

10. The method of claim 1, further comprising:

using the stored information as training data for one or more machine learning modules configured to automatically generate one or more user interfaces.

11. The method of claim 1, wherein the relationships further include a relationship between a third known metadata value and a link type.

12. The method of claim 1, wherein the metadata fields are specified for one or more metadata field types that include:

a field that specifies a placement of one or more elements in a user interface, a field that specifies one or more class names for an element, a field that specifies a direction of text content of an element, a field that indicates whether content of an element is editable, a field that indicates whether an element can be dragged, a field that specifies how dragged data behaves when it is dropped, or a field that specifies a language of the content of an element.

13. A non-transitory computer-readable medium having instructions stored thereon that are capable of execution by a computing device to perform operations comprising:

storing information specifying known metadata values for a plurality of metadata fields and predetermined indications of relationships between ones of the known metadata values and a plurality of types of visible user interface elements, wherein the relationships include a relationship between a first known metadata value and a text type, and a relationship between a second known metadata value and an image type;

determining respective metadata values for a plurality of visible elements of a graphical user interface, wherein the metadata values are included in user interface code that specifies the plurality of visible elements;

scoring ones of the plurality of visible elements, based on the stored predetermined indications of relationships and the determined metadata values, to generate score values for each of the plurality of types of visible elements;

classifying, based on the score values for each of the plurality of types of visible elements, the plurality of visible elements according to the plurality of types of visible elements, including classifying at least one element as a text type, and at least one element as an image type; and

storing information specifying the classified elements.

14. The non-transitory computer-readable medium of claim 13, wherein the operations further comprise:

determining, based on the stored information specifying the classified elements, whether a set of element types for a first group of elements is the same as a set of element types for a second group of elements.

15. The non-transitory computer-readable medium of claim 14, wherein the operations further comprise:

determining, based on the stored information specifying the classified elements, whether a number of elements of each element type is the same for the first group of elements and the second group of elements.

16. The non-transitory computer-readable medium of claim 13, wherein the operations further comprise:

determining and storing a confidence value associated with classification of one of the classified elements, wherein the determining is based on scores for multiple different types of elements generated for the one of the classified elements.

17. An apparatus comprising:

one or more processors configured to;

store information specifying known metadata values for a plurality of metadata fields and predetermined indications of relationships between ones of the known metadata values and a plurality of types of visible user interface elements, wherein the relationships include a relationship between a first known metadata value and a text type, and a relationship between a second known metadata value and an image type;

determine respective metadata values for a plurality of visible elements of a graphical user interface, wherein the metadata values are included in user interface code that specifies the plurality of visible elements;

score ones of the plurality of visible elements, based on the stored predetermined indications of relationships and the determined metadata values, to generate score values for each of the plurality of types of visible elements;

classify, based on the score values for each of the plurality of types of visible elements, the plurality of visible elements according to the plurality of types of visible elements, including classifying at least one element as a text type, and at least one element as an image type; and

store information specifying the classified elements.

18. The apparatus of claim 17, further configured to:

determine, based on the stored information specifying the classified elements, whether a set of element types for a first group of elements is the same as a set of element types for a second group of elements.

19. The apparatus of claim 18, further configured to:

determine, based on the stored information specifying the classified elements, whether a number of elements of each element type is the same for the first group of elements and the second group of elements.

20. The apparatus of claim 17, further configured to:

determine and store a confidence value associated with classification of one of the classified elements, wherein the determining is based on scores for multiple different types of elements generated for the one of the classified elements.