INDEX, SEARCH, AND RETRIEVAL OF USER-INTERFACE CONTENT

• US 20180275751A1
• Filed: 03/21/2017
• Published: 09/27/2018
• Est. Priority Date: 03/21/2017
• Status: Abandoned Application

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Wilson, Andrew D., Mauderer, Michael
• Application Number:
  US15/465,341
• Publication Number:
  US 20180275751A1
• Time in Patent Office: Days
• CPC Class Codes: G06F 16/51 Indexing; Data structures therefor; Storage structures
• CPC Class Codes: G06F 16/5838 using colour
• CPC Class Codes: G06F 16/5846 using extracted text
• CPC Class Codes: G06F 2203/04803 Split screen, i.e. subdividing the display area or the window area into separate subareas
• CPC Class Codes: G06F 3/013 Eye tracking input arrangements G06F3/015 takes precedence
• CPC Class Codes: G06F 3/04842 Selection of displayed objects or displayed text elements G06F3/0482 takes precedence

Abstract

Systems and methods are disclosed for the index, search, and retrieval of user interface content. In one implementation, an image of a user interface as presented to a user via a display device can be captured. The image can be processed to identify a content element depicted within the image. The content element can be associated with the image. The image as associated with the content element can be stored in relation to the user.

Claims

1. A system comprising:

a processing device; and

a memory coupled to the processing device and storing instructions that, when executed by the processing device, cause the system to perform operations comprising;

capturing an image of a user interface as presented to a user via a display device;

processing the image to identify a content element depicted within the image;

associating the content element with the image; and

storing, in relation to the user, the image as associated with the content element.

2. The system of claim 1, wherein the memory further stores instructions for causing the system to perform operations comprising receiving an inputs from a tracking component, the input indicating that one or more eyes of the user are directed to a first region of the user interface.

3. The system of claim 2, wherein capturing the image comprises capturing an image of the first region of the user interface.

4. The system of claim 2, wherein processing the image comprises processing the image to identify a content element depicted within the first region of the user interface.

5. The system of claim 2, wherein processing the image comprises processing the image with respect to the input received from the tracking component to determine a chronological interval during which the one or more eyes of the user are directed to the content element.

6. The system of claim 5, wherein the memory further stores instructions for causing the system to perform operations comprising assigning a weight to the content element based on the chronological interval.

7. The system of claim 1, wherein processing the image comprises processing the image using optical character recognition (OCR) to identify one or more alphanumeric characters depicted within the o image.

8. The system of claim 1, wherein associating the content element comprises associating a content location of the content element with the image.

9. The system of claim 1, wherein associating the content element comprises associating, with the image, an application within which the content element is presented.

10. The system of claim 1, wherein the memory further stores instructions for causing the system to perform operations comprising generating a content index based on the content element.

11. The system of claim 10, wherein the memory further stores instructions for causing the system to perform operations comprising:

receiving a search query from the user;

processing the content index to identify one or more content elements that correspond to the search query;

retrieving at least one image that is associated with at least one of the one or more content elements that correspond to the search query; and

presenting the at least one image to the user via the display device.

12. The system of claim 11, wherein receiving a search query comprises generating a search query based on a selection of a region of the user interface.

13. The system of claim 11, wherein presenting the at least one image comprises presenting the at least one image in conjunction with the one or more content elements that correspond to the search query.

14. A method comprising:

receiving an input from a tracking component, the input indicating that one or more eyes of a user are directed to a first region of a user interface presented to the user via a display device;

capturing an image of the first region of the user interface;

processing the image to identify a content element depicted within the first region of the user interface;

associating the content element with the image; and

storing, in relation to the user, the image as associated with the content element.

15. The method of claim 14, wherein processing the image comprises processing the image with respect to the input received from the tracking component to determine a chronological interval during which the one or more eyes of the user are directed to the content element.

16. The method of claim 15, further comprising assigning a weight to the content element based on the chronological interval.

17. The method of claim 14, further comprising venerating a content index based on the content element.

18. The method of claim 17, further comprising:

receiving a search query from the user;

processing the content index to identify one or more content elements that correspond to the search query;

retrieving at least one image that is associated with at least one of the one or more content elements that correspond to the search query; and

presenting the at least one image to the user via the display device.

19. A non-transitory computer readable medium having instructions stored thereon that, when executed by a processing device, cause the processing device to perform operations comprising:

receiving an input from a tracking component, the input indicating that one or more eyes of a user are directed to a first region of a user interface presented to a user via a display device;

capturing an image of the first region of the user interface;

processing the image to identify a content element depicted within the first region of the user interface;

associating the content element with the image;

storing, in relation to the user, the image as associated with the content element;

generating a content index based on the content element; and

in response to receipt of a search query,

identifying one or more content elements that correspond to the search query, and

presenting at least one image that is associated with at least one the one or more content elements that correspond to the search query.

20. The computer-readable medium of claim 19, wherein the search query comprises a search query generated based on a selection of one or more regions of the user interface.