Contextual overlay for documents

• US 10,621,237 B1
• Filed: 08/01/2016
• Issued: 04/14/2020
• Est. Priority Date: 08/01/2016
• Status: Active Grant

• Current Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Original Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Inventors: Rathnavelu, Kadirvelu, Barbitta, Zach
• Primary Examiner(s): Dang, Thanh-Ha
• Application Number:
  US15/225,529
• Time in Patent Office: 1,352 Days
• Field of Search: 707722
• CPC Class Codes: G06F 16/313 Selection or weighting of terms for indexing
• CPC Class Codes: G06F 16/316 Indexing structures
• CPC Class Codes: G06F 16/332 Query formulation
• CPC Class Codes: G06F 16/338 Presentation of query results
• CPC Class Codes: G06F 16/93 Document management systems
• CPC Class Codes: G06F 16/951 Indexing; Web crawling techniques
• CPC Class Codes: G06F 40/106 Display of layout of documents; Previewing
• CPC Class Codes: G06F 40/169 Annotation, e.g. comment data or footnotes
• CPC Class Codes: G06F 40/216 using statistical methods
• CPC Class Codes: G06F 40/295 Named entity recognition
• CPC Class Codes: G06V 30/10 Character recognition
• CPC Class Codes: G06V 30/224 of printed characters having additional code marks or containing code marks
• CPC Class Codes: G06V 30/40 Document-oriented image-based pattern recognition

Abstract

Approaches provide for analyzing document data to provide contextual overlays. For example, an application executing on a computing device (or at least in communication with the computing device) can analyze document data to determine a set of keywords based on features extracted from the document data. The keywords can be used to query an index of websites based on a relevance function in order to determine websites that are most relevant to the text identified from the document, at least some of which can be analyzed using a search engine to identify contextual information in the websites associated with the document. Thereafter, the contextual information can be provided for display with the document.

Claims

1. A computing device, comprising:

a computing device processor;

a memory device including instructions that, when executed by the computing device processor, cause the computing device to;

receive a document that includes content, the document stored on a computing resource and associated with a user account;

analyze the document to determine a plurality of relevant portions of the content;

display the document on an interface of a display screen of a computing device, the interface including a search field and the content of the document;

receive a query from the search field that includes a search term;

query an index that includes contextual information using the search term;

determine relevant contextual information associated with the search term based at least in part on the search term and at least one weight factor, wherein the at least one weight factor is based at least in part on a first relevance score determined based on the plurality of relevant portions of the document and a second relevance score determined based on the index;

display, on a first area on the interface, a portion of the content of the document associated with the search term; and

display with the portion of the content in a second area of the interface, the relevant contextual information associated with the search term.

2. The computing device of claim 1, wherein the instructions, when executed further enable the computing device to:

use optical character recognition (OCR) techniques on the document to determine a first set of keywords represented in the document;

use object recognition techniques on the document to determine a second set of keywords corresponding to characteristics of an object represented in the document;

use logo recognition techniques on the document to determine a third set of keywords corresponding to one of a logo or brand represented in document; and

use bar code recognition techniques on the document to identify a bar code, wherein the plurality of relevant portions is based at least in part on the first set of keywords, the second set of keywords, the third set of keywords, and the bar code.

3. The computing device of claim 2, wherein the instructions, when executed further enable the computing device to:

analyze a plurality of websites to determine respective keywords associated with each of the plurality of websites;

index information associated with the plurality of web sites and respective keywords;

associate the first set of keywords, the second set of keywords, the third set of keywords, and the barcode with the information associated with the plurality of websites and respective keywords to the relevant contextual information.

4. A computing system, comprising:

a computing device processor;

a memory device including instructions that, when executed by the computing device processor, cause the computing system to;

obtain a document stored on a computing resource and associated with a user account;

analyze the document to identify a relevant portion of the document;

determine contextual information by querying an index using at least some of the relevant portion of the document based at least in part on at least one weight factor, wherein the at least one weight factor is based at least in part on a first relevance score determined based on the relevant portion of the document and a second relevance score determined based on the index;

receive a user input pertaining to the relevant portion of the document;

responsive to the user input, display, on a first area of an interface that includes at least a portion of the document; and

display, on a second area of the interface, one of a graphical element that indicates contextual information associated with the document or contextual information associated with the document.

5. The computing system of claim 4, wherein the document includes at least one of a user manual, a product manual, a scholarly paper, a product listing, a financial report, a receipt, an instruction manual, or a digital book.

6. The computing system of claim 4, wherein the instructions, when executed further enable the computing system to:

use optical character recognition (OCR) techniques and object recognition techniques to determine a set of keywords associated with the document; and

associate the set of keywords with the contextual information stored in the database.

7. The computing system of claim 4, wherein the user input includes a search query, and wherein the instructions, when executed further enable the computing system to:

display, on the first area on the interface, a portion of the document associated with a search term of the search query; and

display with the portion of the document in the second area of the interface, contextual information associated with the search term.

8. The computing system of claim 4, wherein the user input includes scrolling the document from a first viewable area of the document to a second viewable area of the document, and wherein the instructions, when executed to provide information further enable the computing system to:

display updated contextual information based at least in part on the second viewable area of the document being displayed.

9. The computing system of claim 4, wherein the user input includes one of opening a document from an online document storage system, updating a display area of the document, or searching content in the document.

10. The computing system of claim 4, wherein the instructions, when executed further enable the computing system to:

emphasize relevant portions of the document associated with contextual information, wherein emphasizing the relevant portions of the document includes applying a text effect to the relevant portions of the document or displaying a graphical element proximate to the relevant portions of the document.

11. The computing system of claim 4, wherein the instructions, when executed further enable the computing system to:

provide a display of a mapping between the relevant portion of the document and respective contextual information.

12. The computing system of claim 4, wherein the instructions, when executed to provide information further enable the computing system to:

display the contextual information in one of an overlay on the document, a side panel proximate the document, a pop-up window, or in a web browser within the interface.

13. The computing system of claim 4, wherein the instructions, when executed further enable the computing system to:

analyze a plurality of web pages to determine contextual information associated with user uploaded documents; and

index the contextual information, wherein the web pages includes social media pages, electronic message boards, or electronic user generated content.

14. The computing system of claim 4, wherein the instructions, when executed further enable the computing system to:

enable controls to control a type of contextual information displayed and controls to customize a view of the contextual information, wherein customizing the view of the contextual information includes one of changing a layout of the contextual information, minimizing or maximizing a view of the contextual information, or stacking the contextual information.

15. The computing system of claim 4, wherein the instructions, when executed further enable the computing system to:

enable a search engine with the document, wherein the search engine is operable to receive a search query for contextual information associated with the document.

16. A method, comprising:

obtaining a document stored on a computing resource and associated with a user account;

analyzing the document to identify a relevant portion of the document;

determining contextual information by querying an index using at least some of the relevant portion of the document based at least in part on at least one weight factor, wherein the at least one weight factor is based at least in part on a first relevance score determined based on the relevant portion of the document and a second relevance score determined based on the index;

receiving a user input pertaining to the relevant portion of the document;

responsive to the user input, displaying, on a first area of an interface that includes at least a portion of the document; and

displaying, on a second area of the interface, the contextual information.

17. The method of claim 16, further including:

using optical character recognition (OCR) techniques and object recognition techniques to determine a set of keywords associated with the document; and

associating the set of keywords with the contextual information stored in the database.

18. The method of claim 16, wherein the user input includes a search query, the method further including:

displaying, on the first area on the interface, a portion of the document associated with a search term of a search query; and

displaying with the portion of the document in the second area of the interface, contextual information associated with the search term.

19. The method of claim 16, further comprising:

emphasizing relevant portions of the document associated with contextual information, wherein emphasizing the relevant portions of the document includes applying a text effect to the relevant portions of the document or displaying a graphical element proximate to the relevant portions of the document.

20. The method of claim 16, further comprising:

determine a change in a viewing area of the interface; and

update a display of contextual information based on the change in the viewing area.