Providing overlays based on text in a live camera view

• US 9,165,406 B1
• Filed: 09/21/2012
• Issued: 10/20/2015
• Est. Priority Date: 09/21/2012
• Status: Active Grant

• Current Assignee: A9.com Incorporated (Amazon.com, Inc.)
• Original Assignee: A9.com Incorporated (Amazon.com, Inc.)
• Inventors: Gray, Douglas R., Dhua, Arnab S., Lou, Yu, Ramesh, Sunil
• Primary Examiner(s): Yang, Ryan R
• Application Number:
  US13/624,647
• Time in Patent Office: 1,124 Days
• Field of Search: 345/629
• US Class Current: 1/1
• CPC Class Codes: G06F 16/00 Information retrieval; Database structures therefor; File system structures therefor
• CPC Class Codes: G06F 16/5846 using extracted text
• CPC Class Codes: G06F 16/7844 using original textual content or text extracted from visual content or transcript of audio data
• CPC Class Codes: G06F 16/951 Indexing; Web crawling techniques
• CPC Class Codes: G06T 11/00 2D [Two Dimensional] image generation
• CPC Class Codes: G06T 11/60 Editing figures and text; Combining figures or text
• CPC Class Codes: G06V 20/20 in augmented reality scenes
• CPC Class Codes: G06V 20/63 Scene text, e.g. street names
• CPC Class Codes: G06V 30/413 Classification of content, e.g. text, photographs or tables
• CPC Class Codes: H04N 7/183 for receiving images from a single remote source

Abstract

Approaches are described for rendering augmented reality overlays on an interface displaying the active field of view of a camera. The interface can display to a user an image or video, for example, and the overlay can be rendered over, near, or otherwise positioned with respect to any text or other such elements represented in the image. The overlay can have associated therewith at least one function or information, and when an input associated with the overlay is selected, the function can be performed (or caused to be performed) by the portable computing device.

Claims

1. A computer implemented method for providing an overlay in a live camera view, the method comprising:

under control of one or more computer systems configured with executable instructions,

displaying an image on a display element of a computing device, the image being captured by a camera of the computing device;

analyzing the image to locate a region of text in the image;

analyzing the region of text using a character recognition process to determine textual content for the region of text and a text entity type for the region of text;

determining an overlay template associated with the text entity type;

determining functionality associated with the text entity type; and

displaying on the display element, over a current view being captured by the camera, at least a portion of the textual content using the overlay template as an overlay element, the overlay element being rendered with respect to a location of the text recognized, the overlay element including at least one user-selectable element enabling the functionality associated with the text entity type to be performed with respect to the textual content.

2. The computer implemented method of claim 1, further comprising:

detecting an interaction with the at least one user-selectable element of the overlay element; and

performing the functionality associated with the text entity type.

3. The computer implemented method of claim 1, further comprising:

determining an application associated with the text entity type;

associating the application with the at least one user-selectable element of the overlay;

instantiating an instance of the application when the at least one user-selectable element is selected; and

automatically providing the textual content corresponding to the text entity type to the application for use by the application.

4. A computer implemented method, comprising:

under the control of one or more computer systems configured with executable instructions,

recognizing text in an image captured by at least one camera of a computing device, the image being rendered on a display element of the computing device;

determining a text entity type based on the recognized text;

determining an overlay for the text entity type;

determining functionality associated with the text entity type; and

rendering the overlay on the display element over a current view being captured by the camera, wherein detecting an interaction with the overlay causes the associated function to be performed using the recognized text.

5. The computer implemented method of claim 4, wherein the recognizing of text further includes:

analyzing the image to locate a region of text in the image;

cropping the region of the image containing the text to create a region of cropped text;

binarizing the region of cropped text; and

communicating the binarized region of cropped text to a character recognition process for processing.

6. The computer implemented method of claim 5, wherein processing by the character recognition process includes:

decoding the binarized region of cropped text to determine a text entity type for the region of cropped text;

determining the overlay and functionality associated with the text entity type; and

communicating the overlay to the display element of the computing device.

7. The computer implemented method of claim 4, further comprising:

prompting a user to select between a plurality of functions associated with the overlay, the overlay including at least one user-selectable element enabling the functionality associated with the text entity type;

receiving a selection at the at least one user-selectable element indicating one of the plurality of functions associated with the overlay; and

performing the selected function.

8. The computer implemented method of claim 4, further comprising:

detecting an interaction with a user-selectable element of the overlay;

prompting a user to confirm performance of the function associated with the overlay before performance; and

performing the function in response to receiving confirmation.

9. The computer implemented method of claim 4, further comprising:

identifying the text entity type associated with the text based at least in part on an output from an optical character recognition algorithm.

10. The computer implemented method of claim 4, further comprising:

determining an application associated with the text entity type;

associating the application with the overlay;

instantiating an instance of the application when the overlay is selected; and

automatically providing the text corresponding to the text entity type to the application for use by the application.

11. A computing system, comprising:

at least one processor; and

memory including instructions that, when executed by the processor, cause the computing system to;

recognize text in an image captured by at least one camera of a computing device, the image rendered on a display element of the computing device;

determine a text entity type based on the text;

determine an overlay for the text entity type;

determine functionality associated with the text entity type; and

render the overlay on the display element over a current view being captured by the camera, wherein detecting an interaction with the overlay causes an associated function to be performed using the recognized text.

12. The computing system of claim 11, wherein recognizing the text in the image and rendering the overlay are performed locally on the computing device.

13. The computing system of claim 11, wherein when recognizing the text, the at least one processor is further adapted to:

analyze the image to locate a region of text in the image;

crop the region of the image containing the text to create a region of cropped text;

binarize the region of cropped text; and

communicate the binarized region of cropped text to a character recognition process for processing.

14. The computing system of claim 13, wherein the character recognition process is adapted to:

decode the binarized region of cropped text to determine a text entity type for the region of cropped text;

determine an overlay and functionality associated with the text entity type; and

communicate the overlay to the display element of the computing device.

15. The computing system of claim 14, wherein decoding the binarized region of cropped text and associating the text entity type with the overlay is performed by a remote server in communication with the computing device.

16. The computing system of claim 11, wherein when the display element detects an interaction with the overlay, the computing device is adapted to perform the function associated with the text entity type.

17. The computing system of claim 11, wherein the processor is adapted to:

determine an application associated with the text entity type;

associate the application with the overlay;

instantiate an instance of the application when the overlay is selected; and

automatically provide the text corresponding to the text entity type to the application for use by the application.

18. The computing system of claim 11, wherein the display element prompts a user to confirm performance of the function associated with the overlay, and wherein the computing device performs the function in response to receiving confirmation.

19. A non-transitory computer readable storage medium storing one or more sequences of instructions executable by one or more processors to perform a set of operations comprising:

displaying an image on a display element of a computing device;

analyzing the image to locate a region of text in the image;

analyzing the region of text to determine a text entity type for the region of text;

determining an overlay associated with the text entity type;

determining functionality associated with the text entity type; and

displaying the overlay on the display element over a current view being captured by the computing device, the overlay including at least one user-selectable element enabling the functionality associated with the text entity type to be performed with respect to the text.

20. The non-transitory computer readable storage medium of claim 19, further comprising instructions executed by the one or more processors to perform the operations of:

detecting an interaction with the user-selectable element of the overlay; and

performing the functionality associated with the text entity type.

21. The non-transitory computer readable storage medium of claim 19, further comprising instructions executed by the one or more processors to perform the operations of:

determining an application associated with the text entity type;

associating the application with the at least one user-selectable element of the overlay;

instantiating an instance of the application when the at least one user-selectable element is selected; and

automatically providing the text corresponding to the text entity type to the application for use by the application.

22. The non-transitory computer readable storage medium of claim 19, further comprising instructions executed by the one or more processors to perform the operations of:

cropping the region of the image containing the text to create a region of cropped text;

binarizing the region of cropped text; and

communicating the binarized region of cropped text to a character recognition process to determine the text entity type for the region of text.

23. The non-transitory computer readable storage medium of claim 19, further comprising instructions executed by the one or more processors to perform the operations of:

detecting an interaction with the user-selectable element of the overlay; and

prompting a user to confirm performance of the function associated with the overlay, and wherein the computing device performs the function in response to receiving confirmation.

24. The non-transitory computer readable storage medium of claim 19, wherein the text entity type is one of a phone number, an email address, a website URL, a physical address, a business card, a product model number, a book spine, a credit card, a word in a foreign language, a price in a foreign currency, a barcode, a nutrition label and a stock market ticker.