Providing overlays based on text in a live camera view

• US 9,922,431 B2
• Filed: 09/10/2015
• Issued: 03/20/2018
• Est. Priority Date: 09/21/2012
• Status: Active Grant

• Current Assignee: A9.com Incorporated (Amazon.com, Inc.)
• Original Assignee: A9.com Incorporated (Amazon.com, Inc.)
• Inventors: Gray, Douglas R., Dhua, Arnab S., Lou, Yu, Ramesh, Sunil
• Primary Examiner(s): Yang, Ryan R
• Application Number:
  US14/850,813
• Publication Number:
  US 20160005189A1
• Time in Patent Office: 922 Days
• Field of Search: 345629
• CPC Class Codes: G06F 16/00 Information retrieval; Database structures therefor; File system structures therefor
• CPC Class Codes: G06F 16/5846 using extracted text
• CPC Class Codes: G06F 16/7844 using original textual content or text extracted from visual content or transcript of audio data
• CPC Class Codes: G06F 16/951 Indexing; Web crawling techniques
• CPC Class Codes: G06T 11/00 2D [Two Dimensional] image generation
• CPC Class Codes: G06T 11/60 Editing figures and text; Combining figures or text
• CPC Class Codes: G06V 20/20 in augmented reality scenes
• CPC Class Codes: G06V 20/63 Scene text, e.g. street names
• CPC Class Codes: G06V 30/413 Classification of content, e.g. text, photographs or tables
• CPC Class Codes: H04N 7/183 for receiving images from a single remote source

Abstract

Approaches are described for rendering augmented reality overlays on an interface displaying the active field of view of a camera. The interface can display to a user an image or video, for example, and the overlay can be rendered over, near, or otherwise positioned with respect to any text or other such elements represented in the image. The overlay can have associated therewith at least one function or information, and when an input associated with the overlay is selected, the function can be performed (or caused to be performed) by the portable computing device.

Claims

1. A computing system, comprising:

a display element;

a camera;

at least one processor; and

memory including instructions that, when executed by the processor, cause the computing system to;

capture image data by the camera, the image data including a representation of text;

analyze the image data to locate the representation of the text;

analyze the representation of the text to determine an action associated with the text;

display, on the display element, over a view being captured by the camera, a graphical representation of the text, the graphical representation of the text being rendered with respect to a location of the representation of the text and appearing overlaid over the representation of the text; and

perform, in response to a selection of the graphical representation of the text, the action associated with the text.

2. The computing system of claim 1, wherein the instructions when executed by the processor further enable the computing system to:

detect an interaction with the graphical representation of the text; and

perform the action associated with the text.

3. The computing system of claim 1, wherein the instructions when executed by the processor further enable the computing system to:

determine an application associated with the action;

detect an interaction with the graphical representation of the text;

instantiate an instance of the application; and

perform the action by the instance of the application using the representation of the text.

4. The computing system of claim 1, wherein the instructions when executed by the processor further enable the computing system to:

analyze the image data to determine a region that includes the representation of the text;

crop the region to generate a region of cropped text;

binarize the region of cropped text to generate a binarized region of text; and

provide the binarized region of text to a character recognition process for processing.

5. The computing system of claim 4, wherein the instructions when executed by the processor further enable the computing system to:

decode the binarized region of text to identify the text; and

determine the action using the decoded binarized region of text.

6. The computing system of claim 1, wherein the instructions when executed by the processor further enable the computing system to:

detect a selection of the graphical representation of the text;

provide an alert to confirm performance of the action;

receive confirmation; and

perform the action.

7. The computing system of claim 4, wherein the representation of the text corresponds to one of a phone number, an email address, a website URL, a physical address, a business card, a product model number, a book spine, a credit card, a word in a foreign language, a price in a foreign currency, a barcode, a nutrition label and a stock market ticker.

8. The computing system of claim 4, wherein the action correspond to calling a phone number, instantiating an instance of an email client, instantiating an instance of a web browser, instantiating an instance of a navigation application, or instantiating an instance of a shopping application.

9. The computing system of claim 4, wherein the location corresponds to one of directly above the representation of the text or up, down, right, or left of the representation of the text.

10. The computing system of claim 4, wherein appearing more prominent includes causing the graphical representation of the text to appear more bold than the representation of the text, causing a size of the graphical representation of the text to appear larger than the representation of the text, causing a color to the graphical representation of the text to be different from the color of the representation of the text, or causing the graphical representation of the text to move in a path within a predetermined area about the representation of the text.

11. A method, comprising:

capturing image data by a camera of a computing device, the image data including a representation of text;

analyzing the image data to locate the representation of the text;

analyzing the representation of the text to determine an action associated with the text;

displaying, on a display element of the computing device, over a view being captured by the camera, a graphical representation of the text, the graphical representation of the text being rendered with respect to a location of the representation of the text and appearing overlaid over the representation of the text; and

performing, in response to a selection of the graphical representation of the text, the action associated with the text.

12. The method of claim 11, wherein the method is performed locally on the computing device.

13. The method of claim 11, further comprising:

detecting an interaction with the graphical representation of the text; and

performing the action associated with the text.

14. The method of claim 11, further comprising:

determining an application associated with the action;

detecting an interaction with the graphical representation of the text;

instantiating an instance of the application; and

performing the action by the instance of the application using the representation of the text.

15. The method of claim 11, further comprising:

analyzing the image data to determine a region that includes the representation of the text;

cropping the region to generate a region of cropped text;

binarizing the region of cropped text to generate a binarized region of text;

decoding the binarized region of text to identify the representation of the text; and

determining the action using the decoded binarized region of text.

16. The method of claim 11, further comprising:

detecting a selection of the graphical representation of the text;

providing an alert to confirm performance of the action;

receiving confirmation; and

performing the action.

17. A non-transitory computer readable storage medium storing one or more sequences of instructions executable by one or more processors to perform a set of operations comprising:

capturing image data by a camera of a computing device, the image data including a representation of text;

analyzing the image data to locate the representation of the text;

analyzing the representation of the text to determine an action associated with the text;

displaying, on a display element of the computing device, over a view being captured by the camera, a graphical representation of the text, the graphical representation of the text being rendered with respect to a location of the representation of the text and appearing overlaid over the representation of the text; and

performing, in response to a selection of the graphical representation of the text, the action associated with the text.

18. The non-transitory computer readable storage medium of claim 17, further comprising instructions executed by the one or more processors to perform the set of operations of:

determining an application associated with the action;

detecting an interaction with the graphical representation of the text;

instantiating an instance of the application; and

performing the action by the instance of the application using the representation of the text.

19. The non-transitory computer readable storage medium of claim 17, further comprising instructions executed by the one or more processors to perform the set of operations of:

analyzing the image data to determine a region that includes the representation of the text;

cropping the region to generate a region of cropped text;

binarizing the region of cropped text to generate a binarized region of text;

decoding the binarized region of text to identify the text; and

determining the action using the decoded binarized region of text.

20. The non-transitory computer readable storage medium of claim 17, further comprising instructions executed by the one or more processors to perform the set of operations of:

detecting a selection of the graphical representation of the text;

providing an alert to confirm performance of the action;

receiving confirmation; and

performing the action.