Presenting translations of text depicted in images

• US 9,547,644 B2
• Filed: 11/08/2013
• Issued: 01/17/2017
• Est. Priority Date: 11/08/2013
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google Inc. (Alphabet Inc.)
• Inventors: Cuthbert, Alexander J., Estelle, Joshua J.
• Primary Examiner(s): Desir, Pierre-Louis
• Assistant Examiner(s): Tzeng, Forrest F
• Application Number:
  US14/076,029
• Publication Number:
  US 20150134318A1
• Time in Patent Office: 1,166 Days
• Field of Search: 704/2, 704/9, 704/E13.011, 704/3
• US Class Current: 1/1
• CPC Class Codes: G06F 3/018 Input/output arrangements for oriental characters
• CPC Class Codes: G06F 40/47 Machine-assisted translation, e.g. using translation memory
• CPC Class Codes: G06F 40/58 Use of machine translation, e.g. for multi-lingual retrieval, for server-side translation for client devices or for real-time translation
• CPC Class Codes: G06V 20/20 in augmented reality scenes
• CPC Class Codes: G06V 20/62 Text, e.g. of license plates, overlay texts or captions on TV images
• CPC Class Codes: G06V 20/63 Scene text, e.g. street names

Abstract

Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for presenting additional information for text depicted by an image. In one aspect, a method includes receiving an image. Text depicted in the image is identified. The identified text can be in one or more text blocks. A prominence presentation context is selected for the image based on the relative prominence of the one or more text blocks. Each prominence presentation context corresponds to a relative prominence of each text block in which text is presented within images. Each prominence presentation context has a corresponding user interface for presenting additional information related to the identified text depicted in the image. A user interface is identified that corresponds to the selected prominence presentation context. Additional information is presented for at least a portion of the text depicted in the image using the identified user interface.

Claims

1. A method performed by data processing apparatus, the method comprising:

receiving an image from a camera of a user device;

identifying text depicted in the image, the identified text being in two or more text blocks identified in the image, the two or more text blocks including a first text block and a second text block distinct from the first text block, the identified text being in a first language;

processing, by the data processing apparatus, the image to determine a relative prominence between the two or more text blocks and to determine, from a plurality of different prominence presentation contexts, a prominence presentation context for presenting a translation of text depicted in the image based on the relative prominence, wherein each prominence presentation context corresponds to a relative prominence of each text block in which text is presented within images to other text blocks identified in the images, and each prominence presentation context has a corresponding graphical user interface for presenting a translation of a different portion of the identified text than each other prominence presentation context;

determining, based on the selected prominence presentation context, that a translation of a single text block, of the two or more text blocks, will be presented using the graphical user interface corresponding to the selected prominence presentation context;

in response to determining that a translation of a single block of text will be presented using the graphical user interface corresponding to the selected prominence presentation context, selecting, between the first text block and the second text block and based on a size of the text in the first text block and a location of the first text block within the image relative to a size of the text in the second text block and a location of the second text block within the image, first text block as the single text block for which a translation will be presented using the graphical user interface corresponding to the selected prominence presentation context;

presenting, at a display of the user device, the translation of the text in the first text block in an overlay over the image using the graphical user interface corresponding to the selected prominence presentation context, while presenting the text in the second text block in the first language and in the image.

2. The method of claim 1, wherein:

selecting the prominence presentation context for the image comprises;

determining that the first text block is displayed more prominently within the image than the second text block; and

selecting a dominant-secondary block context from the plurality of different prominence presentation contexts in response to the determination, the dominant-secondary block context having at least one text block that is presented more prominently than at least one other text block, the dominant-secondary block context corresponding to a dominant-secondary user interface that presents a language translation of a dominant block of text that has a greatest prominence of text blocks depicted in the image without presenting a language translation of another block of text that does not have the greatest prominence of text blocks depicted in the image.

3. The method of claim 1, wherein the overlay is located over the first text block in the image.

4. The method of claim 1, further comprising:

presenting a selectable user interface element in the user interface corresponding to the selected presentation context at the depiction of the second text block in the image; and

in response to receiving a selection of the selectable user interface element, presenting a language translation of the text in the second text block.

5. The method of claim 2, wherein determining that the first text block is displayed more prominently within the image than the second block of text comprises:

determining that the text in the first text block is larger than the text in the second text block; and

determining that the first text block is located closer to a center of the image than the second text block.

6. The method of claim 1, further comprising identifying the user interface that corresponds to the selected prominence presentation context by:

determining a readability measure for the translation of the text in the first text block based at least on a number of characters included in the translation of the text in the first text block;

determining that the readability measure meets a readability threshold; and

in response to determining that the readability measure meets the readability threshold, selecting a user interface for the selected prominence presentation context that presents the translation of the text in the first text block in an overlay over the first text block.

7. The method of claim 1, further comprising identifying the user interface that corresponds to the selected prominence presentation context by:

determining a readability measure for the translation of the text in the first text block based at least on a number of characters included in the translation of the text in the first text block;

determining that the readability measure does not meet a readability threshold; and

in response to determining that the readability measure does not meet the readability threshold, selecting a user interface for the selected prominence presentation context that presents only a portion of the translation and a user interface element that enables a user to browse to additional portions of the translation.

8. A system, comprising:

a data processing apparatus; and

a memory storage apparatus in data communication with the data processing apparatus, the memory storage apparatus storing instructions executable by the data processing apparatus and that upon such execution cause the data processing apparatus to perform operations comprising;

receiving an image from a camera of a user device;

identifying text depicted in the image, the identified text being in two or more text blocks identified in the image, the two or more text blocks including a first text block and a second text block distinct from the first text block, the identified text being in a first language;

processing, by the data processing apparatus, the image to determine a relative prominence between the two or more text blocks and to determine, from a plurality of different prominence presentation contexts, a prominence presentation context for presenting a translation of text depicted in the image based on the relative prominence, wherein each prominence presentation context corresponds to a relative prominence of each text block in which text is presented within images to other text blocks identified in the images, and each prominence presentation context has a corresponding graphical user interface for presenting a translation of a different portion of the identified text than each other prominence presentation context;

determining, based on the selected prominence presentation context, that a translation of a single text block, of the two or more text blocks, will be presented using the graphical user interface corresponding to the selected prominence presentation context;

in response to determining that a translation of a single block of text will be presented using the graphical user interface corresponding to the selected prominence presentation context, selecting, between the first text block and the second text block and based on a size of the text in the first text block and a location of the first text block within the image relative to a size of the text in the second text block and a location of the second text block within the image, first text block as the single text block for which a translation will be presented using the graphical user interface corresponding to the selected prominence presentation context;

presenting, at a display of the user device, the translation of the text in the first text block in an overlay over the image using the graphical user interface corresponding to the selected prominence presentation context, while presenting the text in the second text block in the first language and in the image.

9. The system of claim 8, wherein:

selecting the prominence presentation context for the image comprises;

determining that the first text block is displayed more prominently within the image than the second text block; and

selecting a dominant-secondary block context from the plurality of different prominence presentation contexts in response to the determination, the dominant-secondary block context having at least one text block that is presented more prominently than at least one other text block, the dominant-secondary block context corresponding to a dominant-secondary user interface that presents a language translation of a dominant block of text that has a greatest prominence of text blocks depicted in the image without presenting a language translation of another block of text that does not have the greatest prominence of text blocks depicted in the image.

10. The system of claim 8, wherein the overlay is located over the first text block in the image.

11. The system of claim 9, wherein the instructions upon execution cause the data processing apparatus to perform further operations comprising:

presenting a selectable user interface element in the user interface corresponding to the selected presentation context at the depiction of the second text block in the image; and

in response to receiving a selection of the selectable user interface element, presenting a language translation of the text in the second text block.

12. A non-transitory computer storage medium encoded with a computer program, the program comprising instructions that when executed by a data processing apparatus cause the data processing apparatus to perform operations comprising:

receiving an image from a camera of a user device;

identifying text depicted in the image, the identified text being in two or more text blocks identified in the image, the two or more text blocks including a first text block and a second text block distinct from the first text block, the identified text being in a first language;

processing, by the data processing apparatus, the image to determine a relative prominence between the two or more text blocks and to determine, from a plurality of different prominence presentation contexts, a prominence presentation context for presenting a translation of text depicted in the image based on the relative prominence, wherein each prominence presentation context corresponds to a relative prominence of each text block in which text is presented within images to other text blocks identified in the images, and each prominence presentation context has a corresponding graphical user interface for presenting a translation of a different portion of the identified text than each other prominence presentation context;

determining, based on the selected prominence presentation context, that a translation of a single text block, of the two or more text blocks, will be presented using the graphical user interface corresponding to the selected prominence presentation context;

in response to determining that a translation of a single block of text will be presented using the graphical user interface corresponding to the selected prominence presentation context, selecting, between the first text block and the second text block and based on a size of the text in the first text block and a location of the first text block within the image relative to a size of the text in the second text block and a location of the second text block within the image, first text block as the single text block for which a translation will be presented using the graphical user interface corresponding to the selected prominence presentation context;

presenting, at a display of the user device, the translation of the text in the first text block in an overlay over the image using the graphical user interface corresponding to the selected prominence presentation context, while presenting the text in the second text block in the first language and in the image.