PRESENTING TRANSLATIONS OF TEXT DEPICTED IN IMAGES

• US 20150134318A1
• Filed: 11/08/2013
• Published: 05/14/2015
• Est. Priority Date: 11/08/2013
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google Inc. (Alphabet Inc.)
• Inventors: Cuthbert, Alexander J., Estelle, Joshua J.
• Granted Patent: US 9,547,644 B2
• Time in Patent Office: Days
• US Class Current: 704/2
• CPC Class Codes: G06F 3/018 Input/output arrangements for oriental characters
• CPC Class Codes: G06F 40/47 Machine-assisted translation, e.g. using translation memory
• CPC Class Codes: G06F 40/58 Use of machine translation, e.g. for multi-lingual retrieval, for server-side translation for client devices or for real-time translation
• CPC Class Codes: G06V 20/20 in augmented reality scenes
• CPC Class Codes: G06V 20/62 Text, e.g. of license plates, overlay texts or captions on TV images
• CPC Class Codes: G06V 20/63 Scene text, e.g. street names

Abstract

Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for presenting additional information for text depicted by an image. In one aspect, a method includes receiving an image. Text depicted in the image is identified. The identified text can be in one or more text blocks. A prominence presentation context is selected for the image based on the relative prominence of the one or more text blocks. Each prominence presentation context corresponds to a relative prominence of each text block in which text is presented within images. Each prominence presentation context has a corresponding user interface for presenting additional information related to the identified text depicted in the image. A user interface is identified that corresponds to the selected prominence presentation context. Additional information is presented for at least a portion of the text depicted in the image using the identified user interface.

Claims

1. A method performed by data processing apparatus, the method comprising:

receiving an image;

identifying text depicted in the image, the identified text being in one or more text blocks identified in the image, each text block being distinct from other text blocks in the image;

selecting, for the image, a prominence presentation context from a plurality of prominence contexts, wherein each prominence presentation context corresponds to a relative prominence of each text block in which text is presented within images and each prominence presentation context has a corresponding user interface for presenting additional information related to the identified text depicted in the image, the selecting of the prominence presentation context being based on the relative prominence of the one or more text blocks in which the identified text is depicted in the image;

identifying a user interface that corresponds to the selected prominence presentation context; and

presenting additional information for at least a portion of the text depicted in the image using the identified user interface, the user interface presenting the additional information in an overlay over the image.

2. The method of claim 1, wherein selecting the prominence presentation context for the image comprises:

determining that the text depicted in the image includes a single text block of prominently displayed text based on a location of the single text block within the image and a size of text included in the single text block; and

selecting a single block context from the plurality of prominence presentation contexts in response to the determination.

3. The method of claim 2, wherein the single block context corresponds to a user interface that presents a language translation of the text included in the single text block of prominently displayed text in an overlay over the single text block of prominently displayed text.

4. The method of claim 1, wherein:

identifying text depicted in the image comprises identifying a first text block and a second text block that is different from the first text block, the first and second text blocks being depicted in the image; and

selecting the prominence presentation context for the image comprises;

determining that the first text block is displayed more prominently within the image than the second text block; and

selecting a dominant-secondary block context from the plurality of prominence presentation contexts in response to the determination, the dominant-secondary block context corresponding to a user interface that presents a language translation of a dominant block of text.

5. The method of claim 4, wherein presenting additional information for at least a portion of the text depicted in the image using the identified user interface comprises presenting a language translation of text included in the first text block in an overlay over the image, the overlay being located over the first text block in the image.

6. The method of claim 4, further comprising:

presenting a selectable user interface element in the dominant-secondary user interface at the depiction of the second text block in the image; and

in response to receiving a selection of the selectable user interface element, presenting a language translation of text included in the second text block.

7. The method of claim 4, wherein determining that the first text block is displayed more prominently within the image than the second block of text comprises:

determining that the text included in the first text block is larger than the text included in the second text block; and

determining that the first text block is located closer to a center of the image than the second text block.

8. The method of claim 1, wherein:

identifying text depicted in the image comprises identifying a first text block and a second text block that is different from the first text block, the first and second text blocks being depicted in the image; and

selecting the prominence presentation context for the image comprises;

determining that the first text block is displayed at a substantially equal prominence within the image as the second text block;

selecting an equal prominence presentation context from the plurality of prominence presentation contexts in response to the determination, the equal prominence presentation context corresponding to a user interface that presents a language translation of multiple blocks of text having substantially equal prominence.

9. The method of claim 8, wherein presenting additional information for at least a portion of the text depicted in the image using the identified user interface comprises:

presenting a language translation of text included in the first text block in a first overlay over the image, the first overlay being located over the first text block in the image; and

presenting a language translation of text included in the second text block in a second overlay over the image, the second overlay being located over the second text block in the image.

10. The method of claim 8, wherein determining that the first text block is displayed at a substantially equal prominence within the image as the second text block comprises determining that a size of the text included in the first text block is substantially equal to a size of the text included in the second text block.

11. The method of claim 1, wherein identifying a user interface that corresponds to the selected prominence presentation context comprises:

determining a readability measure for the additional information based at least on a number of characters included in the additional information;

determining that the readability measure meets a readability threshold; and

in response to determining that the readability measure meets the readability threshold, selecting a user interface for the selected prominence presentation context that presents the additional information in an overlay over text for which the additional information corresponds.

12. The method of claim 1, wherein identifying a user interface that corresponds to the selected prominence presentation context comprises:

determining a readability measure for the additional information based at least on a number of characters included in the additional information;

determining that the readability measure does not meet a readability threshold; and

in response to determining that the readability measure does not meet the readability threshold, selecting a user interface for the selected prominence presentation context that presents a portion of the additional information and a user interface element that enables a user to browse to additional portions of the additional information.

13. A system, comprising:

a data processing apparatus; and

a memory storage apparatus in data communication with the data processing apparatus, the memory storage apparatus storing instructions executable by the data processing apparatus and that upon such execution cause the data processing apparatus to perform operations comprising;

receiving an image;

identifying text depicted in the image, the identified text being in one or more text blocks identified in the image, each text block being distinct from other text blocks in the image;

selecting, for the image, a prominence presentation context from a plurality of prominence contexts, wherein each prominence presentation context corresponds to a relative prominence of each text block in which text is presented within images and each prominence presentation context has a corresponding user interface for presenting additional information related to the identified text depicted in the image, the selecting of the prominence presentation context being based on the relative prominence of the one or more text blocks in which the identified text is depicted in the image;

identifying a user interface that corresponds to the selected prominence presentation context; and

presenting additional information for at least a portion of the text depicted in the image using the identified user interface, the user interface presenting the additional information in an overlay over the image.

14. The system of claim 13, wherein selecting the prominence presentation context for the image comprises:

determining that the text depicted in the image includes a single text block of prominently displayed text based on a location of the single text block within the image and a size of text included in the single text block; and

selecting a single block context from the plurality of prominence presentation contexts in response to the determination.

15. The system of claim 14, wherein the single block context corresponds to a user interface that presents a language translation of the text included in the single text block of prominently displayed text in an overlay over the single text block of prominently displayed text.

16. The system of claim 13, wherein:

identifying text depicted in the image comprises identifying a first text block and a second text block that is different from the first text block, the first and second text blocks being depicted in the image; and

selecting the prominence presentation context for the image comprises;

determining that the first text block is displayed more prominently within the image than the second text block; and

selecting a dominant-secondary block context from the plurality of prominence presentation contexts in response to the determination, the dominant-secondary block context corresponding to a user interface that presents a language translation of a dominant block of text.

17. The system of claim 16, wherein presenting additional information for at least a portion of the text depicted in the image using the identified user interface comprises presenting a language translation of text included in the first text block in an overlay over the image, the overlay being located over the first text block in the image.

18. The system of claim 16, wherein the instructions upon execution cause the data processing apparatus to perform further operations comprising:

presenting a selectable user interface element in the dominant-secondary user interface at the depiction of the second text block in the image; and

in response to receiving a selection of the selectable user interface element, presenting a language translation of text included in the second text block.

19. The system of claim 13, wherein:

identifying text depicted in the image comprises identifying a first text block and a second text block that is different from the first text block, the first and second text blocks being depicted in the image; and

selecting the prominence presentation context for the image comprises;

determining that the first text block is displayed at a substantially equal prominence within the image as the second text block;

selecting an equal prominence presentation context from the plurality of prominence presentation contexts in response to the determination, the equal prominence presentation context corresponding to a user interface that presents a language translation of multiple blocks of text having substantially equal prominence.

20. A computer storage medium encoded with a computer program, the program comprising instructions that when executed by a data processing apparatus cause the data processing apparatus to perform operations comprising:

receiving an image;

identifying text depicted in the image, the identified text being in one or more text blocks identified in the image, each text block being distinct from other text blocks in the image;

selecting, for the image, a prominence presentation context from a plurality of prominence contexts, wherein each prominence presentation context corresponds to a relative prominence of each text block in which text is presented within images and each prominence presentation context has a corresponding user interface for presenting additional information related to the identified text depicted in the image, the selecting of the prominence presentation context being based on the relative prominence of the one or more text blocks in which the identified text is depicted in the image;

identifying a user interface that corresponds to the selected prominence presentation context; and

presenting additional information for at least a portion of the text depicted in the image using the identified user interface, the user interface presenting the additional information in an overlay over the image.