Presenting translations of text depicted in images

• US 10,198,439 B2
• Filed: 07/21/2016
• Issued: 02/05/2019
• Est. Priority Date: 11/08/2013
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google LLC (Alphabet Inc.)
• Inventors: Cuthbert, Alexander J., Estelle, Joshua J.
• Primary Examiner(s): Tzeng, Forrest F
• Application Number:
  US15/216,399
• Publication Number:
  US 20160328394A1
• Time in Patent Office: 929 Days
• Field of Search: 704 2, 704 3, 704 9, 704239
• CPC Class Codes: G06F 3/018 Input/output arrangements for oriental characters
• CPC Class Codes: G06F 40/47 Machine-assisted translation, e.g. using translation memory
• CPC Class Codes: G06F 40/58 Use of machine translation, e.g. for multi-lingual retrieval, for server-side translation for client devices or for real-time translation
• CPC Class Codes: G06V 20/20 in augmented reality scenes
• CPC Class Codes: G06V 20/62 Text, e.g. of license plates, overlay texts or captions on TV images
• CPC Class Codes: G06V 20/63 Scene text, e.g. street names

Abstract

Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for presenting additional information for text depicted by an image. In one aspect, a method includes receiving an image. Text depicted in the image is identified. The identified text can be in one or more text blocks. A prominence presentation context is selected for the image based on the relative prominence of the one or more text blocks. Each prominence presentation context corresponds to a relative prominence of each text block in which text is presented within images. Each prominence presentation context has a corresponding user interface for presenting additional information related to the identified text depicted in the image. A user interface is identified that corresponds to the selected prominence presentation context. Additional information is presented for at least a portion of the text depicted in the image using the identified user interface.

Claims

1. A method performed by data processing apparatus, the method comprising:

obtaining, by the data processing apparatus and from a camera of a user device, an image of a scene that includes (i) first text that, in the image, satisfies at least one criterion of text color, background color, text size, font type, or location-within-image, and (ii) second text that, in the image, does not satisfy any of the criterion of text color, background color, text size, font type, or location-within-image, wherein the location-within-image criterion is a distance between text and a center of the image;

processing, by the data processing apparatus, the image to determine that the first text satisfies at least one of the criterion of text color, background color, text size, font type, or location-within-image and that the second text does not satisfy any of the criterion of the text color, background color, text size, font type, or location-within-image;

determining, by the data processing apparatus, that a readability measure for presentation of a particular user interface that includes a translation of the first text and that does not include a translation of the second text satisfies a readability criterion related to how readable the translation of the first text would be if presented using the particular user interface, wherein the readability measure is based on a number of characters of the translation that would be presented if the translation is presented using the particular user interface; and

in response to determining that (i) the first text satisfies at least one of the criterion of text color, background color, text size, font type, or location-within-image and that the second text does not satisfy any of the criterion of the text color, background color, text size, font type, or location-within-image and (ii) presentation of the particular user interface satisfies the readability criterion, providing, by the data processing apparatus and for display at the user device, the particular user interface that includes the translation of the first text and that does not include a translation of the second text.

2. The method of claim 1, wherein the user interface presents the translation of the first text in an overlay over a location of the first text.

3. The method of claim 1, wherein the user interface presents a user interface element adjacent to the second text, the method further comprising presenting a translation of the second text in response to detecting user interaction with the user interface element.

4. The method of claim 1, wherein the first text has a text color, background color, text size, font type, or location within the image that is determined to be more prominent than a text color, background color, text size, font type, or location within the image of the second text.

5. The method of claim 1, wherein processing the image comprises determining that the first text satisfies the at least one criterion of text color, background color, text size, font type, or location-within-image based on a prominence of the first text being at least a threshold amount greater than a prominence of the second text.

6. The method of claim 1, wherein processing the image comprises determining that the first text satisfies the at least one criterion of text color, background color, text size, font type, or location-within-image in response to determining that each portion of the first text is related based on at least one of text color, background color, text size, front type, or location within the image of each portion the first text.

7. The method of claim 6, further comprising determining that each portion of the first text is related based on similarity of the text color, background color, text size, font type, or location within the image of each portion of the first text.

20. The method of claim 1, wherein determining, by the data processing apparatus, that the readability measure for presentation of the particular user interface that includes a translation of the first text and that does not include a translation of the second text satisfies a readability criterion comprises determining that the readability measure for presentation of the particular user interface exceeds a respective readability measure for each of one or more other user interfaces that includes a translation of the first text and that does not include a translation of the second text, the method further comprising:

selecting the particular user interface over the one or more other user interfaces in response to determining that the readability measure for presentation of the particular user interface exceeds the respective readability measure for each of the one or more other user interfaces.

8. A system, comprising:

a data processing apparatus; and

a memory storage apparatus in data communication with the data processing apparatus, the memory storage apparatus storing instructions executable by the data processing apparatus and that upon such execution cause the data processing apparatus to perform operations comprising;

obtaining, from a camera of a user device, an image of a scene that includes (i) first text that, in the image, satisfies at least one criterion of text color, background color, text size, font type, or location-within-image, and (ii) second text that, in the image, does not satisfy any of the criterion of text color, background color, text size, font type, or location-within-image, wherein the location-within-image criterion is a distance between text and a center of the image;

processing the image to determine that the first text satisfies at least one of the criterion of text color, background color, text size, font type, or location-within-image and that the second text does not satisfy any of the criterion of the text color, background color, text size, font type, or location-within-image;

determining that a readability measure for presentation of a particular user interface that includes a translation of the first text and that does not include a translation of the second text satisfies a readability criterion related to how readable the translation of the first text would be if presented using the particular user interface, wherein the readability measure is based on a number of characters of the translation that would be presented if the translation is presented using the particular user interface; and

in response to determining that (i) the first text satisfies at least one of the criterion of text color, background color, text size, font type, or location-within-image and that the second text does not satisfy any of the criterion of the text color, background color, text size, font type, or location-within-image and (ii) presentation of the particular user interface satisfies the readability criterion, providing, for display at the user device, the user interface that includes the translation of the first text and that does not include a translation of the second text.

9. The system of claim 8, wherein the user interface presents the translation of the first text in an overlay over a location of the first text.

10. The system of claim 8, wherein the user interface presents a user interface element adjacent to the second text, the method further comprising presenting a translation of the second text in response to detecting user interaction with the user interface element.

11. The system of claim 8, wherein the first text has a text color, background color, text size, font type, or location within the image that is determined to be more prominent than a text color, background color, text size, font type, or location within the image of the second text.

12. The system of claim 8, wherein processing the image comprises determining that the first text satisfies the at least one criterion of text color, background color, text size, font type, or location-within-image based on a prominence of the first text being at least a threshold amount greater than a prominence of the second text.

13. The system of claim 8, wherein processing the image comprises determining that the first text satisfies the at least one criterion of text color, background color, text size, font type, or location-within-image in response to determining that each portion of the first text is related based on at least one of text color, background color, text size, front type, or location within the image of each portion the first text.

14. The system of claim 13, wherein the operations further comprise determining that each portion of the first text is related based on similarity of the text color, background color, text size, font type, or location within the image of each portion of the first text.

15. A non-transitory computer-readable storage medium encoded with a computer program, the program comprising instructions that when executed by a data processing apparatus cause the data processing apparatus to perform operations comprising:

obtaining, from a camera of a user device, an image of a scene that includes (i) first text that, in the image, satisfies at least one criterion of text color, background color, text size, font type, or location-within-image, and (ii) second text that, in the image, does not satisfy any of the criterion of text color, background color, text size, font type, or location-within-image, wherein the location-within-image criterion is a distance between text and a center of the image;

processing, by the data processing apparatus, the image to determine that the first text satisfies at least one of the criterion of text color, background color, text size, font type, or location-within-image and that the second text does not satisfy any of the criterion of the text color, background color, text size, font type, or location-within-image;

determining, by the data processing apparatus, that a readability measure for presentation of a particular user interface that includes a translation of the first text and that does not include a translation of the second text satisfies a readability criterion related to how readable the translation of the first text would be if presented using the particular user interface, wherein the readability measure is based on a number of characters of the translation that would be presented if the translation is presented using the particular user interface; and

in response to determining that (i) the first text satisfies at least one of the criterion of text color, background color, text size, font type, or location-within-image and that the second text does not satisfy any of the criterion of the text color, background color, text size, font type, or location-within-image and (ii) presentation of the particular user interface satisfies the readability criterion, providing, for display at the user device, the user interface that includes the translation of the first text and that does not include a translation of the second text.

16. The non-transitory computer-readable storage medium of claim 15, wherein the user interface presents the translation of the first text in an overlay over a location of the first text.

17. The non-transitory computer-readable storage medium of claim 15, wherein the user interface presents a user interface element adjacent to the second text, the method further comprising presenting a translation of the second text in response to detecting user interaction with the user interface element.

18. The non-transitory computer-readable storage medium of claim 15, wherein the first text has a text color, background color, text size, font type, or location within the image that is determined to be more prominent than a text color, background color, text size, font type, or location within the image of the second text.

19. The non-transitory computer-readable storage medium of claim 15, wherein processing the image comprises determining that the first text satisfies the at least one criterion of text color, background color, text size, font type, or location-within-image based on a prominence of the first text being at least a threshold amount greater than a prominence of the second text.