Presenting translations of text depicted in images

• US 10,726,212 B2
• Filed: 01/31/2019
• Issued: 07/28/2020
• Est. Priority Date: 11/08/2013
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google LLC (Alphabet Inc.)
• Inventors: Cuthbert, Alexander Jay, Estelle, Joshua J.
• Primary Examiner(s): Tzeng, Feng-Tzer
• Application Number:
  US16/263,898
• Publication Number:
  US 20190163748A1
• Time in Patent Office: 544 Days
• CPC Class Codes: G06F 3/018 Input/output arrangements for oriental characters
• CPC Class Codes: G06F 40/47 Machine-assisted translation, e.g. using translation memory
• CPC Class Codes: G06F 40/58 Use of machine translation, e.g. for multi-lingual retrieval, for server-side translation for client devices or for real-time translation
• CPC Class Codes: G06V 20/20 in augmented reality scenes
• CPC Class Codes: G06V 20/62 Text, e.g. of license plates, overlay texts or captions on TV images
• CPC Class Codes: G06V 20/63 Scene text, e.g. street names

Abstract

Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for presenting additional information for text depicted by an image. In one aspect, a method includes receiving an image. Text depicted in the image is identified. The identified text can be in one or more text blocks. A prominence presentation context is selected for the image based on the relative prominence of the one or more text blocks. Each prominence presentation context corresponds to a relative prominence of each text block in which text is presented within images. Each prominence presentation context has a corresponding user interface for presenting additional information related to the identified text depicted in the image. A user interface is identified that corresponds to the selected prominence presentation context. Additional information is presented for at least a portion of the text depicted in the image using the identified user interface.

Claims

1. A method performed by data processing apparatus, the method comprising:

receiving, from a user device, an image;

detecting, in the image, an address for a particular geographic location;

determining that the address is in a first language different from a second language specified by a user of the user device, the second language being a language specified by the user as a language to which text found within images in a different language from the second language is to be translated, wherein the second language is specified by the user of the user device using a user interface presented by the user device prior to receiving the image; and

in response to detecting the address for the particular geographic location in the first language and determining that the first language is different from the second language, generating and providing, for presentation by the user device, a user interface that depicts a map that includes the particular geographic location and a translation of the address in the second language specified by the user.

2. The method of claim 1, further comprising determining a geographic location of the user device, wherein the map includes directions from the geographic location of the user device to the particular geographic location.

3. The method of claim 1, further comprising selecting the user interface from multiple user interfaces based on a number of addresses detected in the image.

4. The method of claim 3, further comprising determining that the address for the particular geographic location is the only address depicted by the image, wherein the user interface that depicts the map that includes the particular geographic location and the translation of the address in the second language different is selected in response to determining that the address for the particular geographic location is the only address depicted by the image.

5. The method of claim 3, further comprising detecting one or more additional addresses for one or more additional geographic locations, wherein the user interface presents a control that enables the user to navigate between maps for each detected address.

6. The method of claim 4, wherein each map is presented on a respective user interface card and each user interface card includes a translation, in the second language, of the address for the geographic location corresponding to the map presented on the user interface card.

7. The method of claim 1, wherein the user interface depicts, in the second language, a name of a building located at the particular geographic location.

8. A system, comprising:

a data processing apparatus; and

a memory storage apparatus in data communication with the data processing apparatus, the memory storage apparatus storing instructions executable by the data processing apparatus and that upon such execution cause the data processing apparatus to perform operations comprising;

receiving, from a user device, an image;

detecting, in the image, an address for a particular geographic location;

determining that the address is in a first language different from a second language specified by a user of the user device, the second language being a language specified by the user as a language to which text found within images in a different language from the second language is to be translated, wherein the second language is specified by the user of the user device using a user interface presented by the user device prior to receiving the image; and

in response to detecting the address for the particular geographic location in the first language and determining that the first language is different from the second language, generating and providing, for presentation by the user device, a user interface that depicts a map that includes the particular geographic location and a translation of the address in the second language specified by the user.

9. The system of claim 8, wherein the operations comprise determining a geographic location of the user device, wherein the map includes directions from the geographic location of the user device to the particular geographic location.

10. The system of claim 8, wherein the operations comprise selecting the user interface from multiple user interfaces based on a number of addresses detected in the image.

11. The system of claim 10, wherein the operations comprise determining that the address for the particular geographic location is the only address depicted by the image, wherein the user interface that depicts the map that includes the particular geographic location and the translation of the address in the second language different is selected in response to determining that the address for the particular geographic location is the only address depicted by the image.

12. The system of claim 10, wherein the operations comprise detecting one or more additional addresses for one or more additional geographic locations, wherein the user interface presents a control that enables the user to navigate between maps for each detected address.

13. The system of claim 12, wherein each map is presented on a respective user interface card and each user interface card includes a translation, in the second language, of the address for the geographic location corresponding to the map presented on the user interface card.

14. The system of claim 8, wherein the user interface depicts, in the second language, a name of a building located at the particular geographic location.

15. A non-transitory computer-readable storage medium encoded with a computer program, the program comprising instructions that when executed by a data processing apparatus cause the data processing apparatus to perform operations comprising:

receiving, from a user device, an image;

detecting, in the image, an address for a particular geographic location;

determining that the address is in a first language different from a second language specified by a user of the user device, the second language being a language specified by the user as a language to which text found within images in a different language from the second language is to be translated, wherein the second language is specified by the user of the user device using a user interface presented by the user device prior to receiving the image; and

in response to detecting the address for the particular geographic location in the first language and determining that the first language is different from the second language, generating and providing, for presentation by the user device, a user interface that depicts a map that includes the particular geographic location and a translation of the address in the second language specified by the user.

16. The non-transitory computer-readable storage medium of claim 15, wherein the operations comprise determining a geographic location of the user device, wherein the map includes directions from the geographic location of the user device to the particular geographic location.

17. The non-transitory computer-readable storage medium of claim 15, wherein the operations comprise selecting the user interface from multiple user interfaces based on a number of addresses detected in the image.

18. The non-transitory computer-readable storage medium of claim 17, wherein the operations comprise determining that the address for the particular geographic location is the only address depicted by the image, wherein the user interface that depicts the map that includes the particular geographic location and the translation of the address in the second language different is selected in response to determining that the address for the particular geographic location is the only address depicted by the image.

19. The non-transitory computer-readable storage medium of claim 17, wherein the operations comprise detecting one or more additional addresses for one or more additional geographic locations, wherein the user interface presents a control that enables the user to navigate between maps for each detected address.

20. The non-transitory computer-readable storage medium of claim 19, wherein each map is presented on a respective user interface card and each user interface card includes a translation, in the second language, of the address for the geographic location corresponding to the map presented on the user interface card.