Method For Enhanced Location Based And Context Sensitive Augmented Reality Translation

• US 20120330646A1
• Filed: 06/23/2011
• Published: 12/27/2012
• Est. Priority Date: 06/23/2011
• Status: Active Grant

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: Robbins, Christopher A., Andrade, Rajiv Augusto Santos Galvao de, Franco, Lucas Goncalves
• Granted Patent: US 9,092,674 B2
• Time in Patent Office: Days
• US Class Current: 704/7
• CPC Class Codes: G06F 40/58 Use of machine translation, e.g. for multi-lingual retrieval, for server-side translation for client devices or for real-time translation
• CPC Class Codes: G06V 20/20 in augmented reality scenes
• CPC Class Codes: G06V 20/63 Scene text, e.g. street names
• CPC Class Codes: G06V 30/10 Character recognition

Abstract

A computer system and method where text is recognized from a real world image, and this recognized text is used as input data for a processing program selected by a user. A computer system and method where text is recognized from a real world image, and contextual information is used in conjunction with the text to develop a semantic denotation of the recognized text. The contextual information may include GPS location data. The contextual information may include previous images, captured shortly prior to the image with the recognized text. A computer system and method wherein text is recognized from a real world image, then normalized to be in the plane of the image, then translated and then the translated text is made into an image that is anti-normalized and inserted into the original image (or an image similar to the original image). In this way, the translated text will appear realistically in place of the original untranslated text of the real world image.

Claims

1. A method comprising the following steps:

receiving, by a computer system, an image;

performing character recognition, by the computer system, on at least a portion of the image to yield a recognized character string as text data;

choosing a first further data processing program which is stored on a software storage device;

inputting at least a portion of the recognized character string to the first further data processing program;

processing input data, by the first further data processing program, with the input data including at least a portion of the recognized character string; and

receiving output data resulting from the data processing step.

2. The method of claim 1 further comprising the following step:

performing image preparation, by the computer system, the image for character recognition;

wherein;

the performing-image-preparation step includes the sub-step of normalizing the portion of the image including the characters so that the characters appear as if a plane upon which they are printed is substantially normal to a direction from which the image was captured; and

at the performing-character-recognition step, the character recognition is performed on the image as prepared at the performing-image-preparation step.

3. The method of claim 1 wherein the choosing step includes the sub-steps:

presenting a user with a list of at least one potentially applicable data processing program(s);

receiving choice-indicative user input, by the computer system, from the user to indicate which, if any, potentially applicable data processing programs to use; and

selecting the first further data processing program based upon the choice-indicative user input.

4. The method of claim 1 wherein the choosing step is performed automatically by the computer system.

5. A computer system comprising:

a first data processing program module;

an image receiving module structured and/or programmed to receive an image;

a character recognition module structured and/or programmed to perform character recognition on at least a portion of the image to yield a recognized character string as text data;

a further data processing program chooser module structured and/or programmed to choose the first further data processing program to be used for further processing; and

an input module structured and/or programmed to input at least a portion of the recognized character string to the further first data processing program;

wherein the first further data processing program module is structured and/or programmed to perform data processing on input data with the input data including at least a portion of the recognized character string to output data processing output data resulting from the data processing.

6. The system of claim 5 further comprising an image preparation module structured and/or programmed to perform image preparation for character recognition on the image, with the image preparation including at least normalizing the portion of the image including the characters so that the characters appear as if a plane upon which they are printed is substantially normal to a direction from which the image was captured.

7. The system of claim 5 wherein the further data processing chooser module is further structured and/or programmed to:

present a user with a list of at least one potentially applicable data processing program(s);

receive choice-indicative user input from the user to indicate which, if any, potentially applicable data processing programs to use; and

select the first further data processing program as a program to be used for further data processing based upon the choice-indicative user input.

8. Software stored on a software storage device for use by a computer system, the software comprising:

a first data processing program;

an image receiving software portion programmed to receive an image;

a character recognition software portion programmed to perform character recognition on at least a portion of the image to yield a recognized character string as text data;

a further data processing program chooser software portion programmed to choose the first further data processing program to be used for further processing; and

an input software portion programmed to input at least a portion of the recognized character string to the further first data processing program;

wherein the first further data processing program is programmed to perform data processing on input data with the input data including at least a portion of the recognized character string to output data processing output data resulting from the data processing.

9. A method comprising the following steps:

receiving, by a computer system, an image and associated contextual information data;

performing character recognition, by the computer system, on at least a portion of the image to yield a recognized character string as text data;

determining symbolic denotation data, by the computer system, indicating symbolic denotation of at least a portion of the recognized character string based upon the text data of the character string and the contextual information data; and

performing further processing, by the computer system, on the recognized character string based, at least in part, upon the symbolic denotation data.

10. The method of claim 9 further comprising the step of:

choosing a first further data processing program which is stored on a software storage device;

wherein;

at the determining step, an addressing portion of the recognized character string is determined to have a symbolic denotation in that it corresponds to one of the following address categories;

a telephone number, a physical street address or an email address; and

at the performing-further-processing step, the addressing portion of the recognized character string is input to a further data processing program along with its address category.

11. The method of claim 10 wherein:

at the determining step, an addressable-entity portion of the recognized character string is determined to have a symbolic denotation in that it corresponds to entity having an address indicated by the addressing portion; and

at the performing-further-processing step the addressable-entity portion of the recognized character string is input to the further data processing program as the entity corresponding to the address of the addressing portion.

12. The method of claim 9 wherein:

the contextual information data includes GPS location data obtained through a global positioning satellite system.

13. The method of claim 12 wherein at the determining step includes the following sub-steps:

determining the language of the recognized character string based on contextual information data in the form of the GPS location data; and

determining language-specific symbolic denotation data in the form of a translation for the recognized character string based on the language determined at the determining-the-language sub-step.

14. The method of claim 9 wherein the contextual information data is previous image data.

15. A computer system comprising:

a receiving module structured and/or programmed to receive an image and associated contextual information data;

a character recognition module structured and/or programmed to perform character recognition on at least a portion of the image to yield a recognized character string as text data; and

a symbolic denotation module structured and/or programmed to;

determine symbolic denotation data indicating symbolic denotation of at least a portion of the recognized character string based upon the text data of the character string and the contextual information data, and

perform further processing on the recognized character string based, at least in part, upon the symbolic denotation data.

16. The system of claim 15 further comprising a GPS module structured and/or programmed to determine GPS location data, wherein the contextual information data includes the GPS location data.

17. The system of claim 16 further comprises a translation module structured and/or programmed to:

determine the language of the recognized character string based on contextual information data in the form of the GPS location data, and

determine language-specific symbolic denotation data in the form of a translation for the recognized character string based on the language determined at the determining-the-language sub-step.

18. The system of claim 15 wherein the contextual information data is previous image data.

19. A method comprising the steps of:

isolating, by a computer system, a textual image portion of a raw image which includes a character cluster;

normalizing, by the computer system, the textual image portion to generate a normalized textual image portion with a plane of the character cluster in the normalized textual image portion being at least substantially parallel with the projection plane of the raw image;

performing character recognition, by the computer system, on the normalized textual image portion to yield a recognized character string;

translating, by the computer system, the text into a different language to yield a translated character string;

anti-normalizing, by the computer system, an image of the translated character string to generate a translated textual image portion; and

inserting, by the computer system, the translated textual portion into a target image to form a translated AR image.

20. The method of claim 19 wherein the target image is the same as the raw image.