Method for enhanced location based and context sensitive augmented reality translation

• US 9,092,674 B2
• Filed: 06/23/2011
• Issued: 07/28/2015
• Est. Priority Date: 06/23/2011
• Status: Expired due to Fees

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: Andrade, Rajiv Augusto Santos Galvao de, Franco, Lucas Goncalves, Robbins, Christopher A.
• Primary Examiner(s): Albertalli, Brian
• Application Number:
  US13/166,933
• Publication Number:
  US 20120330646A1
• Time in Patent Office: 1,496 Days
• Field of Search: None
• US Class Current: 1/1
• CPC Class Codes: G06F 40/58 Use of machine translation, e.g. for multi-lingual retrieval, for server-side translation for client devices or for real-time translation
• CPC Class Codes: G06V 20/20 in augmented reality scenes
• CPC Class Codes: G06V 20/63 Scene text, e.g. street names
• CPC Class Codes: G06V 30/10 Character recognition

Abstract

A computer system and method where text is recognized from a real world image, and this recognized text is used as input data for a processing program selected by a user. A computer system and method where text is recognized from a real world image, and contextual information is used in conjunction with the text to develop a semantic denotation of the recognized text. The contextual information may include GPS location data. The contextual information may include previous images, captured shortly prior to the image with the recognized text. A computer system and method wherein text is recognized from a real world image, then normalized to be in the plane of the image, then translated and then the translated text is made into an image that is anti-normalized and inserted into the original image (or an image similar to the original image). In this way, the translated text will appear realistically in place of the original untranslated text of the real world image.

Claims

1. A method for analyzing an image obtained at a location, the method comprising the steps of:

receiving, by a computer system, an image comprising text, and contextual information data associated with the image, wherein the contextual information comprises a plurality of images of the vicinity of the image comprising text, the plurality of images obtained within a predetermined time interval prior to obtaining the image comprising text;

performing character recognition using said image to yield a character string;

sending, to a non-GPS supplemental data module, the received contextual information data;

determining, by the non-GPS supplemental data module, an approximate location of the image based on the received contextual information data; and

providing an analysis of the image based at least in part on the contextual information data and the approximate location of the image, wherein the analysis comprises at least a portion of the character string.

2. The method of claim 1 further comprising the steps of:

performing image preparation, by the computer system, of the image for character recognition;

wherein the performing-image-preparation step includes the sub-step of normalizing the portion of the image including the characters so that the characters appear as if a plane upon which they are printed is substantially normal to a direction from which the image was captured; and

wherein at the performing-character-recognition step, the character recognition is performed on the image as prepared at the performing-image-preparation step.

3. A computer system comprising:

a first data processing program module;

an image receiving module structured and/or programmed to receive an image obtained at a location, and contextual information data associated with the image, wherein the contextual information comprises a plurality of images of the vicinity of the image comprising text, the plurality of images obtained within a predetermined time interval prior to obtaining the image comprising text;

a character recognition module structured and/or programmed to perform character recognition on at least a portion of the image to yield a character string;

a non-GPS supplemental data module, structured and/or programmed to receive the contextual information data, and further structured and/or programmed to determine an approximate location of the image based on the received contextual information data; and

an analysis module structured and/or programmed to analyze the image based at least in part on the contextual information data and the approximate location of the image, wherein the analysis comprises at least a portion of the character string.

4. The system of claim 3 further comprising an image preparation module structured and/or programmed to perform image preparation for character recognition on the image, with the image preparation including at least normalizing the portion of the image including the characters so that the characters appear as if a plane upon which they are printed is substantially normal to a direction from which the image was captured.

5. Software stored on a software storage device for use by a computer system, the software comprising:

a first data processing program;

an image receiving software portion programmed to receive an image obtained at a location, and contextual information data associated with the image, wherein the contextual information comprises a plurality of images of the vicinity of the image comprising text, the plurality of images obtained within a predetermined time interval prior to obtaining the image comprising text;

a character recognition software portion programmed to perform character recognition on at least a portion of the image to yield a character string;

a non-GPS supplemental data receiving software portion programmed to receive the contextual information data, and further programmed to determine an approximate location of the image based on the received contextual information data; and

an analysis software portion programmed to provide an analysis of the image based at least in part on the contextual information data and the approximate location of the image, wherein the analysis comprises at least a portion of the character string.

6. A method comprising the steps of:

receiving, by a computer system, an image obtained at a location, and associated contextual information data, wherein the contextual information comprises a plurality of images of the vicinity of the image comprising text, the plurality of images obtained within a predetermined time interval prior to obtaining the image comprising text;

performing character recognition, by the computer system, on at least a portion of the image to yield a character string as text data;

determining symbolic denotation data, by the computer system, indicating symbolic denotation of at least a portion of the character string based upon the text data and the contextual information data;

sending, to a non-GPS supplemental data module, the received contextual information data;

receiving, from the non-GPS supplemental data module, an approximate location of the image based on the received contextual information data; and

performing further processing, by the computer system, on the recognized character string based, at least in part, upon the symbolic denotation data and the approximate location of the image.

7. The method of claim 6 further comprising the step of:

choosing a first further data processing program which is stored on a software storage device;

wherein at the determining step, an addressing portion of the recognized character string is determined to have a symbolic denotation in that it corresponds to one of the following address categories;

a telephone number, a physical street address or an email address; and

wherein at the performing-further-processing step, the addressing portion of the recognized character string is input to a further data processing program along with its address category.

8. The method of claim 7 wherein:

at the determining step, an addressable-entity portion of the recognized character string is determined to have a symbolic denotation in that it corresponds to entity having an address indicated by the addressing portion; and

at the performing-further-processing step the addressable-entity portion of the recognized character string is input to the further data processing program as the entity corresponding to the address of the addressing portion.

9. The method of claim 6 wherein the contextual information data is previous image data.

10. A computer system comprising:

a receiving module structured and/or programmed to receive;

(i) an image obtained at a location; and

(ii) associated contextual information data, wherein the contextual information comprises a plurality of images of the vicinity of the image comprising text, the plurality of images obtained within a predetermined time interval prior to obtaining the image comprising text;

a character recognition module structured and/or programmed to perform character recognition on at least a portion of the image to yield a character string as text data;

a non-GPS supplemental data module structured and/or programmed to receive the associated contextual information data, and further structured and/or programmed to determine, based on the contextual information data, an approximate location of the image; and

a symbolic denotation module structured and/or programmed to;

(i) determine symbolic denotation data indicating symbolic denotation of at least a portion of the recognized character string based upon the text data of the character string and the contextual information data, and (ii) perform further processing on the recognized character string based, at least in part, upon the symbolic denotation data and the approximate location of the image.

11. The system of claim 10 further comprising a GPS module structured and/or programmed to determine GPS location data.

12. The system of claim 11 further comprises a translation module structured and/or programmed to:

determine the language of the recognized character string based on contextual information data in the form of the GPS location data, and determine language-specific symbolic denotation data in the form of a translation for the recognized character string based on the language determined at the determining-the-language sub-step.