Systems and methods for determining and displaying multi-line foreign language translations in real time on mobile devices

• US 8,965,129 B2
• Filed: 06/14/2014
• Issued: 02/24/2015
• Est. Priority Date: 03/15/2013
• Status: Active Grant

• Current Assignee: Translate Abroad, Inc.
• Original Assignee: Translate Abroad, Inc.
• Inventors: Rogowski, Ryan Leon, Wu, Huan-Yu, Clark, Kevin Anthony
• Primary Examiner(s): SHAH, UTPAL D
• Application Number:
  US14/304,922
• Publication Number:
  US 20140297256A1
• Time in Patent Office: 255 Days
• Field of Search: 382/177, 382/185
• US Class Current: 382/189
• CPC Class Codes: G06F 18/00 Pattern recognition
• CPC Class Codes: G06F 40/51 Translation evaluation
• CPC Class Codes: G06F 40/58 Use of machine translation, e.g. for multi-lingual retrieval, for server-side translation for client devices or for real-time translation
• CPC Class Codes: G06V 10/28 Quantising the image, e.g. histogram thresholding for discrimination between background and foreground patterns
• CPC Class Codes: G06V 10/993 Evaluation of the quality of the acquired pattern
• CPC Class Codes: G06V 30/287 of Kanji, Hiragana or Katakana characters
• CPC Class Codes: G06V 30/414 Extracting the geometrical structure, e.g. layout tree; Block segmentation, e.g. bounding boxes for graphics or text
• CPC Class Codes: G09G 2340/12 Overlay of images, i.e. displayed pixel being the result of switching between the corresponding input pixels
• CPC Class Codes: G09G 3/003 to produce spatial visual effects
• CPC Class Codes: G09G 5/14 Display of multiple viewports

Abstract

The present invention is related to systems and methods for translating language text on a mobile camera device offline without access to the Internet. More specifically, the present invention relates to systems and methods for displaying text of a first language and a translation of the first language text into a second language text which is displayed in real time in augmented reality on the mobile device. The processing can use a single line or a multiline algorithm designed with a plurality of processing innovations to insure accurate real-time translations without motion jitter. The invention may be used to help travelers in a foreign country with difficulties in reading and understanding text written in the local language of that country. The present invention may be utilized with wearable computers or glasses, producing seamless augmented reality foreign language translations. Some embodiments are particularly useful in translations from Asian languages to English. Some embodiments are especially optimized to translate multiple lines of foreign language text.

Claims

1. A method for providing one or more translations in a real-time video feed of a first language into a second language, comprising:

cropping a frame of the real-time video feed of one or more words of the first language to fit inside a bounding box to produce a cropped frame;

performing character segment detection on the cropped frame to produce a plurality character segments;

performing character merging on the character segments to produce a plurality of merged character segments while determining at least a shape score for at least one merged character segment;

performing character recognition on the merged character segments by utilizing at least the shape score of the at least one merged character segment to produce a plurality of recognized characters with high scores;

performing one or more translations on the recognized characters of the first language into one or more translated words of the second language; and

displaying the translated words of the second language.

2. The method of claim 1, wherein one line of the first language is being translated.

3. The method of claim 1, wherein two or more lines of the first language is being translated.

4. The method of claim 1, further comprising additional pre-processing steps before the step of performing the character segment detection, the pre-processing steps comprising:

deciding a threshold type and performing binarization on the cropped frame to produce a binary frame;

performing connected component analysis on the binary frame to produce analyzed connected components; and

de-noising analyzed connected components in the binary frame before performing the character segment detection on the binary frame.

5. The method of claim 4, further comprising:

deciding whether a text precursor is detected after performing the pre-processing steps on the cropped frame before proceeding with the step of character segment detection.

6. The method of claim 1, wherein the step of performing the character segment detection comprises additional steps of:

storing text precursors in a memory device;

performing merging of any two text precursors when the two text precursors are overlapping and/or neighboring;

examining the text precursors'"'"' character size; and

ending character segment detection for small text precursor characters.

7. The method of claim 1, wherein the step of performing the character merging further comprises:

performing vertical and/or horizontal merging on the character segments with recognition feedback to produce the plurality of merged character segments, while also calculating a recognition distance score in addition to the shape score for the at least one merged character segment.

8. The method of claim 1, wherein the step of performing the character recognition on the merged character segments comprises additional steps of:

setting a region of interest on the merged character segments;

normalizing the region of interest of the merged character segments to produce a normalized ROI;

performing feature extraction on the normalized ROI to produce extracted features;

performing dimensionality reduction on the extracted features to produce dimensionally reduced features; and

performing classification on the dimensionally reduced features.

9. The method of claim 1, wherein the step of performing the character recognition further comprises:

calculating a recognition distance score for the at least one merged character segment; and

combining the recognition distance score and the shape score into a combined score for the at least one merged character segment to produce a recognized character with a high combined score.

10. The method of claim 1, wherein the step of performing the one or more translations further comprises:

processing the recognized characters through a translation engine to produce the translation of the recognized characters in the first language into the one or more translated words of the second language, while calculating a translation engine score representing how many and/or how well characters have been translated for each translated line.

11. The method of claim 1, wherein the first language is selected from the group consisting of Chinese, Korean, Japanese, Vietnamese, Khmer, Lao, Thai, English, French, Spanish, German, Italian, Portuguese, Russian, Hindi, Greek, Hebrew, and Arabic.

12. The method of claim 1, wherein the first language is Chinese and the second language is English.

13. The method of claim 1, further comprising:

utilizing a conversion table for converting dialects of the first language into a smaller number of dialects of the first language before translating the first language into the second language.

14. The method of claim 1, further comprising:

utilizing a conversion table for converting traditional Chinese characters to simplified Chinese characters before translating the first language into the second language.

15. The method of claim 1, wherein the second language is selected from the group consisting of Chinese, Korean, Japanese, Vietnamese, Khmer, Lao, Thai, English, French, Spanish, German, Italian, Portuguese, Russian, Hindi, Greek, Hebrew, and Arabic.

16. The method of claim 1, further comprising:

moving the one or more translated words of the second language when a mobile device is moved without recalculating the one or more translations.

17. The method of claim 1, further comprising:

pausing the one or more translations which are displayed to allow a movement of a mobile device without changing displayed language translation.

18. The method of claim 1, further comprising:

storing a paused language translation frame comprising the first language and the second language in memory for later review.

19. The method of claim 1, further comprising:

displaying a phonetic pronunciation of the one or more words of the first language being translated.

20. The method of claim 1, wherein a translated line having a highest translation quality is selected for display, and wherein the translation quality is determined by how many of the one or more words of the first language are translated.

21. The method of claim 1, wherein a translated line having a highest translation quality is selected for display, and wherein the translation quality is determined by how well the one or more words of the first language are translated.

22. The method of claim 1, wherein a translated line having a highest translation quality is selected for display, and wherein the translation quality is determined by how many and how well the one or more words of the first language are translated.

23. The method of claim 1, wherein the step of displaying is performed in augmented reality on a mobile camera device.

24. A mobile device for providing one or more translations in a real-time video feed of a first language into a second language, the mobile device comprising:

a video camera for capturing the video feed of one or more words in the first language which need translation;

a display for displaying the words of the first language and one or more translated words of the second language;

a processor for processing program code; and

one or more memories operatively connected to the processor for storing the program code, which when executed by the processor causes the processor to execute a process to;

crop a frame of the real-time video feed of the words of the first language to fit inside a bounding box to produce a cropped frame;

perform character segment detection on the cropped frame to produce a plurality character segments;

perform character merging on the character segments to produce a plurality of merged character segments while determining at least a shape score for at least one merged character segment;

perform character recognition on the merged character segments by utilizing at least the shape score of the at least one merged character segment to produce a plurality of recognized characters with high scores;

perform one or more translations on the recognized characters of the first language into the one or more translated words of the second language; and

display the translated words of the second language.

25. The mobile device of claim 24, wherein the mobile device is selected from the group consisting of a smartphone, a tablet computer, a wearable computer, and a wearable eye glass.

26. The mobile device of claim 24, wherein the first language is selected from the group consisting of Chinese, Korean, Japanese, Vietnamese, Khmer, Lao, Thai, English, French, Spanish, German, Italian, Portuguese, Russian, Hindi, Greek, Hebrew, and Arabic.

27. The mobile device of claim 24, wherein the second language is selected from the group consisting of Chinese, Korean, Japanese, Vietnamese, Khmer, Lao, Thai, English, French, Spanish, German, Italian, Portuguese, Russian, Hindi, Greek, Hebrew, and Arabic.

28. A non-transitory, computer-readable digital storage medium for storing program code for translating a video feed in real-time from a first language to a second language, the program code comprising:

program code to crop a frame of the real-time video feed of one or more words of the first language to fit inside a bounding box to produce a cropped frame;

program code to perform character segment detection on the cropped frame to produce a plurality character segments;

program code to perform character merging on the character segments to produce a plurality of merged character segments while determining at least a shape score for at least one merged character segment;

program code to perform character recognition on the merged character segments by utilizing at least the shape score of the at least one merged character segment to produce a plurality of recognized characters with high scores;

program code to perform one or more translations on the recognized characters of the first language into one or more translated words of the second language; and

program code to display the translated words of the second language.

29. The storage medium of claim 28, wherein the first language is selected from the group consisting of Chinese, Korean, Japanese, Vietnamese, Khmer, Lao, Thai, English, French, Spanish, German, Italian, Portuguese, Russian, Hindi, Greek, Hebrew, and Arabic.

30. The storage medium of claim 28, wherein the second language is selected from the group consisting of Chinese, Korean, Japanese, Vietnamese, Khmer, Lao, Thai, English, French, Spanish, German, Italian, Portuguese, Russian, Hindi, Greek, Hebrew, and Arabic.