Text overlay techniques in realtime translation

• US 9,690,782 B2
• Filed: 05/02/2015
• Issued: 06/27/2017
• Est. Priority Date: 12/14/2012
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Nijemcevic, Djordje, Vugdelija, Milan, Stojiljkovic, Ivan, Ashman, Matthew S., Josipovi, Pavle, Bhargava, Ruchita, Vukosavljevic, Magdalena
• Primary Examiner(s): MCFADDEN, SUSAN IRIS
• Application Number:
  US14/702,691
• Publication Number:
  US 20150234812A1
• Time in Patent Office: 787 Days
• Field of Search: 704 2
• CPC Class Codes: G06F 40/103 Formatting, i.e. changing of presentation of documents automatic justification G06F40/189; automatic line break hyphenation G06F40/191
• CPC Class Codes: G06F 40/58 Use of machine translation, e.g. for multi-lingual retrieval, for server-side translation for client devices or for real-time translation
• CPC Class Codes: G06V 20/63 Scene text, e.g. street names
• CPC Class Codes: G06V 30/10 Character recognition
• CPC Class Codes: G06V 30/1478 of characters or characters lines
• CPC Class Codes: G06V 30/2445 Alphabet recognition, e.g. Latin, Kanji or Katakana

Abstract

The technology described herein employs techniques for overlaying (superimposing) translated text on top of (over) scanned text in realtime. The technology recognizes text in an image and identifies a first language in which the text is written. The technology translates the text into a second language and generates an overlay in the second language. The overlay is then inserted into the display over the text in the image.

Claims

1. A system, comprising:

a recognition system configured to recognize text in an image, the text of a first language;

a translator configured to translate the text of the first language into text of a second language;

an overlay algorithm configured to determine orientation and alignment between the text of the first language and the text of the second language, and to overlay the text of the second language on the text of the first language, as viewed in a display; and

a microprocessor configured to execute computer-executable instructions in a memory, the instructions executed to enable the recognition system the translator, and the overlay algorithm.

2. The system of claim 1, wherein the recognition system, the translator, and the overlay algorithm operate to overlay the text of the second language on the text of the first language in the display, in realtime.

3. The system of claim 1, wherein the recognition system, the translator, and the overlay algorithm enable application of graphical emphasis to visually differentiate the text of the second language from the text of the first language.

4. The system of claim 1, wherein the recognition system is configured to create a first bounding box that circumscribes the text of the first language and a second bounding box that circumscribes the text of the second language, and the overlay algorithm is configured to align the second bounding box to the first bounding box as part of overlaying the text of the second language on the text of the first language.

5. The system of claim 1, wherein the second language is user selectable and direction of the text of the second language is user configurable.

6. The system of claim 1, wherein the overlay algorithm is configured to compute a relationship of textual portions of the second language to related textual portions of the first language, and to overlay the text of the second language on the text of the first language based on the relationship.

7. The system of claim 1, wherein the overlay algorithm is configured to obtain orientation information associated with the text of the first language in the image, the orientation information relates to overlaying the text of the second language vertically or horizontally.

8. The system of claim 1, wherein the overlay algorithm overlays the text of the second language on the text of the first language in a direction at least one of opposite from which the first language is written, orthogonal to the direction in which the first language is normally written, or in a same direction as the first language is written.

9. A method, comprising acts of:

recognizing text of a first language in an image;

translating the text of the first language into text of a second language; and

visually overlaying the text of the second language on the text of the first language, as viewed on the image in the display.

10. The method of claim 9, further comprising:

creating a first bounding box that circumscribes the text of the first language and a second bounding box that circumscribes the text of the second language; and

aligning the second bounding box to the first bounding box as part of overlaying the text of the second language on the text of the first language.

11. The method of claim 9, further comprising enabling user selection of the first language to be translated and the second language to be translated for overlay on the text of the first language.

12. The method of claim 9, further comprising overlaying the text of the second language on the text of the first language in a line-by-line or word-to-word relationship.

13. The method of claim 9, further comprising overlaying the text of the second language on the text of the first language in a direction opposite from which the first language is written, in a direction orthogonal to the direction in which the first language is normally written, or in a same direction as the first language is written.

14. The method of claim 9, further comprising enabling toggling between the text of the second language and the text of the first language.

15. The method of claim 9, further comprising enabling tagging of objects of the first language in the image and, translation and overlaying of text of the second language on the tagged objects.

16. The method of claim 9, further comprising performing the acts of recognizing, translating, and visually overlaying, in realtime using a video mode of a camera.

17. A method, comprising acts of:

recognizing text in an image in realtime, using a camera operating in video mode and optical character recognition, the text of a first language;

translating the text of the first language into the text of the second language; and

overlaying the text of the second language on the text of the first language, the text of the second language presented in a direction opposite from which the first language is written, in a direction orthogonal to the direction in which the first language is normally written, or in a same direction as the first language is written.

18. The method of claim 17, further comprising enabling selection of the first language and the second language, and configuration of a direction of the second language.

19. The method of claim 17, further comprising enabling toggling between the text of the second language and the text of the first language in a foreground and background way.

20. The method of claim 17, further comprising recognizing and translating a hyperlink of the first language, and overlaying a hyperlink of the second language on the hyperlink of the first language.