TEXT OVERLAY TECHNIQUES IN REALTIME TRANSLATION

• US 20140172408A1
• Filed: 12/14/2012
• Published: 06/19/2014
• Est. Priority Date: 12/14/2012
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Bhargava, Ruchita, Ashman, Matthew S., Stojiljkovic, Ivan, Nijemcevic, Djordje, Vugdelija, Milan, Vukosavljevic, Magdalena, Josipovic, Pavle
• Granted Patent: US 9,037,450 B2
• Time in Patent Office: Days
• US Class Current: 704/2
• CPC Class Codes: G06F 40/103 Formatting, i.e. changing of presentation of documents automatic justification G06F40/189; automatic line break hyphenation G06F40/191
• CPC Class Codes: G06F 40/58 Use of machine translation, e.g. for multi-lingual retrieval, for server-side translation for client devices or for real-time translation
• CPC Class Codes: G06V 20/63 Scene text, e.g. street names
• CPC Class Codes: G06V 30/10 Character recognition
• CPC Class Codes: G06V 30/1478 of characters or characters lines
• CPC Class Codes: G06V 30/2445 Alphabet recognition, e.g. Latin, Kanji or Katakana

Abstract

Architecture that employs techniques for overlaying (superimposing) translated text on top of (over) scanned text in realtime translation to provide clear visual correlation between original text and translated text. Algorithms are provided that overlay text in cases of translated scanned text of language written in first direction to a language written in same direction, translate scanned text from a first language written in a first direction to a second language written in the opposite direction, and translated scanned text from a language written in a first direction to language written in a different direction.

Claims

1. A system, comprising:

a recognition system that recognizes text in an image, the text of a first language written in a first direction;

a translator that translates the text of the first language into text of a second language;

an overlay algorithm that determines orientation and alignment between the text of the first language and the text of the second language, and overlays the text of the second language on the text of the first language as viewed in a display;

wherein the recognition system, the translator, and the overlay algorithm operate to overlay the text of the second language on top of the text of the first language in realtime for viewing; and

a microprocessor configured to execute computer-executable instructions associated with at least one of the recognition system, the translator, or the overlay algorithm.

2. The system of claim 1, wherein the recognition system, the translator, and the overlay algorithm operate to overlay the text of the second language on the text of the first language in the display in realtime using a video mode.

3. The system of claim 1, wherein the recognition system creates a first bounding box that circumscribes the text of the first language and a second bounding box that circumscribes the text of the second language, and the overlay algorithm aligns the second bounding box to the first bounding box as part of overlaying the text of the second language on the text of the first language.

4. The system of claim 3, wherein the overlay algorithm aligns the first and second bounding boxes to each other based on bounding box centers.

5. The system of claim 1, wherein the overlay algorithm computes a relationship of textual portions of the second language to related textual portions of the first language and overlays the related textual portions.

6. The system of claim 1, wherein the overlay algorithm obtains orientation information associated with the text of the first language in the image, the orientation information relates to writing vertically or horizontally.

7. The system of claim 1, wherein the overlay algorithm overlays the text of the second language on the text of the first language in a direction opposite from which the first language is written or in a same direction as the first language is written.

8. The system of claim 1, wherein the overlay algorithm overlays the text of the second language on the text of the first language in a direction orthogonal to the direction in which the first language is normally written and new lines are added.

9. A method, comprising acts of:

recognizing text in an image, the text of a first language written in a first direction;

translating the text of the first language into text of a second language;

aligning the text between the first language and the second language;

visually overlaying the text of the second language on top of the text of the first language, the text of the second language presented according to a direction in which the second language is written; and

configuring a processor to execute instructions related to at least one of the acts of recognizing, translating, aligning, or visually overlaying.

10. The method of claim 9, further comprising:

creating a first bounding box that circumscribes the text of the first language and a second bounding box that circumscribes the text of the second language; and

aligning the second bounding box to the first bounding box as part of overlaying the text of the second language on the text of the first language.

11. The method of claim 9, further comprising obtaining orientation information associated with the text in the image.

12. The method of claim 9, further comprising overlaying the text of the second language on the text of the first language in a line-by-line or word-to-word relationship.

13. The method of claim 9, further comprising overlaying the text of the second language on the text of the first language in a direction opposite from which the first language is written.

14. The method of claim 9, further comprising overlaying the text of the second language on the text of the first language in a same direction as the first language is written.

15. The method of claim 9, further comprising overlaying the text of the second language on the text of the first language in a direction orthogonal to the direction in which the first language is normally written.

16. The method of claim 9, further comprising performing the acts of recognizing, translating, aligning, and visually overlaying, in realtime during a video mode of a camera.

17. A method, comprising acts of:

recognizing text in an image using a camera operating in video mode and optical character recognition, the text of a first language written in a first direction;

creating a first bounding box that circumscribes the text of the first language and a second bounding box that circumscribes the text of the second language;

obtaining orientation information associated with the text in the image;

translating the text of the first language into text of a second language;

aligning and overlapping the second bounding box to the first bounding box as part of overlaying the text of the second language on the text of the first language;

overlaying the text of the second language on the text of the first language, the text of the second language presented according to a direction in which the second language is written;

performing the acts of recognizing, translating, aligning, and visually overlaying, in realtime during the video mode of the camera; and

configuring a processor to execute instructions related to at least one of the acts of recognizing, creating, obtaining, translating, aligning, overlaying, or performing.

18. The method of claim 17, further comprising overlaying the text of the second language on the text of the first language in a line-by-line relationship.

19. The method of claim 17, further comprising overlaying the text of the second language on the text of the first language in a direction opposite from which the first language is written or in a same direction as the first language is written.

20. The method of claim 17, further comprising overlaying the text of the second language on the text of the first language in a direction orthogonal to the direction in which the first language is normally written.