LANGUAGE TRANSLATION OF VISUAL AND AUDIO INPUT

• US 20130185052A1
• Filed: 12/28/2012
• Published: 07/18/2013
• Est. Priority Date: 03/29/2007
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Boyd, Jonathan J., Pathak, Binay K.
• Granted Patent: US 8,645,121 B2
• Time in Patent Office: Days
• US Class Current: 704/3
• CPC Class Codes: G06F 40/40 Processing or translation of natural language natural language analysis G06F40/20; semantic analysis G06F40/30
• CPC Class Codes: G06F 40/58 Use of machine translation, e.g. for multi-lingual retrieval, for server-side translation for client devices or for real-time translation

Abstract

The present translation system translates visual input and/or audio input from one language into another language. Some implementations incorporate a context-based translation that uses information obtained from visual input or audio input to aid in the translation of the other input. Other implementations combine the visual and audio translation. The translation system includes visual components and/or audio components. The visual components analyze visual input to identify a textual element and translate the textual element into a translated textual element. The visual image represents a captured image of a target scene. The visual components may further substitute the translated textual element for the textual element in the captured image. The audio components convert audio input into translated audio.

Claims

1. A method comprising:

receiving audio input;

receiving visual input; and

translating the audio input based upon a contextual hint determined based upon a non-textual element identified based upon the visual input.

2. The method of claim 1, the method performed at least in part by glasses.

3. The method of claim 1, the translating comprising translating the audio input into translated audio, the audio input associated with a first language and the translated audio associated with a second language different than the first language.

4. The method of claim 1, comprising extracting the non-textual element using a scale invariant feature transformation.

5. The method of claim 1, comprising analyzing the visual input to identify a textual element and translating the textual element.

6. The method of claim 1, comprising translating the audio input based upon a second contextual hint determined based upon a second non-textual element identified based upon received second audio input instead of the contextual hint.

7. The method of claim 1, comprising translating the audio input based upon a second contextual hint determined based upon a second non-textual element identified based upon received second audio input.

8. A system comprising:

a component configured to;

receive audio input;

receive visual input; and

translate the audio input based upon a contextual hint determined based upon a non-textual element identified based upon the visual input.

9. The system of claim 8, the system comprised at least partially in glasses.

10. The system of claim 8, the translating comprising translating the audio input into translated audio, the audio input associated with a first language and the translated audio associated with a second language different than the first language.

11. The system of claim 8, the component configured to extract the non-textual element using a scale invariant feature transformation.

12. The system of claim 8, the component configured to analyze the visual input to identify a textual element.

13. The system of claim 12, the component configured to translate the textual element.

14. The system of claim 8, the visual input corresponding to a target scene.

15. A computer-readable medium comprising instructions which when executed perform actions, comprising:

receiving audio input;

receiving visual input;

analyzing the visual input to identify a non-textual element; and

translating the audio input based upon a contextual hint determined based upon the non-textual element.

16. The computer-readable medium of claim 15, the actions performed at least in part by glasses.

17. The computer-readable medium of claim 15, the translating comprising translating the audio input into translated audio, the audio input associated with a first language and the translated audio associated with a second language different than the first language.

18. The computer-readable medium of claim 15, the actions comprising extracting the non-textual element using a scale invariant feature transformation.

19. The computer-readable medium of claim 15, the actions comprising analyzing the visual input to identify a textual element.

20. The computer-readable medium of claim 19, the actions comprising translating the textual element.