Language translation of visual and audio input

• US 8,515,728 B2
• Filed: 03/29/2007
• Issued: 08/20/2013
• Est. Priority Date: 03/29/2007
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Boyd, Jonathan J., Pathak, Binay K.
• Primary Examiner(s): SPOONER, LAMONT M
• Application Number:
  US11/731,282
• Publication Number:
  US 20080243473A1
• Time in Patent Office: 2,336 Days
• Field of Search: 704 2- 8, 704/231, 704/251, 715/264
• US Class Current: 704/2
• CPC Class Codes: G06F 40/40 Processing or translation of natural language natural language analysis G06F40/20; semantic analysis G06F40/30
• CPC Class Codes: G06F 40/58 Use of machine translation, e.g. for multi-lingual retrieval, for server-side translation for client devices or for real-time translation

Abstract

The present translation system translates visual input and/or audio input from one language into another language. Some implementations incorporate a context-based translation that uses information obtained from visual input or audio input to aid in the translation of the other input. Other implementations combine the visual and audio translation. The translation system includes visual components and/or audio components. The visual components analyze visual input to identify a textual element and translate the textual element into a translated textual element. The visual image represents a captured image of a target scene. The visual components may further substitute the translated textual element for the textual element in the captured image. The audio components convert audio input into translated audio.

Claims

1. Glasses configured to translate, comprising:

a language selection component configured to determine at least one of a first language or a second language based at least in part on at least one of a selection by a user or a logic for automatically determining a language;

a visual capture component configured to receive visual input of a target scene, the visual input originating from at least one of streaming video or captured video;

a visual analysis component configured to analyze, using neural network based optical character recognition, the visual input to identify one or more locations within the visual input that comprise a textual element associated with the first language;

a text translator component configured to translate the textual element into a translated textual element associated with the second language based at least in part on a first contextual hint determined based at least in part on an audio input associated with the first language;

a visual rendering component configured to;

at least one of substitute the translated textual element for the textual element in an image of the target scene or add the translated textual element to the textual element in the image; and

display the image of the target scene comprising at least one of the substituted or added translated textual element to a user wearing the glasses; and

an audio capture component that is voice-activated and is configured to;

receive the audio input associated with the first language;

translate the audio input into translated audio associated with the second language based at least in part on;

hidden Markov model based speech synthesis;

one or more pauses comprised within the audio input;

a sentence structure associated with at least some of the audio input;

a number of syllables of the audio input; and

a second contextual hint determined based at least in part on the visual input; and

play the translated audio to the user via a speaker.

2. The glasses of claim 1, the audio capture component comprising a microphone.

3. The glasses of claim 2, the speaker comprising an earphone operatively coupled to the glasses.

4. The glasses of claim 1, the speaker comprising an earphone operatively coupled to the glasses.

5. The glasses of claim 1, the visual rendering component configured to display the translated textual element at a location corresponding to the textual element in the received visual input.

6. The glasses of claim 1, at least some of at least one of the visual capture component, the visual analysis component, the text translator component, or and the visual rendering component implemented at least in part via a remote computing component operatively coupled to the glasses.

7. The glasses of claim 6, the remote computing component coupled to the glasses via a network.

8. The glasses of claim 1, the visual capture component configured to receive visual input from streaming media.

9. The glasses of claim 1, the audio capture component configured to capture audio input from a spoken communication.

10. The glasses of claim 1, the visual rendering component configured to display the translated textual element on a bottom of a display associated with the glasses.

11. A tangible computer-readable storage device comprising computer-executable instructions that when executed, perform a method comprising:

determining at least one of a first language or a second language based at least in part on at least one of a selection by a user or a logic for automatically determining a language;

receiving visual input of a target scene, the visual input originating from at least one of streaming video, a digital photograph or captured video;

analyzing, using neural network based optical character recognition, the visual input to identify one or more locations within the visual input that comprise a textual element associated with the first language;

translating the textual element into a translated textual element associated with the second language based at least in part on a first contextual hint determined based at least in part on an audio stream; and

translating the audio stream into a translated audio stream associated with the second language based at least in part on;

hidden Markov model based speech synthesis;

one or more pauses comprised within the audio stream;

a sentence structure associated with at least some of the audio stream;

a number of syllables of the audio stream; and

a second contextual hint determined based at least in part on the visual input.

12. The tangible computer-readable storage device of claim 11, comprising translating the audio stream based at least in part on a transcribed audio segment.

13. The tangible computer-readable storage device of claim 12, at least one of the transcribed audio segment or the translated audio stream providing the first contextual hint.

14. The tangible computer-readable storage device of claim 11, the visual input associated with one or more file types.

15. The tangible computer-readable storage device of claim 11, the method performed by a kiosk-type device.

16. The tangible computer-readable storage device of claim 15, the kiosk-type device configured to scan a printed version of the digital photograph.

17. The tangible computer-readable storage device of claim 11, comprising:

substituting the translated textual element for the textual element in an image of the visual input.

18. A kiosk configured to perform a computer-implemented method, comprising:

determining at least one of a first language or a second language based at least in part on at least one of a selection by a user or a logic for automatically determining a language;

receiving visual input of a target scene, the visual input representative of a captured image of the target scene, the visual input originating from a printed picture that is scanned;

analyzing, using neural network based optical character recognition, the visual input to identify one or more locations within the visual input that comprise a textual element associated with the first language;

translating the textual element into a translated textual element associated with the second language based at least in part on a first contextual hint determined based at least in part on an audio stream; and

translating the audio stream into a transcribed audio segment based at least in part on;

hidden Markov model based speech synthesis;

one or more pauses comprised within the audio stream;

a sentence structure associated with at least some of the audio stream;

a number of syllables of the audio stream; and

a second contextual hint determined based at least in part on the visual input.

19. The kiosk of claim 18, the method comprising scanning, by the kiosk, the digital picture.

20. The kiosk of claim 18, the method comprising receiving, by a microphone of the kiosk, the audio stream.