INTELLIGENT TRANSLATIONS IN PERSONAL SEE THROUGH DISPLAY

• US 20130293577A1
• Filed: 05/04/2012
• Published: 11/07/2013
• Est. Priority Date: 05/04/2012
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Perez, Kathryn Stone, Clavin, John, Geisner, Kevin A., Latta, Stephen G., Mount, Brian J., Tomlin, Arthur C., Poulos, Adam G.
• Granted Patent: US 9,519,640 B2
• Time in Patent Office: Days
• US Class Current: 345/633
• CPC Class Codes: G02B 2027/014 comprising information/image processing systems
• CPC Class Codes: G02B 2027/0178 Eyeglass type eyeglass details G02C
• CPC Class Codes: G02B 2027/0187 slaved to motion of at least a part of the body of the user, e.g. head, eye
• CPC Class Codes: G02B 27/017 Head mounted
• CPC Class Codes: G06F 1/163 Wearable computers, e.g. on a belt
• CPC Class Codes: G06F 40/58 Use of machine translation, e.g. for multi-lingual retrieval, for server-side translation for client devices or for real-time translation
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00

Abstract

A see-through, near-eye, mixed reality display apparatus for providing translations of real world data for a user. A wearer'"'"'s location and orientation with the apparatus is determined and input data for translation is selected using sensors of the apparatus. Input data can be audio or visual in nature, and selected by reference to the gaze of a wearer. The input data is translated for the user relative to user profile information bearing on accuracy of a translation and determining from the input data whether a linguistic translation, knowledge addition translation or context translation is useful.

Claims

1. A method for presenting a translation of a real world expression to a wearer of a see through head mounted display apparatus, comprising:

determining a gaze of a wearer looking through the see-through display of the apparatus;

determining a three dimensional location of one or more objects in the field of view of the user through the see-through display, the determining of the three dimensional location of the object is performed using the one or more sensors;

receiving a selection of data for translation in the field of view of the wearer by reference to the gaze of the wearer at one of the objects;

analyzing the data for translation to provide input data;

translating the input data into a translated form for the user; and

rendering the translation in an audio or visual format in the see through head mounted display.

2. The method of claim 1 further including accessing a user profile for user information bearing on accuracy of the translation and wherein translating the data comprises evaluating input data for translation against the user information.

3. The method of claim 1 wherein the step of translating comprises converting the input data from a first language to a second language.

4. The method of claim 1 wherein the step of translating comprises providing supplemental knowledge for the input data on a subject matter identified in the input data.

5. The method of claim 4 wherein the supplemental knowledge includes a selectable link for the user to provide additional supplemental knowledge on the subject matter.

6. The method of claim 1 wherein the step of translating comprises evaluating input data in the form of a person and a context, determining a location of the apparatus, and providing contextual information based on at least the location and the input data.

7. The method of claim 2 further including the step of determining one or more upcoming translation events based on user profile information and preparing translation information in advance of the event.

8. The method of claim 1 wherein the sensors include audio sensors and that step of receiving a selection of data includes receiving audio data and analyzing the data including isolating audio data based on the gaze to provide the input data.

9. The method of claim 1 further including a step of providing input data from a wearer in written or audio form;

performing the receiving, analyzing, translating and rendering steps; and

receiving from the wearer a corrected translation of the training data.

10. A see through head mounted display apparatus presenting translations of input data to a wearer, comprising:

a see-through, near-eye, augmented reality display that is worn by a wearer, the apparatus includes one or more input sensors;

one or more processing devices in communication with apparatus, the one or more processing devices automatically determine that the wearer is at a location, the one or more processing devices access a wearer profile for the wearer and identify one or more characteristics of the wearer that match characteristics affecting translation of input data, the one or more processing devices determine input data from real world objects and persons in the field of view of the wearer and provide a translation the input data to provide contextual data, knowledge augmentation or a linguistic translation on the input data;

the translation based on known characteristics of the wearer used to improve the translation, the translation rendered by the one or more processors in the see-though near eye display apparatus by the one or more processors.

11. The apparatus of claim 10 wherein the one or more processors may provide a translation wherein:

the input data comprises audio data and the translation is rendered in an audio format;

the input data comprises audio data and the translation is rendered in an visual format;

the input data comprises visual data and the translation is rendered in an audio format; and

the input data comprises visual data and the translation is rendered in an visual format.

12. The apparatus of claim 10 wherein the apparatus includes one or more camera sensors and the input data is retrieved from data provided by the camera sensor, the data including written language information provided as input data, or image data provided as input data.

13. The apparatus of claim 12 wherein the one or more processors perform optical character recognition on input data to determine written information for translation.

14. The apparatus of claim 12 wherein the one or more processors render supplemental knowledge on the input data on a subject matter identified in the input data for a knowledge translation.

15. The apparatus of claim 12 wherein the supplemental knowledge includes a selectable link for the user to provide additional supplemental knowledge on the subject matter.

16. The method of claim 10 wherein the one or more processors evaluate input data in the form of a person and a context, and provide contextual information on the person for a contextual data translation.

17. A method for presenting a translation of information at a location to a wearer of a see through head mounted display apparatus, comprising:

receiving from the wearer a selection of input data for translation at the location of a user by reference to the gaze of a user, including determining a gaze of a wearer looking through the see-through display apparatus, the apparatus includes one or more sensors and a see-through display;

determining three dimensional locations of objects within a field of view of the wearer at the location;

translating the input data into a translated form for the user, the translating including

retrieving user profile information bearing on accuracy of a translation;

determining from the input data whether a linguistic translation, knowledge addition translation or context translation is useful;

performing one or more of a linguistic translation, knowledge addition translation or context translation on the input data; and

rendering the translation in an audio or visual format in the see through head mounted display.

18. The method of claim 17 wherein a knowledge addition translation includes determining subject matter in the input data, retrieving supplementation information on the subject matter;

determining user knowledge of the subject matter from the user profile data;

determining an appropriate complexity of supplemental knowledge information and outputting the information.

19. The method of claim 17 wherein a context translation includes determining bodily characteristics of a person in the field of view of the wearer, determining the context of the person in the field of view of the wearer;

and comparing the bodily characteristics and context to known characteristics and contexts to provide an emotional translation of the person in the field of view of the wearer.

20. The method of claim 17 further including a step of rendering the translation output to the wearer and receiving from the wearer a corrected translation of the input data.