SENSORY EYEWEAR

• US 20180075659A1
• Filed: 09/12/2017
• Published: 03/15/2018
• Est. Priority Date: 09/13/2016
• Status: Active Grant

• Current Assignee: Magic Leap, Inc.
• Original Assignee: Magic Leap, Inc.
• Inventors: Browy, Eric, Woods, Michael Janusz, Rabinovich, Andrew
• Granted Patent: US 10,580,213 B2
• Time in Patent Office: Days
• CPC Class Codes: G02B 2027/0138 comprising image capture systems, e.g. camera
• CPC Class Codes: G02B 2027/014 comprising information/image processing systems
• CPC Class Codes: G02B 2027/0141 characterised by the informative content of the display
• CPC Class Codes: G02B 2027/0178 Eyeglass type eyeglass details G02C
• CPC Class Codes: G02B 2027/0187 slaved to motion of at least a part of the body of the user, e.g. head, eye
• CPC Class Codes: G02B 27/017 Head mounted
• CPC Class Codes: G02B 27/0172 characterised by optical features
• CPC Class Codes: G02B 27/0179 Display position adjusting means not related to the information to be displayed
• CPC Class Codes: G06F 1/163 Wearable computers, e.g. on a belt
• CPC Class Codes: G06F 1/1686 the I/O peripheral being an integrated camera
• CPC Class Codes: G06F 1/1688 the I/O peripheral being integrated loudspeakers
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/013 Eye tracking input arrangements G06F3/015 takes precedence
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0304 Detection arrangements using opto-electronic means constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042
• CPC Class Codes: G06F 3/16 Sound input; Sound output speech processing G10L
• CPC Class Codes: G06F 40/58 Use of machine translation, e.g. for multi-lingual retrieval, for server-side translation for client devices or for real-time translation
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G06T 2207/20084 Artificial neural networks [ANN]
• CPC Class Codes: G06T 7/70 Determining position or orientation of objects or cameras camera calibration G06T7/80
• CPC Class Codes: G06V 20/20 : in augmented reality scenes
• CPC Class Codes: G06V 30/10 : Character recognition
• CPC Class Codes: G06V 40/28 : Recognition of hand or arm movements, e.g. recognition of deaf sign language static hand signs G06V40/113

Abstract

A sensory eyewear system for a mixed reality device can facilitate user'"'"'s interactions with the other people or with the environment. As one example, the sensory eyewear system can recognize and interpret a sign language, and present the translated information to a user of the mixed reality device. The wearable system can also recognize text in the user'"'"'s environment, modify the text (e.g., by changing the content or display characteristics of the text), and render the modified text to occlude the original text.

Claims

1. A wearable system for sign language recognition, the wearable system comprising:

a head-mounted display configured to present virtual content to a user;

an imaging system configured to image an environment of the user; and

a hardware processor in communication with the head-mounted display and the imaging system, and programmed to;

receive an image captured by the imaging system;

detect a gesture in the image with an object recognizer;

recognize a meaning of the gesture in a sign language;

identify a target language based on contextual information associated with the user;

translate the gesture into the target language based on the recognized meaning;

generate virtual content based at least partly on a translation of the gesture into the target language; and

cause the head-mounted display to render the virtual content to the user.

2. The wearable system of claim 1, wherein the imaging system comprises one or more of wide-angle cameras configured to image a surrounding of the user.

3. The wearable system of claim 1, where in the hardware processor is further programmed to access auxiliary information associated with the gesture;

and where the virtual content rendered by the head-mounted display comprises the auxiliary information.

4. The wearable system of claim 1, wherein to identify a target language based on contextual information associated with the user, the hardware processor is programmed to:

set the target language as a language understood by the user based on at least one of;

the user'"'"'s speech as captured by the wearable system, the user'"'"'s location, or an input from the user selecting the language as the target language.

5. The wearable system of claim 1, wherein the hardware processor is programmed to determine whether the target language is a spoken language;

and in response to a determination that the target language is a spoken language, to play an audio stream of a speech associated with the translated gesture in the target language.

6. The wearable system of claim 1, wherein the hardware processor is programmed to determine whether the target language is another sign language;

and in response to a determination that the target language is another sign language, present a graphic of another gesture in the other sign language as the translation of the gesture.

7. The wearable system of claim 1, wherein to recognize the meaning of the gesture in the sign language, the hardware processor is programmed to apply a deep neutral network technique on at a portion of the image captured by the imaging system.

8. The wearable system of claim 1, wherein the hardware processor is further programmed to identify the sign language from a list of candidate sign language based at least partly on a location of the user.

9. The wearable system of claim 1, wherein to translate the gesture into the target language based on the recognized meaning, the hardware processor is programmed to convert the gesture to a text expression in the target language.

10. The wearable system of claim 1, wherein the hardware processor is programmed to determine a source of the detected gesture;

and upon determining the source of the detected gesture to be the user of the wearable system, communicate the translation of the gesture in the target language to a wearable device of another user.

11. The wearable system of claim 1, wherein the hardware processor is programmed to detect a condition from dismissing the virtual content from display by the head-mounted display, and remove the virtual content from the display by the head-mounted display in response to a detection of the condition.

12. The wearable system of claim 11, wherein the condition comprises at least one of:

a duration of time, a user'"'"'s hand gesture, or an input from a user input device.

13. The wearable system of claim 1, wherein the image comprises one or more frames of a video.

14. A method for sign language recognition, the method comprising:

receiving an image captured by an imaging system;

analyzing the image to detect a gesture of a user;

detecting a presence of a communication in a sign language based at least partly on the detected gesture;

recognizing a meaning of the gesture in the sign language;

identifying a target language to which the gesture will be translated into;

translating the gesture into the target language based on the recognized meaning;

generating virtual content based at least partly on a translation of the gesture into the target language; and

causing a head-mounted display to render the virtual content to the user.

15. The method of claim 14, wherein the image is received from a first wearable device configured to present mixed reality content while the virtual content is communicated to a second wearable device for rendering, wherein the first wearable device and the second wearable device are configured to present mixed reality content to the user.

16. The method of claim 14, wherein translating the gesture into the target language based on the recognized meaning comprises converting the gesture to a text expression in the target language.

17. The method of claim 14, wherein the virtual content comprises a text expression in the target language or a graphic illustrating another in the target language.

18. The method of claim 14, wherein recognizing the meaning of the gesture in the sign language comprises applying a deep neutral network technique on at a portion of the image captured by the imaging system.

19. The method of claim 14, wherein detecting the presence of the communication in the sign language comprises:

identifying the sign language from a list of candidate sign languages; and

determining the detected gesture correspond to an expression in the sign language.

20. The method of claim 19, wherein determining the detected gesture correspond to an expression in the sign language comprises analyzing the gesture in connection with the lip movements of a person making the gesture and audio data captures while the user is making the gesture.