SYSTEM AND METHOD FOR COMBINING THREE-DIMENSIONAL TRACKING WITH A THREE-DIMENSIONAL DISPLAY FOR A USER INTERFACE

• US 20130265220A1
• Filed: 04/04/2013
• Published: 10/10/2013
• Est. Priority Date: 04/09/2012
• Status: Active Grant

• Current Assignee: Intel Corporation
• Original Assignee: Omek Interactive Ltd. (Intel Corporation)
• Inventors: Kutliroff, Gershom, Yanai, Yaron, Fleischmann, Shahar
• Granted Patent: US 9,477,303 B2
• Time in Patent Office: Days
• US Class Current: 345/156
• CPC Class Codes: G06F 3/005 Input arrangements through a video camera
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/04815 Interaction with a metaphor-based environment or interaction object displayed as three-dimensional, e.g. changing the user viewpoint with respect to the environment or object
• CPC Class Codes: G06F 3/04842 Selection of displayed objects or displayed text elements G06F3/0482 takes precedence
• CPC Class Codes: G06F 3/04883 for inputting data by handwriting, e.g. gesture or text
• CPC Class Codes: G06T 13/40 of characters, e.g. humans, animals or virtual beings
• CPC Class Codes: G06T 17/00 Three dimensional [3D] modelling, e.g. data description of 3D objects
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G06V 40/28 Recognition of hand or arm movements, e.g. recognition of deaf sign language static hand signs G06V40/113

Abstract

Systems and methods for combining three-dimensional tracking of a user'"'"'s movements with a three-dimensional user interface display is described. A tracking module processes depth data of a user performing movements, for example, movements of the user'"'"'s hand and fingers. The tracked movements are used to animate a representation of the hand and fingers, and the animated representation is displayed to the user using a three-dimensional display. Also displayed are one or more virtual objects with which the user can interact. In some embodiments, the interaction of the user with the virtual objects controls an electronic device.

Claims

1. A method comprising:

acquiring depth data of a subject with a depth sensor;

tracking the subject'"'"'s movements using the acquired depth data;

causing to be displayed in a three-dimensional display the subject'"'"'s movements.

2. The method of claim 1, wherein the acquired depth data of the subject includes at least one of the subject'"'"'s hands.

3. The method of claim 2, wherein movements of the at least one of the subject'"'"'s hands are mapped to a first virtual object, and corresponding movements of the first virtual object are shown in the three-dimensional display.

4. The method of claim 1, further comprising causing to be displayed in the three-dimensional display a second virtual object, wherein the subject interacts with the second virtual object through the subject'"'"'s movements, and further wherein interaction of the subject with the second virtual object permits the subject to interact with an electronic device.

5. The method of claim 4, wherein at least some of the subject'"'"'s movements occlude the second virtual object in the three-dimensional display.

6. The method of claim 1, wherein tracking the subject'"'"'s movements comprises:

identifying a plurality of features of the subject in the acquired depth data;

obtaining three-dimensional positions corresponding to the identified plurality of features from the depth data;

constructing a model from the three-dimensional positions.

7. The method of claim 6, wherein tracking the subject'"'"'s movements further comprises classifying the subject'"'"'s movements by qualitatively comparing the subject'"'"'s movements to known gestures.

8. The method of claim 6, wherein the identified plurality of features are a plurality of joints of the subject'"'"'s hand.

9. The method of claim 1, wherein the three-dimensional display is generated by stereoscopy.

10. The method of claim 1, wherein the three-dimensional display is generated by holography.

11. The method of claim 1, wherein the three-dimensional display is a head-mounted display.

12. The method of claim 1, further comprising causing to be displayed a semi-transparent wall in front of a screen of the three-dimensional display, wherein when portions of the subject'"'"'s body intersect the wall, indications of relative locations of intersections with the wall are caused to be displayed.

13. The method of claim 1, further comprising changing a depth of field of the three-dimensional display based on a distance of a tracked body part of the subject from a screen of the three-dimensional display.

14. The method of claim 1, further comprising changing a depth of field of the three-dimensional display based on a distance between fingertips of a tracked hand of the subject from a palm of the hand.

15. The method of claim 1, further comprising dynamically rendering a virtual object caused to be displayed in the three-dimensional display to be transparent or visible, wherein the virtual object is rendered to be transparent when an average virtual distance between the virtual object and a screen of the three-dimensional display is greater than a distance between a hand of the subject and the screen, and further wherein the virtual object is rendered to be visible when the average virtual distance between the virtual object and the screen of the three-dimensional display is less than the distance between the hand of the subject and the screen.

16. The method of claim 1, further comprising causing to be displayed a user interface element as a projection from a screen of the three-dimensional display toward the subject, wherein a perceived distance of the projection of the user interface element is set based on a value of a parameter associated with the user interface element.

17. The method of claim 16, wherein when the subject'"'"'s movement corresponds to pushing the user interface element, the parameter associated with the user interface element is changed in a first direction, and when the subject'"'"'s movement corresponds to pulling the user interface element, the parameter associated with the user interface element is changed in an opposite direction.

18. A system comprising:

a sensing module configured to acquire a sequence of depth images of one or more body parts of a subject;

a tracking module configured to track movements of the one or more body parts in the sequence of depth images;

a display configured to provide feedback images perceived as three-dimensional to the subject,

wherein the feedback images include a first virtual object animated with movements corresponding to the tracked movements of the one or more body parts, and

further wherein the feedback images include as second virtual object with which the subject can interact.

19. The system of claim 18, wherein the feedback images comprise a user interface configured to allow the subject to control an electronic device based upon interactions of the subject with the second virtual object.

20. The system of claim 18, wherein the one or more body parts is a hand and fingers of the hand, and wherein tracking movements of the hand and fingers comprises identifying features of the hand and fingers in the depth images;

obtaining three-dimensional positions corresponding to the identified features; and

constructing a model form the three-dimensional positions.

21. A system comprising:

means for acquiring depth data of a subject with a depth sensor;

a tracking module configured to track the subject'"'"'s movements using the acquired depth data;

means for providing a display of the subject'"'"'s movements and a virtual object, wherein the display is perceived to be three-dimensional to the subject, and further wherein the subject'"'"'s movements relative to the virtual object permit the subject to control an electronic device.