System and method for close-range movement tracking

• US 9,910,498 B2
• Filed: 06/25/2012
• Issued: 03/06/2018
• Est. Priority Date: 06/23/2011
• Status: Active Grant

• Current Assignee: Intel Corporation
• Original Assignee: Intel Corporation
• Inventors: Kutliroff, Gershom, Yanai, Yaron, Bleiweiss, Amit, Fleishman, Shahar, Livny, Yotam, Epstein, Jonathan
• Primary Examiner(s): Wilson, Nicholas R
• Application Number:
  US13/532,609
• Publication Number:
  US 20120327125A1
• Time in Patent Office: 2,080 Days
• Field of Search: None
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0304 Detection arrangements using opto-electronic means constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042
• CPC Class Codes: G06F 3/04815 Interaction with a metaphor-based environment or interaction object displayed as three-dimensional, e.g. changing the user viewpoint with respect to the environment or object
• CPC Class Codes: G06F 3/04842 Selection of displayed objects or displayed text elements G06F3/0482 takes precedence
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00

Abstract

A system and method for close range object tracking are described. Close range depth images of a user'"'"'s hands and fingers or other objects are acquired using a depth sensor. Using depth image data obtained from the depth sensor, movements of the user'"'"'s hands and fingers or other objects are identified and tracked, thus permitting the user to interact with an object displayed on a screen, by using the positions and movements of his hands and fingers or other objects.

Claims

1. A method for operating a user interface comprising:

acquiring close range depth images of a user'"'"'s hand with a depth sensor;

constructing a skeletal model of the user'"'"'s hand based on the acquired depth images, the skeletal model including one or more designated zones of joints;

identifying from the acquired depth images movement within the one or more designated zones of joints in the skeletal model of the user'"'"'s hand, the movement including movement of fingers of the user'"'"'s hand within the one or more designated zones of joints based on the constructed skeletal model of the user'"'"'s hand;

tracking the movement of fingers of the user'"'"'s hand within the one or more designated zones of joints in the skeletal model of the user'"'"'s hand using a 3D joint data corresponding to a position of a joint within the one or more designated zones of joints in the skeletal mode of the user'"'"'s hand;

mapping the 3D joint data corresponding to the position of the joint in the constructed skeletal model to project the user'"'"'s hand on a screen as a first object performing a gesture corresponding to the movement of fingers of the user'"'"'s hand within the designated zones of joints in the skeletal model of the user'"'"'s hand; and

wherein the fingers on the user'"'"'s hand are represented as individual virtual cursor components of the first object, each individual virtual cursor component configured to;

simultaneously interact with one or more separate objects on the screen, and have a different function for interacting with a same object of the one or more separate objects on the screen,

including;

an index finger on the user'"'"'s hand having a first function to open the same object,

a middle finger on the user'"'"'s hand having a second function to erase the same object, and

a ring finger on the user'"'"'s hand having a third function to maximize the same object.

2. The method of claim 1, wherein performing the gesture corresponding to the movement of fingers of the user'"'"'s hand within the designated zones of joints in the skeletal model of the user'"'"'s hand generates a force that interacts with the one or more of the separate objects on the screen.

3. The method of claim 1, wherein identified movement of fingers of the user'"'"'s hand within the designated zones of joints in the skeletal model of the user'"'"'s hand corresponds to a selection gesture, wherein the first object on the screen selects a second object on the screen.

4. The method of claim 1, wherein identified movement of fingers of the user'"'"'s hand within the designated zones of joints in the skeletal model of the user'"'"'s hand corresponds to a manipulation gesture, wherein a second object on the screen is manipulated according to a predefined action associated with the manipulation gesture.

5. The method of claim 1, further comprising:

determining a distance of the user'"'"'s hand from the screen;

zooming the screen in and out based on changes in the distance.

6. The method of claim 5, further comprising moving a cursor on the screen to indicate a selected object from a stack of objects on the screen, wherein the cursor is moved based upon the distance of the user'"'"'s hand from the screen, wherein moving the cursor comprises moving the cursor towards a bottom of the stack of objects as the user'"'"'s hand moves closer to the screen and moving the cursor towards a top of the stack of objects as the user'"'"'s hand moves farther from the screen.

7. The method of claim 1, further comprising scaling a size of the first object based on the distance of the user'"'"'s hand from the screen.

8. A system comprising:

a depth sensor to acquire at close range depth images of a user'"'"'s hand;

a processor to execute;

a tracking module to construct a skeletal model of the user'"'"'s hand based on the acquired depth images, the skeletal model including one or more designated zones of joints, identify from the acquired depth images movement within the one or more designated zones of joints in the skeletal model of the user'"'"'s hand, the movement including movement of fingers of the user'"'"'s hand within the one or more designated zones of joints based on the constructed skeletal model of the user'"'"'s hand, and track the movement of fingers of the user'"'"'s hand within the one or more designated zones of joints in the skeletal model of the user'"'"'s hand using a 3D joint data corresponding to a position of a joint within the one or more designated zones of joints in the skeletal mode of the user'"'"'s hand;

an output module

to map the 3D joint data corresponding to the position of the joint in the constructed skeletal model to project the user'"'"'s hand on a screen as a first object performing a gesture corresponding to the movement of fingers of the user'"'"'s hand within the designated zones of joints in the skeletal model of the user'"'"'s hand, wherein the fingers on the user'"'"'s hand are represented as individual virtual cursor components of the first object, each individual virtual cursor component configured to;

simultaneously interact with one or more separate objects on the screen, and have a different function for interacting with a same object of the one or more separate objects on the screen,

including;

an index finger on the user'"'"'s hand having a first function to open the same object,

a middle finger on the user'"'"'s hand having a second function to erase the same object, and

a ring finger on the user'"'"'s hand having a third function to maximize the same object.

9. The system of claim 8, wherein performing the gesture corresponding to the movement of fingers of the user'"'"'s hand within the designated zones of joints in the skeletal model of the user'"'"'s hand generates a force that interacts with the one or more of the separate objects on the screen.

10. The system of claim 8, wherein identified movement of fingers of the user'"'"'s hand within the designated zones of joints in the skeletal model of the user'"'"'s hand corresponds to a selection gesture, wherein the first object on the screen selects a second object on the screen.

11. The system of claim 8, wherein identified movement of the fingers of user'"'"'s hand within the designated zones of joints in the skeletal model of the user'"'"'s hand corresponds to a manipulation gesture, wherein a second object on the screen is manipulated according to a predefined action associated with the manipulation gesture.

12. The system of claim 8, wherein the output module determines a distance of the user'"'"'s hand from the screen and zooms the screen in and out based on changes in the distance.

13. The system of claim 12, wherein the output module moves a cursor on the screen to indicate a selected object from a stack of objects on the screen, wherein the cursor is moved based upon the distance of the user'"'"'s hand from the screen, wherein moving the cursor comprises moving the cursor towards a bottom of the stack of objects as the user'"'"'s hand moves closer to the screen and moving the cursor towards a top of the stack of objects as the user'"'"'s hand moves farther from the screen.

14. The system of claim 8, wherein the output module scales a size of the first object based on the distance of the user'"'"'s hand from the screen.

15. A non-transitory computer readable medium comprising instructions, which when executed by a processor perform operations, comprising:

acquiring close range depth images of a user'"'"'s hand with a depth sensor;

constructing a skeletal model of the user'"'"'s hand based on the acquired depth images, the skeletal model including one or more designated zones of joints;

identifying from the acquired depth images movement within the one or more designated zones of joints in the skeletal model of the user'"'"'s hand, the movement including movement of fingers of the user'"'"'s hand within the one or more designated zones of joints based on the constructed skeletal model of the user'"'"'s hand;

tracking the movement of fingers of the user'"'"'s hand within the one or more designated zones of joints in the skeletal model of the user'"'"'s hand using a 3D joint data corresponding to a position of a joint within the one or more designated zones of joints in the skeletal mode of the user'"'"'s hand;

mapping the 3D joint data corresponding to the position of the joint in the constructed skeletal model to project the user'"'"'s hand on a screen as a first object performing a gesture corresponding to the movement of fingers of the user'"'"'s hand within the designated zones of joints in the skeletal model of the user'"'"'s hand; and

wherein the fingers on the user'"'"'s hand are represented as individual virtual cursor components of the first object, each individual virtual cursor component configured to;

simultaneously interact with one or more separate objects on the screen, and

have a different function for interacting with a same object of the one or more separate objects on the screen,

including;

an index finger on the user'"'"'s hand having a first function to open the same object,

a middle finger on the user'"'"'s hand having a second function to erase the same object, and

a ring finger on the user'"'"'s hand having a third function to maximize the same object.

16. The computer readable medium of claim 15, wherein performing the gesture corresponding to the movement of fingers of the user'"'"'s hand within the designated zones of joints in the skeletal model of the user'"'"'s hand generates a force that interacts with the one or more of the separate objects on the screen.

17. The computer readable medium of claim 16, wherein identified movement of fingers of the user'"'"'s hand within the designated zones of joints in the skeletal model of the user'"'"'s hand corresponds to a selection gesture, wherein the first object on the screen selects a second object on the screen.