System for finger recognition and tracking

• US 8,897,491 B2
• Filed: 10/19/2011
• Issued: 11/25/2014
• Est. Priority Date: 06/06/2011
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Ambrus, Anthony, Lee, Kyungsuk David, Campbell, Andrew, Haley, David, Mount, Brian, Robles, Albert, Osborn, Daniel, Wright, Shawn, Sharkasi, Nahil, Hill, Dave, McCulloch, Daniel
• Primary Examiner(s): Lu, Tom Y
• Application Number:
  US13/277,011
• Publication Number:
  US 20120309532A1
• Time in Patent Office: 1,133 Days
• Field of Search: 382/100, 382/103, 382/107
• US Class Current: 382/103
• CPC Class Codes: A63F 13/213 comprising photodetecting means, e.g. cameras, photodiodes or infrared cells A63F13/219, A63F13/655 take precedence
• CPC Class Codes: A63F 13/428 involving motion or position input signals, e.g. signals representing the rotation of an input controller or a player's arm motions sensed by accelerometers or gyroscopes
• CPC Class Codes: A63F 13/44 involving timing of operations, e.g. performing an action within a time slot
• CPC Class Codes: A63F 13/812 Ball games, e.g. soccer or baseball
• CPC Class Codes: A63F 2300/1087 comprising photodetecting means, e.g. a camera
• CPC Class Codes: A63F 2300/6045 for mapping control signals received from the input arrangement into game commands
• CPC Class Codes: A63F 2300/6607 for animating game characters, e.g. skeleton kinematics
• CPC Class Codes: A63F 2300/6615 using models with different levels of detail [LOD]
• CPC Class Codes: G06F 3/0425 using a single imaging device like a video camera for tracking the absolute position of a single or a plurality of objects with respect to an imaged reference surface, e.g. video camera imaging a display or a projection screen, a table or a wall surface, on which a computer generated image is displayed or projected tracking a projected light spot to determine a position on a display surface G06F3/0386
• CPC Class Codes: G06F 3/04883 for inputting data by handwriting, e.g. gesture or text
• CPC Class Codes: G06T 11/20 Drawing from basic elements, e.g. lines or circles
• CPC Class Codes: G06T 2207/10028 Range image; Depth image; 3D point clouds
• CPC Class Codes: G06T 2207/30196 Human being; Person
• CPC Class Codes: G06T 7/251 involving models
• CPC Class Codes: G06V 40/113 Recognition of static hand signs
• CPC Class Codes: G06V 40/28 Recognition of hand or arm movements, e.g. recognition of deaf sign language static hand signs G06V40/113

Abstract

A system and method are disclosed relating to a pipeline for generating a computer model of a target user, including a hand model of the user'"'"'s hands and fingers, captured by an image sensor in a NUI system. The computer model represents a best estimate of the position and orientation of a user'"'"'s hand or hands. The generated hand model may be used by a gaming or other application to determine such things as user gestures and control actions.

Claims

1. A method for generating a model of a user'"'"'s hand including one or more fingers, comprising:

(a) receiving position data representing a position of a user interacting with a sensor, the position data including at least one of depth and image data representing the user'"'"'s hand; and

(b) analyzing the position data to identify the user'"'"'s hand in the position data, said step (b) including the steps of;

(b)(1) analyzing depth data from the position data captured in said step (a) to segment the position data into data of the hand, and

(b)(2) extracting a set of feature descriptors by applying one or more filters to the image data of the hand identified in said step (b)(1), the one or more filters analyzing image data of the hand as compared to image data outside of a boundary of the hand to discern features of the hand including a shape and orientation of the hand.

2. The method of claim 1, further comprising the steps of running an application receiving commands via the sensing mechanism, and affecting a control action in the application based on a position of the hand identified in said step (b).

3. The method of claim 1, further comprising the steps of running a gaming application receiving commands via the sensing mechanism, and affecting an action in the gaming application based on a position of the hand identified in said step (b).

4. The method of claim 1, wherein said step (b)(1) comprises the step of analyzing centroids constructed from the image data to locate a best candidate of a hand.

5. The method of claim 4, wherein said step (b)(1) further comprises the step of analyzing the best candidate of a hand to determine a best candidate of the center of the hand.

6. The method of claim 1, wherein said step (b)(2) comprises the steps of applying a pixel classifier including the steps of:

selecting pixels within a boundary of the shape descriptor of the hand,

constructing a box of predetermined size around each pixel, each box constructed in a plane of the shape descriptor,

determining intersection points with each box where the image data transitions between a foreground point and a background point, and

identifying hand and fingers from analyzing the intersection points of each box for each examined pixel.

7. The method of claim 1, wherein said step (b)(2) comprises the steps of applying a curvature analysis filter including the steps of:

selecting pixels along a boundary of the shape descriptor of the hand,

examining a plurality of pixels surrounding a selected pixel, and assigning a value to the selected pixel indicating which surrounding pixel is also along a boundary of the shape descriptor,

converting the values into angles and contours around the hand including peaks and valleys, and

determining which of the peaks represent fingers of the hand.

8. The method of claim 1, wherein said step (b)(2) comprises the steps of applying a histogram filter including the step of constructing a histogram of distances between a plurality of points in the shape descriptor and a device capturing the image data.

9. A system for generating a model of a user'"'"'s hand including one or more fingers, the system including a sensing mechanism operatively coupled to a computing device, the system comprising:

a skeletal recognition engine for recognizing at least a portion of a skeleton of a user from received data including at least one of image and depth data;

an image segmentation engine for segmenting one or more regions of the body into a region representing a hand of the user; and

a descriptor extraction engine for extracting data representative of a hand including one or more fingers and an orientation of the hand, the descriptor extraction engine applying a plurality of filters for analyzing pixels in the region representing the hand, each filter in the plurality of filters determining a position and orientation of the hand, the descriptor extraction engine combining the results of each filter to arrive at a best estimate of the position and orientation of the hand.

10. The system of claim 9, wherein the plurality of filters of the descriptor extraction engine include one or more filters optimized for identifying the position and orientation of the hand as a silhouette relative to a device capturing the received data.

11. The system of claim 9, wherein the plurality of filters of the descriptor extraction engine include one or more filters optimized for identifying the position and orientation of the hand when pointed toward or away from a device capturing the received data.

12. The system of claim 9, wherein the plurality of filters of the descriptor extraction engine include a classifier engine for analyzing a hand as a silhouette relative to the sensing mechanism, the classifier engine selecting pixels within a region representing a hand of the user, constructing a box of predetermined size around each pixel, each box constructed in a plane of the silhouette, determining intersection points with each box where the image data transitions between a foreground point and a background point, and identifying hand and fingers from analyzing the intersection points of each box for each examined pixel.

13. The system of claim 12, wherein the classifier engine identifies a centroid representing a fingertip where two intersection points are identified on a box and a distance between the intersection points is too small to represent a palm of the hand.

14. The system of claim 13, wherein a location of the two intersection points on the same or different sides of a box indicates an orientation of the identified fingertip.

15. The system of claim 12, wherein the classifier engine identifies a centroid representing a finger where four intersection points are identified on a box.

16. The system of claim 12, wherein the classifier engine identifies a centroid representing a palm of the hand where two intersection points are identified on a box and a distance between the intersection points is too large to represent a fingertip.

17. The system of claim 12, wherein the box constructed around a given pixel is a first box of a first size, the pixel classifier engine further constructing a second box around the given pixel of a second size larger than the first size to detect a condition where fingers of the hand are together with each other.