Hand and hand pose detection

• US 9,870,056 B1
• Filed: 10/08/2012
• Issued: 01/16/2018
• Est. Priority Date: 10/08/2012
• Status: Expired due to Fees

• Current Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Original Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Inventors: Yao, Ning
• Primary Examiner(s): Mehmood, Jennifer
• Assistant Examiner(s): Gupta, Parul
• Application Number:
  US13/647,188
• Time in Patent Office: 1,926 Days
• Field of Search: None
• CPC Class Codes: G06F 1/1694 the I/O peripheral being a single or a set of motion sensors for pointer control or gesture input obtained by sensing movements of the portable computer
• CPC Class Codes: G06F 16/436 using biological or physiological data of a human being, e.g. blood pressure, facial expression, gestures
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0304 Detection arrangements using opto-electronic means constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042
• CPC Class Codes: G06V 40/107 Static hand or arm
• CPC Class Codes: G06V 40/28 Recognition of hand or arm movements, e.g. recognition of deaf sign language static hand signs G06V40/113
• CPC Class Codes: H04N 9/3147 Multi-projection systems displays in general H04N9/12; video walls G06F3/1446, G09G2300/026
• CPC Class Codes: H04N 9/3185 Geometric adjustment, e.g. keystone or convergence optical or mechanical adjustment of convergence H04N9/317; using scanning means H04N3/22; optical or mechanical adjustments of projectors not peculiar to the presence of an electronic spatial light modulator G03B21/14

Abstract

Techniques for detecting a hand of a user and a pose of the hand are described herein. In some instances, as a user interacts with a computing system through various hand signals, the hand of the user and pose of the hand (e.g., position and orientation) may be detected. To detect a hand, a point set may be analyzed to identify local extremum points (e.g., peaks and valleys) along an outer boundary of the point set. The local extremum points may then be used to identify a potential hand segment including one or more potential finger segments and/or a potential palm segment. When the potential hand segment satisfies one or more criteria, a hand may be detected. To detect a hand pose, an observed pose of the hand segment may be compared to one or more hand pose models.

Claims

1. A system comprising:

one or more processors;

an imaging sensor; and

one or more computer-readable media storing computer-executable instructions that, when executed by the one or more processors, cause the one or more processors to perform acts comprising;

obtaining a point set using the imaging sensor;

identifying a first local extremum point from the point set along a first portion of an outer boundary of the point set, the first portion associated with a first curvature value that satisfies one or more curvature criteria;

identifying a second local extremum point from the point set along a second portion of the outer boundary of the point set, the second portion associated with a second curvature value that satisfies the one or more curvature criteria;

identifying a potential hand segment within the point set;

identifying a first line segment from the point set, the first line segment extending from the first local extremum point and terminating at a first endpoint, the first endpoint different than the second local extremum point;

identifying a second line segment from the point set, the second line segment extending from the second local extremum point and terminating at a second endpoint, the first endpoint different than the second endpoint, and the second endpoint different than the first local extremum point;

identifying a centroid of a shape within the point set, the centroid based at least in part on the first endpoint and the second endpoint;

applying a shape model to the potential hand segment based at least in part on a location of the centroid;

determining that the potential hand segment is a potential palm segment based at least in part on the shape model and the location of the centroid;

designating the point set as corresponding to a hand of a user; and

comparing the potential hand segment to one or more hand pose models to identify a pose of the hand of the user.

2. The system of claim 1, wherein the identifying the potential hand segment includes:

identifying one or more potential finger segments, a potential finger segment of the one or more potential finger segments including points of the point set along and within a contour that extends from the first local extremum point to a third local extremum point, from the third local extremum point to a particular point along the outer boundary of the point set, and from the particular point to the first local extremum point; and

determining that the potential palm segment comprises points of the point set that are at least partly bordered by the one or more potential finger segments.

3. The system of claim 2, the acts further comprising determining that at least one of:

a length of at least one potential finger segment of the one or more potential finger segments satisfies one or more predetermined finger lengths, a width of the at least one potential finger segment satisfies one or more predetermined finger widths, the at least one potential finger segment includes a particular curvature characteristic, or a quantity of the one or more potential finger segments satisfies one or more predetermined finger quantities.

4. The system of claim 1, wherein the determining that the potential hand segment is the potential palm segment comprises determining that at least one of:

a length of the potential palm segment satisfies one or more predetermined palm lengths, a width of the potential palm segment satisfies one or more predetermined palm widths, or a shape of the potential palm segment satisfies one or more predetermined shapes.

5. A system comprising:

one or more processors; and

one or more computer-readable media storing computer-executable instructions that, when executed, cause the one or more processors to perform acts comprising;

obtaining a point set;

identifying a local extremum point along an outer boundary of the point set that has a curvature value that satisfies one or more curvature criteria;

identifying a potential finger segment of one or more potential finger segments comprising points from the point set that are based at least in part on the local extremum point, the local extremum point representing a distal portion of the potential finger segment;

determining that the potential finger segment has a length that satisfies a predetermined finger length;

determining that the potential finger segment has a width that satisfies a predetermined finger width;

determining that the potential finger segment has a centerline;

identifying a line segment from the point set based at least in part on the centerline, the line segment extending from the local extremum point of the potential finger segment and terminating at an endpoint;

identifying a centroid of a shape within the point set, the centroid based at least in part on the endpoint of the line segment, wherein the centroid is different than the endpoint;

identifying a potential palm segment based at least in part on a location of the centroid; and

determining that the point set represents, at least in part, a hand of a user.

6. The system of claim 5, the acts further comprising determining that the potential finger segment includes a particular curvature characteristic, or a quantity of the one or more potential finger segments satisfies one or more predetermined finger quantities.

7. The system of claim 5, wherein the local extremum point is one of a plurality of local extremum points, the acts further comprising determining that a point of the plurality of local extremum points is located within a predetermined distance to another point of the plurality of local extremum points.

8. The system of claim 5, the acts further comprising:

determining that the potential palm segment satisfies one or more palm criteria;

and wherein determining that the point set represents the hand of the user is based at least in part on determining that the potential palm segment satisfies the one or more palm criteria.

9. The system of claim 8, wherein determining that the potential palm segment satisfies the one or more palm criteria comprises determining that at least one of:

a length of the potential palm segment satisfies one or more predetermined palm lengths, a width of the potential palm segment satisfies one or more predetermined palm widths, or a shape of the potential palm segment satisfies one or more predetermined shapes.

10. The system of claim 5, wherein the potential palm segment is identified by aligning a shape model to the point set by aligning the shape model to be substantially centered on the centroid, the potential palm segment comprising points of the point set that are within the aligned shape model or are within a predetermined distance of the aligned shape model.

11. The system of claim 5, the acts further comprising:

identifying a line that extends through the centroid in a direction that the one or more potential finger segments are pointing;

identifying a potential wrist line that extends from a first side of the point set to a second side of the point set, the potential wrist line being substantially perpendicular to the line that extends in the direction that the one or more potential finger segments are pointing; and

determining that a length of the potential wrist line satisfies one or more predetermined wrist lengths;

and wherein determining that the point set represents the hand of the user is based at least in part on determining that the length of the potential wrist line satisfies the one or more predetermined wrist lengths.

12. The system of claim 5, the acts further comprising:

determining that the potential palm segment satisfies one or more palm criteria; and

segmenting the potential palm segment from the point set.

13. The system of claim 5, the acts further comprising:

determining that the potential finger segment satisfies one or more finger criteria; and

segmenting the potential finger segment from the point set.

14. The system of claim 5, the acts further comprising upon determining that the point set describes the hand of the user, comparing the point set to one or more hand pose models to identify a pose of the hand.

15. One or more non-transitory computer-readable storage media storing computer-readable instructions that, when executed, instruct one or more processors to perform operations comprising:

obtaining a point set;

identifying a local extremum point along an outer boundary of the point set that has a curvature value that satisfies one or more curvature characteristics;

identifying a potential finger segment comprising points from the point set that are based at least in part on the local extremum point;

identifying a centerline of the potential finger segment;

identifying a line segment from the point set based at least in part on the centerline, the line segment extending from the local extremum point of the potential finger segment and terminating at an endpoint;

identifying a centroid of a shape within the point set, the centroid based at least in part on the endpoint of the line segment, the centroid different than the endpoint;

identifying a potential palm segment comprising points from the point set that are at least partly bordered by the potential finger segment, the potential palm segment based at least in part on a location of the centroid;

determining that the potential palm segment satisfies one or more palm criteria; and

determining that the point set represents, at least in part, a hand of a user.

16. The one or more non-transitory computer-readable storage media of claim 15, the operations further comprising:

identifying a plurality of potential finger segments, the potential finger segment one of the plurality of potential finger segments; and

determining that at least one of;

a length of at least one potential finger segment of the plurality of potential finger segments satisfies one or more predetermined finger lengths, a width of the at least one potential finger segment satisfies one or more finger widths, the at least one potential finger segment includes a particular curvature characteristic, or a quantity of the plurality of potential finger segments satisfies one or more finger quantities.

17. The one or more non-transitory computer-readable storage media of claim 15, wherein:

the local extremum point is one of a plurality of local extremum points; and

the operations further comprise determining that a point of the plurality of local extremum points is located within a predetermined distance to another point of the plurality of local extremum points.

18. The one or more non-transitory computer-readable storage media of claim 15, wherein determining that the point set represents the hand of the user is based at least in part on a shape model based on the location of the centroid.

19. The one or more non-transitory computer-readable storage media of claim 15, wherein determining that the potential palm segment satisfies the one or more palm criteria comprises determining that at least one of:

a length of the potential palm segment satisfies one or more predetermined palm lengths, a width of the potential palm segment satisfies one or more predetermined palm widths, or a shape of the potential palm segment satisfies one or more predetermined shapes.

20. The one or more non-transitory computer-readable storage media of claim 15, wherein the operations further comprise aligning a shape model to the point set by aligning the shape model to be substantially centered on the centroid, the potential palm segment comprising points of the point set that are within the shape model or are within a predetermined distance of the shape model.

21. The one or more non-transitory computer-readable storage media of claim 15, wherein the operations further comprise:

identifying a line that extends through the centroid in a direction that the potential finger segment is pointing;

identifying a potential wrist line that extends from a first side of the point set to a second side of the point set, the potential wrist line being substantially perpendicular to the line that extends in the direction that the potential finger segment is pointing; and

determining that a length of the potential wrist line satisfies one or more predetermined wrist lengths;

and wherein determining that the point set represents the hand of the user is based at least in part on determining that the length of the potential wrist line satisfies the one or more predetermined wrist lengths.

22. A method comprising:

under control of one or more computing systems configured with executable instructions,

obtaining a point set;

determining that the point set describes, at least in part, a hand of a user, by;

determining that at least some of the points of the point set correspond to a finger segment based at least in part on a curvature value of a local extremum point from the point set, the local extremum point representing a distal portion of the finger segment;

determining a centerline of the finger segment;

determining a line segment from the point set based at least in part on the centerline, the line segment extending from the local extremum point of the finger segment and terminating at an endpoint;

determining a centroid of a shape within the point set, the centroid based at least in part on the endpoint of the line segment, wherein the centroid of the shape is different than the endpoint;

determining that at least some of the points of the point set correspond to a palm segment based at least in part on the centroid; and

determining a wrist line associated with the point set;

identifying an observed pose of the hand by;

identifying a pose for the finger segment, the pose of the finger segment comprising a pose from a group of poses including an extended pose, a half-folded pose, and a folded pose; and

identifying a pose of the palm segment;

comparing the observed pose of the hand to one or more hand pose models; and

determining a pose of the hand based at least in part on the comparing.

23. The method of claim 22, wherein the extended pose is associated with a state in which a finger of the user is extended, the half-folded pose is associated with a state in which the finger is bent toward a palm and a distance between a tip of the finger and the palm is greater than a predetermined distance, and the folded pose is associated with a state in which the finger is bent toward the palm and a distance between the tip of the finger and the palm is less than the predetermined distance.

24. The method of claim 22, further comprising, at a time prior to obtaining the point set, identifying the local extremum point.

25. The method of claim 22, further comprising determining a length of the finger segment, the length of the finger segment satisfying a predetermined finger length, wherein the endpoint of the line segment is based at least in part on the length of the finger segment.