Pointing finger detection

• US 9,390,500 B1
• Filed: 03/14/2013
• Issued: 07/12/2016
• Est. Priority Date: 03/14/2013
• Status: Expired due to Fees

• Current Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Original Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Inventors: Chang, Samuel Henry, Yao, Ning
• Primary Examiner(s): Mehta, Bhavesh
• Assistant Examiner(s): Mistry, Oneal R
• Application Number:
  US13/830,564
• Time in Patent Office: 1,216 Days
• Field of Search: None
• US Class Current: 1/1
• CPC Class Codes: G06F 2203/012 Walk-in-place systems for allowing a user to walk in a virtual environment while constraining him to a given position in the physical environment
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0304 Detection arrangements using opto-electronic means constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042
• CPC Class Codes: G06V 10/46 Descriptors for shape, contour or point-related descriptors, e.g. scale invariant feature transform [SIFT] or bags of words [BoW]; Salient regional features colour feature extraction G06V10/56
• CPC Class Codes: G06V 40/113 Recognition of static hand signs
• CPC Class Codes: G06V 40/28 Recognition of hand or arm movements, e.g. recognition of deaf sign language static hand signs G06V40/113

Abstract

A number of images of an environment may be obtained over time by a number of sensors. The data obtained by the sensors may be used to generate contours of objects in the environment. In some cases, hands with pointing fingers in the environment may be identified based on the contours of the objects in the environment. In particular, contours having a palm segment and a pointing finger segment may be identified as hands with pointing fingers. In an implementation, 1-dimensional curvature and distance information may be used to identify contours having a palm segment and a pointing finger segment.

Claims

1. A system comprising:

one or more processors;

one or more cameras to obtain one or more images of an environment, the environment including one or more objects;

one or more computer-readable media storing computer-executable instructions that, when executed by the one or more processors, cause the one or more processors to perform acts comprising;

determining a contour of a surface of an object of the one or more objects based, at least in part, on data obtained from the one or more images;

generating a curvature map for the contour according to a first function that expresses the contour in terms of changes in direction along the contour;

generating a distance map for the contour according to a second function that expresses an absolute value of a distance between respective points along the contour;

identifying a plurality of peak points of the contour based at least in part on the curvature map and the distance map, each of the plurality of peak points indicating a portion of the contour where curvature of the contour is at a local maximum and distance between adjacent points of the contour is at a local minimum;

selecting a peak point of the plurality of peak points, the peak point being associated with the contour;

determining a sub-contour of the contour by identifying a portion of the contour that includes the peak point;

determining that the sub-contour includes a palm segment;

determining that the sub-contour includes a pointing finger segment; and

determining that the particular contour includes a hand with a pointing finger at least partly in response to determining that the sub-contour includes the palm segment and the pointing finger segment.

2. The system of claim 1, wherein determining the sub-contour of the contour by identifying a portion of the sub-contour that includes the peak point includes:

defining a first circular region in the environment, wherein the peak point is a center of the first circular region and the diameter of the first circular region is based, at least in part, on at least one of an average size of an adult human hand or a median size of an adult human hand, wherein the sub-contour includes the portion of the contour included in the first circular region.

3. The system of claim 2, wherein determining that the sub-contour includes the palm segment includes:

defining one or more second circular regions within the sub-contour;

identifying a particular second circular region of the one or more circular regions within the sub-contour that satisfies one or more criteria; and

determining that the sub-contour includes the palm segment when a radius of the particular second circular is included in a predetermined range of values for the radius.

4. The system of claim 3, wherein determining that the sub-contour includes the pointing finger segment includes:

determining a distance between the peak point and a center of the particular second circular region; and

determining that the sub-contour includes the pointing finger segment when the distance between the peak point and the center of the particular second circular region is included in a predetermined range of values for the distance.

5. A method comprising:

obtaining, by one or more sensors of a computing device, one or more images of an environment;

determining a contour of a surface in the environment based, at least in part, on the one or more images;

determining that the contour represents a palm segment and a pointing finger segment based, at least in part, on an absolute value of one or more distances between respective points along the contour, wherein determining that the contour represents a palm segment and a pointing finger segment includes;

identifying a peak point of the contour;

identifying a sub-contour of the contour within a region defined, at least in part, by the peak point;

defining a geometric shape within the sub-contour, wherein the geometric shape has features that satisfy a first criteria;

determining that the sub-contour includes the palm segment when a dimension of the geometric shape satisfies a second criteria;

determining a distance between the peak point and a point of the geometric shape; and

determining that the sub-contour includes the pointing finger segment when the distance satisfies an additional criteria, wherein the additional criteria includes the distance being within a predetermined range of values for a distance between a tip of an extended index finger and a center of a palm of an average or median adult human hand; and

determining that the contour represents a hand with a pointing finger based, at least in part, on the contour representing the palm segment and the pointing finger segment.

6. The method of claim 5, wherein the first criteria includes the geometric shape having boundaries that contact a maximum amount of the sub-contour, the geometric shape having a maximum dimension, or both.

7. The method of claim 5, wherein the geometric shape is a circular region defined within the sub-contour.

8. The method of claim 5, further comprising:

displaying a user interface into the environment;

determining a location of the pointing finger with respect to the user interface; and

determining an input indicated by the pointing finger based, at least in part, on the location of the pointing finger with respect to the user interface.

9. One or more non-transitory computer-readable media storing computer-executable instructions that, when executed by one or more processors, cause the one or more processors to perform acts comprising:

selecting a first contour of a plurality of contours determined from one or more images of an environment;

determining that the first contour does not represent a hand with a pointing finger;

selecting a second contour of the plurality of contours;

generating a curvature map comprising 1-dimensional (1D) curvature information contour according to a first function that expresses the second contour in terms of changes in direction along the second contour;

generating a distance map comprising 1D distance information according to a second function that expresses an absolute value of a distance between respective points along the second contour;

identifying a palm segment and a pointing finger segment within a portion of the second contour based at least in part on the absolute value of the distance; and

determining that the second contour represents the hand with the pointing finger.

10. The one or more non-transitory computer-readable media of claim 9, wherein each of the plurality of contours are determined from 3-dimensional (3D) information obtained from the one or more images.

11. The one or more non-transitory computer-readable media of claim 10, wherein each of the plurality of contours from 3D information obtained from the one or more images are determined by:

segmenting surfaces in the environment to identify a plurality of continuous surfaces in the environment;

determining boundaries of each of the plurality of continuous surfaces;

determining the first contour by translating first 3D information of a first continuous surface to first 2-dimensional (2D) information according to a function; and

determining the second contour by translating second 3D information of a second continuous surface to second 2D information according to the function.

12. The one or more non-transitory computer-readable media of claim 9, wherein the palm segment and the pointing finger segment are identified using the 1-dimensional (1D) curvature information and the 1D distance information obtained from the second contour.

13. The one or more non-transitory computer-readable media of claim 12, wherein the acts further comprise:

identifying peak points of the second contour based, at least in part, on the 1D curvature information, wherein the peak points are defined as regions on the second contour corresponding to portions of the curvature map that have a local maximum; and

identifying the palm segment and the pointing finger segment within the portion of the second contour based, at least in part, on at least one of the peak points.

14. The one or more non-transitory computer-readable media of claim 9, wherein determining whether the first contour represents a hand with a pointing finger includes:

identifying a peak point of the first contour;

identifying a sub-contour of the first contour within a region defined, at least in part, by the peak point of the first contour;

defining a shape within the sub-contour of the first contour based, at least in part, on one or more criteria; and

determining that the first contour does not represent the hand with the pointing finger based, at least in part, on the shape defined within the sub-contour of the first contour including a dimension having a value outside of a predetermined range of values for the dimension.

15. The one or more non-transitory computer-readable media of claim 9, wherein determining whether the first contour represents a hand with a pointing finger includes:

identifying a peak point of the first contour;

identifying a sub-contour of the first contour within a region defined, at least in part, by the peak point of the first contour;

defining a shape within the sub-contour of the first contour based, at least in part, on one or more criteria; and

determining that the first contour does not represent the hand with the pointing finger based, at least in part, on a distance from the peak point to a center of the shape being outside of a predetermined range of distances.