Fingertip tracking for touchless user interface

• US 9,135,503 B2
• Filed: 04/07/2011
• Issued: 09/15/2015
• Est. Priority Date: 11/09/2010
• Status: Active Grant

• Current Assignee: Qualcomm, Inc.
• Original Assignee: Qualcomm, Inc.
• Inventors: Sundaresan, Sairam, Bi, Ning, Aleksic, Milivoje, Qi, Yingyong, Zhang, Xuerui
• Primary Examiner(s): Shibru, Helen
• Application Number:
  US13/082,295
• Publication Number:
  US 20120113241A1
• Time in Patent Office: 1,622 Days
• Field of Search: 715/863, 382/203, 348/77
• US Class Current: 1/1
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06T 2207/30196 Human being; Person
• CPC Class Codes: G06T 7/251 involving models
• CPC Class Codes: G06V 40/107 Static hand or arm
• CPC Class Codes: G06V 40/28 Recognition of hand or arm movements, e.g. recognition of deaf sign language static hand signs G06V40/113

Abstract

In general, this disclosure describes techniques for providing a gesture-based user interface. For example, according to some aspects of the disclosure, a user interface generally includes a camera and a computing device that identifies and tracks the motion of one or more fingertips of a user. In some examples, the user interface is configured to identify predefined gestures (e.g., patterns of motion) associated with certain motions of the user'"'"'s fingertips. In another example, the user interface is configured to identify hand postures (e.g., patterns of showing up of fingertips). Accordingly, the user can interact with the computing device by performing the gestures.

Claims

1. A method for tracking one or more fingertips in video, comprising:

generating, by one or more processors, a set of points associated with curvatures of a hand in a video frame;

generating, by the one or more processors, a set of candidate fingertips in the video frame based on the set of points;

merging, by the one or more processors, candidate fingertips of the set of candidate fingertips in the video frame based on an initial target number of candidate fingertips in the video frame and a relative proximity of the candidate fingertips to one another; and

designating, by the one or more processors, one or more of the merged candidate fingertips as fingertips for tracking.

2. The method of claim 1, further comprising fitting an elliptical shape to one or more merged candidate fingertips, and tracking the relative position of the elliptical shape in a plurality of frames.

3. The method of claim 1, wherein generating the set of candidate fingertips includes identifying peak curvatures associated with the hand, and allowing the peak curvatures to remain in the set of candidate fingertips.

4. The method of claim 1, further comprising verifying the set of candidate fingertips by identifying valley curvatures associated with the hand, and removing the valley curvatures from being considered in the set of candidate fingertips.

5. The method of claim 1, wherein generating the set of points associated with curvatures of the hand includes generating a set of points using a K-curvature algorithm.

6. The method of claim 1, wherein merging candidate fingertips based on the initial target number of candidate fingertips includes setting the initial target number of candidate fingertips to five fingertips.

7. The method of claim 1, wherein merging candidate fingertips comprises merging the candidate fingertips using a K-means algorithm.

8. The method of claim 7, wherein the K-means algorithm is configured to merge the set of points into five candidate fingertips.

9. The method of claim 1, further comprising tracking the relative position of one or more designated fingertips in a plurality of frames, recognizing a gesture based on the tracking of the one or more designated fingertips, and performing a computing function based on the gesture.

10. A computer program product for tracking one or more fingertips in video comprising a non-transitory computer-readable storage medium having stored thereon instructions that, when executed, cause a processor to:

generate a set of points associated with curvatures of a hand in a video frame;

generate a set of candidate fingertips in the video frame based on the set of points;

merge candidate fingertips of the set of candidate fingertips in the video frame based on an initial target number of candidate fingertips in the video frame and a relative proximity of the candidate fingertips to one another; and

designate one or more of the merged candidate fingertips as fingertips for tracking.

11. The computer program product of claim 10, further comprising instructions causing the processor to fit an elliptical shape to one or more merged candidate fingertips, and tracking the relative position of the elliptical shape in a plurality of frames.

12. The computer program product of claim 10, wherein generating the set of candidate fingertips includes identifying peak curvatures associated with the hand, and allowing the peak curvatures to remain in the set of candidate fingertips.

13. The computer program product of claim 10, further comprising instructions causing the processor to verify the set of candidate fingertips by identifying valley curvatures associated with the hand, and removing the valley curvatures from being considered in the set of candidate fingertips.

14. The computer program product of claim 10, wherein generating the set of points associated with curvatures of the hand includes generating a set of points using a K-curvature algorithm.

15. The computer program product of claim 10, wherein merging candidate fingertips based on the initial target number of candidate fingertips includes setting the initial target number of candidate fingertips to five fingertips.

16. The computer program product of claim 10, wherein merging candidate fingertips comprises merging the candidate fingertips using a K-means algorithm.

17. The computer program product of claim 16, wherein the K-means algorithm is configured to merge the set of points into five candidate fingertips.

18. The computer program product of claim 10, further comprising instructions causing the processor to track the relative position of one or more designated fingertips in a plurality of frames, recognize a gesture based on the tracking of the one or more designated fingertips, and perform a function based on the gesture.

19. An apparatus for tracking one or more fingertips in video comprising:

a video camera configured to capture video frames;

a fingertip tracking module configured to;

generate a set of points associated with curvatures of a hand in the video frame;

generate a set of candidate fingertips in the video frame based on the set of points;

merge candidate fingertips of the set of candidate fingertips in the video frame based on an initial target number of candidate fingertips in the video frame and a relative proximity of the candidate fingertips to one another; and

designate one or more of the merged candidate fingertips as fingertips for tracking.

20. The apparatus of claim 19, wherein the fingertip tracking module is further configured to fit an elliptical shape to one or more merged candidate fingertips, and tracking the relative position of the elliptical shape in a plurality of frames.

21. The apparatus of claim 19, wherein generating the set of candidate fingertips includes identifying peak curvatures associated with the hand, and allowing the peak curvatures to remain in the set of candidate fingertips.

22. The apparatus of claim 19, wherein the fingertip tracking module is further configured to verify the set of candidate fingertips by identifying valley curvatures associated with the hand, and removing the valley curvatures from being considered in the set of candidate fingertips.

23. The apparatus of claim 19, wherein generating the set of points associated with curvatures of the hand includes generating a set of points using a K-curvature algorithm.

24. The apparatus of claim 19, wherein merging candidate fingertips based on the initial target number of candidate fingertips includes setting the initial target number of candidate fingertips to five fingertips.

25. The apparatus of claim 19, wherein merging candidate fingertips comprises merging the candidate fingertips using a K-means algorithm.

26. The apparatus of claim 25, wherein the K-means algorithm is configured to merge the set of points into five candidate fingertips.

27. The apparatus of claim 19, wherein the fingertip tracking module is further configured to track the relative position of one or more designated fingertips in a plurality of frames, recognize a gesture based on the tracking of the one or more designated fingertips, and perform a function based on the gesture.

28. An apparatus for tracking one or more fingertips in video comprising:

means for generating a set of points associated with curvatures of a hand in a video frame;

means for generating a set of candidate fingertips in the video frame based on the set of points;

means for merging candidate fingertips of the set of candidate fingertips in the video frame based on an initial target number of candidate fingertips in the video frame and a relative proximity of the candidate fingertips to one another; and

means for designating one or more of the merged candidate fingertips as fingertips for tracking.

29. The apparatus of claim 28, further comprising a means for fitting an elliptical shape to one or more merged candidate fingertips, and tracking the relative position of the elliptical shape in a plurality of frames.

30. The apparatus of claim 28, wherein the means for generating the set of candidate fingertips further comprises a means for identifying peak curvatures associated with the hand, and allowing the peak curvatures to remain in the set of candidate fingertips.

31. The apparatus of claim 28, further comprising a means for verifying the set of candidate fingertips by identifying valley curvatures associated with the hand, and removing the valley curvatures from being considered in the set of candidate fingertips.

32. The apparatus of claim 28, wherein the means for generating the set of points associated with curvatures of the hand comprises a means for generating a set of points using a K-curvature algorithm.

33. The apparatus of claim 28, wherein the means for merging candidate fingertips based on the initial target number of candidate fingertips comprises a means for setting the initial target number of candidate fingertips to five fingertips.

34. The apparatus of claim 28, wherein the means for merging candidate fingertips comprises a means for merging the candidate fingertips using a K-means algorithm.

35. The apparatus of claim 34, wherein the means for using the K-means algorithm comprises a means for merging the set of points into five candidate fingertips.

36. The apparatus of claim 28, further comprising a means for tracking the relative position of one or more designated fingertips in a plurality of frames, recognizing a gesture based on the tracking of the one or more designated fingertips, and performing a function based on the gesture.