Fast fingertip detection for initializing a vision-based hand tracker

• US 8,941,588 B2
• Filed: 03/26/2012
• Issued: 01/27/2015
• Est. Priority Date: 04/24/2008
• Status: Expired due to Fees

• Current Assignee: Oblong Industries, Inc. (Oblong, Inc.)
• Original Assignee: Oblong Industries, Inc. (Oblong, Inc.)
• Inventors: Minnen, David
• Primary Examiner(s): Shankar, Vijay
• Application Number:
  US13/430,509
• Publication Number:
  US 20120268364A1
• Time in Patent Office: 1,037 Days
• Field of Search: 345156-165, 345173-179, 345204-215
• US Class Current: 345/158
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0304 Detection arrangements using opto-electronic means constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042
• CPC Class Codes: G06F 3/0325 using a plurality of light emitters or reflectors or a plurality of detectors forming a reference frame from which to derive the orientation of the object, e.g. by triangulation or on the basis of reference deformation in the picked up image
• CPC Class Codes: G06V 10/245 by locating a pattern; Special marks for positioning
• CPC Class Codes: G06V 40/107 Static hand or arm

Abstract

Systems and methods for initializing real-time, vision-based hand tracking systems are described. The systems and methods for initializing the vision-based hand tracking systems image a body and receive gesture data that is absolute three-space data of an instantaneous state of the body at a point in time and space, and at least one of determine an orientation of the body using an appendage of the body and track the body using at least one of the orientation and the gesture data.

Claims

1. A method comprising:

detecting a gesture of a body from gesture data received via an optical detector, wherein the gesture data is absolute three-space location data of an instantaneous state of the body at a point in time and space, the detecting comprising detecting a segment of an appendage of the body and generating an estimated orientation of the body using the segment;

tracking the body using at least one of the estimated orientation and the gesture data and, while tracking, identifying the gesture using the gesture data; and

generating a gesture signal that represents the gesture and providing the gesture signal as a control signal.

2. A method comprising:

receiving via an optical detector gesture data that is absolute three-space data of an instantaneous state of a body at a point in time and space;

at least one of generating an estimated orientation of the body using an appendage of the body and tracking the body using at least one of the estimated orientation and the gesture data;

detecting and identifying a gesture of the body using the gesture data, and translating the gesture to a gesture signal; and

controlling a component coupled to a computer in response to the gesture signal.

3. A method comprising:

receiving via an optical detector gesture data that is absolute three-space data of an instantaneous state of the body at a point in time and space;

detecting an appendage of the body and identifying a segment of the appendage using the gesture data, and determining an orientation of the body using the segment;

tracking the body using at least one of the orientation and the gesture data;

detecting and identifying a gesture of the body using only the gesture data;

translating the gesture to a gesture signal; and

controlling a computer component using the gesture signal.

4. A method comprising:

receiving via an optical detector gesture data that is absolute three-space data of an instantaneous state of the body at a point in time and space;

detecting a segment of an appendage of the body and generating an estimated orientation of the body using the segment; and

tracking the body using at least one of estimated orientation and the gesture data.

5. A method comprising:

receiving via an optical detector gesture data that is absolute three-space data of an instantaneous state of the body at a point in time and space;

detecting an appendage of the body and identifying a segment of the appendage using the gesture data;

generating an estimated orientation of the body using data of the segment; and

tracking the body using at least one of the estimated orientation and the gesture data.

6. A method comprising:

receiving via an optical detector gesture data that is absolute three-space data of an instantaneous state of a body at a point in time and space;

detecting an appendage of the body and identifying a region of the appendage using the gesture data; and

at least one of generating an estimated orientation of the body using the appendage and tracking the body using at least one of the estimated orientation and the gesture data.

7. The method of claim 6, wherein the optical detector comprises a video detector outputting a plurality of video frames comprising the gesture data.

8. The method of claim 7, comprising detecting motion in the plurality of video frames.

9. The method of claim 8, comprising detecting the appendage by discriminating between at least one first pixel region of a video frame that includes the appendage and at least one second pixel region of the video frame from which the appendage is absent.

10. The method of claim 9, comprising detecting edge pixels of the video frame.

11. The method of claim 10, comprising generating an edge image for the video frame.

12. The method of claim 11, comprising generating an edge orientation image for each edge pixel of the edge image.

13. The method of claim 12, wherein the generating of the edge orientation image comprises calculating an angle implied by local partial derivatives of the video frame.

14. The method of claim 12, comprising generating a body prediction using tracking data when the tracking data is available.

15. The method of claim 14, comprising generating a body prediction using at least one of the edge pixels, the edge orientation image, and the appendage.

16. The method of claim 15, comprising detecting the body by comparing the body prediction to a set of body models.

17. The method of claim 16, comprising tracking the body.

18. The method of claim 17, comprising detecting a gesture of the body using the gesture data.

19. The method of claim 18, wherein the gesture comprises at least one of poses and motion of the body.

20. The method of claim 18, comprising translating the gesture into a gesture signal using a gesture notation.

21. The method of claim 20, comprising controlling a computer application using the gesture signal.

22. A method comprising:

receiving via an optical detector gesture data that is absolute three-space data of an instantaneous state of a body at a point in time and space; and

at least one of generating an estimated orientation of the body using an appendage of the body and tracking the body using at least one of the estimated orientation and the gesture data.

23. The method of claim 22, comprising detecting the appendage of the body and identifying a segment of the appendage using the gesture data.

24. The method of claim 23, comprising detecting the appendage of the body by discriminating between at least one first pixel region that includes the appendage and at least one second pixel region from which the appendage is absent.

25. The method of claim 23, wherein the optical detector comprises a video detector outputting a plurality of video frames comprising the gesture data.

26. The method of claim 25, comprising converting each video frame into a binary image.

27. The method of claim 25, comprising pre-processing each video frame of the plurality of video frames, the pre-processing comprising the detecting of the appendage and the identifying of the segment of the appendage.

28. The method of claim 27, comprising generating an estimated orientation of the body using data of the segment.

29. The method of claim 25, comprising comparing each video frame of the plurality of video frames to a background model.

30. The method of claim 29, comprising generating the background model by computing an adapting running average of the plurality of video frames.

31. The method of claim 30, comprising converting the background model into a background binary image.

32. The method of claim 29, comprising calculating for each pixel of the video frame a pixel value difference between a first pixel value of the respective pixel of the video frame and a second pixel value of each respective corresponding pixel of the background model.

33. The method of claim 32, comprising generating marked pixels by marking each pixel of the video frame for which the pixel value difference exceeds a threshold value.

34. The method of claim 33, comprising generating a foreground image comprising the marked pixels.

35. The method of claim 34, comprising extracting at least one feature from the gesture data, wherein the at least one feature comprises at least one pixel of the foreground image.

36. The method of claim 35, wherein the detecting comprises applying the at least one feature to a plurality of filters in turn, and identifying the at least one feature as including the appendage when the at least one feature is accepted by all filters of the plurality of filters.

37. The method of claim 35, wherein the at least one feature comprises at least one box, wherein the at least one box is specified using a plurality of coordinates within at least one query region of the gesture data.

38. The method of claim 37, comprising specifying the at least one box using an upper-left coordinate and a lower-right coordinate of the at least one box.

39. The method of claim 37, wherein the at least one box is characterized by a number of pixels of the at least one foreground image.

40. The method of claim 37, wherein the at least one feature comprises at least one ring, wherein the at least one ring is an unfilled rectangle region within a box.

41. The method of claim 40, wherein the detecting comprises applying the at least one feature to a plurality of filters in turn, and identifying the at least one feature as including the appendage when the at least one feature is accepted by all filters of the plurality of filters.

42. The method of claim 41, wherein the appendage comprises a fingertip.

43. The method of claim 42, wherein the body comprises a finger.

44. The method of claim 41, wherein the body comprises a hand.

45. The method of claim 41, wherein the body comprises an arm.

46. The method of claim 41, comprising, for each detected instance of the appendage, generating a body prediction corresponding to the appendage.

47. The method of claim 46, wherein the body prediction comprises a predicted position corresponding to the appendage.

48. The method of claim 46, wherein the body prediction comprises the estimated orientation corresponding to the detected region.

49. The method of claim 46, comprising generating a body prediction list and storing the body prediction in the body prediction list, wherein the body prediction list comprises a plurality of body predictions.

50. The method of claim 49, comprising generating an edge image for the video frame.

51. The method of claim 50, wherein the generating of the edge image is provided by an edge detector.

52. The method of claim 50, comprising generating an edge orientation image for each edge pixel of the edge image.

53. The method of claim 52, wherein the generating of the edge orientation image comprises calculating an angle implied by local partial derivatives.

54. The method of claim 52, comprising, when tracking of the body is in progress, generating a predicted estimate for the body prediction using a previous estimate of the body prediction from at least one previous video frame.

55. The method of claim 54, comprising, when tracking of a plurality of body hypotheses is in progress, generating for each body hypothesis a predicted estimate for the body prediction using a previous estimate of the body prediction from at least one previous video frame.

56. The method of claim 54, comprising removing from the body prediction list duplicates of the body predictions corresponding to the video frame.

57. The method of claim 54, comprising, for each predicted estimate, identifying a best parameter value.

58. The method of claim 57, comprising identifying the best parameter value using particle swarm optimization.

59. The method of claim 57, comprising identifying the best parameter value using a set of body models.

60. The method of claim 59, wherein the set of body models comprises a set of hand models.

61. The method of claim 59, comprising generating the set of hand models by creating a contour that matches a shape of a hand in each pose of a plurality of poses.

62. The method of claim 61, comprising representing the contour using a quadratic b-spline.

63. The method of claim 59, wherein the best parameter value comprises at least one parameter value of a body prediction that best matches a body model of the set of body models, wherein the body prediction corresponds to the gesture data.

64. The method of claim 63, wherein the best parameter value comprises at least one of position, scale, and orientation.

65. The method of claim 63, wherein the best parameter value comprises position.

66. The method of claim 63, wherein the best parameter value comprises scale.

67. The method of claim 63, wherein the best parameter value comprises orientation.

68. The method of claim 63, comprising, for each parameter value, updating control points of a hand spline of the body model.

69. The method of claim 68, wherein the updating comprises updating via at least one of translation, scaling, and rotation.

70. The method of claim 68, wherein the updating comprises updating via translation.

71. The method of claim 68, wherein the updating comprises updating via scaling.

72. The method of claim 68, wherein the updating comprises updating via rotation.

73. The method of claim 68, comprising sampling along the hand spline.

74. The method of claim 73, comprising comparing local contour orientation of the hand spline to oriented edges of the edge orientation image of the video frame.

75. The method of claim 74, comprising comparing by searching along a normal of the hand spline for a closest edge with a matching orientation to the body prediction.

76. The method of claim 74, comprising updating the plurality of predicted estimates using the best parameter values.

77. The method of claim 76, comprising reporting as a current body location a highest scoring predicted estimate when the highest scoring predicted estimate exceeds a threshold.

78. The method of claim 77, comprising reporting absence of a body in a corresponding frame when the highest scoring predicted estimate when the highest scoring predicted estimates is less than the threshold.

79. The method of claim 25, comprising detecting a gesture of the body using the gesture data.

80. The method of claim 79, wherein the gesture comprises at least one of poses and motion of the body.

81. The method of claim 79, comprising translating the gesture into a gesture signal using a gesture notation.

82. The method of claim 81, comprising controlling a component coupled to a computer using the gesture signal.

83. The method of claim 82, comprising identifying the gesture, wherein the identifying includes identifying a pose and an orientation of a portion of the body.

84. The method of claim 83, wherein the detecting comprises generating three-dimensional space point data representing the gesture and labeling the space point data.

85. The method of claim 84, wherein the translating includes translating the space point data into commands appropriate to a configuration of the computer.

86. The method of claim 82, wherein the translating comprises translating information of the gesture to a gesture notation.

87. The method of claim 86, wherein the gesture notation represents a gesture vocabulary, and the gesture signal comprises communications of the gesture vocabulary.

88. The method of claim 87, wherein the gesture vocabulary represents in textual form instantaneous pose states of kinematic linkages of the body.

89. The method of claim 87, wherein the gesture vocabulary represents in textual form an orientation of kinematic linkages of the body.

90. The method of claim 87, wherein the gesture vocabulary represents in textual form a combination of orientations of kinematic linkages of the body.

91. The method of claim 87, wherein the gesture vocabulary includes a string of characters that represent a state of kinematic linkages of the body.

92. The method of claim 91, comprising assigning each position in the string to the appendage.

93. The method of claim 92, comprising assigning characters of a plurality of characters to each of a plurality of positions of the appendage.

94. The method of claim 93, wherein the plurality of positions are established relative to a coordinate origin.

95. The method of claim 94, comprising establishing the coordinate origin using a position selected from a group consisting of an absolute position and orientation in space, a fixed position and orientation relative to the body irrespective of an overall position and heading of the body, and interactively in response to an action of the body.

96. The method of claim 93, comprising assigning characters of the plurality of characters to each of a plurality of orientations of the appendage.

97. The method of claim 82, wherein controlling the component comprises controlling a three-space object in six degrees of freedom simultaneously by mapping the gesture of the appendage to the three-space object.

98. The method of claim 82, wherein controlling the component comprises controlling a three-space object through three translational degrees of freedom and three rotational degrees of freedom.

99. The method of claim 98, wherein the three-space object is presented on a display device coupled to the computer.

100. The method of claim 98, wherein the three-space object is coupled to the computer.

101. The method of claim 98, wherein the detecting comprises detecting when an extrapolated position of the object intersects virtual space, wherein the virtual space comprises space depicted on a display device coupled to the computer.

102. The method of claim 101, wherein controlling the component comprises controlling a virtual object in the virtual space when the extrapolated position intersects the virtual object.

103. The method of claim 79, comprising controlling scaling of the detecting and controlling to generate coincidence between virtual space and physical space, wherein the virtual space comprises space depicted on a display device coupled to the computer, wherein the physical space comprises space inhabited by the body.

104. The method of claim 103, comprising determining dimensions, orientations, and positions in the physical space of a display device coupled to the computer.

105. The method of claim 104, comprising dynamically mapping the physical space in which the display device is located as a projection into the virtual space of at least one application coupled to the computer.

106. The method of claim 103, comprising translating scale, angle, depth, and dimension between the virtual space and the physical space as appropriate to at least one application coupled to the computer.

107. The method of claim 103, comprising controlling at least one virtual object in the virtual space in response to movement of at least one physical object in the physical space.

108. The method of claim 103, comprising controlling rendering of graphics on the display device in response to position of the body in physical space relative to position of the display device.