Fast fingertip detection for initializing a vision-based hand tracker

• US 8,896,531 B2
• Filed: 03/26/2012
• Issued: 11/25/2014
• Est. Priority Date: 04/24/2008
• Status: Active Grant

• Current Assignee: Oblong Industries, Inc. (Oblong, Inc.)
• Original Assignee: Oblong Industries, Inc. (Oblong, Inc.)
• Inventors: Minnen, David
• Primary Examiner(s): Shankar, Vijay
• Application Number:
  US13/430,626
• Publication Number:
  US 20120326963A1
• Time in Patent Office: 974 Days
• Field of Search: 345156-166, 345173-179, 345204-215
• US Class Current: 345/158
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0304 Detection arrangements using opto-electronic means constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042
• CPC Class Codes: G06F 3/0325 using a plurality of light emitters or reflectors or a plurality of detectors forming a reference frame from which to derive the orientation of the object, e.g. by triangulation or on the basis of reference deformation in the picked up image
• CPC Class Codes: G06V 10/245 by locating a pattern; Special marks for positioning
• CPC Class Codes: G06V 40/107 Static hand or arm

Abstract

Systems and methods for initializing real-time, vision-based hand tracking systems are described. The systems and methods for initializing the vision-based hand tracking systems image a body and receive gesture data that is absolute three-space data of an instantaneous state of the body at a point in time and space, and at least one of determine an orientation of the body using an appendage of the body and track the body using at least one of the orientation and the gesture data.

Claims

1. A system comprising:

at least one optical detector coupled to a processor executing a gesture component;

wherein the at least one optical detector images a body and receives gesture data that is absolute three-space data of an instantaneous state of the body at a point in time and space;

wherein the gesture component detects an appendage of the body and identifies a segment of the appendage using the gesture data, and determines an orientation of the body using the segment;

wherein the gesture component tracks the body using at least one of the orientation and the gesture data;

wherein the gesture component detects and identifies a gesture of the body using only the gesture data;

wherein the gesture component translates the gesture to a gesture signal; and

wherein the gesture component controls a computer component using the gesture signal.

2. A system comprising:

at least one optical detector coupled to a processor executing a gesture component;

wherein the at least one optical detector images a body and receives gesture data that is absolute three-space data of an instantaneous state of the body at a point in time and space;

wherein the gesture component detects a gesture of a body from the gesture data by detecting a segment of an appendage of the body and determining an orientation of the body using the segment;

wherein the gesture component tracks the body using at least one of the orientation and the gesture data and, while tracking, identifies the gesture using the gesture data; and

wherein the gesture component generates a gesture signal that represents the gesture and provides the gesture signal as a control signal.

3. A system comprising:

at least one optical detector coupled to a processor executing a gesture component;

wherein the at least one optical detector images a body and receives gesture data that is absolute three-space data of an instantaneous state of the body at a point in time and space;

wherein the gesture component at least one of generates an estimated orientation of the body using an appendage of the body and tracks the body using at least one of the estimated orientation and the gesture data;

wherein the gesture component detects and identifies a gesture of the body using the gesture data, and translates the gesture to a gesture signal; and

wherein the gesture component controls a component coupled to a computer in response to the gesture signal.

4. A system comprising:

at least one optical detector coupled to a processor executing a gesture component;

wherein the at least one optical detector images a body and receives gesture data that is absolute three-space data of an instantaneous state of the body at a point in time and space;

wherein the gesture component at least one of determines an orientation of the body using an appendage of the body and tracks the body using at least one of the orientation and the gesture data;

wherein the gesture component detects and identifies a gesture of the body using the gesture data; and

wherein the gesture component translates the gesture to a gesture signal and controls a computer component using the gesture signal.

5. A system comprising:

at least one optical detector coupled to a processor executing a gesture component;

wherein the at least one optical detector images a body and receives gesture data that is absolute three-space data of an instantaneous state of the body at a point in time and space;

wherein the gesture component detects a segment of an appendage of the body and determines an orientation of the body using the segment;

wherein the gesture component tracks the body using at least one of the orientation and the gesture data.

6. A system comprising:

at least one optical detector coupled to a processor executing a gesture component;

wherein the at least one optical detector images a body and receives gesture data that is absolute three-space data of an instantaneous state of the body at a point in time and space;

wherein the gesture component detects a segment of an appendage of the body and determines an orientation of the body using the segment;

wherein the gesture component tracks the body using at least one of the orientation and the gesture data.

7. A system comprising:

at least one optical detector coupled to a processor executing a gesture component;

wherein the at least one optical detector images a body and receives gesture data that is absolute three-space data of an instantaneous state of the body at a point in time and space;

wherein the gesture component detects an appendage of the body and identifies a segment of the appendage using the gesture data;

wherein the gesture component determines an orientation of the body using data of the segment;

wherein the gesture component tracks the body using at least one of the orientation and the gesture data.

8. A system comprising:

at least one optical detector coupled to a processor executing a gesture component;

wherein the at least one optical detector images a body and receives gesture data that is absolute three-space data of an instantaneous state of the body at a point in time and space;

wherein the at least one gesture component detects an appendage of the body and identifies a region of the appendage using the gesture data; and

wherein the at least one gesture component at least one of generates an estimated orientation of the body using the appendage and tracks the body using at least one of the estimated orientation and the gesture data.

9. The system of claim 8, wherein the optical detector comprises a video detector outputting a plurality of video frames comprising the gesture data.

10. The system of claim 9, wherein the at least one gesture component detects motion in the plurality of video frames.

11. The system of claim 10, wherein the at least one gesture component detects the appendage by discriminating between at least one first pixel region of a video frame that includes the appendage and at least one second pixel region of the video frame from which the appendage is absent.

12. The system of claim 11, wherein the at least one gesture component detects edge pixels of the video frame.

13. The system of claim 12, wherein the at least one gesture component generates an edge image for the video frame.

14. The system of claim 13, wherein the at least one gesture component generates an edge orientation image for each edge pixel of the edge image.

15. The system of claim 14, wherein the generating of the edge orientation image comprises calculating an angle implied by local partial derivatives of the video frame.

16. The system of claim 14, wherein the at least one gesture component generates a body prediction using tracking data when the tracking data is available.

17. The system of claim 16, wherein the at least one gesture component generates a body prediction using at least one of the edge pixels, the edge orientation image, and the appendage.

18. The system of claim 17, wherein the at least one gesture component detects the body by comparing the body prediction to a set of body models.

19. The system of claim 18, wherein the at least one gesture component tracks the body.

20. The system of claim 19, wherein the at least one gesture component detects a gesture of the body using the gesture data.

21. The system of claim 20, wherein the gesture comprises at least one of poses and motion of the body.

22. The system of claim 20, wherein the at least one gesture component translates the gesture into a gesture signal using a gesture notation.

23. The system of claim 22, wherein the at least one gesture component controls a computer application using the gesture signal.

24. A system comprising:

at least one optical detector coupled to a processor executing a gesture component;

wherein the at least one optical detector images a body and receives gesture data that is absolute three-space data of an instantaneous state of the body at a point in time and space;

wherein the gesture component at least one of determines an orientation of the body using an appendage of the body and tracks the body using at least one of the orientation and the gesture data.

25. The system of claim 24, wherein the gesture component detects the appendage of the body and identifies a segment of the appendage using the gesture data.

26. The system of claim 25, wherein the gesture component detects the appendage of the body by discriminating between at least one first pixel region that includes the appendage and at least one second pixel region from which the appendage is absent.

27. The system of claim 25, wherein the optical detector comprises a video detector outputting a plurality of video frames comprising the gesture data.

28. The system of claim 27, comprising converting each video frame into a binary image.

29. The system of claim 27, comprising pre-processing each video frame of the plurality of video frames, the pre-processing comprising the detecting of the appendage and the identifying of the segment of the appendage.

30. The system of claim 29, wherein the gesture component generates an estimated orientation of the body using data of the segment.

31. The system of claim 27, wherein the gesture component compares each video frame of the plurality of video frames to a background model.

32. The system of claim 31, wherein the gesture component generates the background model by computing an adapting running average of the plurality of video frames.

33. The system of claim 32, wherein the gesture component converts the background model into a background binary image.

34. The system of claim 31, wherein the gesture component calculates for each pixel of the video frame a pixel value difference between a first pixel value of the respective pixel of the video frame and a second pixel value of each respective corresponding pixel of the background model.

35. The system of claim 34, wherein the gesture component generates marked pixels by marking each pixel of the video frame for which the pixel value difference exceeds a threshold value.

36. The system of claim 35, wherein the gesture component generates a foreground image comprising the marked pixels.

37. The system of claim 36, wherein the gesture component extracts at least one feature from the gesture data, wherein the at least one feature comprises at least one pixel of the foreground image.

38. The system of claim 37, wherein the detecting comprises applying the at least one feature to a plurality of filters in turn, and identifying the at least one feature as including the appendage when the at least one feature is accepted by all filters of the plurality of filters.

39. The system of claim 37, wherein the at least one feature comprises at least one box, wherein the at least one box is specified using a plurality of coordinates within at least one query region of the gesture data.

40. The system of claim 39, wherein the gesture component specifies the at least one box using an upper-left coordinate and a lower-right coordinate of the at least one box.

41. The system of claim 39, wherein the at least one box is characterized by a number of pixels of the at least one foreground image.

42. The system of claim 39, wherein the at least one feature comprises at least one ring, wherein the at least one ring is an unfilled rectangle region within a box.

43. The system of claim 42, wherein the gesture component applies the at least one feature to a plurality of filters in turn, and identifies the at least one feature as including the appendage when the at least one feature is accepted by all filters of the plurality of filters.

44. The system of claim 43, wherein the appendage comprises a fingertip.

45. The system of claim 44, wherein the body comprises a finger.

46. The system of claim 43, wherein the body comprises a hand.

47. The system of claim 43, wherein the body comprises an arm.

48. The system of claim 43, wherein the gesture component, for each detected instance of the appendage, generates a body prediction corresponding to the appendage.

49. The system of claim 48, wherein the body prediction comprises a predicted position corresponding to the appendage.

50. The system of claim 48, wherein the body prediction comprises the estimated orientation corresponding to the detected region.

51. The system of claim 48, wherein the gesture component generates a body prediction list and includes the body prediction in the body prediction list, wherein the body prediction list comprises a plurality of body predictions.

52. The system of claim 51, wherein the gesture component generates an edge image for the video frame.

53. The system of claim 52, wherein the generating of the edge image is provided by an edge detector.

54. The system of claim 52, wherein the gesture component generates an edge orientation image for each edge pixel of the edge image.

55. The system of claim 54, wherein the generating of the edge orientation image comprises calculating an angle implied by local partial derivatives.

56. The system of claim 54, wherein the gesture component, when tracking of the body is in progress, generates a predicted estimate for the body prediction using a previous estimate of the body prediction from at least one previous video frame.

57. The system of claim 56, wherein the gesture component, when tracking of a plurality of body hypotheses is in progress, generates for each body hypothesis a predicted estimate for the body prediction using a previous estimate of the body prediction from at least one previous video frame.

58. The system of claim 56, wherein the gesture component removes from the body prediction list duplicates of the body predictions corresponding to the video frame.

59. The system of claim 56, wherein the gesture component, for each predicted estimate, identifies a best parameter value.

60. The system of claim 59, wherein the gesture component identifies the best parameter value using particle swarm optimization.

61. The system of claim 59, wherein the gesture component identifies the best parameter value using a set of body models.

62. The system of claim 61, wherein the set of body models comprises a set of hand models.

63. The system of claim 61, comprising generating the set of hand models by creating a contour that matches a shape of a hand in each pose of a plurality of poses.

64. The system of claim 63, comprising representing the contour using a quadratic b-spline.

65. The system of claim 61, wherein the best parameter value comprises at least one parameter value of a body prediction that best matches a body model of the set of body models, wherein the body prediction corresponds to the gesture data.

66. The system of claim 65, wherein the best parameter value comprises at least one of position, scale, and orientation.

67. The system of claim 65, wherein the best parameter value comprises position.

68. The system of claim 65, wherein the best parameter value comprises scale.

69. The system of claim 65, wherein the best parameter value comprises orientation.

70. The system of claim 65, wherein the gesture component, for each parameter value, updates control points of a hand spline of the body model.

71. The system of claim 70, wherein the updating comprises updating via at least one of translation, scaling, and rotation.

72. The system of claim 70, wherein the updating comprises updating via translation.

73. The system of claim 70, wherein the updating comprises updating via scaling.

74. The system of claim 70, wherein the updating comprises updating via rotation.

75. The system of claim 70, comprising sampling along the hand spline.

76. The system of claim 75, wherein the gesture component compares local contour orientation of the hand spline to oriented edges of the edge orientation image of the video frame.

77. The system of claim 76, wherein the gesture component compares by searching along a normal of the hand spline for a closest edge with a matching orientation to the body prediction.

78. The system of claim 76, wherein the gesture component updates the plurality of predicted estimates using the best parameter values.

79. The system of claim 78, wherein the gesture component reports as a current body location a highest scoring predicted estimate when the highest scoring predicted estimate exceeds a threshold.

80. The system of claim 79, wherein the gesture component reports absence of a body in a corresponding frame when the highest scoring predicted estimate when the highest scoring predicted estimates is less than the threshold.

81. The system of claim 27, wherein the gesture component detects a gesture of the body using the gesture data.

82. The system of claim 81, wherein the gesture comprises at least one of poses and motion of the body.

83. The system of claim 81, wherein the gesture component translates the gesture into a gesture signal using a gesture notation.

84. The system of claim 83, wherein the gesture component controls a component coupled to a computer using the gesture signal.

85. The system of claim 84, wherein the gesture component identifies the gesture, wherein the identifying includes identifying a pose and an orientation of a portion of the body.

86. The system of claim 85, wherein the detecting comprises generating three-dimensional space point data representing the gesture and labeling the space point data.

87. The system of claim 86, wherein the translating includes translating the space point data into commands appropriate to a configuration of the computer.

88. The system of claim 84, wherein the translating comprises translating information of the gesture to a gesture notation.

89. The system of claim 88, wherein the gesture notation represents a gesture vocabulary, and the gesture signal comprises communications of the gesture vocabulary.

90. The system of claim 89, wherein the gesture vocabulary represents in textual form instantaneous pose states of kinematic linkages of the body.

91. The system of claim 89, wherein the gesture vocabulary represents in textual form an orientation of kinematic linkages of the body.

92. The system of claim 89, wherein the gesture vocabulary represents in textual form a combination of orientations of kinematic linkages of the body.

93. The system of claim 89, wherein the gesture vocabulary includes a string of characters that represent a state of kinematic linkages of the body.

94. The system of claim 93, wherein the gesture component assigns each position in the string to the appendage.

95. The system of claim 94, wherein the gesture component assigns characters of a plurality of characters to each of a plurality of positions of the appendage.

96. The system of claim 95, wherein the plurality of positions are established relative to a coordinate origin.

97. The system of claim 96, wherein the gesture component establishes the coordinate origin using a position selected from a group consisting of an absolute position and orientation in space, a fixed position and orientation relative to the body irrespective of an overall position and heading of the body, and interactively in response to an action of the body.

98. The system of claim 92, wherein the gesture component assigns characters of the plurality of characters to each of a plurality of orientations of the appendage.

99. The system of claim 84, wherein controlling the component comprises controlling a three-space object in six degrees of freedom simultaneously by mapping the gesture of the appendage to the three-space object.

100. The system of claim 84, wherein controlling the component comprises controlling a three-space object through three translational degrees of freedom and three rotational degrees of freedom.

101. The system of claim 100, wherein the three-space object is presented on a display device coupled to the computer.

102. The system of claim 100, wherein the three-space object is coupled to the computer.

103. The system of claim 100, wherein the detecting comprises detecting when an extrapolated position of the object intersects virtual space, wherein the virtual space comprises space depicted on a display device coupled to the computer.

104. The system of claim 103, wherein controlling the component comprises controlling a virtual object in the virtual space when the extrapolated position intersects the virtual object.

105. The system of claim 81, comprising controlling scaling of the detecting and controlling to generate coincidence between virtual space and physical space, wherein the virtual space comprises space depicted on a display device coupled to the computer, wherein the physical space comprises space inhabited by the body.

106. The system of claim 105, comprising determining dimensions, orientations, and positions in the physical space of a display device coupled to the computer.

107. The system of claim 106, comprising dynamically mapping the physical space in which the display device is located as a projection into the virtual space of at least one application coupled to the computer.

108. The system of claim 105, comprising translating scale, angle, depth, and dimension between the virtual space and the physical space as appropriate to at least one application coupled to the computer.

109. The system of claim 105, comprising controlling at least one virtual object in the virtual space in response to movement of at least one physical object in the physical space.

110. The system of claim 105, comprising controlling rendering of graphics on the display device in response to position of the body in physical space relative to position of the display device.