Vision-based pointer tracking and object classification method and apparatus

• US 20030095140A1
• Filed: 10/12/2002
• Published: 05/22/2003
• Est. Priority Date: 10/12/2001
• Status: Active Grant

• Current Assignee: HRL Laboratories LLC (The Boeing Co.)
• Original Assignee: HRL Laboratories LLC (The Boeing Co.)
• Inventors: Keaton, Patricia (Trish), Sayed, Ali H., Dominguez, Sylvia
• Granted Patent: US 7,148,913 B2
• Time in Patent Office: Days
• US Class Current: 715/700
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/0304 Detection arrangements using opto-electronic means constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042
• CPC Class Codes: G06F 3/042 by opto-electronic means
• CPC Class Codes: G06V 40/107 Static hand or arm

Abstract

A vision-based pointer tracking and object recognizing system is presented. The system comprises a computer system including a processor, a memory coupled with the processor, an input coupled with the processor for receiving a series of image frames from a camera, and an output coupled with the processor for outputting a path of a pointer. The computer system includes a tracking system and a recognition system which are configured to accurately and efficiently track the fingertip of a user in order to allow the user to perform coarse segmentation of an object in preparation for object classification. The tracking system includes a Robust Tracking Filter that accommodates uncertainties. The object recognition system incorporates an invariant object recognition technique that allows objects to be recognized despite changes in user viewpoint.

Claims

1. A vision-based pointer tracking system comprising:

a computer system including a processor, a memory coupled with the processor, an input coupled with the processor for receiving a series of image frames from a camera, and an output coupled with the processor for outputting a path of a pointer, the computer system further comprising means, residing in its processor and memory for;

searching a search window within the image frame for a portion of the window indicative of a pointer;

recognizing the pointer;

determining the position and trajectory of the pointer;

robustly predicting the position of the pointer in the next image frame in the series of image frames;

adjusting the search window within the next image frame such that the search window comprises a subset of the next image frame positioned so that the actual position of the pointer is likely to be within the search window in the next image frame;

tracking the path of the pointer through an area represented by the series of image frames; and

outputting the path of the pointer, whereby the means for robustly predicting the position of the pointer, combined with the adjusted search window enables the system to track the path of a pointer in a computationally efficient manner.

2. A vision-based pointer tracking system as set forth in claim 1, wherein the means for recognizing the pointer is a combination color segmenter and shape identifier, whereby the color segmenter operates on the search window to determine whether pointer candidates are present, and the shape identifier filters the pointer candidates to recognize the pointer within the search window.

3. A vision-based pointer tracking system as set forth in claim 2, wherein the means for recognizing the pointer is tailored for recognizing a finger of a user, whereby the color segmenter operates on the search window to find finger-colored portions of the search window, and the shape identifier filters the finger-colored portions of the search window in order to find the finger.

4. A vision-based pointer tracking system as set forth in claim 2, wherein the color segmenter is selected from a group consisting of a mean shift theorem algorithm used to adaptively quantize a color space to reduce the number of colors;

a Gaussian mixture modeling algorithm;

a thresholding algorithm used to limit the number of colors to a set designated apriori; and

a Markov-random field algorithm.

5. A vision-based pointer tracking system as set forth in claim 4, wherein the means for recognizing the pointer is tailored for recognizing a finger of a user, whereby the color segmenter operates on the search window to find finger-colored portions of the search window, and the shape identifier filters the finger-colored portions of the search window in order to find the finger.

6. A vision-based pointer tracking system as set forth in claim 1, wherein the means for robustly predicting the position of the pointer is an algorithm selected from a group consisting of Kalman filters, Extended Kalman filters, and Particle filters and adapted to accommodate uncertainty.

7. A vision-based pointer tracking system as set forth in claim 1, wherein the means for adjusting the search window within the next frame centers a search window of predetermined size on the predicted position of the pointer.

8. A vision-based pointer tracking system as set forth in claim 7, wherein the size of the search window is determined as a function of the uncertainty in the motion of the pointer.

9. A vision-based pointer tracking system as set forth in claim 1, wherein the search window has a size and a shape, and wherein the means for adjusting the search window adjusts size and shape of the search window based on historical pointer movements.

10. A vision-based pointer tracking system as set forth in claim 1, where when a pointer is not recognized in the search window, the search window is redefined to encompass the entire image frame so that recognition may be attempted again in a larger area.

11. A vision-based pointer tracking system as set forth in claim 1, wherein the means for robustly predicting the position of the pointer is a Robust Tracking Filter.

12. A vision-based pointer tracking system as set forth in claim 11, wherein the means for recognizing the pointer is tailored for recognizing a finger of a user, whereby a color segmenter operates on the search window to find finger-colored portions of the search window, and a shape identifier filters finger-colored portions of the search window in order to find the finger.

13. A vision-based pointer tracking system as set forth in claim 11, wherein the means for recognizing the pointer is a combination color segmenter and shape identifier, whereby the color segmenter operates on the search window to determine the presence of pointer candidates, and the shape identifier filters the pointer candidates to recognize the pointer within the search window.

14. A vision-based pointer tracking system as set forth in claim 13, wherein the color segmenter selected from a group consisting of a mean shift theorem algorithm used to adaptively quantize a color space to reduce the number of colors;

a Gaussian mixture modeling algorithm;

a thresholding algorithm used to limit the number of colors to a set designated apriori; and

a Markov-random field algorithm.

15. A vision-based pointer tracking system as set forth in claim 14, wherein the means for recognizing the pointer is tailored for recognizing a finger of a user, whereby the color segmenter operates on the search window to find finger-colored portions of the search window, and the shape identifier filters the finger-colored portions of the search window in order to find the finger.

16. A vision-based pointer tracking system as set forth in claim 15, wherein the search window has a size and a shape, and wherein the means for adjusting the search window adjusts size and shape of the search window based on historical pointer movements.

17. A vision-based pointer tracking system as set forth in claim 15, wherein the means for adjusting the search window within the next frame centers a search window of predetermined size on the predicted position of the pointer.

18. A vision-based pointer tracking system as set forth in claim 17, wherein the size of the search window is determined as a function of the uncertainty in the motion of the pointer.

19. A vision-based pointer tracking system as set forth in claim 18, where when a pointer is not recognized in the search window, the search window is redefined to encompass the entire image frame so that recognition may be attempted again in a larger area.

20. A vision-based pointer tracking system as set forth in claim 1, wherein the output is configured to provide the path of the pointer to an object recognizer system residing in the processor and memory, with the object recognizer comprising means for:

segmenting an object indicated by the path of the pointer; and

recognizing the object, whereby the pointer may be used to indicate an area in which an object to be recognized resides, and the system can then segment and recognize the object from a scene.

21. A vision-based pointer tracking system as set forth in claim 20, wherein the means for recognizing the object comprises means for:

extracting features from the segmented object; and

classifying the object based on the extracted features to generate an object classification.

22. A vision-based pointer tracking system as set forth in claim 21, wherein the output of the system is configured to output the object classification.

23. A vision-based pointer tracking system as set forth in claim 22, wherein the output is connected with a database system, such that the object classification is communicated with the database system, and wherein the input is connected with the database system to receive supplementary information regarding the classified object;

whereby once an object is classified, information regarding its classification is sent to a database in order to retrieve stored information regarding the object.

24. A vision-based pointer tracking system as set forth in claim 23, wherein the output of the system is configured to provide the user with at least one type of feedback selected from a group consisting of audio feedback, visual feedback, and kinesthetic feedback based on the path of the pointer, whereby the system can indicate when a path is completed for the recognition process to begin.

25. A vision-based pointer tracking system as set forth in claim 23, wherein the output of the system is configured to provide the user with at least one type of feedback selected from a group consisting of audio feedback, visual feedback, and kinesthetic feedback based on the classification of the object.

26. A vision-based pointer tracking system as set forth in claim 21, wherein the output of the system is configured to provide the user with at least one type of feedback selected from a group consisting of audio feedback, visual feedback, and kinesthetic feedback based on the path of the pointer, whereby the system can indicate when a path is completed for the recognition process to begin.

27. A vision-based pointer tracking system as set forth in claim 21, wherein the output of the system is configured to provide the user with at least one type of feedback selected from a group consisting of audio feedback, visual feedback, and kinesthetic feedback based on the classification of the object.

28. A vision-based pointer tracking system as set forth in claim 21, wherein the system further comprises a database, and wherein the classification of the object is communicated with the database and the database, in turn, recalls supplementary information regarding the object.

29. A vision-based pointer tracking system as set forth in claim 28, wherein the output of the system is configured to provide at least one type of feedback selected from a group consisting of audio feedback, visual feedback, and kinesthetic feedback based on the path of the pointer, whereby the system can indicate when a path is completed for the recognition process to begin.

30. A vision-based pointer tracking system as set forth in claim 28, wherein the output of the system is configured to provide the user with at least one type of feedback selected from a group consisting of audio feedback, visual feedback, and kinesthetic feedback based on the classification of the object.

31. A vision-based pointer tracking system as set forth in claim 21, wherein the means for extracting features from the segmented object is selected from a group consisting of a combination of factor analyzing algorithms and an EM algorithm, a color histogram matching algorithm, a wavelet algorithm, a DCT coefficient algorithm, a texture-based algorithm, and an edge-based algorithm.

32. A vision-based pointer tracking system as set forth in claim 31, wherein the means for classifying the object based on the extracted features is selected from a group consisting of a K-nearest neighbor algorithm, a neural network, and a support vector machine.

33. A vision-based pointer tracking system as set forth in claim 32, wherein the output of the system is configured to output the object classification.

34. A vision-based pointer tracking system as set forth in claim 33, wherein the output is connected with a database system, such that the object classification is communicated with the database system, and wherein the input is connected with the database system to receive supplementary information regarding the classified object;

whereby once an object is classified, information regarding its classification is sent to a database in order to retrieve stored information regarding the object.

35. A vision-based pointer tracking system as set forth in claim 34, wherein the output of the system is configured to provide the user with at least one type of feedback selected from a group consisting of audio feedback, visual feedback, and kinesthetic feedback based on the path of the pointer, whereby the system can indicate when a path is completed for the recognition process to begin.

36. A vision-based pointer tracking system as set forth in claim 34, wherein the output of the system is configured to provide the user with at least one type of feedback selected from a group consisting of audio feedback, visual feedback, and kinesthetic feedback based on the classification of the object.

37. A vision-based pointer tracking system as set forth in claim 32, wherein the output of the system is configured to provide the user with at least one type of feedback selected from a group consisting of audio feedback, visual feedback, and kinesthetic feedback based on the path of the pointer, whereby the system can indicate when a path is completed for the recognition process to begin.

38. A vision-based pointer tracking system as set forth in claim 32, wherein the output of the system is configured to provide the user with at least one type of feedback selected from a group consisting of audio feedback, visual feedback, and kinesthetic feedback based on the classification of the object.

39. A vision-based pointer tracking system as set forth in claim 32, wherein the system further comprises a database, and wherein the classification of the object is communicated with the database and the database, in turn, recalls supplementary information regarding the object.

40. A vision-based pointer tracking system as set forth in claim 39, wherein the output of the system is configured to provide the user with at least one type of feedback selected from a group consisting of audio feedback, visual feedback, and kinesthetic feedback based on the path of the pointer, whereby the system can indicate when a path is completed for the recognition process to begin.

41. A vision-based pointer tracking system as set forth in claim 39, wherein the output of the system is configured to provide the user with at least one type of feedback selected from a group consisting of audio feedback, visual feedback, and kinesthetic feedback based on the classification of the object.

42. A vision-based pointer tracking system as set forth in claim 21, wherein the means for classifying the object based on the extracted features is selected from a group consisting of a K-nearest neighbor algorithm, a neural network, and a support vector machine.

43. A vision-based pointer tracking system as set forth in claim 42, wherein the output of the system is configured to output the object classification.

44. A vision-based pointer tracking system as set forth in claim 43, wherein the output is connected with a database system, such that the object classification is communicated with the database system, and wherein the input is connected with the database system to receive supplementary information regarding the classified object;

whereby once an object is classified, information regarding its classification is sent to a database in order to retrieve stored information regarding the object.

45. A vision-based pointer tracking system as set forth in claim 44, wherein the output of the system is configured to provide the user with at least one type of feedback selected from a group consisting of audio feedback, visual feedback, and kinesthetic feedback based on the path of the pointer, whereby the system can indicate when a path is completed for the recognition process to begin.

46. A vision-based pointer tracking system as set forth in claim 44, wherein the output of the system is configured to provide the user with at least one type of feedback selected from a group consisting of audio feedback, visual feedback, and kinesthetic feedback based on the classification of the object.

47. A vision-based pointer tracking system as set forth in claim 42, wherein the output of the system is configured to provide the user with at least one type of feedback selected from a group consisting of audio feedback, visual feedback, and kinesthetic feedback based on the path of the pointer, whereby the system can indicate when a path is completed for the recognition process to begin.

48. A vision-based pointer tracking system as set forth in claim 42, wherein the output of the system is configured to provide the user with at least one type of feedback selected from a group consisting of audio feedback, visual feedback, and kinesthetic feedback based on the classification of the object.

49. A vision-based pointer tracking system as set forth in claim 42, wherein the system further comprises a database, and wherein the classification of the object is communicated with the database and the database, in turn, recalls supplementary information regarding the object.

50. A vision-based pointer tracking system as set forth in claim 49, wherein the output of the system is configured to provide the user with at least one type of feedback selected from a group consisting of audio feedback, visual feedback, and kinesthetic feedback based on the path of the pointer, whereby the system can indicate when a path is completed for the recognition process to begin.

51. A vision-based pointer tracking system as set forth in claim 49, wherein the output of the system is configured to provide the user with at least one type of feedback selected from a group consisting of audio feedback, visual feedback, and kinesthetic feedback based on the classification of the object.

52. A vision-based pointer tracking system as set forth in claim 20, wherein the means for recognizing the object comprises a means for extracting features from the segmented object, and wherein the output of the system is configured to output the extracted features.

53. A vision-based pointer tracking system as set forth in claim 20, wherein the means for robustly predicting the position of the pointer is a Robust Tracking Filter.

54. A vision-based pointer tracking system as set forth in claim 53, wherein the means for recognizing the pointer is tailored for recognizing a finger of a user, whereby the color segmenter operates on the search window to find finger-colored portions of the search window, and the shape identifier filters the finger-colored portions of the search window in order to find the finger.

55. A vision-based pointer tracking system as set forth in claim 53, wherein the means for recognizing the pointer is a combination color segmenter and shape identifier, whereby the color segmenter operates on the search window to determine the presence of pointer candidates, and the shape identifier filters the pointer candidates to recognize the pointer within the search window.

56. A vision-based pointer tracking system as set forth in claim 55, wherein the color segmenter selected from a group consisting of a mean shift theorem algorithm used to adaptively quantize a color space to reduce the number of colors;

a Gaussian mixture modeling algorithm;

a thresholding algorithm used to limit the number of colors to a set designated apriori; and

a Markov-random field algorithm.

57. A vision-based pointer tracking system as set forth in claim 56, wherein the means for recognizing the pointer is tailored for recognizing a finger of a user, whereby the color segmenter operates on the search window to find finger-colored portions of the search window, and the shape identifier filters the finger-colored portions of the search window in order to find the finger.

58. A vision-based pointer tracking system as set forth in claim 57, wherein the search window has a size and a shape, and wherein the means for adjusting the search window adjusts size and shape of the search window based on historical pointer movements.

59. A vision-based pointer tracking system as set forth in claim 57, wherein the means for adjusting the search window within the next frame centers a search window of predetermined size on the predicted position of the pointer.

60. A vision-based pointer tracking system as set forth in claim 59, wherein the size of the search window is determined as a function of the uncertainty in the motion of the pointer.

61. A vision-based pointer tracking system as set forth in claim 60, where when a pointer is not recognized in the search window, the search window is redefined to encompass the entire image frame so that recognition may be attempted again in a larger area.

81. A method for vision-based pointer tracking as set forth in claim 1, wherein the output is configured to provide the path of the pointer to an object recognizer system residing in the processor and memory, with the object recognizer comprising step of:

segmenting an object indicated by the path of the pointer; and

recognizing the object, whereby the pointer may be used to indicate an area in which an object to be recognized resides, and the system can then segment and recognize the object from a scene.

82. A method for vision-based pointer tracking as set forth in claim 81, wherein the step of recognizing the object comprises the steps of:

extracting features from the segmented object; and

classifying the object based on the extracted features to generate an object classification.

83. A method for vision-based pointer tracking as set forth in claim 82, wherein the outputting step outputs the object classification.

84. A method for vision-based pointer tracking as set forth in claim 83, further comprising steps of communicating the object classification with a database system, and receiving supplementary information regarding the classified object;

whereby once an object is classified, information regarding its classification is sent to a database in order to retrieve stored information regarding the object.

85. A method for vision-based pointer tracking as set forth in claim 84, wherein the in the step of outputting, the user is provided with at least one type of feedback selected from a group consisting of audio feedback, visual feedback, and kinesthetic feedback based on the path of the pointer, whereby the method indicates when a path is completed for the recognition process to begin.

86. A method for vision-based pointer tracking as set forth in claim 84, wherein in the step of outputting, the user is provided with at least one type of feedback selected from a group consisting of audio feedback, visual feedback, and kinesthetic feedback based on the classification of the object.

87. A method for vision-based pointer tracking as set forth in claim 82, wherein the in the step of outputting, the user is provided with at least one type of feedback selected from a group consisting of audio feedback, visual feedback, and kinesthetic feedback based on the path of the pointer, whereby the method indicates when a path is completed for the recognition process to begin.

88. A method for vision-based pointer tracking as set forth in claim 82, wherein in the step of outputting, the user is provided with at least one type of feedback selected from a group consisting of audio feedback, visual feedback, and kinesthetic feedback based on the classification of the object.

89. A method for vision-based pointer tracking as set forth in claim 82, wherein the step of extracting features from the segmented object is an algorithm selected from a group consisting of a combination of factor analyzing algorithms and an EM algorithm, a color histogram matching algorithm, a wavelet algorithm, a DCT coefficient algorithm, a texture-based algorithm, and an edge-based algorithm.

90. A method for vision-based pointer tracking as set forth in claim 89, wherein the step of classifying the object based on the extracted features is a technique selected from a group consisting of a K-nearest neighbor algorithm, a neural network, and a support vector machine.

91. A method for vision-based pointer tracking as set forth in claim 90, wherein the outputting step outputs the object classification.

92. A method for vision-based pointer tracking as set forth in claim 91, further comprising steps of communicating the object classification with a database system, and receiving supplementary information regarding the classified object;

whereby once an object is classified, information regarding its classification is sent to a database in order to retrieve stored information regarding the object.

93. A method for vision-based pointer tracking as set forth in claim 92, wherein the in the step of outputting, the user is provided with at least one type of feedback selected from a group consisting of audio feedback, visual feedback, and kinesthetic feedback based on the path of the pointer, whereby the method indicates when a path is completed for the recognition process to begin.

94. A method for vision-based pointer tracking as set forth in claim 92, wherein in the step of outputting, the user is provided with at least one type of feedback selected from a group consisting of audio feedback, visual feedback, and kinesthetic feedback based on the classification of the object.

95. A method for vision-based pointer tracking as set forth in claim 32, wherein the in the step of outputting, the user is provided with at least one type of feedback selected from a group consisting of audio feedback, visual feedback, and kinesthetic feedback based on the path of the pointer, whereby the method indicates when a path is completed for the recognition process to begin.

96. A method for vision-based pointer tracking as set forth in claim 32, wherein in the step of outputting, the user is provided with at least one type of feedback selected from a group consisting of audio feedback, visual feedback, and kinesthetic feedback based on the classification of the object.

97. A method for vision-based pointer tracking as set forth in claim 82, wherein the step of classifying the object based on the extracted features is a technique selected from a group consisting of a K-nearest neighbor algorithm, a neural network, and a support vector machine.

98. A method for vision-based pointer tracking as set forth in claim 42, wherein the outputting step outputs the object classification.

99. A method for vision-based pointer tracking as set forth in claim 98, further comprising steps of communicating the object classification with a database system, and receiving supplementary information regarding the classified object;

whereby once an object is classified, information regarding its classification is sent to a database in order to retrieve stored information regarding the object.

100. A method for vision-based pointer tracking as set forth in claim 99, wherein the in the step of outputting, the user is provided with at least one type of feedback selected from a group consisting of audio feedback, visual feedback, and kinesthetic feedback based on the path of the pointer, whereby the method indicates when a path is completed for the recognition process to begin.

101. A method for vision-based pointer tracking as set forth in claim 99, wherein in the step of outputting, the user is provided with at least one type of feedback selected from a group consisting of audio feedback, visual feedback, and kinesthetic feedback based on the classification of the object.

102. A method for vision-based pointer tracking as set forth in claim 42, wherein the in the step of outputting, the user is provided with at least one type of feedback selected from a group consisting of audio feedback, visual feedback, and kinesthetic feedback based on the path of the pointer, whereby the method indicates when a path is completed for the recognition process to begin.

103. A method for vision-based pointer tracking as set forth in claim 42, wherein in the step of outputting, the user is provided with at least one type of feedback selected from a group consisting of audio feedback, visual feedback, and kinesthetic feedback based on the classification of the object.

104. A method for vision-based pointer tracking as set forth in claim 81, wherein the step of recognizing the object comprises a step of extracting features from the segmented object, and wherein in the outputting step, the extracted features are outputted.

105. A method for vision-based pointer tracking as set forth in claim 20, wherein the step of robustly predicting the position of the pointer is performed by a Robust Tracking Filter based on state-space estimation.

106. A method for vision-based pointer tracking as set forth in claim 105, wherein the step of recognizing the pointer is tailored for recognizing a finger of a user, whereby the color segmenting step operates on the search window to find finger-colored portions of the search window, and the shape identifying step filters the finger-colored portions of the search window in order to find the finger.

107. A method for vision-based pointer tracking as set forth in claim 105, wherein the step of recognizing the pointer performs a combination of a color segmenting step and a shape identifying step, whereby the color segmenting step operates on the search window to determine the presence of pointer candidates, and the shape identifying step filters the pointer candidates to recognize the pointer within the search window.

108. A method for vision-based pointer tracking as set forth in claim 107, wherein the color segmenting step performs an algorithm selected from a group consisting of a mean shift theorem algorithm used to adaptively quantize a color space to reduce the number of colors;

a Gaussian mixture modeling algorithm;

a thresholding algorithm used to limit the number of colors to a set designated apriori; and

a Markov-random field algorithm.

109. A method for vision-based pointer tracking as set forth in claim 108, wherein the step of recognizing the pointer is tailored for recognizing a finger of a user, whereby the color segmenter operates on the search window to find finger-colored portions of the search window, and the shape identifier filters the finger-colored portions of the search window in order to find the finger.

110. A method for vision-based pointer tracking as set forth in claim 109, wherein the search window has a size and a shape, and wherein the step of adjusting the search window adjusts size and shape of the search window based on historical pointer movements.

111. A method for vision-based pointer tracking as set forth in claim 109, wherein the step of adjusting the search window within the next frame centers a search window of predetermined size on the predicted position of the pointer.

112. A method for vision-based pointer tracking as set forth in claim 111, wherein further comprising a step of determining the size of the search window as a function of the uncertainty in the motion of the pointer.

113. A method for vision-based pointer tracking as set forth in claim 112, where when a pointer is not recognized in the search window, a step of redefining the search window is performed, redefining the search window to encompass the entire image frame so that recognition may be attempted again in a larger area.

124. A computer program product for vision-based pointer tracking as set forth in claim 1, wherein the means for robustly predicting the position of the pointer is a Robust Tracking Filter.

125. A computer program product for vision-based pointer tracking as set forth in claim 124, wherein the means for recognizing the pointer is tailored for recognizing a finger of a user, whereby the color segmenter operates on the search window to find finger-colored portions of the search window, and the shape identifier filters the finger-colored portions of the search window in order to find the finger.

126. A computer program product for vision-based pointer tracking as set forth in claim 124, wherein the means for recognizing the pointer is a combination color segmenter and shape identifier, whereby the color segmenter operates on the search window to determine the presence of pointer candidates, and the shape identifier filters the pointer candidates to recognize the pointer within the search window.

127. A computer program product for vision-based pointer tracking as set forth in claim 126, wherein the color segmenter selected from a group consisting of a mean shift theorem algorithm used to adaptively quantize a color space to reduce the number of colors;

a Gaussian mixture modeling algorithm;

a thresholding algorithm used to limit the number of colors to a set designated apriori; and

a Markov-random field algorithm.

128. A computer program product for vision-based pointer tracking as set forth in claim 127, wherein the means for recognizing the pointer is tailored for recognizing a finger of a user, whereby the color segmenter operates on the search window to find finger-colored portions of the search window, and the shape identifier filters the finger-colored portions of the search window in order to find the finger.

129. A computer program product for vision-based pointer tracking as set forth in claim 128, wherein the search window has a size and a shape, and wherein the means for adjusting the search window adjusts size and shape of the search window based on historical pointer movements.

130. A computer program product for vision-based pointer tracking as set forth in claim 129, wherein the means for adjusting the search window within the next frame centers a search window of predetermined size on the predicted position of the pointer.

131. A computer program product for vision-based pointer tracking as set forth in claim 130, wherein the size of the search window is determined as a function of the uncertainty in the motion of the pointer.

132. A computer program product for vision-based pointer tracking as set forth in claim 131, where when a pointer is not recognized in the search window, the search window is redefined to encompass the entire image frame so that recognition may be attempted again in a larger area.

133. A computer program product for vision-based pointer tracking as set forth in claim 1, wherein the output is configured to provide the path of the pointer to an object recognizer system residing in the processor and memory, with the object recognizer comprising means for:

segmenting an object indicated by the path of the pointer; and

recognizing the object, whereby the pointer may be used to indicate an area in which an object to be recognized resides, and the system can then segment and recognize the object from a scene.

134. A computer program product for vision-based pointer tracking as set forth in claim 133, wherein the means for recognizing the object comprises means for:

extracting features from the segmented object; and

classifying the object based on the extracted features to generate an object classification.

135. A computer program product for vision-based pointer tracking as set forth in claim 134, wherein the means for outputting is configured to output the object classification.

136. A computer program product for vision-based pointer tracking as set forth in claim 135, wherein means for outputting is configured to connect with a database system, so that the object classification is communicated with the database system, and wherein a means for inputting is configured to connect with the database system to receive supplementary information regarding the classified object;

whereby once an object is classified, information regarding its classification is sent to a database in order to retrieve stored information regarding the object.

137. A computer program product for vision-based pointer tracking as set forth in claim 136, wherein the means for outputting is configured to provide the user with at least one type of feedback selected from a group consisting of audio feedback, visual feedback, and kinesthetic feedback based on the path of the pointer, whereby the system can indicate when a path is completed for the recognition process to begin.

138. A computer program product for vision-based pointer tracking as set forth in claim 136, wherein the means for outputting is configured to provide the user with at least one type of feedback selected from a group consisting of audio feedback, visual feedback, and kinesthetic feedback based on the classification of the object.

139. A computer program product for vision-based pointer tracking as set forth in claim 134, wherein the means for outputting is configured to provide the user with at least one type of feedback selected from a group consisting of audio feedback, visual feedback, and kinesthetic feedback based on the path of the pointer, whereby the system can indicate when a path is completed for the recognition process to begin.

140. A computer program product for vision-based pointer tracking as set forth in claim 134, wherein the means for outputting is configured to provide the user with at least one type of feedback selected from a group consisting of audio feedback, visual feedback, and kinesthetic feedback based on the classification of the object.

141. A computer program product for vision-based pointer tracking as set forth in claim 134, wherein the computer program product further comprises a database, and wherein the classification of the object may be communicated with the database and the database, in turn, may recall supplementary information regarding the object.

142. A computer program product for vision-based pointer tracking as set forth in claim 141, wherein the means for outputting is configured to provide the user with at least one type of feedback selected from a group consisting of audio feedback, visual feedback, and kinesthetic feedback based on the path of the pointer, whereby the system can indicate when a path is completed for the recognition process to begin.

143. A computer program product for vision-based pointer tracking as set forth in claim 141, wherein the means for outputting is configured to provide the user with at least one type of feedback selected from a group consisting of audio feedback, visual feedback, and kinesthetic feedback based on the classification of the object.

144. A computer program product for vision-based pointer tracking as set forth in claim 134, wherein the means for extracting features from the segmented object is selected from a group consisting of a combination of factor analyzing algorithms and an EM algorithm, a color histogram matching algorithm, a wavelet algorithm, a DCT coefficient algorithm, a texture-based algorithm, and an edge-based algorithm.

145. A computer program product for vision-based pointer tracking as set forth in claim 144, wherein the means for classifying the object based on the extracted features is selected from a group consisting of a K-nearest neighbor algorithm, a neural network, and a support vector machine.

146. A computer program product for vision-based pointer tracking as set forth in claim 145, wherein the means for outputting is configured to output the object classification.

147. A computer program product for vision-based pointer tracking as set forth in claim 146, wherein means for outputting is configured to connect with a database system, so that the object classification is communicated with the database system, and wherein a means for inputting is configured to connect with the database system to receive supplementary information regarding the classified object;

whereby once an object is classified, information regarding its classification is sent to a database in order to retrieve stored information regarding the object.

148. A computer program product for vision-based pointer tracking as set forth in claim 147, wherein the means for outputting is configured to provide the user with at least one type of feedback selected from a group consisting of audio feedback, visual feedback, and kinesthetic feedback based on the path of the pointer, whereby the system can indicate when a path is completed for the recognition process to begin.

149. A computer program product for vision-based pointer tracking as set forth in claim 147, wherein the means for outputting is configured to provide the user with at least one type of feedback selected from a group consisting of audio feedback, visual feedback, and kinesthetic feedback based on the classification of the object.

150. A computer program product for vision-based pointer tracking as set forth in claim 145, wherein the means for outputting is configured to provide the user with at least one type of feedback selected from a group consisting of audio feedback, visual feedback, and kinesthetic feedback based on the path of the pointer, whereby the system can indicate when a path is completed for the recognition process to begin.

151. A computer program product for vision-based pointer tracking as set forth in claim 145, wherein the means for outputting is configured to provide the user with at least one type of feedback selected from a group consisting of audio feedback, visual feedback, and kinesthetic feedback based on the classification of the object.

152. A computer program product for vision-based pointer tracking as set forth in claim 145, wherein the computer program product further comprises a database, and wherein the classification of the object may be communicated with the database and the database, in turn, may recall supplementary information regarding the object.

153. A computer program product for vision-based pointer tracking as set forth in claim 152, wherein the means for outputting is configured to provide the user with at least one type of feedback selected from a group consisting of audio feedback, visual feedback, and kinesthetic feedback based on the path of the pointer, whereby the system can indicate when a path is completed for the recognition process to begin.

154. A computer program product for vision-based pointer tracking as set forth in claim 152, wherein the means for outputting is configured to provide the user with at least one type of feedback selected from a group consisting of audio feedback, visual feedback, and kinesthetic feedback based on the classification of the object.

155. A computer program product for vision-based pointer tracking as set forth in claim 134, wherein the means for classifying the object based on the extracted features is selected from a group consisting of a K-nearest neighbor algorithm, a neural network, and a support vector machine.

156. A computer program product for vision-based pointer tracking as set forth in claim 155, wherein the means for outputting is configured to output the object classification.

157. A computer program product for vision-based pointer tracking as set forth in claim 156, wherein means for outputting is configured to connect with a database system, so that the object classification is communicated with the database system, and wherein a means for inputting is configured to connect with the database system to receive supplementary information regarding the classified object;

whereby once an object is classified, information regarding its classification is sent to a database in order to retrieve stored information regarding the object.

158. A computer program product for vision-based pointer tracking as set forth in claim 157, wherein the means for outputting is configured to provide the user with at least one type of feedback selected from a group consisting of audio feedback, visual feedback, and kinesthetic feedback based on the path of the pointer, whereby the system can indicate when a path is completed for the recognition process to begin.

159. A computer program product for vision-based pointer tracking as set forth in claim 157, wherein the means for outputting is configured to provide the user with at least one type of feedback selected from a group consisting of audio feedback, visual feedback, and kinesthetic feedback based on the classification of the object.

160. A computer program product for vision-based pointer tracking as set forth in claim 155, wherein the means for outputting is configured to provide the user with at least one type of feedback selected from a group consisting of audio feedback, visual feedback, and kinesthetic feedback based on the path of the pointer, whereby the system can indicate when a path is completed for the recognition process to begin.

161. A computer program product for vision-based pointer tracking as set forth in claim 155, wherein the means for outputting is configured to provide the user with at least one type of feedback selected from a group consisting of audio feedback, visual feedback, and kinesthetic feedback based on the classification of the object.

162. A computer program product for vision-based pointer tracking as set forth in claim 155, wherein the computer program product further comprises a database, and wherein the classification of the object may be communicated with the database and the database, in turn, may recall supplementary information regarding the object.

163. A computer program product for vision-based pointer tracking as set forth in claim 162, wherein the means for outputting is configured to provide the user with at least one type of feedback selected from a group consisting of audio feedback, visual feedback, and kinesthetic feedback based on the path of the pointer, whereby the system can indicate when a path is completed for the recognition process to begin.

164. A computer program product for vision-based pointer tracking as set forth in claim 162, wherein the means for outputting is configured to provide the user with at least one type of feedback selected from a group consisting of audio feedback, visual feedback, and kinesthetic feedback based on the classification of the object.

165. A computer program product for vision-based pointer tracking as set forth in claim 33, wherein the means for recognizing the object comprises a means for extracting features from the segmented object, and wherein the means for outputting is configured to output the extracted features.

166. A computer program product for vision-based pointer tracking as set forth in claim 20, wherein the means for robustly predicting the position of the pointer is a Robust Tracking Filter.

167. A computer program product for vision-based pointer tracking as set forth in claim 166, wherein the means for recognizing the pointer is tailored for recognizing a finger of a user, whereby the color segmenter operates on the search window to find finger-colored portions of the search window, and the shape identifier filters the finger-colored portions of the search window in order to find the finger.

168. A computer program product for vision-based pointer tracking as set forth in claim 166, wherein the means for recognizing the pointer is a combination color segmenter and shape identifier, whereby the color segmenter operates on the search window to determine the presence of pointer candidates, and the shape identifier filters the pointer candidates to recognize the pointer within the search window.

169. A computer program product for vision-based pointer tracking as set forth in claim 168, wherein the color segmenter selected from a group consisting of a mean shift theorem algorithm used to adaptively quantize a color space to reduce the number of colors;

a Gaussian mixture modeling algorithm;

a thresholding algorithm used to limit the number of colors to a set designated apriori; and

a Markov-random field algorithm.

170. A computer program product for vision-based pointer tracking as set forth in claim 169, wherein the means for recognizing the pointer is tailored for recognizing a finger of a user, whereby the color segmenter operates on the search window to find finger-colored portions of the search window, and the shape identifier filters the finger-colored portions of the search window in order to find the finger.

171. A computer program product for vision-based pointer tracking as set forth in claim 170, wherein the search window has a size and a shape, and wherein the means for adjusting the search window adjusts size and shape of the search window based on historical pointer movements.

172. A computer program product for vision-based pointer tracking as set forth in claim 170, wherein the means for adjusting the search window within the next frame centers a search window of predetermined size on the predicted position of the pointer.

173. A computer program product for vision-based pointer tracking as set forth in claim 172, wherein the size of the search window is determined as a function of the uncertainty in the motion of the pointer.

174. A computer program product for vision-based pointer tracking as set forth in claim 173, where when a pointer is not recognized in the search window, the search window is redefined to encompass the entire image frame so that recognition may be attempted again in a larger area.

62. A method for vision-based pointer tracking comprising the steps of:

receiving a series of image frames from a camera;

searching a search window within the frame for a portion of the window indicative of a pointer;

recognizing the pointer;

determining the position and trajectory of the pointer;

robustly predicting the position of the pointer in the next image frame in the series of image frames;

adjusting the search window within the next image frame such that the search window comprises a subset of the next image frame positioned so that the actual position of the pointer is likely to be within the search window in the next image frame;

tracking the path of the pointer through an area represented by the series of image frames; and

outputting the path of the pointer, whereby the step of robustly predicting the position of the pointer, combined with the adjusted search window enables tracking of the path of a pointer in a computationally efficient manner.

63. A method for vision-based pointer tracking as set forth in claim 62, wherein the step of recognizing the pointer performs a combination of a color segmenting step and a shape identifying step, whereby the color segmenting step operates on the search window to determine the presence of pointer candidates, and the shape identifying step filters the pointer candidates to recognize the pointer within the search window.

64. A method for vision-based pointer tracking as set forth in claim 63, wherein the step of recognizing the pointer is tailored for recognizing a finger of a user, whereby the color segmenting step operates on the search window to find finger-colored portions of the search window, and the shape identifying step filters the finger-colored portions of the search window in order to find the finger.

65. A method for vision-based pointer tracking as set forth in claim 63, wherein the color segmenting step performs an algorithm selected from a group consisting of a mean shift theorem algorithm used to adaptively quantize a color space to reduce the number of colors;

a Gaussian mixture modeling algorithm;

a thresholding algorithm used to limit the number of colors to a set designated apriori; and

a Markov-random field algorithm.

66. A method for vision-based pointer tracking as set forth in claim 63, wherein the step of recognizing the pointer is tailored for recognizing a finger of a user, whereby the color segmenting step operates on the search window to find finger-colored portions of the search window, and the shape identifying step filters the finger-colored portions of the search window in order to find the finger.

67. A method for vision-based pointer tracking as set forth in claim 62, wherein the step of robustly predicting the position of the pointer performs an algorithm selected from a group consisting of Kalman filters, Extended Kalman filters, and Particle filters and adapted to accommodate uncertainty.

68. A method for vision-based pointer tracking as set forth in claim 62, wherein the step of adjusting the search window within the next frame centers a search window of predetermined size on the predicted position of the pointer.

69. A method for vision-based pointer tracking as set forth in claim 68, wherein further comprising a step of determining the size of the search window as a function of the uncertainty in the motion of the pointer.

70. A method for vision-based pointer tracking as set forth in claim 62, wherein the search window has a size and a shape, and wherein the step of adjusting the search window adjusts size and shape of the search window based on historical pointer movements.

71. A method for vision-based pointer tracking as set forth in claim 62, where when a pointer is not recognized in the search window, a step of redefining the search window is performed, redefining the search window to encompass the entire image frame so that recognition may be attempted again in a larger area.

72. A method for vision-based pointer tracking as set forth in claim 62, wherein the step of robustly predicting the position of the pointer is performed by a Robust Tracking Filter based on state-space estimation.

73. A method for vision-based pointer tracking as set forth in claim 72, wherein the step of recognizing the pointer is tailored for recognizing a finger of a user, whereby the color segmenting step operates on the search window to find finger-colored portions of the search window, and the shape identifying step filters the finger-colored portions of the search window in order to find the finger.

74. A method for vision-based pointer tracking as set forth in claim 72, wherein the step of recognizing the pointer performs a combination of a color segmenting step and a shape identifying step, whereby the color segmenting step operates on the search window to determine the presence of pointer candidates, and the shape identifying step filters the pointer candidates to recognize the pointer within the search window.

75. A method for vision-based pointer tracking as set forth in claim 74, wherein the color segmenting step performs an algorithm selected from a group consisting of a mean shift theorem algorithm used to adaptively quantize a color space to reduce the number of colors;

a Gaussian mixture modeling algorithm;

a thresholding algorithm used to limit the number of colors to a set designated apriori; and

a Markov-random field algorithm.

76. A method for vision-based pointer tracking as set forth in claim 75, wherein the step of recognizing the pointer is tailored for recognizing a finger of a user, whereby the color segmenter operates on the search window to find finger-colored portions of the search window, and the shape identifier filters the finger-colored portions of the search window in order to find the finger.

77. A method for vision-based pointer tracking as set forth in claim 76, wherein the search window has a size and a shape, and wherein the step of adjusting the search window adjusts size and shape of the search window based on historical pointer movements.

78. A method for vision-based pointer tracking as set forth in claim 76, wherein the step of adjusting the search window within the next frame centers a search window of predetermined size on the predicted position of the pointer.

79. A method for vision-based pointer tracking as set forth in claim 78, wherein further comprising a step of determining the size of the search window as a function of the uncertainty in the motion of the pointer.

80. A method for vision-based pointer tracking as set forth in claim 79, where when a pointer is not recognized in the search window, a step of redefining the search window is performed, redefining the search window to encompass the entire image frame so that recognition may be attempted again in a larger area.

114. A computer program product for vision-based pointer tracking, the computer program product comprising means, stored on a computer readable medium, for:

receiving a series of image frames from a camera;

searching a search window within the frame for a portion of the window indicative of a pointer;

recognizing the pointer;

determining the position and trajectory of the pointer;

robustly predicting the position of the pointer in the next image frame in the series of image frames;

adjusting the search window within the next image frame such that the search window comprises a subset of the next image frame positioned so that the actual position of the pointer is likely to be within the search window in the next image frame;

tracking the path of the pointer through an area represented by the series of image frames; and

outputting the path of the pointer, whereby the means for robustly predicting the position of the pointer, combined with the adjusted search window enables the tracking of the path of a pointer in a computationally efficient manner.

115. A computer program product for vision-based pointer tracking as set forth in claim 114, wherein the means for recognizing the pointer is a combination color segmenter and shape identifier, whereby the color segmenter operates on the search window to determine the presence of pointer candidates, and the shape identifier filters the pointer candidates to recognize the pointer within the search window.

116. A computer program product for vision-based pointer tracking as set forth in claim 115, wherein the means for recognizing the pointer is tailored for recognizing a finger of a user, whereby the color segmenter operates on the search window to find finger-colored portions of the search window, and the shape identifier filters the finger-colored portions of the search window in order to find the finger.

117. A computer program product for vision-based pointer tracking as set forth in claim 115, wherein the color segmenter selected from a group consisting of a mean shift theorem algorithm used to adaptively quantize a color space to reduce the number of colors;

a Gaussian mixture modeling algorithm;

a thresholding algorithm used to limit the number of colors to a set designated apriori; and

a Markov-random field algorithm.

118. A computer program product for vision-based pointer tracking as set forth in claim 117, wherein the means for recognizing the pointer is tailored for recognizing a finger of a user, whereby the color segmenter operates on the search window to find finger-colored portions of the search window, and the shape identifier filters the finger-colored portions of the search window in order to find the finger.

119. A computer program product for vision-based pointer tracking as set forth in claim 114, wherein the means for robustly predicting the position of the pointer is an algorithm selected from a group consisting of Kalman filters, Extended Kalman filters, and Particle filters and adapted to accommodate uncertainty.

120. A computer program product for vision-based pointer tracking as set forth in claim 114, wherein the means for adjusting the search window within the next frame centers a search window of predetermined size on the predicted position of the pointer.

121. A computer program product for vision-based pointer tracking as set forth in claim 120, wherein the size of the search window is determined as a function of the uncertainty in the motion of the pointer.

122. A computer program product for vision-based pointer tracking as set forth in claim 114, wherein the search window has a size and a shape, and wherein the means for adjusting the search window adjusts size and shape of the search window based on historical pointer movements.

123. A computer program product for vision-based pointer tracking as set forth in claim 114, where when a pointer is not recognized in the search window, the search window is redefined to encompass the entire image frame so that recognition may be attempted again in a larger area.