Motion-Assisted Visual Language For Human Computer Interfaces

• US 20140347263A1
• Filed: 11/20/2013
• Published: 11/27/2014
• Est. Priority Date: 05/23/2013
• Status: Active Grant

• Current Assignee: FastVDO LLC
• Original Assignee: FastVDO LLC
• Inventors: Dai, Wei, Krishnan, Madhu Peringassery, Topiwala, Pankaj
• Granted Patent: US 9,829,984 B2
• Time in Patent Office: Days
• US Class Current: 345/156
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0482 Interaction with lists of selectable items, e.g. menus
• CPC Class Codes: G06F 3/04842 Selection of displayed objects or displayed text elements G06F3/0482 takes precedence
• CPC Class Codes: G06V 40/28 Recognition of hand or arm movements, e.g. recognition of deaf sign language static hand signs G06V40/113

Abstract

Embodiments of the invention recognize human visual gestures, as captured by image and video sensors, to develop a visual language for a variety of human computer interfaces. One embodiment of the invention provides a computer-implement method for recognizing a visual gesture portrayed by a part of human body such as a human hand, face or body. The method includes steps of receiving the visual signature captured in a video having multiple video frames, determining a gesture recognition type from multiple gesture recognition types including shaped-based gesture, position-based gesture, motion-assisted and mixed gesture that combining two different gesture types. The method further includes steps of selecting a visual gesture recognition process based on the determined gesture type and applying the selected visual gesture recognition process to the multiple video frames capturing the visual gesture to recognize the visual gesture.

Claims

1. A computer-implemented method for recognizing a visual gesture, the method comprising:

receiving a visual gesture formed by a part of human body, the visual gesture being captured in a video having a plurality of video frames; and

applying a visual gesture recognition process to the plurality of video frames to recognize the visual gesture.

2. The method of claim 1, further comprising:

selecting the visual gesture recognition process from a plurality of visual gesture recognition processes.

3. The method of claim 2, wherein the plurality of visual gesture recognition processes include shaped-based gesture recognition process, position-based gesture recognition process, motion-assisted gesture recognition process and mixed gesture recognition process.

4. The method of claim 1, wherein applying a visual gesture recognition process comprises:

applying a shape-based gesture recognition process to the plurality of the video frames of the visual gesture.

5. The method of claim 4, wherein applying the shape-based gesture recognition process to the plurality of the video frames of the visual gesture comprises:

selecting a video frame of the plurality of the video frames of the video of the visual gesture as a reference frame, the video frame representing a digital color image of the visual gesture at a time instance;

applying a general parametric model to the selected video frame of the visual gesture to generate a specific parameter template of the visual gesture;

subsequently receiving one or more video frames of the video of the visual gesture, wherein the visual gesture in a subsequently received video frame is the visual gesture at a subsequent time instance;

for each subsequently received video frame, detecting a visual contour of the visual gesture based at least in part on the specific parametric template of the visual gesture; and

recognizing the visual gesture based at least in part on the detected visual contours.

6. The method of claim 1, wherein applying a visual gesture recognition process comprises:

applying a motion-assisted gesture recognition process to the plurality of the video frames of the visual gesture.

7. The method of claim 6, wherein applying the motion-assisted gesture recognition process to the plurality of the video frames of the visual gesture comprises:

detecting motion of an object contained in the plurality of video frames capturing the visual gesture, the object being bounded in a region of interest (ROI) of a video frame of the plurality of video frames of the video; and

recognizing the visual gesture based at least in part on the detected motion of the object.

8. The method of claim 7, further comprising:

tracking the motion of the object across a subset of the plurality of video frames;

obtaining estimated motion parameters of the object based on the tracking;

refining the estimated motion parameters of the object; and

recognizing the visual gesture based at least in part on the refined estimation of motion parameters of the object.

9. The method of claim 1, wherein applying a visual gesture recognition process comprises:

applying a position-based gesture recognition process to the plurality of the video frames of the visual gesture, wherein the position-based gesture recognition process uses one of motion-assisted object tracking process, a secondary object tracking process and a mixed process applying the motion-assisted object tracking process and the secondary object tracking process in parallel.

10. The method of claim 9, wherein applying the position-based gesture recognition process to the plurality of the video frames of the visual gesture comprises:

detecting changes of position of an object contained in the plurality of video frames capturing the visual gesture, a change of position of the object representing the visual gesture moving from a previous position to a current position; and

recognizing the visual gesture based on the changes of position of the object.

77. The method of claim 1, further comprising:

providing the visual gesture to a computer system having a video sensor; and

activating one or more computer commands associated with the computer system using the visual gesture.

78. The method of claim 1, wherein the visual gesture is a visual code representing one of a plurality of visual gestures and corresponding user input commands.

79. The method of claim 1, further comprising generating the visual code based on one or more user input commands and a plurality of visual gestures.

80. The method of claim 79, wherein the visual code is generated based at least in part on a plurality predefined visual codes, a plurality of user prerecorded visual codes or a plurality of visual codes created in real time by a user.

81. The method of claim 80, wherein creating the visual codes in real time by the user comprises:

receiving one or more user input commands and a plurality of human visual gestures;

encoding the plurality of human visual gestures into one or more visual codes based on the user input commands; and

storing the encoded visual codes in a computer storage medium.

82. The method of claim 81, wherein the user input commands are received using at least one of following:

tactile input by the user;

live feed from a camera;

voice commands;

hand configurations of the user;

eye movements of the user;

handwritings of the user;

finger or hand movements of the user;

facial movements of the user; and

body movements of the user.

83. The method of claim 81, wherein the plurality of the human visual gestures comprise at least one of:

a plurality of hand gestures;

a plurality of facial gestures; and

a plurality of body gestures.

84. The method of claim 77, wherein the visual gesture is a detected hand motion, and the detected hand motion is discretized into a small number of directions including Left, Right, Up, and Down, and wherein the detected discrete hand motion is used to indicate a command in a computer application.

85. The method of claim 84, wherein the detected discrete motion is used to indicate a command in navigating an application by moving one step, including moving a step forward or back in a presentation, document, webpage, list, or media gallery.

86. The method of claim 84, wherein a combination of two detected discrete motions in quick succession constitutes another gesture, a Wave hand gesture.

87. The method of claim 84, wherein a special gesture is used to launch or close a computer application.

88. The method of claim 87, wherein the special gesture is comprised of a Wave hand gesture.

89. The method of claim 87, further comprising:

detecting and tracking continuously a subsequent motion or position of the hand responsive to the computer application being launched via a hand gesture, wherein the detected hand motion or position is converted to corresponding motion or position of an on-screen object.

90. The method of claim 89, wherein holding the hand still within a specified tolerance for a specified number of frames is used to indicate readiness to make a selection.

95. The method of claim 77, wherein a visual gesture is used in an interactive application.

96. The method of claim 77, wherein a visual gesture corresponds to an action of an on-screen object in a computer application.

97. The method of claim 77, wherein detected hand shapes or figures are interpreted as numbers.

98. The method of claim 77, wherein detected hand shapes or figures are interpreted as letters, words, or phrases, as in a sign language.

99. The method of claim 77, wherein a visual gesture is used to indicate a zoom function in a computer application.

100. The method of claim 99, wherein the visual gesture for zoom indication is comprised of moving the hand smoothly closer to the sensor indicating “

zoom in” and

moving the hand further away from the sensor indicating “

zoom out”

.

101. The method of claim 77, wherein a special gesture is comprised of a specified shape of a hand including:

a fist;

thumbs up;

thumbs down;

a “

V”

sign with two fingers up, other fingers closed;

an “

O”

sign with the hand shaped like a circle; and

a “

C”

or “

U”

sign with the hand partly open, thumb parallel to remaining fingers, and

pointing up, down, left or right.

102. The method of claim 77, wherein the visual gesture is applied to or in a computer application residing in a computer system of one of the following types:

a desktop, laptop, personal computer, printer, copier, or other peripheral;

a tablet, smartphone, wearable computer, or other mobile device;

an embedded computer system, as in a car, train, plane, television or home appliance; and

an enterprise computing system.

103. The method of claim 102, wherein the computer application includes at least one of the following types:

a hardware device application including a keyboard, mouse, camera or other peripheral;

a software application including a presentation, spreadsheet, word processor, browser, media player or media editor;

an art or design application;

a communications application;

an entertainment application or game;

a virtual reality application; and

a health monitoring application.

104. The method of claim 77, wherein the visual gesture is comprised of a current position of the hand relative to an initial or reference position of the hand.

105. The method of claim 77, wherein the visual gesture is comprised of a motion of the hand.

106. The method of claim 77, wherein the visual gesture is comprised of a shape of the hand.

107. The method of claim 77, wherein the visual gesture is comprised of a composite gesture comprising more than one gesture of type position, motion, or shape.

108. The method of claim 77, wherein the visual gesture is composed with two hands.

109. The method of claim 77, wherein the visual gesture is composed using the face, or a part of the face, the part of face including an eye or both eyes.

110. The method of claim 77, wherein the visual gesture is composed with arms.

111. The method of claim 77, wherein the visual gesture is composed with the human body.

11. A computer-implemented method for recognizing a visual gesture, the method comprising:

receiving a visual gesture formed by a part of human body, the visual gesture being captured in a video having a plurality of video frames;

selecting a visual gesture recognition process from a plurality of visual gesture recognition processes; and

applying the selected visual gesture recognition process to the plurality of video frames to recognize the visual gesture.

12. The method of claim 11, wherein the plurality of visual gesture recognition processes comprises shaped-based gesture recognition process, position-based gesture recognition process, motion-assisted gesture recognition process and mixed gesture recognition process.

13. The method of claim 11, further comprising:

selecting a shape-based visual gesture recognition process; and

applying the selected shape-based gesture recognition process to the plurality of the video frames of the visual gesture.

14. The method of claim 11, further comprising:

selecting a motion-assisted visual gesture recognition process; and

applying the selected motion-assisted gesture recognition process to the plurality of the video frames of the visual gesture.

15. The method of claim 11, further comprising:

selecting a position-based visual gesture recognition process; and

applying the selected position-based gesture recognition process to the plurality of the video frames of the visual gesture.

16. The method of claim 11, further comprising:

selecting a mixed visual gesture recognition process, the mixed visual gesture recognition process comprising two or more gesture recognition processes; and

applying the selected mixed gesture recognition process to the plurality of the video frames of the visual gesture.

17. A computer-implemented method for recognizing a visual gesture, the method comprising:

receiving a visual gesture formed by a part of human body, the visual gesture being captured in a video having a plurality of video frames;

determining a type of the visual gesture formed by the part of human body;

selecting a visual gesture recognition process based on the determined visual gesture type; and

applying the selected visual gesture recognition process to the plurality of video frames to recognize the visual gesture.

18. The method of claim 17, further comprising:

presenting a plurality of visual gesture recognition types on a computer display.

19. The method of claim 18, wherein the plurality of visual gesture types include shaped-based gesture, position-based gesture, motion-assisted gesture and mixed gesture.

20. The method of claim 17, wherein a part of the human body forming the visual gesture is at least one of human hand, face and body.

21. The method of claim 17, wherein selecting a visual gesture recognition process comprises:

responsive to the determination that the type of the visual gesture is shaped-based, selecting a shape-based gesture recognition process to the plurality of the video frames of the visual gesture.

22. The method of claim 17, wherein selecting a visual gesture recognition process further comprises:

responsive to the determination that the type of the visual gesture is position-based, selecting a position-based gesture recognition process to the plurality of the video frames of the visual gesture.

23. The method of claim 17, wherein selecting a visual gesture recognition process further comprises:

responsive to the determination that the type of the visual gesture is motion-assisted, selecting a motion-assisted gesture recognition process to the plurality of the video frames of the visual gesture.

24. The method of claim 17, wherein applying the selected visual gesture recognition process to the plurality of video frames comprises:

applying a shape-based gesture recognition process to the plurality of the video frames of the visual gesture.

25. The method of claim 24, wherein applying the shape-based gesture recognition process to the plurality of the video frames of the visual gesture comprises:

selecting a video frame of the plurality of the video frames of the video of the visual gesture as a reference frame, the video frame representing a digital color image of the visual gesture at a time instance;

applying a general parametric model to the selected video frame of the visual gesture to generate a specific parameter template of the visual gesture;

subsequently receiving one or more video frames of the video of the visual gesture, wherein the visual gesture in a subsequently received video frame is the visual gesture at a subsequent time instance;

for each subsequently received video frame, detecting a visual contour of the visual gesture based at least in part on the specific parametric template of the visual gesture; and

recognizing the visual gesture based at least in part on the detected visual contours.

26. The method of claim 17, wherein applying the selected visual gesture recognition process to the plurality of video frames comprises:

applying a motion-assisted gesture recognition process to the plurality of the video frames of the visual gesture.

27. The method of claim 26, wherein applying the motion-assisted gesture recognition process to the plurality of the video frames of the visual gesture comprises:

detecting motion of an object contained in the plurality of video frames capturing the visual gesture, the object being bounded in a region of interest (ROI) of a video frame of the plurality of video frames of the video; and

recognizing the visual gesture based at least in part on the detected motion of the object.

28. The method of claim 27, further comprising:

tracking the motion of the object across a subset of the plurality of video frames;

obtaining estimated motion parameters of the object based on the tracking;

refining the estimated motion parameters of the object; and

recognizing the visual gesture based at least in part on the detected motion of the object.

29. The method of claim 27, wherein determining motion of an object comprises:

identifying a first instance of the object contained in the ROI in the video frame; and

obtaining center point position of the ROI and size of the ROI within the video frames;

30. The method of claim 28, wherein tracking the motion of the object comprises:

obtaining motion vectors of the object, a motion vector of the object indicating a position of the object in the video frame relative to its corresponding position in a previous or reference video frame of the video.

31. The method of claim 28, wherein obtaining estimated motion parameters of the object based on the tracking comprises:

clustering motion vectors representing the tracked motion of the object; and

obtaining one or more estimates of the center point position of the ROI and the size of the ROI based on the clustering of the motion vectors.

32. The method of claim 28, wherein refining the estimated motion parameters of the object comprises:

selecting a plurality of candidate ROIs based on a local search centered around estimated position of the ROI;

applying a similarity measure to the plurality of candidate ROIs of the object; and

selecting at least one ROI based on the similarity measure.

33. The method of claim 32, wherein the similarity measure is chosen from a group of sum of absolute difference, mean squared error and normalized correlation coefficient, and the selected ROI optimizes the similarity measure among the plurality of candidate ROIs.

34. The method of claim 32, further comprising adding another video frame as a second reference frame containing an estimated ROI, wherein the estimated ROI has a similarity score exceeding a threshold value.

35. The method of claim 17, wherein applying the selected visual gesture recognition process to the plurality of video frames comprises:

applying a position-based gesture recognition process to the plurality of the video frames of the visual gesture.

36. The method of claim 35, wherein applying the position-based gesture recognition process to the plurality of the video frames of the visual gesture comprises:

detecting changes of position of an object contained in the plurality of video frames capturing the visual gesture, a change of position of the object representing the visual gesture moving from a previous position to a current position; and

recognizing the visual gesture based on the changes of position of the object.

37. The method of claim 36, further comprising:

recording the changes of position of the object;

quantizing the changes of the position of the object; and

recognizing the visual gesture based on quantized changes of position of the object.

38. The method of claim 37, wherein recording the changes of position of the object comprises:

recording a change of position of object by instantaneous position change of the object, wherein the instantaneous position change of the object indicates the change over an immediately previous position of the object.

39. The method of claim 37, wherein recording the changes of position of the object comprises:

recording a change of position of object by reference position change of the object, wherein the reference position change of the object indicates the change over a pre-defined reference position of the object.

40. The method of claim 37, wherein recording the changes of position of the object further comprises:

determining at least one characteristic of the visual gesture;

selecting a method to record the changes of position of the object based on the determined characteristic of the visual gesture; and

recording the changes of position of the object using the selected method.

41. The method of claim 17, wherein applying the selected visual gesture recognition process to the plurality of video frames comprises:

responsive to the determination that the type of the visual gesture is one of motion-assisted, shape-based and a mixture of both motion-assisted and shape-based, applying a motion-assisted object tracking process and a secondary object tracking process in parallel to the plurality of video frames of the visual gesture.

42. The method of claim 41, wherein tracking an object contained in the plurality of video frames capturing the visual gesture by the motion-assisted object tracking process and the secondary object tracking process in parallel comprises:

obtaining a first set of estimates of the center point position of the ROI containing the object and the size of the ROI based on the object tracking by the motion-assisted object tracking process;

obtaining a second set of estimates of the center point position of the ROI containing the object and the size of the ROI based on the object tracking by the secondary object tracking process; and

merging the first set of estimation and the second set of estimation.

43. The method of claim 42, wherein merging the first set of estimation and the second set of estimation comprises:

calculating a first confidence score for the motion-assisted object tracking process;

calculating a second confidence score for the secondary object tracking process; and

selecting the set of estimates based on comparison of the first confidence score and the second confidence score.

44. The method of claim 42, further comprising:

allocating computing resource to one tracking process with a priority higher than the other tracking process; and

responsive to total computer resource allocated to both the motion-assisted object tracking process and the secondary object tracking process exceeding a predetermined threshold, disabling one of the object tracking processes temporally.

45. The method of claim 42, further comprising:

responsive to total computer resource exceeding a predetermined threshold, scaling the complexity of at least one of the object tracking processes temporally.

46. The method of claim 17, wherein applying the selected visual gesture recognition process to the plurality of video frames comprises:

analyzing a plurality of features extracted from the plurality of video frames of the visual gesture;

determining whether to apply a shape-based visual gesture recognition process or motion-assisted gesture recognition process based on the analyzing of the plurality of the features;

responsive to the determination of applying the shape-based visual gesture recognition process, applying the shape-based visual gesture recognition process to the plurality of the video frames capturing the visual gesture to recognize a shape gesture; and

responsive to the determining of applying a motion-assisted visual gesture recognition process, applying a motion-assisted visual gesture recognition process to the plurality of the video frames capturing the visual gesture to recognize a motion gesture.

47. A non-transitory computer-readable storage medium storing executable computer program instructions for recognizing a visual gesture, the computer programs instructions comprising code for:

receiving a visual gesture formed by a part of human body, the visual gesture being captured in a video having a plurality of video frames;

determining a type of the visual gesture formed by the part of human body;

selecting a visual gesture recognition process based on the determined visual gesture type; and

applying the selected visual gesture recognition process to the plurality of video frames to recognize the visual gesture.

48. The computer-readable storage medium of claim 47, further comprising computer program instructions for:

presenting a plurality of visual gesture recognition types on a computer display.

49. The computer-readable storage medium of claim 48, wherein the plurality of visual gesture types include shaped-based gesture, position-based gesture, motion-assisted gesture and mixed gesture.

50. The computer-readable storage medium of claim 47, wherein a part of the human body forming the visual gesture is at least one of human hand, face and body.

51. The computer-readable storage medium of claim 47, wherein the computer program instructions for selecting a visual gesture recognition process comprise computer program instructions for:

responsive to the determination that the type of the visual gesture is shaped-based, selecting a shape-based gesture recognition process to the plurality of the video frames of the visual gesture.

52. The computer-readable storage medium of claim 47, wherein the computer program instructions for selecting a visual gesture recognition process further comprise computer program instructions for:

responsive to the determination that the type of the visual gesture is position-based, selecting a position-based gesture recognition process to the plurality of the video frames of the visual gesture.

53. The computer-readable storage medium of claim 47, wherein the computer program instructions for selecting a visual gesture recognition process further comprise computer program instructions for:

responsive to the determination that the type of the visual gesture is motion-assisted, selecting a motion-assisted gesture recognition process to the plurality of the video frames of the visual gesture.

54. The computer-readable storage medium of claim 47, wherein the computer program instructions for applying the selected visual gesture recognition process to the plurality of video frames comprise computer program instructions for:

applying a shape-based gesture recognition process to the plurality of the video frames of the visual gesture.

55. The computer-readable storage medium of claim 54, wherein the computer program instructions for applying the shape-based gesture recognition process to the plurality of the video frames of the visual gesture comprise computer program instructions for:

selecting a video frame of the plurality of the video frames of the video of the visual gesture as a reference frame, the video frame representing a digital color image of the visual gesture at a time instance;

applying a general parametric model to the selected video frame of the visual gesture to generate a specific parameter template of the visual gesture;

subsequently receiving one or more video frames of the video of the visual gesture, wherein the visual gesture in a subsequently received video frame is the visual gesture at a subsequent time instance;

for each subsequently received video frame, detecting a visual contour of the visual gesture based at least in part on the specific parametric template of the visual gesture; and

recognizing the visual gesture based at least in part on the detected visual contours.

56. The computer-readable storage medium of claim 47, wherein the computer program instructions for applying the selected visual gesture recognition process to the plurality of video frames comprise computer program instructions for:

applying a motion-assisted gesture recognition process to the plurality of the video frames of the visual gesture.

57. The computer-readable storage medium of claim 56, wherein the computer program instructions for applying the motion-assisted gesture recognition process to the plurality of the video frames of the visual gesture comprise computer program instructions for:

detecting motion of an object contained in the plurality of video frames capturing the visual gesture, the object being bounded in a region of interest (ROI) of a video frame of the plurality of video frames of the video; and

recognizing the visual gesture based at least in part on the detected motion of the object.

58. The computer-readable storage medium of claim 57, further comprising computer program instructions for:

tracking the motion of the object across a subset of the plurality of video frames;

obtaining estimated motion parameters of the object based on the tracking;

refining the estimated motion parameters of the object; and

recognizing the visual gesture based at least in part on the detected motion of the object.

59. The computer-readable storage medium of claim 57, wherein the computer program instructions for determining motion of an object comprise computer program instructions for:

identifying a first instance of the object contained in the ROI in the video frame; and

obtaining center point position of the ROI and size of the ROI within the video frames;

60. The computer-readable storage medium of claim 58, wherein the computer program instructions for tracking the motion of the object comprise computer program instructions for:

obtaining motion vectors of the object, a motion vector of the object indicating a position of the object in the video frame relative to its corresponding position in a previous or reference video frame of the video.

61. The computer-readable storage medium of claim 58, wherein the computer program instructions for obtaining estimated motion parameters of the object based on the tracking comprise computer program instructions for:

clustering motion vectors representing the tracked motion of the object; and

obtaining one or more estimates of the center point position of the ROI and the size of the ROI based on the clustering of the motion vectors.

62. The computer-readable storage medium of claim 58, wherein the computer program instructions for refining the estimated motion parameters of the object comprise computer program instructions for:

selecting a plurality of candidate ROIs based on a local search centered around estimated position of the ROI;

applying a similarity measure to the plurality of candidate ROIs of the object; and

selecting at least one ROI based on the similarity measure.

63. The computer-readable storage medium of claim 62, wherein the similarity measure is chosen from a group of sum of absolute difference, mean squared error and normalized correlation coefficient, and the selected ROI optimizes the similarity measure among the plurality of candidate ROIs.

64. The computer-readable storage medium of claim 62, further comprising computer program instructions for adding another video frame as a second reference frame containing an estimated ROI, wherein the estimated ROI has a similarity score exceeding a threshold value.

65. The computer-readable storage medium of claim 47, wherein the computer program instructions for applying the selected visual gesture recognition process to the plurality of video frames comprise computer program instructions for:

applying a position-based gesture recognition process to the plurality of the video frames of the visual gesture.

66. The computer-readable storage medium of claim 65, wherein the computer program instructions for applying the position-based gesture recognition process to the plurality of the video frames of the visual gesture comprise computer program instructions for:

detecting changes of position of an object contained in the plurality of video frames capturing the visual gesture, a change of position of the object representing the visual gesture moving from a previous position to a current position; and

recognizing the visual gesture based on the changes of position of the object.

67. The computer-readable storage medium of claim 66, further comprising computer program instructions for:

recording the changes of position of the object;

quantizing the changes of the position of the object; and

recognizing the visual gesture based on quantized changes of position of the object.

68. The computer-readable storage medium of claim 67, wherein the computer program instructions for recording the changes of position of the object comprise computer program instructions for:

recording a change of position of object by instantaneous position change of the object, wherein the instantaneous position change of the object indicates the change over an immediately previous position of the object.

69. The computer-readable storage medium of claim 67, wherein the computer program instructions for recording the changes of position of the object comprise computer program instructions for:

recording a change of position of object by reference position change of the object, wherein the reference position change of the object indicates the change over a pre-defined reference position of the object.

70. The computer-readable storage medium of claim 67, wherein the computer program instructions for recording the changes of position of the object further comprise computer program instructions for:

determining at least one characteristic of the visual gesture;

selecting a method to record the changes of position of the object based on the determined characteristic of the visual gesture; and

recording the changes of position of the object using the selected method.

71. The computer-readable storage medium of claim 47, wherein the computer program instructions for applying the selected visual gesture recognition process to the plurality of video frames comprise computer program instructions for:

responsive to the determination that the type of the visual gesture is one of motion-assisted, shape-based and a mixture of both motion-assisted and shape-based, applying a motion-assisted object tracking process and a secondary object tracking process in parallel to the plurality of video frames of the visual gesture.

72. The computer-readable storage medium of claim 71, wherein the computer program instructions for tracking an object contained in the plurality of video frames capturing the visual gesture by the motion-assisted object tracking process and the secondary object tracking process in parallel comprise computer program instructions for:

obtaining a first set of estimates of the center point position of the ROI containing the object and the size of the ROI based on the object tracking by the motion-assisted object tracking process;

obtaining a second set of estimates of the center point position of the ROI containing the object and the size of the ROI based on the object tracking by the secondary object tracking process; and

merging the first set of estimation and the second set of estimation.

73. The computer-readable storage medium of claim 72, wherein the computer program instructions for merging the first set of estimation and the second set of estimation comprise computer program instructions for:

calculating a first confidence score for the motion-assisted object tracking process;

calculating a second confidence score for the secondary object tracking process; and

selecting the set of estimates based on comparison of the first confidence score and the second confidence score.

74. The computer-readable storage medium of claim 72, further comprising computer program instructions for:

allocating computing resource to one tracking process with a priority higher than the other tracking process; and

responsive to total computer resource allocated to both the motion-assisted object tracking process and the secondary object tracking process exceeding a predetermined threshold, disabling one of the object tracking processes temporally.

75. The computer-readable storage medium of claim 72, further comprising computer program instructions for:

responsive to total computer resource exceeding a predetermined threshold, scaling the complexity of at least one of the object tracking processes temporally.

76. The computer-readable storage medium of claim 47, wherein the computer program instructions for applying the selected visual gesture recognition process to the plurality of video frames comprise computer program instructions for:

analyzing a plurality of features extracted from the plurality of video frames of the visual gesture;

determining whether to apply a shape-based visual gesture recognition process or a motion-assisted recognition process based on the analyzing of the plurality of the features;

responsive to the determination of applying the shape-based visual gesture recognition process, applying the shape-based visual gesture recognition process to the plurality of the video frames capturing the visual gesture to recognize a shape gesture; and

responsive to the determining of applying a motion-assisted visual gesture recognition process, applying a motion-assisted visual gesture recognition process to the plurality of the video frames capturing the visual gesture to recognize a motion gesture.

91. The method of 77, wherein a special visual gesture is used to indicate selection or deselection in a computer application, including selecting a button, window, object, or link.

92. The method of claim 91, wherein the visual gesture for selection or deselection is comprised of closing an open hand and making a first or opening a fist.

93. The method of claim 91, wherein the visual gesture for selection or deselection is comprised of pointing a finger at the screen or retracting a finger.

94. The method of claim 91, wherein the visual gesture for selection or deselection is comprised of a quick Tap of the hand towards the screen or retracting the hand.