TRACKING BIMANUAL MOVEMENTS

• US 20080219502A1
• Filed: 10/31/2007
• Published: 09/11/2008
• Est. Priority Date: 04/15/2004
• Status: Active Grant

• Current Assignee: Qualcomm, Inc.
• Original Assignee: GestureTek Incorporated (Winning Brands Corp.)
• Inventors: Shamaie, Atid
• Granted Patent: US 8,259,996 B2
• Time in Patent Office: Days
• US Class Current: 382/103
• CPC Class Codes: G06V 40/20 Movements or behaviour, e.g. gesture recognition recognition of facial expressions G06V40/16
• CPC Class Codes: G06V 40/28 Recognition of hand or arm movements, e.g. recognition of deaf sign language static hand signs G06V40/113

Abstract

Hands may be tracked before, during, and after occlusion, and a gesture may be recognized. Movement of two occluded hands may be tracked as a unit during an occlusion period. A type of synchronization characterizing the two occluded hands during the occlusion period may be determined based on the tracked movement of the occluded hands. Based on the determined type of synchronization, it may be determined whether directions of travel for each of the two occluded hands change during the occlusion period. Implementations may determine that a first hand and a second hand are occluded during an occlusion period, the first hand having come from a first direction and the second hand having come from a second direction. The first hand may be distinguished from the second hand after the occlusion period based on a determined type of synchronization characterizing the two hands, and a behavior of the two hands.

Claims

1-20. -20. (canceled)

21. A method for determining whether directions of travel for each of two occluded hands changes, the method comprising:

tracking movement of two occluded hands as a unit during an occlusion period; and

determining whether directions of travel for each of the two occluded hands change during the occlusion period.

22. A method of identifying an observed bimanual hand gesture, the method comprising:

tracking movement of a first hand during an observed bimanual gesture;

tracking movement of a second hand during the observed bimanual gesture;

determining a first set of] measures of likelihoods that the tracked movement of the first hand corresponds to one or more known bimanual gestures;

determining a second set of measures of likelihoods that the tracked movement of the second hand corresponds to the one or more known bimanual gestures;

based on the first set of measures of the likelihoods that the tracked movement of the first hand corresponds to the one or more known bimanual gestures and the second set of measures of the likelihoods that the tracked movement of the second hand corresponds to the one or more known bimanual gestures, identifying a particular one of the one or more known bimanual gestures as corresponding to the observed bimanual gesture.

23. The method of claim 22 wherein:

tracking movement of a first hand during an observed bimanual gesture comprises extracting a sequence of images of the first hand from a sequence of images of the observed bimanual gesture;

tracking movement of a second hand during an observed bimanual gesture comprises extracting a sequence of images of the second hand from the sequence of images of the observed bimanual gesture;

determining a first set of measures of likelihoods that the tracked movement of the first hand corresponds to one or more known bimanual gestures comprises;

identifying a sequence of principal components corresponding to the sequence of images of the first hand, and

based on the identified sequence of principal components corresponding to the sequence of images of the first hand, using trained Hidden Markov Models that model first-hand movements of the one or more known bimanual gestures to determine the first set of measures of likelihoods that the tracked movement of the first hand corresponds to the one or more known bimanual gestures; and

determining a second set of measures of likelihoods that the tracked movement of the second hand corresponds to one or more known bimanual gestures comprises;

identifying a sequence of principal components corresponding to the sequence of images of the second hand, and

based on the identified sequence of principal components corresponding to the sequence of images of the second hand, using trained Hidden Markov Models that model second-hand movements of the one or more known bimanual gestures to determine the second set of measures of likelihoods that the tracked movement of the second hand corresponds to the one or more known bimanual gestures.

24. The method of claim 23 wherein:

identifying a sequence of principal components corresponding to the sequence of images of the first hand comprises;

projecting the images from the sequence of images of the first hand into a first-hand eigenspace formed based on principal components determined for a set of training images of first-hand movements of the one or more known bimanual gestures, and extracting, from the first-hand eigenspace, a first sequence of codewords associated with the projections of the images of the first hand; and

identifying a sequence of principal components corresponding to the sequence of images of the second hand comprises;

projecting the images from the sequence of images of the second hand into a second-hand eigenspace formed based on principal components determined for a set of training images of second-hand movements of the one or more known bimanual gestures, and

extracting, from the second-hand eigenspace, a second sequence of codewords associated with the projections of the images of the second hand.

25. The method of claim 22 further comprising maintaining a Bayesian network for identifying bimanual gestures that includes a first-hand node, a second-hand node, and a bimanual gesture node, wherein identifying a particular one of the one or more known bimanual gestures as corresponding to the observed bimanual gesture based on the first set of measures of the likelihoods that the tracked movement of the first hand corresponds to the one or more known bimanual gestures and the second set of measures of the likelihoods that the tracked movement of the second hand corresponds to the one or more known bimanual gestures comprises:

inputting, to the first-hand node of the Bayesian network, the first set of measures of the likelihoods that the tracked movement of the first hand corresponds to the one or more known bimanual gestures;

inputting, to the second-hand node of the Bayesian network, the second set of measures of the likelihoods that the tracked movement of the second hand corresponds to the one or more known bimanual gestures;

using the first-hand node of the Bayesian network to determine a first belief of a known bimanual gesture that corresponds to the tracked movement of the first hand based on the input first set of measures of the likelihoods that the tracked movement of the first hand corresponds to the one or more known bimanual gestures;

using the second-hand node of the Bayesian network to determine a second belief of a known bimanual gesture that corresponds to the tracked movement of the second hand based on the input second set of measures of the likelihoods that the tracked movement of the second hand corresponds to the one or more known bimanual gestures;

inputting the determined first belief and the determined second belief to the bimanual gesture root node; and

using the bimanual gesture root node to identify a particular one of the one or more known bimanual gestures as corresponding to the observed bimanual gesture based on the determined first belief and the determined second belief.

26. The method of claim 22 further comprising:

tracking movement of the first and second hands as a unit during an occlusion period of the bimanual gesture; and

determining a third set of measures of likelihoods that the tracked movement of the first and second hands as a unit during the occlusion period corresponds to the one or more known bimanual gestures, wherein;

tracking movement of a first hand during an observed bimanual gesture comprises tracking movement of the first hand during a non-occlusion period of the observed bimanual gesture;

tracking movement of a second hand during the observed bimanual gesture comprises tracking movement of the second hand during the non-occlusion period of the observed bimanual gesture;

determining a first set of measures of likelihoods that the tracked movement of the first hand corresponds to one or more known bimanual gestures comprises determining a first set of measures of likelihoods that the tracked movement of the first hand during the non-occlusion period corresponds to the one or more known bimanual gestures;

determining a second set of measures of likelihoods that the tracked movement of the second hand corresponds to the one or more known bimanual gestures comprises determining a second set of measures of likelihoods that the tracked movement of the second hand during the non-occlusion period corresponds to the one or more known bimanual gestures; and

identifying a particular one of the one or more known bimanual gestures as corresponding to the observed bimanual gesture comprises identifying a particular one of the one or more known bimanual gestures as corresponding to the observed bimanual gesture based on the first set of measures of the likelihoods that the tracked movement of the first hand corresponds to the one or more known bimanual gestures, the second set of measures of the likelihoods that the tracked movement of the second hand corresponds to the one or more known bimanual gestures, and the third set of measures of the likelihoods that the tracked movement of the first and second hands as a unit during the occlusion period corresponds to the one or more known bimanual gestures.

27. The method of claim 26 wherein:

tracking movement of the first hand during the non-occlusion period of the observed bimanual gesture comprises extracting a sequence of images of the first hand during the non-occlusion period from a sequence of images of the observed bimanual gesture;

tracking movement of a second hand during the non-occlusion period of the observed bimanual gesture comprises extracting a sequence of images of the second hand during the non-occlusion period from the sequence of images of the observed bimanual gesture;

tracking movement of the first and second hands as a unit during an occlusion period of the bimanual gesture comprises extracting a sequence of images of the first and second hands during the occlusion period from the sequence of images of the observed bimanual gesture;

determining a first set of measures of likelihoods that the tracked movement of the first hand during the non-occlusion period corresponds to one or more known bimanual gestures comprises;

identifying a sequence of principal components corresponding to the sequence of images of the first hand during the non-occlusion period, and

based on the identified sequence of principal components corresponding to the sequence of images of the first hand during the non-occlusion period, using trained Hidden Markov Models that model first-hand movements of the one or more known bimanual gestures to determine the first set of measures of likelihoods that the tracked movement of the first hand during the non-occlusion period corresponds to the one or more known bimanual gestures;

determining the second set of measures of likelihoods that the tracked movement of the second hand during the non-occlusion period corresponds to one or more known bimanual gestures comprises;

identifying a sequence of principal components corresponding to the sequence of images of the second hand during the non-occlusion period, and

based on the identified sequence of principal components corresponding to the sequence of images of the second hand during the non-occlusion period, using trained Hidden Markov Models that model second-hand movements of the one or more known bimanual gestures to determine the second set of measures of likelihoods that the tracked movement of the second hand during the non-occlusion period corresponds to the one or more known bimanual gestures; and

determining a third set of measures of likelihoods that the tracked movement of the first hand and second hands as a unit during the occlusion period corresponds to the one or more known bimanual gestures comprises;

identifying a sequence of principal components corresponding to the sequence of images of the first and second hands during the occlusion period, and

based on the identified sequence of principal components corresponding to the sequence of images of the first and second hands during the occlusion period, using trained Hidden Markov Models that model occluded-hand movements of the one or more known bimanual gestures to determine a third set of measures of likelihoods that the tracked movement of the first and second hands as a unit during the occlusion period corresponds to the one or more known bimanual gestures.

28. The method of claim 27 wherein:

identifying a sequence of principal components corresponding to the sequence of images of the first hand during the non-occlusion period comprises;

projecting the images from the sequence of images of the first hand during the non-occlusion period into a first-hand eigenspace formed based on principal components determined for a set of training images of first-hand movements of the one or more known bimanual gestures, and

extracting, from the first-hand eigenspace, a first sequence of codewords associated with the projections of the images of the first hand during the non-occlusion period;

identifying a sequence of principal components corresponding to the sequence of images of the second hand during the non-occlusion period comprises;

projecting the images from the sequence of images of the second hand during the non-occlusion period into a second-hand eigenspace formed based on principal components determined for a set of training images of second-hand movements of the one or more known bimanual gestures, and

extracting, from the second-hand eigenspace, a second sequence of codewords associated with the projections of the images of the second hand during the non-occlusion period; and

identifying a sequence of principal components corresponding to the sequence of images of the first and second hands during the occlusion period comprises;

projecting the images from the sequence of images of the first and second hands during the occlusion period into an occluded-hand eigenspace formed based on principal components determined for a set of training images of occluded-hand movements of the one or more known bimanual gestures, and

extracting, from the occluded-hand eigenspace, a third sequence of codewords associated with the projections of the images of the first and second hands during the occlusion period.

29. The method of claim 26 further comprising maintaining a Bayesian network for identifying bimanual gestures that includes a first-hand node, a second-hand node, an occluded-hand node, and a bimanual gesture node, wherein identifying a particular one of the one or more known bimanual gestures as corresponding to the observed bimanual gesture based on the first set of measures of the likelihoods that the tracked movement of the first hand during the non-occlusion period corresponds to the one or more known bimanual gestures, the second set of measures of the likelihoods that the tracked movement of the second hand during the non-occlusion period corresponds to the one or more known bimanual gestures, and the third set of measures of the likelihoods that the tracked movement of the first and second hands as a unit during the occlusion period corresponds to the one or more known bimanual gestures comprises:

inputting, to the first-hand node of the Bayesian network, the first set of measures of the likelihoods that the tracked movement of the first hand during the non-occlusion period corresponds to the one or more known bimanual gestures;

inputting, to the second-hand node of the Bayesian network, the second set of measures of the likelihoods that the tracked movement of the second hand during the non-occlusion period corresponds to the one or more known bimanual gestures;

inputting, to the occluded-hand node of the Bayesian network, the third set of measures of the likelihoods that the tracked movement of the first and second hands during the occlusion period corresponds to the one or more known bimanual gestures;

using the first-hand node of the Bayesian network to determine a first belief of a known bimanual gesture that corresponds to the tracked movement of the first hand during the non-occlusion period based on the input first set of measures of the likelihoods that the tracked movement of the first hand during the non-occlusion period corresponds to the one or more known bimanual gestures;

using the second-hand node of the Bayesian network to determine a second belief of a known bimanual gesture that corresponds to the tracked movement of the second hand during the non-occlusion period based on the input second set of measures of the likelihoods that the tracked movement of the second hand during the non-occlusion period corresponds to the one or more known bimanual gestures;

using the occluded-hand node of the Bayesian network to determine a third belief of a known bimanual gesture that corresponds to the tracked movement of the first and second hands during the occlusion period based on the input third set of measures of the likelihoods that the tracked movement of the first and second hands during the occlusion period corresponds to the one or more known bimanual gestures;

inputting the determined first belief, the determined second belief, and the determined third belief to the bimanual gesture root node; and

using the bimanual gesture root node to identify a particular one of the one or more known bimanual gestures as corresponding to the observed bimanual gesture based on the determined first belief, the determined second belief, and the determined third belief.

30. A method of identifying an observed bimanual hand gesture, the method comprising:

tracking movement of a first hand during a first non-occluded segment of an observed bimanual gesture;

tracking movement of a second hand during the first non-occluded segment of the observed bimanual gesture;

determining that the first and second hands are occluded, wherein a determination that the first and second hands are occluded signals a beginning of an occluded segment of the observed bimanual gesture;

tracking movement of the first and second hands as a unit during the occluded segment of the observed bimanual gesture;

determining that the first and second hands are no longer occluded, wherein a determination that the first and second hands are no longer included signals a beginning of a second non-occluded segment of the observed bimanual gesture;

distinguishing, after determining that the first and second hands are no longer occluded, the first hand from the second hand;

after distinguishing the first hand from the second hand, tracking movement of the first hand during the second non-occluded end segment of the observed bimanual gesture;

after distinguishing the first hand from the second hand, tracking movement of the second hand during the second non-occluded segment of the observed bimanual gesture;

determining a first set of measures of likelihoods that the tracked movement of the first hand during the first non-occluded segment corresponds to one or more known bimanual gestures;

determining a second set of measures of likelihoods that the tracked movement of the second hand during the first non-occluded segment corresponds to the one or more known bimanual gestures;

determining a third set of measures of likelihoods that the tracked movement of the first and second hands as a unit during the occluded segment corresponds to the one or more known bimanual gestures;

determining a fourth set of measures of likelihoods that the tracked movement of the first hand during the second non-occluded segment corresponds to the one or more known bimanual gestures;

determining a fifth set of measures of likelihoods that the tracked movement of the second hand during the second non-occluded segment corresponds to the one or more known bimanual gestures; and

identifying a particular one of the one or more known bimanual gestures as corresponding to the observed bimanual gesture based on the first set of measures of the likelihoods that the tracked movement of the first hand during the first non-occluded segment corresponds to the one or more known bimanual gestures, the second set of measures of the likelihoods that the tracked movement of the second hand during the first non-occluded segment corresponds to the one or more known bimanual gestures, the third set of measures of the likelihoods that the tracked movement of the first and second hands as a unit during the occluded segment corresponds to the one or more known bimanual gestures, the fourth set of measures of likelihoods that the tracked movement of the first hand during the second non-occluded segment corresponds to the one or more known bimanual gestures, and the fifth set of measures of likelihoods that the tracked movement of the second hand during the second non-occluded segment corresponds to the one or more known bimanual gestures.

31. The method of claim 30 further comprising determining a type of synchronization characterizing the first and second hands during the occluded segment based on the tracked movement of the first and second hands as a unit during the occluded segment, wherein distinguishing the first hand from the second hand comprises distinguishing the first hand from the second hand based on the determined type of synchronization.

32. The method of claim 31 wherein:

tracking movement of a first hand during a first non-occluded segment of an observed bimanual gesture comprises extracting a sequence of images of the first hand during the first non-occluded segment from a sequence of images of the observed bimanual gesture;

tracking movement of a second hand during a second non-occluded segment of an observed bimanual gesture comprises extracting a sequence of images of the second hand during the first non-occluded segment from the sequence of images of the observed bimanual gesture;

tracking movement of the first and second hands as a unit during the occluded segment of the observed bimanual gesture comprises extracting a sequence of images of the first and second hands during the occluded segment from the sequence of images of the observed bimanual gesture;

tracking movement of the first hand during a second non-occluded segment of the observed bimanual gesture comprises extracting a sequence of images of the first hand during the second non-occluded segment from a sequence of images of the observed bimanual gesture;

tracking movement of the second hand during the second non-occluded segment of the observed bimanual gesture comprises extracting a sequence of images of the second hand during the second non-occluded segment from the sequence of images of the observed bimanual gesture;

determining a first set of measures of likelihoods that the tracked movement of the first hand during the first non-occluded segment corresponds to one or more known bimanual gestures comprises;

identifying a sequence of principal components corresponding to the sequence of images of the first hand during the first non-occluded segment, and

based on the identified sequence of principal components corresponding to the sequence of images of the first hand during the first non-occluded segment, using trained Hidden Markov Models that model first-hand movements of first non-occluded segments of the one or more known bimanual gestures to determine the first set of measures of likelihoods that the tracked movement of the first hand during the first non-occluded segment corresponds to the one or more known bimanual gestures;

determining a second set of measures of likelihoods that the tracked movement of the second hand during the first non-occluded segment corresponds to one or more known bimanual gestures comprises;

identifying a sequence of principal components corresponding to the sequence of images of the second hand during the first non-occluded segment, and

based on the identified sequence of principal components corresponding to the sequence of images of the second hand during the first non-occluded segment, using trained Hidden Markov Models that model second-hand movements of first non-occluded segments of the one or more known bimanual gestures to determine the second set of measures of likelihoods that the tracked movement of the second hand during the first non-occluded segment corresponds to the one or more known bimanual gestures;

determining a third set of measures of likelihoods that the tracked movement of the first and second hands during the occluded segment corresponds to one or more known bimanual gestures comprises;

identifying a sequence of principal components corresponding to the sequence of images of the first and second hands during the occluded segment, and

based on the identified sequence of principal components corresponding to the sequence of images of the first and second hands during the occluded segment, using trained Hidden Markov Models that model occluded-hand movements of occluded segments of the one or more known bimanual gestures to determine the third set of measures of likelihoods that the tracked movement of the first and second hands during the occluded segment corresponds to the one or more known bimanual gestures;

determining a fourth set of measures of likelihoods that the tracked movement of the first hand during the second non-occluded segment corresponds to one or more known bimanual gestures comprises;

identifying a sequence of principal components corresponding to the sequence of images of the first hand during the second non-occluded segment, and

based on the identified sequence of principal components corresponding to the sequence of images of the first hand during the second non-occluded segment, using trained Hidden Markov Models that model first-hand movements of second non-occluded segments of the one or more known bimanual gestures to determine the fourth set of measures of likelihoods that the tracked movement of the first hand during the second non-occluded segment corresponds to the one or more known bimanual gestures; and

determining a fifth set of measures of likelihoods that the tracked movement of the second hand during the second non-occluded segment corresponds to one or more known bimanual gestures comprises;

identifying a sequence of principal components corresponding to the sequence of images of the second hand during the second non-occluded segment, and

based on the identified sequence of principal components corresponding to the sequence of images of the second hand during the second non-occluded segment, using trained Hidden Markov Models that model second-hand movements of second non-occluded segments of the one or more known bimanual gestures to determine the fifth set of measures of likelihoods that the tracked movement of the second hand during the second non-occluded segment corresponds to the one or more known bimanual gestures.

33. The method of claim 32 wherein:

identifying a sequence of principal components corresponding to the sequence of images of the first hand during the first non-occluded segment comprises;

projecting the images from the sequence of images of the first hand during the first non-occluded segment into a first first-hand eigenspace formed based on principal components determined for a set of training images of first-hand movements of first non-occluded segments of the one or more known bimanual gestures, and

extracting, from the first first-hand eigenspace, a first sequence of codewords associated with the projections of the images of the first hand during the first non-occluded segment;

identifying a sequence of principal components corresponding to the sequence of images of the second hand during the first non-occluded segment comprises;

projecting the images from the sequence of images of the second hand during the first non-occluded segment into a first second-hand eigenspace formed based on principal components determined for a set of training images of second-hand movements of first non-occluded segments of the one or more known bimanual gestures, and

extracting, from the first second-hand eigenspace, a second sequence of codewords associated with the projections of the images of the second hand during the first non-occluded segment;

identifying a sequence of principal components corresponding to the sequence of images of the first and second hands during the occluded segment comprises;

projecting the images from the sequence of images of the first and second hands during the occluded segment into an occluded-hand eigenspace formed based on principal components determined for a set of training images of first and second hand movements of occluded segments of the one or more known bimanual gestures, and

extracting, from the occluded-hand eigenspace, a third sequence of codewords associated with the projections of the images of the first and second hands during the occluded segment;

identifying a sequence of principal components corresponding to the sequence of images of the first hand during the second non-occluded segment comprises;

projecting the images from the sequence of images of the first hand during the second non-occluded segment into a second first-hand eigenspace formed based on principal components determined for a set of training images of first-hand movements of second non-occluded segments of the one or more known bimanual gestures, and

extracting, from the second first-hand eigenspace, a fourth sequence of codewords associated with the projections of the images of the first hand during the second non-occluded segment; and

identifying a sequence of principal components corresponding to the sequence of images of the second hand during the second non-occluded segment comprises;

projecting the images from the sequence of images of the second hand during the second non-occluded segment into a second second-hand eigenspace formed based on principal components determined for a set of training images of second-hand movements of second non-occluded segments of the one or more known bimanual gestures, and

extracting, from the second second-hand eigenspace, a fifth sequence of codewords associated with the projections of the images of the second hand during the second non-occluded segment.

34. The method of claim 30 further comprising maintaining a Bayesian network for identifying bimanual gestures that includes a first non-occluded segment first-hand node, a second non-occluded segment first-hand node, a first non-occluded segment second-hand node, a second non-occluded segment second-hand node, an occluded segment node, and a bimanual gesture node, wherein identifying a particular one of the one or more known bimanual gestures as corresponding to the observed bimanual gesture based on the first set of measures of the likelihoods that the tracked movement of the first hand during the first non-occluded segment corresponds to the one or more known bimanual gestures, the second set of measures of the likelihoods that the tracked movement of the second hand during the first non-occluded segment corresponds to the one or more known bimanual gestures, the third set of measures of the likelihoods that the tracked movement of the first and second hands as a unit during the occluded segment corresponds to the one or more known bimanual gestures, the fourth set of measures of the likelihoods that the tracked movement of the first hand during the second non-occluded segment corresponds to the one or more known bimanual gestures, and the fifth set of measures of the likelihoods that the tracked movement of the second hand during the second non-occluded segment corresponds to the one or more known bimanual gestures comprises:

inputting, to the first non-occluded segment first-hand node of the Bayesian network, the first set of measures of the likelihoods that the tracked movement of the first hand during the first non-occluded segment corresponds to the one or more known bimanual gestures;

inputting, to the first non-occluded segment second-hand node of the Bayesian network, the second set of measures of the likelihoods that the tracked movement of the second hand during the first non-occluded segment corresponds to the one or more known bimanual gestures;

inputting, to the occluded segment node of the Bayesian network, the third set of measures of the likelihoods that the tracked movement of the first and second hands during the occluded segment corresponds to the one or more known bimanual gestures;

inputting, to the second non-occluded segment first-hand node of the Bayesian network, the fourth set of measures of the likelihoods that the tracked movement of the first hand during the second non-occluded segment corresponds to the one or more known bimanual gestures;

inputting, to the second non-occluded segment second-hand node of the Bayesian network, the fifth set of measures of the likelihoods that the tracked movement of the second hand during the second non-occluded segment corresponds to the one or more known bimanual gestures;

using the first non-occluded segment first-hand node of the Bayesian network to determine a first belief of a known bimanual gesture that corresponds to the tracked movement of the first hand during the first non-occluded segment based on the input first set of measures of the likelihoods that the tracked movement of the first hand during the first non-occluded segment corresponds to the one or more known bimanual gestures;

using the first non-occluded segment second-hand node of the Bayesian network to determine a second belief of a known bimanual gesture that corresponds to the tracked movement of the second hand during the first non-occluded segment based on the input second set of measures of the likelihoods that the tracked movement of the second hand during the first non-occluded segment corresponds to the one or more known bimanual gestures;

using the occluded segment node of the Bayesian network to determine a third belief of a known bimanual gesture that corresponds to the tracked movement of the first and second hands during the occluded segment based on the input third set of measures of the likelihoods that the tracked movement of the first and second hands during the occluded segment corresponds to the one or more known bimanual gestures;

using the second non-occluded segment first-hand node of the Bayesian network to determine a fourth belief of a known bimanual gesture that corresponds to the tracked movement of the first hand during the second non-occluded segment based on the input fourth set of measures of the likelihoods that the tracked movement of the first hand during the second non-occluded segment corresponds to the one or more known bimanual gestures;

using the second non-occluded segment second-hand node of the Bayesian network to determine a fifth belief of a known bimanual gesture that corresponds to the tracked movement of the second hand during the second non-occluded segment based on the input fifth set of measures of the likelihoods that the tracked movement of the second hand during the second non-occluded segment corresponds to the one or more known bimanual gestures;

inputting the determined first belief, the determined second belief, the determined third belief, the determined fourth belief, and the determined fifth belief to the bimanual gesture root node; and

using the bimanual gesture root node to identify a particular one of the one or more known bimanual gestures as corresponding to the observed bimanual gesture based on the determined first belief, the determined second belief, the determined third belief, the determined fourth belief, and the determined fifth belief.

35. A computer program product for identifying an observed bimanual hand gesture, the computer program product being tangibly embodied in a machine readable medium and comprising instructions that, when read by a machine, operate to cause a data processing apparatus to:

track movement of a first hand during an observed bimanual gesture;

track movement of a second hand during the observed bimanual gesture;

determine a first set of measures of likelihoods that the tracked movement of the first hand corresponds to one or more known bimanual gestures;

determine a second set of measures of likelihoods that the tracked movement of the second hand corresponds to the one or more known bimanual gestures; and

identify a particular one of the one or more known bimanual gestures as corresponding to the observed bimanual gesture based on the first set of measures of the likelihoods that the tracked movement of the first hand corresponds to the one or more known bimanual gestures and the second set of measures of the likelihoods that the tracked movement of the second hand corresponds to the one or more known bimanual gestures.

36. A system for identifying an observed bimanual hand gesture, the system comprising:

a first hand tracking means for tracking movement of a first hand during an observed bimanual gesture;

a second hand tracking means for tracking movement of a second hand during the observed bimanual gesture;

a first determining means for determining a first set of measures of likelihoods that the tracked movement of the first hand corresponds to one or more known bimanual gestures;

a second determining means for determining a second set of measures of likelihoods that the tracked movement of the second hand corresponds to the one or more known bimanual gestures; and

a first bimanual gesture identifying means for identifying a particular one of the one or more known bimanual gestures as corresponding to the observed bimanual gesture based on the first set of measures of the likelihoods that the tracked movement of the first hand corresponds to the one or more known bimanual gestures and the second set of measures of the likelihoods that the tracked movement of the second hand corresponds to the one or more known bimanual gestures.

37. A computer program product, tangibly embodied in a machine readable medium, the computer program product comprising instructions that, when read by a machine, operate to cause a data processing apparatus to:

recognize an occlusion segment and a non-occlusion segment of a bimanual gesture;

recognize, from a non-occlusion segment, separate left hand and right hand gestures;

input the recognized left hand and right hand gestures and the recognized occlusion segment into nodes of a Bayesian network; and

automatically recognize the bimanual gesture based on inputting the recognized left hand and right hand gestures and the recognized occlusion segment into nodes of a Bayesian network.

38. A computer program product, tangibly embodied in a machine readable medium, the computer program product comprising instructions that, when read by a machine, operate to cause a data processing apparatus to:

track movements of two hands;

predict an occlusion period based on the tracked movements; and

automatically recognize a bimanual gesture based on the predicted occlusion period.

39. A computer program product, tangibly embodied in a machine readable medium, the computer program product comprising instructions that, when read by a machine, operate to cause a data processing apparatus to:

track movements of two hands;

segment the movement into a beginning non-occluded segment, at least one middle non-occluded segment, an end non-occluded segment, and at least one occluded segment; and

automatically recognize a bimanual gesture based on the segmented movement.

40. A computer program product, tangibly embodied in a machine readable medium, the computer program product comprising instructions that, when read by a machine, operate to cause a data processing apparatus to:

recognize a non-occluded bimanual gesture;

recognize separate left hand and right hand gestures from the non-occluded bimanual gesture;

input the recognized left hand and right hand gestures into nodes of a Bayesian network; and

automatically recognize the bimanual gesture based on inputting the recognized left hand and right hand gestures into nodes of a Bayesian network.

41. A computer program product, tangibly embodied in a machine readable medium, the computer program product comprising instructions that, when read by a machine, operate to cause a data processing apparatus to:

track a gesture, further comprising;

predicting an occlusion period,

detecting a movement of two occluded hands during the occlusion period, the two occluded hands being tracked as a unit,

determining a type of synchronization characterizing the two occluded hands during the occlusion period based on the tracked movement of the two occluded hands, and

determining, based on the determined type of synchronization, whether directions of travel for each of the two occluded hands change during the occlusion period; and

automatically recognize the tracked gesture.