Multimodal interactions based on body postures

• US 20140132505A1
• Filed: 05/23/2011
• Published: 05/15/2014
• Est. Priority Date: 05/23/2011
• Status: Active Grant

• Current Assignee: Hewlett-Packard Development Company, L.P. (HP Inc.)
• Original Assignee: Hewlett-Packard Development Company, L.P. (HP Inc.)
• Inventors: Vennelakanti, Ramadevi, Subramanian, Anbumani, Dey, Prasenjit, Madhvanath, Sriganesh, Mandalapu, Dinesh
• Granted Patent: US 9,619,018 B2
• Time in Patent Office: Days
• US Class Current: 345/156
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883

Abstract

In one example, a method for multimodal human-machine interaction includes sensing a body posture of a participant using a camera (605) and evaluating the body posture to determine a posture-based probability of communication modalities from the participant (610). The method further includes detecting control input through a communication modality from the participant to the multimedia device (615) and weighting the control input by the posture-based probability (620).

Claims

1. A method for multimodal human-machine interaction comprising:

sensing a body posture of a participant using a camera (605);

evaluating the body posture to determine a posture-based probability of communication modalities from the participant, the posture-based probability being calculated using a processor in a multimedia device (610);

detecting control input through a communication modality from the participant using a sensing device in the multimedia device (615); and

weighting the control input by the posture-based probability (620).

2. The method of claim 1, further comprising receiving application state information (725).

3. The method of claim 2, further comprising evaluating the application state information determine to which commands and communication modalities are available to receive commands (725).

4. The method of claim 1, further comprising:

receiving application content information (730); and

evaluating a probability of a posture of the participant when presented with application content described in the application content information (730).

5. The method of claim 4, further comprising:

receiving an output confidence from a posture recognizer (515); and

weighting the probability of a posture of the participant by the output confidence.

6. The method of claim 1, further comprising:

recognizing the participant (705);

retrieving a record of past multimedia interactions of the participant (510); and

weighting input control based on past the multimedia transactions.

7. The method of claim 1, further comprising calculating a probability of a modality according to the following formula:

8. A method for human-machine interaction implemented by a processor in a multimedia device comprising:

recognizing a participant in a co-present multi-user multimodal human-machine interaction (705) with a multimedia device (200);

initializing a skeletal model of the participant (710);

tracking a posture of the participant during the multimodal human-machine interaction using the skeletal model (715);

evaluating the participant'"'"'s posture to determine a posture-based probability of communication modalities from the participant (720);

receiving application state information and evaluating application state information to determine which communication modalities are available (725);

receiving application content information and evaluating a probability of a posture of a participant when presented with the application content (730);

detecting control input through a communication modality from the participant (735); and

weighting the control input by the posture-based probability (740).

9. A multimedia device comprising:

a sensing device (220);

an application (235) for presenting a multimedia experience to a participant (115);

a recognition module (230) for accepting input from the participant through the sensing device (220);

a posture analyzer (M3) for outputting a participant posture;

a multimodal combiner (225) for evaluating a probability of a communication modality given the participant posture and for weighting input from the participant by the probability of a communication modality.

10. The device of claim 9, in which the sensing device (220) comprises a color camera and a depth camera.

11. The device of claim 9, further comprising application state information (540) and application content information (505) passed from the application to the multimodal combiner (225).

12. The device of claim 11, in which the application content information (505) comprises identification of a type of media being displayed and the length of the media presentation.

13. The device of claim 12, further comprising a probability of a posture calculated from a confidence level of the posture analyzer (515) and application content information (505).

14. The device of claim 11, in which the application state information (540) comprises identification of which application is currently in the observable foreground of the multimedia device (200) and which commands are currently active in the application.

15. The device of claim 9, in which the recognizer module comprises a speech recognizer (M1) and a gesture recognizer (M2), the multimodal combiner (225) for weighting input from the speech recognizer (M1) and gesture recognizer (M2) by the probability of a communication modality derived from the participant posture.