MULTI-MODAL MODELING OF TEMPORAL INTERACTION SEQUENCES

• US 20140212853A1
• Filed: 01/31/2013
• Published: 07/31/2014
• Est. Priority Date: 01/31/2013
• Status: Active Grant

• Current Assignee: SRI International, Inc.
• Original Assignee: SRI International, Inc.
• Inventors: Divakaran, Ajay, Khan, Saad, Lubin, Jeffrey, Siddiquie, Behjat, Sawhney, Harpreet S.
• Granted Patent: US 9,734,730 B2
• Time in Patent Office: Days
• US Class Current: 434/236
• CPC Class Codes: G09B 19/00 Teaching not covered by other main groups of this subclass teaching or practice apparatus for gun-aiming or gun-laying F41G3/26

Abstract

A multi-modal interaction modeling system can model a number of different aspects of a human interaction across one or more temporal interaction sequences. Some versions of the system can generate assessments of the nature or quality of the interaction or portions thereof, which can be used to, among other things, provide assistance to one or more of the participants in the interaction.

Claims

1. A method for assessing an interaction involving at least two participants, at least one of the participants being a person, the method comprising, with a computing system:

detecting, from multi-modal data captured by at least one sensing device during the interaction, a plurality of different behavioral cues expressed by the participants;

analyzing the detected behavioral cues with respect to a plurality of different time scales, each of the time scales being defined by a time interval whose size can be compared to the size of other time intervals of the interaction;

recognizing, based on the analysis of the detected behavioral cues, a temporal interaction sequence comprising a pattern of the behavioral cues corresponding to one or more of the time scales; and

deriving, from the temporal interaction sequence, an assessment of the efficacy of the interaction.

2. The method of claim 1, wherein the plurality of different behavioral cues of the temporal interaction sequence involves at least one non-verbal cue.

3. The method of claim 1, comprising recognizing a plurality of temporal interaction sequences occurring over different time intervals, at least one of the temporal interaction sequences involving behavioral cues expressed by the person and behavioral cues expressed by another participant, and deriving the assessment from the plurality of temporal interaction sequences.

4. The method of claim 1, wherein each of the plurality of different behavioral cues comprises a cue relating to one or more of:

a gesture, a body pose, a head pose, an eye gaze, a facial expression, a voice tone, a voice loudness, another non-verbal vocal feature, and verbal content.

5. The method of claim 1, comprising semantically analyzing verbal content of at least one of the behavioral cues and recognizing the temporal interaction sequence based on the semantic analysis of the verbal content.

6. The method of claim 1, comprising semantically analyzing the verbal content of at least one of the behavioral cues and deriving the assessment based on the semantic analysis of the verbal content.

7. The method of claim 1, comprising analyzing the relative significance of each behavioral cue in relation to the temporal interaction sequence as a whole, and deriving the assessment based on the relative significance of each behavioral cue to the temporal interaction sequence as a whole.

8. The method of claim 1, comprising analyzing the relative significance of each behavioral cue in relation to the other behavioral cues of the temporal interaction sequence, and deriving the assessment based on the relative significance of each of the behavioral cues of the temporal interaction sequence.

9. The method of claim 1, comprising recognizing a plurality of temporal interaction sequences, deriving an assessment of the nature of each temporal interaction sequence, and determining changes in the efficacy of the interaction over time based on the assessments of each of the temporal interaction sequences.

10. The method of claim 1, comprising recognizing a plurality of temporal interaction sequences, deriving an assessment of the nature of each temporal interaction sequence, and evaluating the overall efficacy of the interaction based on the assessments of the temporal interaction sequences.

11. The method of claim 1, comprising recognizing a plurality of temporal interaction sequences having overlapping time intervals and deriving the assessment based on the plurality of temporal interaction sequences having overlapping time intervals.

12. The method of claim 1, wherein the plurality of behavioral cues of the temporal interaction sequence comprises at least one behavioral cue that indicates a positive interaction and at least one behavioral cue that indicates a negative interaction.

13. The method of claim 1, comprising presenting a suggestion to one or more of the participants based on the assessment.

14. The method of claim 1, comprising generating a description of the interaction based on the assessment.

15. The method of claim 14, wherein the description comprises a recounting of the interaction, and the recounting comprises one or more behavioral cues having evidentiary value to the assessment.

16. The method of claim 1, comprising determining a semantic meaning of the pattern of related behavioral cues and using the semantic meaning to derive the assessment.

17. The method of claim 1, comprising determining the pattern of related behavioral cues using a graphical model.

18. The method of claim 17, wherein the graphical model comprises a discriminative probabilistic model.

19. The method of claim 18, wherein the discriminative probabilistic model comprises conditional random fields.

20. The method of claim 1, comprising developing the assessment of the interaction based on changes in the behavioral cues of at least one of the participants over time during the interaction.

21. The method of claim 1, comprising inferring one or more events from the temporal interaction sequence, the one or more events each comprising a semantic characterization of at least a portion of the temporal interaction sequence, deriving an assessment of the one or more events based on the behavioral cues, and deriving the assessment of the efficacy of the interaction based on the assessment of the one or more events.

22. The method of claim 1, comprising determining relationships between or among the behavioral cues of the temporal interaction sequence.

23. The method of claim 1, wherein the duration of each of the plurality of different time scales is defined by the detection of at least two of the behavioral cues.

24. A method for assessing an interaction involving at least two participants, at least one of the participants being a person, the method comprising, with a computing system:

detecting, from multi-modal data captured by at least one sensing device, a plurality of different behavioral cues expressed by the participants during the interaction, the behavioral cues comprising one or more non-verbal cues and verbal content;

recognizing a temporal interaction sequence comprising a pattern of the behavioral cues occurring over a time interval during the interaction; and

deriving, from the temporal interaction sequence, an assessment of the efficacy of the interaction.

25. A method for assessing a person'"'"'s emotional state during an interaction involving the person and at least one other participant, the method comprising, with a computing system:

detecting, from multi-modal data captured by one or more sensing devices, a plurality of different behavioral cues expressed by the participants during the interaction;

recognizing a plurality of temporal interaction sequences, each temporal interaction sequence comprising a pattern of the behavioral cues occurring over a time interval during the interaction, and at least one of the temporal interaction sequences involving behavioral cues of the person and behavioral cues of at least one other participant;

assessing the person'"'"'s emotional state during each of the temporal interaction sequences based on the behavioral cues involved in the temporal interaction sequence;

determining a relative significance of each of the temporal interaction sequences to the interaction; and

assessing the person'"'"'s emotional state during the interaction as a whole based on the person'"'"'s assessed emotional state during each of the temporal interaction sequences and the determined relative significance of each of the temporal interaction sequences to the interaction as a whole.

26. The method of claim 25, comprising assessing at least one other participant'"'"'s emotional state during the temporal interaction sequences based on the behavioral cues involved in the temporal interaction sequences, and using the assessment of the other participant'"'"'s emotional state to evaluate the emotional state of the person.

27. The method of claim 25, comprising determining an aspect of the person'"'"'s behavior based on the person'"'"'s assessed emotional state during one or more of the temporal interaction sequences and the time interval in which it occurred relative to the total duration of the interaction.

28. The method of claim 25, comprising determining a degree of intensity of a behavioral cue of the person and using the degree of intensity to recognize one or more of the temporal interaction sequences.

29. The method of claim 25, comprising determining the relative significance of each of the temporal interaction sequences by comparing the behavioral cues involved in the temporal interaction sequence to the behavioral cues involved in the other temporal interaction sequences.

30. The method of claim 25, wherein the behavioral cues involved in the temporal interaction sequences comprise verbal content and non-verbal cues.

31. The method of claim 25, wherein the behavioral cues involved in the temporal interaction sequences comprise cues relating to one or more of:

a gesture, a body pose, a head pose, an eye gaze, a facial expression, a voice tone, a voice loudness, another non-verbal vocal feature, and verbal content.

32. The method of claim 25, comprising assessing the person'"'"'s emotional state by analyzing changes in the person'"'"'s behavior over time during one or more of the temporal interaction sequences.

33. The method of claim 25, comprising assessing the person'"'"'s emotional state by analyzing changes in the person'"'"'s behavior in response to the behavior of other participants during one or more of the temporal interaction sequences.

34. The method of claim 25, wherein at least two of the temporal interaction sequences occur over different time intervals, the method comprising assessing the person'"'"'s emotional state by analyzing the person'"'"'s behavior over the different time intervals during the interaction.

35. The method of claim 25, comprising assessing the person'"'"'s emotional state by analyzing the relative degree of emotion expressed by the person during one or more of the temporal interaction sequences.

36. The method of claim 25, comprising assessing the person'"'"'s emotional state by analyzing the semantic meaning of verbal content of the behavioral cues.

37. The method of claim 25, comprising assessing the person'"'"'s emotional state using conditional random fields and by analyzing one or more hidden states of the conditional random fields.