Mobile device executing face-to-face interaction monitoring, method of monitoring face-to-face interaction using the same, and interaction monitoring system including the same, and mobile interaction monitoring application executed on the same

• US 9,813,879 B2
• Filed: 06/12/2014
• Issued: 11/07/2017
• Est. Priority Date: 04/17/2014
• Status: Active Grant

• Current Assignee: Korea Advanced Institute of Science and Technology
• Original Assignee: Korea Advanced Institute of Science and Technology
• Inventors: Song, Junehwa, Lee, Youngki, Min, Chulhong, Hwang, Chanyou, Lee, Jaeung, Hwang, Inseok, Ju, Younghyun, Yoo, Chungkuk, Moon, Miri, Lee, Uichin
• Primary Examiner(s): Yang, Qian
• Application Number:
  US14/302,440
• Publication Number:
  US 20150302846A1
• Time in Patent Office: 1,244 Days
• Field of Search: None
• CPC Class Codes: G10L 17/00 Speaker identification or verification techniques
• CPC Class Codes: G10L 25/48 specially adapted for particular use
• CPC Class Codes: G10L 25/78 Detection of presence or absence of voice signals switching of direction of transmission by voice frequency in two-way loud-speaking telephone systems H04M9/10
• CPC Class Codes: H04W 4/08 User group management
• CPC Class Codes: H04W 56/001 Synchronization between nodes

Abstract

Disclosed herein is a mobile face-to-face interaction monitoring device and method using the same and system including the same, for supporting accurate and efficient turn monitoring. One embodiment of the mobile face-to-face interaction monitoring device may comprise a conversation group detector for scanning mobile devices in a surrounding area and setting a conversation group, a turn detector for determining (conversational) turn using volume topography created based on sound signals detected in the mobile devices in the conversation group, and a meta-linguistic information processor for extracting meta-linguistic context of participants or interactants in the conversation group based on the turn. Other embodiments are described and shown.

Claims

1. A mobile face-to-face interaction monitoring device, comprising:

a conversation group detector for scanning one or more mobile devices in a surrounding area and setting a conversation group;

a turn detector for determining turn by using volume topography created based on sound signals detected in the mobile devices in the conversation group,

wherein the turn detector comprises;

a volume topography builder for creating the volume topography based on the sound signals in the mobile devices in the conversation group, during a training period; and

a volume topography matcher for determining current turn by matching current sound signals with the volume topography, after the training period; and

a meta-linguistic information processor for extracting meta-linguistic context of interactants in the conversation group based on the turn;

wherein the turn is a sound-signal segment where an interactant in the conversation group starts and ends the interactant'"'"'s speech, and

wherein the volume topography is created by using a signature or feature vector of volume value(s) for the sound signals detected,

wherein the meta-linguistic context of interactants in the conversation group is extracted by extracting features of the turn and determining the features of the turn categorized as;

individual for each of the interactants, including speaking time, a quantity and duration of speech and non-speech turns, and associated statistics, for each of the interactants;

relational among the interactants, including orders and frequencies of exchanges of the turns and ratio thereof, among the interactants in the conversation group; and

cumulative for one or more conversation sessions and the conversation group, including a number of the interactants in the conversation sessions, a quantity and duration of the conversation sessions, and a quantity and duration of the turns in the conversation sessions and the conversation group.

2. The mobile face-to-face interaction monitoring device according to claim 1, wherein the conversation group detector comprises:

a voice detector for detecting sound signals, and

an interactant manager for storing mobile device addresses and forming the conversation group when the mobile devices having the addresses are scanned.

3. The mobile face-to-face interaction monitoring device according to claim 2, wherein the conversation group detector further comprises

a subgroup detector for determining subgroup formation by analyzing overlapping speech pattern in the sound signals.

4. The mobile face-to-face interaction monitoring device according to claim 2, wherein the conversation group detector further comprises:

a head selector for selecting a head mobile device among a plurality of mobile devices in the conversation group, wherein

the head mobile device collects the sound signals in the mobile devices in the conversation group, determines the turn, and transmits the turn to the mobile devices.

5. The mobile face-to-face interaction monitoring device according to claim 1, further comprising

a monitoring planner for controlling monitoring of the turn based on information of the conversation group, wherein the monitoring planner comprises:

a source selector for determining a quantity of the mobile devices in the conversation group and selecting one or more source mobile devices for collaborative turn monitoring among the mobile devices in the conversation group, and

an execution planner for controlling the monitoring in the turn detector.

6. The mobile face-to-face interaction monitoring device according to claim 5, wherein the monitoring planner further comprises

a sync manager for synchronizing the mobile devices in the conversation group.

7. The mobile face-to-face interaction monitoring device according to claim 5, wherein the monitoring planner further comprises

a resource monitor for monitoring resources and resource utilization in the mobile devices.

8. The mobile face-to-face interaction monitoring device according to claim 1, wherein the volume topography is created by using a feature vector P(t), wherein

the feature vector P(t) is defined as P(t)=(p(t,1), p(t,2), . . . , p(t,np)),

where p(t, i) is an average of a square of the sound signals in each mobile device i of the mobile devices at a given time t, and

where np is a quantity of the mobile devices in the conversation group.

9. The mobile face-to-face interaction monitoring device according to claim 8, wherein the quantity np of the mobile devices is equal to or less than a number of the interactants in the conversation group.

10. The mobile face-to-face interaction monitoring device according to claim 8, wherein the volume topography is created by using a normalized vector P′

(t), wherein

the normalized feature vector P′

(t) is defined as P′

(t)=P(t)/E(t),

where E(t) is an average of the feature vector P(t).

11. The mobile face-to-face interaction monitoring device according to claim 10, wherein the volume topography is created by using a transformed vector P″

(t), wherein

the transformed vector P″

(t) is defined as

P″

(t)=D(t)*P′

(t)={D(t,1)*p(t,1)/E(t),D(t,2)*p(t,2)/E(t), . . . ,D(t,np)*p(t,np)/E(t)},

where a decibel D(t) measured on the mobile device i, at the given time t, is defined as D(t,i)=20*log 10 (p(t,i)/p.ref),

where p.ref is a standard reference sound pressure level.

12. The mobile face-to-face interaction monitoring device according to claim 1, wherein the volume topography builder creates a quantity of n+1 of the volume topography, where

n is a number of the interactants in the conversation group, and wherein:

each of the n-quantity of the volume topography created respectively corresponds to each of the n-number of the interactants, and

the l-quantity of the volume topography created corresponds to non-speech in the conversation group.

13. The mobile face-to-face interaction monitoring device according to claim 1, wherein the volume topography builder recreates the volume topography when there is a change in the conversation group and the surrounding area.

14. The mobile face-to-face interaction monitoring device according to claim 1, wherein the turn has factors of speaker, start time, and end time.

15. The mobile face-to-face interaction monitoring device according to claim 1, wherein the meta-linguistic information processor comprises:

a feature extractor for extracting prosodic features of the sound signals; and

a pattern analyzer for determining interaction pattern among the interactants in the conversation group, based on the turn and at least one of the prosodic features.

16. The mobile face-to-face interaction monitoring device according to claim 15, wherein the pattern analyzer determines interactivity based on a quantity of the turn in a given time.

17. The mobile face-to-face interaction monitoring device according to claim 15, wherein the pattern analyzer determines sparseness based on a quantity of non-speech turn in a given time.

18. The mobile face-to-face interaction monitoring device according to claim 15, wherein the pattern analyzer determines skewness based on a standard deviation of a quantity of the turn for all of the interactants.

19. The mobile face-to-face interaction monitoring device according to claim 1, further comprising

an interaction history manager for storing information of the turn and a conversation session and supporting user query and generating an interaction history based on the information.

20. A mobile face-to-face interaction monitoring method, comprising:

scanning one or more mobile devices in a surrounding area and setting a conversation group;

detecting sound signals in the mobile devices in the conversation group, relative to differences in positions of the mobile devices;

creating volume topography based on the sound signals for each interactant in the conversation group;

determining turn by using the volume topography; and

extracting meta-linguistic context of interactants in the conversation group based on the turn,

wherein the turn is a sound-signal segment where each interactant in the conversation group starts and ends the interactant'"'"'s speech,

wherein the volume topography is created by using a signature or feature vector of volume value(s) for the sound signals detected, and

wherein the creating the volume topography based on the sound signals is performed by creating a quantity of n+l of the volume topography, where

n is a number of the interactants in the conversation group, and wherein;

each of the n-quantity of the volume topography created respectively corresponds to each of the n-number of the interactants, and

the l-quantity of the volume topography created corresponds to non-speech or silent turns in the conversation group.

21. The mobile face-to-face interaction monitoring method according to claim 20, further comprising:

determining a quantity of the mobile devices in the conversation group; and

selecting one or more source devices among the mobile devices in the conversation group, for collaborative turn monitoring among the mobile devices in the conversation group.

22. The mobile face-to-face interaction monitoring method according to claim 21, further comprising

synchronizing the mobile devices in the conversation group.

23. The mobile face-to-face interaction monitoring method according to claim 20, wherein:

the creating the volume topography based on the sound signals is performed during a training period; and

the determining the turn by using the volume topography comprises

determining current turn by matching current sound signals with the volume topography, after the training period.

24. The mobile face-to-face interaction monitoring method according to claim 23, wherein the creating the volume topography based on the sound signals is performed by using a feature vector P(t), wherein

the feature vector P(t) is defined as P(t)=(p(t,1), p(t,2), . . . , p(t,np)),

where p(t, i) is an average of a square of the sound signals in each mobile device i of the mobile devices at a given time t, and

where np is a quantity of the mobile devices in the conversation group.

25. The mobile face-to-face interaction monitoring method according to claim 24, wherein the quantity np of the mobile devices is equal to or less than a number of the interactants in the conversation group.

26. The mobile face-to-face interaction monitoring method according to claim 24, wherein the creating the volume topography based on the sound signals is performed by using a normalized vector P′

(t), wherein

the normalized feature vector P′

(t) is defined as P′

(t)=P(t)/E(t),

where E(t) is an average of the feature vector P(t).

27. The mobile face-to-face interaction monitoring method according to claim 26, wherein the creating the volume topography based on the sound signals is performed by using a transformed vector P″

(t), wherein

the transformed vector P″

(t) is defined as

P″

(t)=D(t)*P′

(t)={D(t,1)*p(t,1)/E(t),D(t,2)*p(t,2)/E(t), . . . ,D(t,np)*p(t,np)/E(t)},

where a decibel D(t) measured on the mobile device i, at the given time t, is defined as D(t,i)=20*log 10 (p(t,i)/p.ref),

where p.ref is a standard reference sound pressure level.

28. The mobile face-to-face interaction monitoring method according to claim 23, further comprising:

recreating the volume topography when there is a change in the conversation group and the surrounding area.

29. The mobile face-to-face interaction monitoring method according to claim 23, wherein the turn has factors of speaker, start time, and end time.

30. The mobile face-to-face interaction monitoring method according to claim 20, wherein the extracting the meta-linguistic context of interactants in the conversation group based on the turn comprises:

extracting features of the turn;

extracting prosodic features of the sound signals; and

determining interaction pattern among the interactants in the conversation group, based on the features of the turn and at least one of the prosodic features.

31. The mobile face-to-face interaction monitoring method according to claim 30, wherein the extracting features of the turn is performed by determining the features of the turn categorized as:

individual for each of the interactants, including speaking time, a quantity and duration of speech and non-speech turns, and associated statistics, for each of the interactants;

relational among the interactants, including orders and frequencies of exchanges of the turns and ratio thereof, among the interactants in the conversation group; and

cumulative for one or more conversation sessions and the conversation group, including a number of the interactants in the conversation sessions, a quantity and duration of the conversation sessions, and a quantity and duration of the turns in the conversation sessions and the conversation group.

32. An interaction monitoring system, comprising:

a first mobile device, which determines turn, and

at least one second mobile device, which receives the turn determined from the first mobile device;

wherein the first mobile device comprises;

a first conversation group detector for scanning one or more mobile devices in a surrounding area and setting a conversation group,

a turn detector for determining turn for interactants in the conversation group by using volume topography created by using a signature or feature vector of volume value(s) based on sound signals detected in the mobile devices, relative to different positions of the first and second mobile devices, and represented as a set of sound-signal signatures for each of the interactants, and

a first meta-linguistic information processor for extracting meta-linguistic context of the interactants in the conversation group based on the turn; and

wherein the second mobile device comprises;

a second conversation group detector for scanning the mobile devices in the surrounding area and setting the conversation group, and

a second meta-linguistic information processor for extracting the meta-linguistic context of the interactants in the conversation group based on the turn received from the first mobile device;

wherein the volume topography is created by using a transformed vector P″

(t),

wherein the transformed vector P″

(t) is defined as

P″

(t)=D(t)*P′

(t)={D(t,1)*p(t,1)/E(t),D(t,2)*p(t,2)/E(t), . . . ,D(t,np)*p(t,np)/E(t)},

where a decibel D(t) measured on the mobile device i, at the given time t, is defined as D(t,i)=20*log 10 (p(t,i)/p.ref), where p.ref is a standard reference sound pressure level, and

where a normalized feature vector P′

(t) is defined as P′

(t)=P(t)/E(t), where E(t) is an average of a feature vector P(t),

where the feature vector P(t) is defined as P(t)=(p(t,1), p(t,2), . . . , p(t,np)),

where p(t, i) is an average of a square of the sound signals in each mobile device i of the mobile devices at a given time t, and np is a quantity of the mobile devices in the conversation group.