Audio User Interaction Recognition and Context Refinement

• US 20130304476A1
• Filed: 11/12/2012
• Published: 11/14/2013
• Est. Priority Date: 05/11/2012
• Status: Active Application

• Current Assignee: Qualcomm, Inc.
• Original Assignee: Qualcomm, Inc.
• Inventors: Kim, Lae-Hoon, Shin, Jongwon, Visser, Erik
• Application Number:
  US13/674,773
• Publication Number:
  US 20130304476A1
• Time in Patent Office: Days
• US Class Current: 704/270
• CPC Class Codes: G01S 3/80 using ultrasonic, sonic or infrasonic waves
• CPC Class Codes: G10L 17/00 Speaker identification or verification techniques
• CPC Class Codes: G10L 2021/02166 Microphone arrays; Beamforming
• CPC Class Codes: G10L 21/00 Speech or voice signal processing techniques to produce another audible or non-audible signal, e.g. visual or tactile, in order to modify its quality or its intelligibility G10L19/00 takes precedence
• CPC Class Codes: G10L 25/06 the extracted parameters being correlation coefficients
• CPC Class Codes: G10L 25/48 specially adapted for particular use
• CPC Class Codes: H04L 65/403 Arrangements for multi-party communication, e.g. for conferences data switching systems for conference H04L12/18; arrangements for connecting several subscribers to a common circuit, i.e. affording conference facilities H04M3/56; television conferencing systems H04N7/15
• CPC Class Codes: H04N 7/15 Conference systems
• CPC Class Codes: H04R 1/406 microphones
• CPC Class Codes: H04R 2430/20 Processing of the output signals of the acoustic transducers of an array for obtaining a desired directivity characteristic H04R2203/12 takes precedence
• CPC Class Codes: H04R 2460/01 Hearing devices using active noise cancellation
• CPC Class Codes: H04R 2499/11 Transducers incorporated or for use in hand-held devices, e.g. mobile phones, PDA's, camera's
• CPC Class Codes: H04R 29/005 Microphone arrays
• CPC Class Codes: H04R 29/008 Visual indication of individual signal levels visual indication of stereophonic sound image H04S7/40
• CPC Class Codes: H04R 3/005 for combining the signals of two or more microphones specially adapted for hearing aids H04R25/407

Abstract

A system which performs social interaction analysis for a plurality of participants includes a processor. The processor is configured to determine a similarity between a first spatially filtered output and each of a plurality of second spatially filtered outputs. The processor is configured to determine the social interaction between the participants based on the similarities between the first spatially filtered output and each of the second spatially filtered outputs and display an output that is representative of the social interaction between the participants. The first spatially filtered output is received from a fixed microphone array, and the second spatially filtered outputs are received from a plurality of steerable microphone arrays each corresponding to a different participant.

Claims

1. A system which performs social interaction analysis for a plurality of participants, comprising:

a processor configured to;

determine a similarity between a first spatially filtered output and each of a plurality of second spatially filtered outputs,

determine a social interaction between the participants based on the similarity between the first spatially filtered output and each of the second spatially filtered outputs, and

display an output representative of the social interaction between the participants;

wherein the first spatially filtered output is received from a fixed microphone array, and the second spatially filtered outputs are received from a plurality of steerable microphone arrays each corresponding to a different participant.

2. The system of claim 1, wherein the output is displayed in real-time as the participants are interacting with each other.

3. The system of claim 1, wherein the output comprises an interaction graph comprising:

a plurality of identifiers, each identifier corresponding to a respective participant; and

a plurality of indicators, each indicator providing information relating to at least one of;

a participant looking at another participant, a strength of an interaction between two participants, a participation level of a participant, or a leader of a group of participants.

4. The system of claim 3, wherein the strength of the interaction between two participants is based on a time that the two participants have interacted.

5. The system of claim 3, wherein the indicators have at least one of a direction, a thickness, or a color, wherein the direction indicates which participant is looking at another participant, the thickness indicates the strength of the interaction between two participants, and the color indicates the leader of the group of participants.

6. The system of claim 3, wherein each of the participants is a speaker.

7. The system of claim 3, wherein the interaction graph is used to assess group dynamics or topic dynamics.

8. The system of claim 3, wherein the interaction graph indicates social interaction information among the participants.

9. The system of claim 8, wherein the social interaction information is accumulated over a period of time.

10. The system of claim 3, wherein the interaction graph is displayed on a smartphone.

11. The system of claim 3, wherein the interaction graph is displayed on at least one from among the group comprising a handset, a laptop, a tablet, a computer, and a netbook.

12. The system of claim 3, wherein each indicator represents active participant location and energy.

13. The system of claim 12, further comprising an additional indicator that represents a refined active participant location and energy.

14. The system of claim 12, wherein the indicators comprise beam patterns.

15. The system of claim 1, wherein the processor is further configured to perform real-time meeting analysis of a meeting the participants are participating in.

16. The system of claim 1, wherein the processor is further configured to generate a personal time line for a participant that shows an interaction history of the participant with respect to the other participants, a meeting topic, or a subject matter.

17. The system of claim 1, wherein the processor is further configured to generate participant interaction statistics over time.

18. The system of claim 1, wherein the processor is further configured to generate an evolution of interaction between participants over time.

19. The system of claim 1, wherein the processor is further configured to generate an interaction graph among the participants.

20. The system of claim 1, further comprising a user interface that is configured for collaboratively zooming into one of the participants in real-time.

21. A method for performing social interaction analysis for a plurality of participants, comprising:

determining a similarity between a first spatially filtered output and each of a plurality of second spatially filtered outputs;

determining a social interaction between the participants based on the similarity between the first spatially filtered output and each of the second spatially filtered outputs; and

displaying an output representative of the social interaction between the participants;

wherein the first spatially filtered output is received from a fixed microphone array, and the second spatially filtered outputs are received from a plurality of steerable microphone arrays each corresponding to a different participant.

22. The method of claim 21, further comprising displaying the output in real-time as the participants are interacting with each other.

23. The method of claim 21, wherein the output comprises an interaction graph comprising:

a plurality of identifiers, each identifier corresponding to a respective participant; and

a plurality of indicators, each indicator providing information relating to at least one of;

a participant looking at another participant, a strength of an interaction between two participants, a participation level of a participant, or a leader of a group of participants.

24. The method of claim 23, wherein the strength of the interaction between two participants is based on a time that the two participants have interacted.

25. The method of claim 23, wherein the indicators have at least one of a direction, a thickness, or a color, wherein the direction indicates which participant is looking at another participant, the thickness indicates the strength of the interaction between two participants, and the color indicates the leader of the group of participants.

26. The method of claim 23, wherein each of the participants is a speaker.

27. The method of claim 23, further comprising using the interaction graph to assess group dynamics or topic dynamics.

28. The method of claim 23, wherein the interaction graph indicates social interaction information among the participants.

29. The method of claim 28, further comprising accumulating the social interaction information over a period of time.

30. The method of claim 23, further comprising displaying the interaction graph on a smartphone.

31. The method of claim 23, further comprising displaying the interaction graph on at least one from among the group comprising a handset, a laptop, a tablet, a computer, and a netbook.

32. The method of claim 23, wherein each indicator represents active participant location and energy.

33. The method of claim 23, further comprising an additional indicator that represents a refined active participant location and energy.

34. The method of claim 23, wherein the indicators comprise beam patterns.

35. The method of claim 21, further comprising performing real-time meeting analysis of a meeting the participants are participating in.

36. The method of claim 21, further comprising generating a personal time line for a participant that shows an interaction history of the participant with respect to other participants, a meeting topic, or a subject matter.

37. The method of claim 21, further comprising generating participant interaction statistics over time.

38. The method of claim 21, further comprising generating an evolution of interaction between participants over time.

39. The method of claim 21, further comprising generating an interaction graph among the participants.

40. The method of claim 21, further comprising collaboratively zooming into one of the participants in real-time.

41. An apparatus for performing social interaction analysis for a plurality of participants, comprising:

means for determining a similarity between a first spatially filtered output and each of a plurality of second spatially filtered outputs;

means for determining a social interaction between the participants based on the similarity between the first spatially filtered output and each of the second spatially filtered outputs; and

means for displaying an output representative of the social interaction between the participants;

wherein the first spatially filtered output is received from a fixed microphone array, and the second spatially filtered outputs are received from a plurality of steerable microphone arrays each corresponding to a different participant.

42. The apparatus of claim 41, further comprising means for displaying the output in real-time as the participants are interacting with each other.

43. The apparatus of claim 41, wherein the output comprises an interaction graph comprising:

a plurality of identifiers, each identifier corresponding to a respective participant; and

a plurality of indicators, each indicator providing information relating to at least one of;

a participant looking at another participant, a strength of an interaction between two participants, a participation level of a participant, or a leader of a group of participants.

44. The apparatus of claim 43, wherein the strength of the interaction between two participants is based on a time that the two participants have interacted.

45. The apparatus of claim 43, wherein the indicators have at least one of a direction, a thickness, or a color, wherein the direction indicates which participant is looking at another participant, the thickness indicates the strength of the interaction between two participants, and the color indicates the leader of the group of participants.

46. The apparatus of claim 43, wherein each of the participants is a speaker.

47. The apparatus of claim 43, further comprising means for using the interaction graph to assess group dynamics or topic dynamics.

48. The apparatus of claim 43, wherein the interaction graph indicates social interaction information among the participants.

49. The apparatus of claim 48, further comprising means for accumulating the social interaction information over a period of time.

50. The apparatus of claim 43, further comprising means for displaying the interaction graph on a smartphone.

51. The apparatus of claim 43, further comprising means for displaying the interaction graph on at least one from among the group comprising a handset, a laptop, a tablet, a computer, and a netbook.

52. The apparatus of claim 43, wherein each indicator represents active participant location and energy.

53. The apparatus of claim 52, further comprising an additional indicator that represents a refined active participant location and energy.

54. The apparatus of claim 52, wherein the indicators comprise beam patterns.

55. The apparatus of claim 41, further comprising means for performing real-time meeting analysis of a meeting the participants are participating in.

56. The apparatus of claim 41, further comprising means for generating a personal time line for a participant that shows an interaction history of the participant with respect to other participants, a meeting topic, or a subject matter.

57. The apparatus of claim 41, further comprising means for generating participant interaction statistics over time.

58. The apparatus of claim 41, further comprising means for generating an evolution of interaction between participants over time.

59. The apparatus of claim 41, further comprising means for generating an interaction graph among the participants.

60. The apparatus of claim 41, further comprising means for collaboratively zooming into one of the participants in real-time.

61. A non-transitory computer-readable medium comprising computer-readable instructions for causing a processor to:

determine a similarity between a first spatially filtered output and each of a plurality of second spatially filtered outputs;

determine a social interaction between a plurality of participants based on the similarity between the first spatially filtered output and each of the second spatially filtered outputs; and

display an output representative of the social interaction between the plurality of participants;

wherein the first spatially filtered output is received from a fixed microphone array, and the second spatially filtered outputs are received from a plurality of steerable microphone arrays each corresponding to a different participant.

62. The computer-readable medium of claim 61, further comprising instructions for causing the processor to display the output in real-time as the participants are interacting with each other.

63. The computer-readable medium of claim 61, wherein the output comprises an interaction graph comprising:

a plurality of identifiers, each identifier corresponding to a respective participant; and

a plurality of indicators, each indicator providing information relating to at least one of;

a participant looking at another participant, a strength of an interaction between two participants, a participation level of a participant, or a leader of a group of participants.

64. The computer-readable medium of claim 63, wherein the strength of the interaction between two participants is based on a time that the two participants have interacted.

65. The computer-readable medium of claim 63, wherein the indicators have at least one of a direction, a thickness, or a color, wherein the direction indicates which participant is looking at another participant, the thickness indicates the strength of the interaction between two participants, and the color indicates the leader of the group of participants.

66. The computer-readable medium of claim 63, wherein each of the participants is a speaker.

67. The computer-readable medium of claim 63, further comprising instructions for causing the processor to use the interaction graph to assess group dynamics or topic dynamics.

68. The computer-readable medium of claim 63, wherein the interaction graph indicates social interaction information among the participants.

69. The computer-readable medium of claim 68, further comprising instructions for causing the processor to accumulate the social interaction information over a period of time.

70. The computer-readable medium of claim 63, further comprising instructions for causing the processor to display the interaction graph on a smartphone.

71. The computer-readable medium of claim 63, further comprising instructions for causing the processor to display the interaction graph on at least one from among the group comprising a handset, a laptop, a tablet, a computer, and a netbook.

72. The computer-readable medium of claim 63, wherein each indicator represents active participant location and energy.

73. The computer-readable medium of claim 72, further comprising an additional indicator that represents a refined active participant location and energy.

74. The computer-readable medium of claim 72, wherein the indicators comprise beam patterns.

75. The computer-readable medium of claim 61, further comprising instructions for causing the processor to perform real-time meeting analysis of a meeting the participants are participating in.

76. The computer-readable medium of claim 61, further comprising instructions for causing the processor to generate a personal time line for a participant that shows an interaction history of the participant with respect to other participants, a meeting topic, or a subject matter.

77. The computer-readable medium of claim 61, further comprising instructions for causing the processor to generate participant interaction statistics over time.

78. The computer-readable medium of claim 61, further comprising instructions for causing the processor to generate an evolution of interaction between participants over time.

79. The computer-readable medium of claim 61, further comprising instructions for causing the processor to generate an interaction graph among the participants.

80. The computer-readable medium of claim 61, further comprising instructions for causing the processor to collaboratively zoom into one of the participants in real-time.