Audio user interaction recognition and application interface

• US 10,073,521 B2
• Filed: 07/10/2017
• Issued: 09/11/2018
• Est. Priority Date: 05/11/2012
• Status: Active Grant

• Current Assignee: Qualcomm, Inc.
• Original Assignee: Qualcomm, Inc.
• Inventors: Kim, Lae-Hoon, Shin, Jongwon, Visser, Erik
• Primary Examiner(s): Tzeng, Fred
• Application Number:
  US15/645,365
• Publication Number:
  US 20170308164A1
• Time in Patent Office: 428 Days
• Field of Search: 345156
• CPC Class Codes: G01S 3/8083 determining direction of source
• CPC Class Codes: G04G 21/00 Input or output devices integrated in time-pieces
• CPC Class Codes: G06F 1/1613 for portable computers cooling arrangements therefor G06F1/203; constructional details or arrangements for pocket calculators, electronic agendas or books G06F15/0216; constructional details of portable telephone sets: with several bodies H04M1/0202
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/013 Eye tracking input arrangements G06F3/015 takes precedence
• CPC Class Codes: G10L 17/00 Speaker identification or verification techniques
• CPC Class Codes: G10L 2021/02166 Microphone arrays; Beamforming
• CPC Class Codes: G10L 25/48 specially adapted for particular use
• CPC Class Codes: H04M 3/568 audio processing specific to telephonic conferencing, e.g. spatial distribution, mixing of participants echo suppression in two-way loud-speaking telephone systems H04M9/02; sound field processing per se H04S7/30
• CPC Class Codes: H04N 7/15 Conference systems
• CPC Class Codes: H04R 1/406 microphones
• CPC Class Codes: H04R 29/005 Microphone arrays
• CPC Class Codes: H04R 3/005 for combining the signals of two or more microphones specially adapted for hearing aids H04R25/407
• CPC Class Codes: H04S 7/304 For headphones

Abstract

Disclosed is an application interface that takes into account the user'"'"'s gaze direction relative to who is speaking in an interactive multi-participant environment where audio-based contextual information and/or visual-based semantic information is being presented. Among these various implementations, two different types of microphone array devices (MADs) may be used. The first type of MAD is a steerable microphone array (a.k.a. a steerable array) which is worn by a user in a known orientation with regard to the user'"'"'s eyes, and wherein multiple users may each wear a steerable array. The second type of MAD is a fixed-location microphone array (a.k.a. a fixed array) which is placed in the same acoustic space as the users (one or more of which are using steerable arrays).

Claims

1. An apparatus comprising:

a display of a wearable watch-type device configured to display a visual output;

an interface of the wearable watch-type device configured to receive a first image from a mobile device; and

a processor of the wearable watch-type device configured to;

determine whether the wearable watch-type device is in a gaze direction of a user; and

in response to determining that the wearable watch-type display device is in the gaze direction of the user, initiate presentation of the first image at the display, wherein the first image indicates a social interaction among a plurality of participants.

2. The apparatus of claim 1, wherein the display is further configured to present a second image, wherein the first image is displayed concurrently with the second image, wherein the second image comprises a date and a time, and wherein the gaze direction corresponds to a look direction of the user.

3. The apparatus of claim 1, wherein the social interaction includes looking toward a speaker of the plurality of participants.

4. The apparatus of claim 3, wherein the first image indicates identities of the plurality of participants and indicates an interaction measurement corresponding to, for the social interaction, amounts of interaction between a particular participant and each of the other participants of the plurality of participants.

5. The apparatus of claim 1, wherein the first image indicates multiple interaction measurements associated with each participant of the plurality of participants.

6. The apparatus of claim 1, wherein the first image indicates a plurality of topics corresponding to the social interaction, and wherein the first image indicates a participation amount of a particular participant.

7. The apparatus of claim 1, wherein the social interaction comprises a plurality of meetings.

8. A method comprising:

receiving, at a wearable watch-type device, a first image from a mobile device;

determining whether the wearable watch-type device is in a gaze direction of a user; and

in response to determining that the wearable watch-type device is in the gaze direction of the user, presenting the first image, at a display of the wearable watch-type device, wherein the first image indicates a social interaction among a plurality of participants.

9. The method of claim 8, wherein the social interaction includes looking toward a speaker, and wherein the first image further indicates a percentage of the plurality of participants other than a particular participant of the plurality of participants that are looking at the particular participant during a timeframe associated with the social interaction.

10. The method of claim 8, wherein the first image indicates a plurality of topics discussed during the social interaction.

11. The method of claim 8, wherein the first image indicates a percentage of time that each participant of the plurality of participants expand discussing each topic of a plurality of topics discussed during the social interaction.

12. The method of claim 11, wherein the first image comprises an interaction graph, and wherein the interaction graph includes a plurality of indicators, each indicator identifying information relating to at least one of:

a participant of the plurality of participants looking at another participant of the plurality of participants, a participation level of the participant, or a leader of a group of participants of the plurality of participants.

13. The method of claim 8, wherein the first image indicates, for the social interaction, amounts of interaction between a particular participant and each of the other participants of the plurality of participants, and wherein an interaction measurement is selected from a range of interaction values.

14. The method of claim 8, wherein one or more social interactions include looking toward a speaker.

15. The method of claim 8, wherein the first image indicates a social interaction summary accumulated throughout a period of time corresponding to the social interaction for a particular participant of the plurality of participants.

16. The method of claim 15, wherein the social interaction summary includes:

a participant column listing the plurality of participants other than the particular participant; and

a specific month column of a plurality of monthly columns, wherein each row of a plurality of rows in the specific month column indicates an interaction value corresponding to the social interaction between the particular participant and each participant listed in the participant column.

17. The method of claim 8, wherein the first image indicates a personal timeline for a first participant of the plurality of participants, and wherein the personal timeline includes social interaction data for each of a plurality of points in time, the social interaction data based on the social interaction.

18. The method of claim 8, further comprising generating a graphical user interface (GUI) at the wearable watch-type device, wherein the GUI includes the first image.

19. The method of claim 18, wherein the GUI includes a second image presented concurrently with the first image, and wherein the second image comprises a date and a time.

20. The method of claim 8, wherein the first image indicates an identifier associated with a particular participant of the plurality of participants and data indicating a look direction of the particular participant.

21. An apparatus comprising:

means for displaying a visual output at a wearable watch-type device;

means for receiving a first image at the wearable watch-type device from a mobile device; and

means for initiating presentation of the first image at the means for displaying the visual output in response to a determination that the wearable watch-type device is in a gaze direction of a user, wherein the first image indicates a social interaction among a plurality of participants.

22. The apparatus of claim 21, wherein the first image indicates a social interaction summary, and wherein the social interaction summary includes a participant column listing the plurality of participants other than a particular participant of the plurality of participants.

23. The apparatus of claim 22, wherein the social interaction summary further includes a specific month column of a plurality of monthly columns, wherein each row of a plurality of rows in the specific month column indicates an interaction value corresponding to the social interaction between the particular participant and each participant listed in the participant column.

24. The apparatus of claim 21, wherein the means for initiating the presentation of the first image is configured to generate a personal timeline for a first participant of the plurality of participants, and wherein the personal timeline includes social interaction data for each of a plurality of points in time, the social interaction data based on the social interaction.

25. The apparatus of claim 21, wherein the social interaction includes a look direction of a non-speaking participant, and wherein the gaze direction is determined based on an audio signal.

26. A computer-readable storage device storing instructions that, when executed, cause a processor to perform operations comprising

receiving, at a wearable watch-type device, a first image from a mobile device;

determining whether the wearable watch-type device is in a gaze direction of a user; and

in response to determining that the wearable watch-type device is in the gaze direction of the user, presenting the first image, at a display of the wearable watch-type device, wherein the first image indicates a social interaction among a plurality of participants.

27. The computer-readable storage device of claim 26, wherein the first image comprises an interaction graph, wherein a strength of an interaction between two participants of the plurality of participants is based on a duration of time that the two participants have interacted, and wherein the interaction graph indicates the strength of the interaction.

28. The computer-readable storage device of claim 27, wherein the interaction graph further includes a plurality of indicators, and wherein each indicator has a particular dimension corresponding to a strength of interaction between two participants of the plurality of participants.

29. The computer-readable storage device of claim 26, wherein the social interaction corresponds to a meeting that includes the plurality of participants.

30. The computer-readable storage device of claim 26, wherein the first image indicates participant interaction statistics during the social interaction.