AUDIO USER INTERACTION RECOGNITION AND APPLICATION INTERFACE

• US 20130300648A1
• Filed: 11/12/2012
• Published: 11/14/2013
• Est. Priority Date: 05/11/2012
• Status: Active Grant

• Current Assignee: Qualcomm, Inc.
• Original Assignee: Qualcomm, Inc.
• Inventors: Visser, Erik, Kim, Lae-Hoon, Shin, Jongwon
• Granted Patent: US 9,746,916 B2
• Time in Patent Office: Days
• US Class Current: 345/156
• CPC Class Codes: G01S 3/8083 determining direction of source
• CPC Class Codes: G04G 21/00 Input or output devices integrated in time-pieces
• CPC Class Codes: G06F 1/1613 for portable computers cooling arrangements therefor G06F1/203; constructional details or arrangements for pocket calculators, electronic agendas or books G06F15/0216; constructional details of portable telephone sets: with several bodies H04M1/0202
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/013 Eye tracking input arrangements G06F3/015 takes precedence
• CPC Class Codes: G10L 17/00 Speaker identification or verification techniques
• CPC Class Codes: G10L 2021/02166 Microphone arrays; Beamforming
• CPC Class Codes: G10L 25/48 specially adapted for particular use
• CPC Class Codes: H04M 3/568 audio processing specific to telephonic conferencing, e.g. spatial distribution, mixing of participants echo suppression in two-way loud-speaking telephone systems H04M9/02; sound field processing per se H04S7/30
• CPC Class Codes: H04N 7/15 Conference systems
• CPC Class Codes: H04R 1/406 microphones
• CPC Class Codes: H04R 29/005 Microphone arrays
• CPC Class Codes: H04R 3/005 for combining the signals of two or more microphones specially adapted for hearing aids H04R25/407
• CPC Class Codes: H04S 7/304 For headphones

Abstract

Disclosed is an application interface that takes into account the user'"'"'s gaze direction relative to who is speaking in an interactive multi-participant environment where audio-based contextual information and/or visual-based semantic information is being presented. Among these various implementations, two different types of microphone array devices (MADs) may be used. The first type of MAD is a steerable microphone array (a.k.a. a steerable array) which is worn by a user in a known orientation with regard to the user'"'"'s eyes, and wherein multiple users may each wear a steerable array. The second type of MAD is a fixed-location microphone array (a.k.a. a fixed array) which is placed in the same acoustic space as the users (one or more of which are using steerable arrays).

Claims

1. A method for displaying information, the method comprising:

detecting a gaze direction line-of-sight for a user based on at least an audio detection component from a fixed array and at one steerable array;

determining if a display device is in the line-of-sight of the user;

if the display device is in the line-of-sight of the user, then displaying a first image on the display device; and

if the display device is not in the line-of-sight of the user, then displaying a second image on the display.

2. The method of claim 1, wherein the display device is a personal display device that is movable into the line-of-sight of the user.

3. The method of claim 2, wherein the first image specifically pertains to that which the user is looking at based on the gaze direction.

4. The method of claim 1, wherein the second image is one of a blank display or a non-specific display.

5. The method of claim 1, wherein the line-of-sight determination comprises both a horizontal component corresponding to a target and a vertical component corresponding to whether the device is being held up to be viewed, wherein both components must be true in order for the determination to be true.

6. A method for displaying information, the method comprising:

detecting a collective gaze direction target for a plurality of participants based on a plurality of audio detection components from a fixed array and at least one steerable array;

determining if the target is a display device; and

if the target is a display device, then activating the display device to complete a display presentation during which the presentation is paused for periods during which the display device is not the target and resumed when the display device is again the target, and deactivating the display device once the display presentation is completed.

7. The method of claim 6, wherein the display presentation includes graphics to accompany an audio presentation by a first participant from among the plurality of participants.

8. The method of claim 6, wherein the display presentation is an audio/visual presentation, wherein the display presentation is also paused whenever an active speaker is detected, and wherein the display presentation is also muted whenever an active speaker is detected.

9. A method for displaying information on a wearable watch-type display device, the method comprising:

detecting an ultrasound emission from the device in order to determine when the device is in a range; and

sending a signal back to the device.

10. The method of claim 9, wherein the detecting and the sending are performed by a steerable array.

11. The method of claim 9, further comprising:

displaying a first image on the device when the device is in a range; and

displaying a second image on the device when the device is not in a range.

12. The method of claim 11, wherein the first image is determined by a gaze direction of a user, and wherein the determination is made utilizing a steerable array and a fixed array.

13. The method of claim 11, wherein the first image is a lighted display if the device determines a low-lighting environment.

14. The method of claim 11, wherein display of the second image corresponds to a low power state to save energy.

15. The method of claim 9, further comprising determining an angle offset from a user.

16. A method for displaying information on a wearable watch-type display device, the method comprising:

detecting when the wearable watch-type display device is placed in line-of-sight with a wearable array;

receiving a first image from a mobile device; and

reorienting and redisplaying the first image as a second image on the wearable watch-type display device such that the second image is aligned with the first image on the mobile device.

17. A method for displaying information, the method comprising:

detecting, via a steerable array, when a display device is in a range; and

receiving, at the display device, a signal back from the steerable array.

18. The method of claim 17, wherein the range corresponds to a line-of-sight of a user.

19. The method of claim 17, further comprising:

displaying a first image on the display when the steerable array is in the range; and

displaying a second image on the display when the steerable array is not in the range.

20. The method of claim 17, further comprising detecting, via a microphone, an ultrasound transmission from the steerable array when in a line-of-sight of the user.

21. A system for displaying information, the system comprising a processor configured to:

detect a gaze direction line-of-sight for a user based on at least an audio detection component from a fixed array and at one steerable array;

determine if a display device is in the line-of-sight of the user;

if the display device is in the line-of-sight of the user, then display a first image on the display device; and

if the display device is not in the line-of-sight of the user, then display a second image on the display device.

22. The system of claim 21, wherein the display device is a personal display device that is movable into the line-of-sight of the user.

23. The system of claim 22, wherein the first image specifically pertains to an item the user is looking at based on the gaze direction.

24. The system of claim 21, wherein the second image is one of a blank display or a non-specific display.

25. The system of claim 21, wherein the line-of-sight determination comprises both a horizontal component corresponding to a target and a vertical component corresponding to whether the device is being held up to be viewed, wherein both components must be true in order for the determination to be true.

26. A system for displaying information, the system comprising a processor configured to:

detect a collective gaze direction target for a plurality of participants based on a plurality of audio detection components from a fixed array and at least one steerable array;

determine if the target is a display device; and

if the target is a display device, then activate the display device to complete a display presentation during which the presentation is paused for periods during which the display device is not the target and resumed when the display device is again the target, and deactivate the display device once the display presentation is completed.

27. The system of claim 26, wherein the display presentation comprises graphics to accompany an audio presentation by a first participant from among the plurality of participants.

28. The system of claim 26, wherein the display presentation is an audio/visual presentation, wherein the display presentation is also paused whenever an active speaker is detected, and wherein the display presentation is also muted whenever an active speaker is detected.

29. A system for displaying information on a wearable watch-type display device, the system comprising a processor configured to:

detect an ultrasound emission from the device in order to determine when the device is in a range; and

send a signal back to the device.

30. The system of claim 29, wherein the detection of the ultrasound emission and the sending of the signal are performed by a steerable array.

31. The system of claim 29, wherein the processor is further configured to:

display a first image on the device when the device is in the range; and

display a second image on the device when the device is not in the range.

32. The system of claim 31, wherein the first image is determined by a gaze direction of a user, and wherein the determination is made utilizing a steerable array and a fixed array.

33. The system of claim 31, wherein the first image is a lighted display if the device determines a low-lighting environment.

34. The system of claim 31, wherein the second image corresponds to a low power state to save energy.

35. The system of claim 29, wherein the processor is further configured to determine an angle offset from a user.

36. A system for displaying information on a wearable watch-type display device, the system comprising a processor configured to:

detect when the wearable watch-type display device is placed in line-of-sight with a wearable array;

receive a first image from a mobile device; and

reorient and redisplay the first image as a second image on the wearable watch-type display device such that the second image is aligned with the first image on the mobile device.

37. A system for displaying information, the system comprising a processor configured to:

detect, via a steerable array, when a display device is in a range; and

receive a signal back from the steerable array.

38. The system of claim 37, wherein the detection is performed by an ultrasonic detector that is part of the steerable array.

39. The system of claim 37, wherein the receiving is performed by a fixed array coupled to the processor.

40. The system of claim 37, wherein the range corresponds to a line-of-sight of a user.

41. The system of claim 37, wherein the processor is further configured to display a first image on the display when the steerable array is in the range.

42. The system of claim 37, wherein the processor is further configured to display a second image on the display when the steerable array is not in the range.

43. The system of claim 37, wherein the processor is further configured to detect, via a microphone, an ultrasound transmission from the steerable array when in line-of-sight of a user.

44. An apparatus for displaying information, the apparatus comprising:

means for detecting a gaze direction line-of-sight for a user based on at least an audio detection component from a fixed array and at one steerable array;

means for determining if a display device is in the line-of-sight of the user;

means for displaying a first image on the display device if the display device is in the line-of-sight of the user; and

means for displaying a second image if the display device is not in the line-of-sight of the user.

45. The apparatus of claim 44, further comprising means for determining whether the display device is a personal display device that is movable into the line-of-sight of the user.

46. The apparatus of claim 45, further comprising means for determining whether the first image specifically pertains to an item the user is looking at based on the gaze direction.

47. The apparatus of claim 44, further comprising means for determining whether the second image should be one of a blank display or a non-specific display.

48. The apparatus of claim 44, further comprising:

means for determining whether the line-of-sight determination comprises both a horizontal component corresponding to a target and a vertical component corresponding to whether the device is being held up to be viewed; and

means for determining whether both components must be true in order for the determination to be true.

49. An apparatus for displaying information, the apparatus comprising:

means for detecting a collective gaze direction target for a plurality of participants based on a plurality of audio detection components from a fixed array and at least one steerable array; and

means for displaying an image.

50. The apparatus of claim 49, further comprising means for determining if the target is a display device.

51. The apparatus of claim 50, further comprising means for activating the display device to complete a display presentation during which the presentation is paused for periods during which the display device is not the target and resumed when the display device is again the target.

52. The apparatus of claim 51, further comprising means for deactivating the display device once the display presentation is completed.

53. The apparatus of claim 51, further comprising means for determining whether the display presentation comprises graphics to accompany an audio presentation by a first participant from among the plurality of participants.

54. The apparatus of claim 51, further comprising means for determining whether the display presentation is an audio/visual presentation, whether the display presentation should be paused whenever an active speaker is detected, and whether the display presentation should be muted whenever an active speaker is detected.

55. A apparatus for displaying information on a wearable display device, the apparatus comprising:

means for detecting an ultrasound emission from the device in order to determine when the device is in a range; and

means for sending a signal back to the device.

56. The apparatus of claim 55, wherein the wearable display device is a watch-type device.

57. The apparatus of claim 55, wherein the wearable display device is one of a bracelet-type device, necklace-type device, ring-type device, or eyeglass-type device.

58. The apparatus of claim 55, further comprising a steerable array that comprises the means for detecting and the means for sending.

59. The apparatus of claim 55, further comprising:

means for displaying a first image on the display device when the device is in a range; and

means for displaying a second image on the display device when the device is not in the range.

60. The apparatus of claim 59, wherein the first image is determined by a gaze direction of a user, and wherein the determination is made utilizing a steerable array and a fixed array.

61. The apparatus of claim 59, further comprising means for detecting a low-lighting environment, and wherein the first image is a lighted display upon detection of determines the low-lighting environment.

62. The apparatus of claim 59, further comprising means for determining whether the second image corresponds to a low power state to save energy.

63. The apparatus of claim 55, further comprising means for determining an angle offset from a user.

64. An apparatus for displaying information on a wearable watch-type display device, the apparatus comprising:

means for detecting when the wearable watch-type display device is placed in line-of-sight with a wearable array;

means for receiving a first image from a mobile device; and

means for reorienting and redisplaying the first image as a second image on the wearable watch-type display device such that the second image is aligned with the first image on the mobile device.

65. An apparatus for displaying information, the apparatus comprising:

means for detecting, via a steerable array, when a display device is in a range; and

means for receiving a signal back from the steerable array.

66. The apparatus of claim 65, wherein the range corresponds to a line-of-sight of a user.

67. The apparatus of claim 65, further comprising:

means for displaying a first image on the display device when the steerable array is in the range; and

means for displaying a second image on the display device when the steerable array is not in the range.

68. The apparatus of claim 65, further comprising means for detecting, via a microphone, an ultrasound transmission from the steerable array when in line-of-sight of a user.

69. A wearable watch-type device for displaying information, the wearable watch-type device comprising:

a steerable array, configured to be in communication with a different device comprising a fixed array, for detecting a gaze direction line-of-sight for a user based on at least an audio detection component from a fixed array and at one steerable array;

a proximity detector for determining if a display device is in the line-of-sight of the user; and

a display for displaying a first image on the display device if the display device is in the line-of-sight of the user, and for displaying a second image if the display device is not in the line-of-sight of the user.

70. The wearable watch-type device of claim 69, wherein the display device is a personal display device that is movable into the line-of-sight of the user.

71. The wearable watch-type device of claim 70, wherein the first image specifically pertains to an item the user is looking at based on the gaze direction.

72. The wearable watch-type device of claim 69, wherein the second image should be one of a blank display or a non-specific display.

73. The wearable watch-type device of claim 69, further comprising:

a processor for determining whether the line-of-sight determination comprises both a horizontal component corresponding to a target and a vertical component corresponding to whether the device is being held up to be viewed, wherein both components must be true in order for the determination to be true.

74. A wearable watch-type device comprising a receiver for receiving information pertaining to a system, the system including logic for:

detecting a collective gaze direction target for a plurality of participants based on a plurality of audio detection components from a fixed array and at least one steerable array;

determining if the target is a display device; and

if the target is a display device, then activating the display device to complete a display presentation during which the presentation is paused for periods during which the display device is not the target and resumed when the display device is again the target, and deactivating the display device once the display presentation is completed.

75. The wearable watch-type device of claim 74, wherein the display presentation includes graphics to accompany an audio presentation by a first participant from among the plurality of participants.

76. The wearable watch-type device of claim 74, wherein the display presentation is an audio/visual presentation, wherein the display presentation is also paused whenever an active speaker is detected, and wherein the display presentation is also muted whenever an active speaker is detected.

77. A wearable watch-type device comprising:

a display capable of providing an output to a user;

an ultrasound emitter capable of enabling a detector to detect when the device is in a range; and

a receiver for receiving a signal back from the detector.

78. The wearable watch-type device of claim 77, further comprising a steerable array comprising the detector.

79. The wearable watch-type device of claim 77, further comprising a display processor capable of displaying a first image on the display when the device is in the range and displaying a second image on the display when the device is not in the range.

80. The wearable watch-type device of claim 79, wherein the first image is determined by a gaze direction of a user, and wherein the determination is made utilizing a steerable array and a fixed array.

81. The wearable watch-type device of claim 79, wherein the first image is a lighted display if the device determines a low-lighting environment.

82. The wearable watch-type device of claim 79, wherein the second image corresponds to a low power state to save energy.

83. The wearable watch-type device of claim 77, further comprising a microphone capable of enabling the device to determine an angle offset from the user.

84. A wearable watch-type display device comprising:

means for detecting when the wearable watch-type display device is placed in line-of-sight with a wearable array;

means for receiving a first image from a mobile device; and

means for reorienting and redisplaying the first image as a second image on the wearable watch-type display device such that the second image is aligned with the first image on the mobile device.

85. A wearable watch-type device for displaying information, the wearable watch-type device comprising:

a display;

an ultrasound emitter capable of enabling a steerable array to detect when the device is in a range; and

a receiver for receiving a signal back from the steerable array.

86. The wearable watch-type device of claim 85, wherein the range corresponds to a line-of-sight of a user.

87. The wearable watch-type device of claim 85, further comprising a display processor capable of displaying a first image on the display when the steerable array is in the range and displaying a second image on the display when the steerable array is not in the range.

88. The wearable watch-type device of claim 85, further comprising a microphone capable of detecting an ultrasound transmission from the steerable array when in line-of-sight of a user.

89. A non-transitory computer-readable medium comprising computer-readable instructions for causing a processor of a content/context system to:

detect a gaze direction line-of-sight for a user based on at least an audio detection component from a fixed array and at one steerable array;

determine if a display device is in the line-of-sight of the user;

if the display device is in the line-of-sight of the user, then display a first image on the display device; and

if the display device is not in the line-of-sight of the user, then display a second image.

90. The computer readable medium of claim 89, further comprising instructions for causing the processor to determine whether the display device is a personal display device that is movable into the line-of-sight of the user.

91. The computer readable medium of claim 90, further comprising instructions for causing the processor to display the first image based on the gaze direction.

92. The computer readable medium of claim 89, further comprising instructions for causing the processor to display a blank display or a non-specific display based on the gaze direction.

93. The computer readable medium of claim 89, further comprising instructions for causing the processor to determine the line-of-sight using both a horizontal component corresponding to a target and a vertical component corresponding to whether the device is being held up to be viewed, wherein both components must be true in order for the determination to be true.

94. A non-transitory computer-readable medium comprising computer-readable instructions for causing a processor of a content/context system to:

detect a collective gaze direction target for a plurality of participants based on a plurality of audio detection components from a fixed array and at least one steerable array;

determine if the target is a display device; and

if the target is a display device, then activate the display device to complete a display presentation during which the presentation is paused for periods during which the display device is not the target and resumed when the display device is again the target, and deactivate the display device once the display presentation is completed.

95. The computer readable medium of claim 94, further comprising instructions for causing the processor to display graphics to accompany an audio presentation by a first participant from among the plurality of participants.

96. The computer readable medium of claim 94, wherein the display presentation is an audio/visual presentation, wherein the display presentation is also paused whenever an active speaker is detected, and wherein the display presentation is also muted whenever an active speaker is detected.

97. A non-transitory computer-readable medium comprising computer-readable instructions for causing a processor of a content/context system to:

detect an ultrasound emission to determine when a wearable watch-type display device is in a range; and

send a signal to the wearable watch-type display device.

98. The computer readable medium of claim 97, wherein the detecting and the sending are performed by a steerable array.

99. The computer readable medium of claim 97, further comprising instructions for causing the processor to:

display a first image on a display when the device is in a range; and

display a second image on the display when the device is not in the range.

100. The computer readable medium of claim 99, further comprising instructions for causing the processor to determine a gaze direction of a user utilizing a steerable array and a fixed array.

101. The computer readable medium of claim 99, further comprising instructions for causing the processor to:

determine a low-lighting environment; and

lighting the first image.

102. The computer readable medium of claim 99, further comprising instructions for causing the processor to send the second image to the display while the device is in a low power state to save energy.

103. The computer readable medium of claim 97, further comprising instructions for causing the processor to determine an angle offset from a user.

104. A non-transitory computer-readable medium comprising computer-readable instructions for causing a processor of a content/context system to:

detect when the wearable watch-type display device is placed in line-of-sight with a wearable array;

receive a first image from a mobile device; and

reorient and redisplay the first image as a second image on the wearable watch-type display device such that the second image is aligned with the first image on the mobile device.

105. A non-transitory computer-readable medium comprising computer-readable instructions for causing a processor of a content/context system to:

detect, via a steerable array, when a display device is in a range; and

receive a signal back from the steerable array.

106. The computer readable medium of claim 105, wherein the range corresponds to a line-of-sight of a user.

107. The computer readable medium of claim 105, further comprising instructions for causing the processor to:

display a first image on a display when the steerable array is in the range; and

display a second image on the display when the steerable array is not in the range.

108. The computer readable medium of claim 105, further comprising instructions for causing the processor to detect, via a microphone, an ultrasound transmission from the steerable array when in line-of-sight of the user.

109. A system for displaying information, the system comprising a processor configured to:

detect a gaze direction line-of-sight for a user based on at least an audio detection component from a fixed array and at one steerable array;

determine if a display device is in the line-of-sight of the user;

if the display device is in the line-of-sight of the user, then display a first image on the display device; and

if the display device is not in the line-of-sight of the user, then display a second image on the display device.

110. The system of claim 109, wherein the display device is a personal display device that is movable into the line-of-sight of the user.

111. The system of claim 110, wherein the first image specifically pertains to an item the user is looking at based on the gaze direction.

112. The system of claim 109, wherein the second image is one of a blank display or a non-specific display.

113. The system of claim 109, wherein the line-of-sight determination comprises both a horizontal component corresponding to a target and a vertical component corresponding to whether the device is being held up to be viewed, wherein both components must be true in order for the determination to be true.

114. A system for displaying information, the system comprising a processor configured to:

detect a collective gaze direction target for a plurality of participants based on a plurality of audio detection components from a fixed array and at least one steerable array;

determine if the target is a display device; and

if the target is a display device, then activate the display device to complete a display presentation during which the presentation is paused for periods during which the display device is not the target and resumed when the display device is again the target, and deactivate the display device once the display presentation is completed.

115. The system of claim 114, wherein the display presentation comprises graphics to accompany an audio presentation by a first participant from among the plurality of participants.

116. The system of claim 114, wherein the display presentation is an audio/visual presentation, wherein the display presentation is also paused whenever an active speaker is detected, and wherein the display presentation is also muted whenever an active speaker is detected.

117. A system for displaying information on a wearable watch-type display device, the system comprising a processor configured to:

detect an ultrasound emission from the device in order to determine when the device is in a range; and

send a signal back to the device.

118. The system of claim 117, wherein the detection and the sending are performed by a steerable array.

119. The system of claim 117, wherein the processor is further configured to:

display a first image on the device when the device is in the range; and

display a second image on the device when the device is not in the range.

120. The system of claim 119, wherein the first image is determined by a gaze direction of a user, and wherein the determination is made utilizing a steerable array and a fixed array.

121. The system of claim 119, wherein the first image is a lighted display if the device determines a low-lighting environment.

122. The system of claim 119, wherein the display of the second image corresponds to a low power state to save energy.

123. The system of claim 117, wherein the processor is further configured to determine an angle offset from the user.

124. A system for displaying information on a wearable watch-type display device, the system comprising a processor configured to:

detect when the wearable watch-type display device is placed in line-of-sight with a wearable array;

receive a first image from a mobile device; and

reorient and redisplay the first image as a second image on the wearable watch-type display device such that the second image is aligned with the first image on the mobile device.

125. A system for displaying information, the system comprising a processor configured to:

detect, via a steerable array, when the display device is in a range; and

receive a signal back from the steerable array.

126. The system of claim 125, wherein the range corresponds to a line-of-sight of a user.

127. The system of claim 125, wherein the processor is further configured to:

display a first image on the display when the steerable array is in the range; and

display a second image on the display when the steerable array is not in the range.

128. The system of claim 125, wherein the processor is further configured to detect, via a microphone, an ultrasound transmission from the steerable array when in line-of-sight of the user.