Personalized, real-time audio processing

• US 10,304,476 B2
• Filed: 02/27/2018
• Issued: 05/28/2019
• Est. Priority Date: 02/02/2016
• Status: Active Grant

• Current Assignee: eBay Inc.
• Original Assignee: eBay Inc.
• Inventors: Glasgow, Dane, Ramadge, David, MacLaurin, Matthew Bret, Van Winkle, Justin, Sherman, Corinne Elizabeth
• Primary Examiner(s): Islam, Mohammad K
• Application Number:
  US15/906,836
• Publication Number:
  US 20180190309A1
• Time in Patent Office: 455 Days
• Field of Search: 381 1408, 381313, 381 1416, 348169
• CPC Class Codes: G02B 27/0093 with means for monitoring data relating to the user, e.g. head-tracking, eye-tracking
• CPC Class Codes: G02B 27/022 Viewing apparatus G02B27/04, G02B27/06, G02B27/08 take precedence
• CPC Class Codes: G06F 3/013 Eye tracking input arrangements G06F3/015 takes precedence
• CPC Class Codes: G06F 3/16 Sound input; Sound output speech processing G10L
• CPC Class Codes: G06V 40/19 Sensors therefor
• CPC Class Codes: G10L 21/028 using properties of sound source
• CPC Class Codes: G10L 21/0364 for improving intelligibility
• CPC Class Codes: H04R 1/406 microphones
• CPC Class Codes: H04R 2201/403 Linear arrays of transducers
• CPC Class Codes: H04R 29/005 Microphone arrays
• CPC Class Codes: H04R 3/005 for combining the signals of two or more microphones specially adapted for hearing aids H04R25/407

Abstract

An apparatus and method for real-time audio processing employs a gaze detection sensor to detect a direction of a user'"'"'s gaze and output a gaze signal corresponding to the detected direction of the user'"'"'s gaze. A digital signal processing unit responds to a plurality of signals corresponding to a plurality of sounds received at the apparatus, and the determined direction of gaze to identify a signal of interest from the plurality of signals using the gaze signal. The signal of interest is processed for output to the user. In embodiments, a microphone array provides the plurality of signals. An imaging sensor may work with either the microphone array or the gaze detection sensor to identify the signal of interest.

Claims

1. An apparatus, comprising:

a sensor configured to detect a first direction of a gaze of a user and output a first gaze signal corresponding to the detected first direction of the gaze of the user;

a digital signal processing unit configured to;

receive the first gaze signal from the sensor;

read the first gaze signal to determine the first detected direction of the gaze of the user;

determine a second gaze signal using a motion input signal, the second gaze signal corresponding to a second direction of the gaze of the user, the second direction of the gaze of the user being different than the first detected direction of the gaze of the user;

select the second direction of the gaze of the user over the first detected direction of the gaze of the user;

responsive to determining the detected second direction of the gaze of the user, identify, in an electronic data storage, one or more available actions that correspond to the detected second direction of the gaze of the user;

select a first action from the one or more available actions; and

execute a set of computer-readable instructions that correspond to the first action.

2. The apparatus of claim 1, further comprising a microphone positioned relative to a position of the sensor to provide information indicating directionality of one or more sounds, wherein the first action is selected based on the detected second direction of the gaze of the user and the one or more sounds.

3. The apparatus of claim 2, wherein the digital signal processing unit is further configured to determine that a person is speaking in a direction of the user based on one or more of the second gaze signal and the one or more sounds, wherein the first action is selected based at least in part on a determination that the person is speaking in the direction of the user.

4. The apparatus of claim 1, further comprising an imaging sensor configured to detect an image, wherein the first action is selected based at least in part on the image.

5. The apparatus of claim 1, wherein the first action includes outputting an audio signal via a speaker.

6. The apparatus of claim 5, wherein the first action further includes adjusting the audio signal by emphasizing a first portion of the audio signal, deemphasizing a second portion of the audio signal, adding audio to the audio signal, or removing audio from the audio signal.

7. The apparatus of claim 1, wherein the first action includes enabling a feature or disabling a feature for an application during runtime of the application.

8. The apparatus of claim 1, wherein the first action includes adjusting playback of a media item by starting, stopping, pausing, rewinding, or advancing playback of the media item.

9. The apparatus of claim 1, further comprising an override switch to override or adjust the second gaze signal.

10. The apparatus of claim 9, wherein, in response to a signal from the override switch, the digital signal processing unit ignores input from the sensor for a finite period of time.

11. The apparatus of claim 1, wherein the digital signal processing unit is further configured to receive an identification signal from a sound source, the identification signal including metadata that identifies either the sound source itself, or a person associated with the sound source.

12. A method, comprising:

receiving, from a sensor, a first gaze signal corresponding to a detected first direction of a gaze of a user;

reading the first gaze signal to determine the detected first direction of the gaze of the user;

determining a second gaze signal using a motion input signal, the second gaze signal corresponding to a second direction of the gaze of the user;

selecting the second direction of the gaze of the user over the first detected direction of the gaze of the user;

responsive to determining the detected second direction of the gaze of the user, identifying, in an electronic data storage, one or more available actions that correspond to the detected second direction of the gaze of the user;

selecting a first action from the one or more available actions; and

executing a set of computer-readable instructions that correspond to the first action.

13. The method of claim 12, wherein the first action is selected based on the detected second direction of the gaze of the user and one or more sounds detected by a microphone.

14. The method of claim 13, further comprising determining that a person is speaking in a direction of the user based on one or more of the second gaze signal and the one or more sounds, wherein the first action is selected based at least in part on a determination that the person is speaking in the direction of the user.

15. The method of claim 12, wherein the first action further includes adjusting an output of an audio signal by emphasizing a first portion of the audio signal, deemphasizing a second portion of the audio signal, adding audio to the audio signal, or removing audio from the audio signal.

16. The method of claim 12, wherein the first action includes enabling a feature or disabling a feature for an application during runtime of the application.

17. A non-transitory machine-readable medium having stored thereon machine-readable instructions executable to cause a machine to perform operations comprising:

receive, from a sensor, a first gaze signal corresponding to a detected first direction of a gaze of a user;

read the first gaze signal to determine the detected first direction of the gaze of the user;

determine a second gaze signal using a motion input signal, the second gaze signal corresponding to a second direction of the gaze of the user;

select the second direction of the gaze of the user over the first detected direction of the gaze of the user;

responsive to determining the detected second direction of the gaze of the user, identify, in an electronic data storage, one or more available actions that correspond to the detected second direction of the gaze of the user;

select a first action from the one or more available actions; and

execute a set of computer-readable instructions that correspond to the first action.

18. The non-transitory machine-readable medium of claim 17, wherein the operations further comprise determining that a person is speaking in a direction of the user based on one or more of the second gaze signal and one or more sounds, wherein the first action is selected based at least in part on a determination that the person is speaking in the direction of the user.

19. The non-transitory machine-readable medium of claim 17, wherein the first action further includes adjusting an output of an audio signal by emphasizing a first portion of the audio signal, deemphasizing a second portion of the audio signal, adding audio to the audio signal, or removing audio from the audio signal.

20. The non-transitory machine-readable medium of claim 17, wherein the first action includes enabling a feature or disabling a feature for an application during runtime of the application.