Amplifying audio-visual data based on user's head orientation

• US 9,554,229 B2
• Filed: 01/13/2012
• Issued: 01/24/2017
• Est. Priority Date: 10/31/2011
• Status: Active Grant

• Current Assignee: Sony Corporation (Sony Group Corp.)
• Original Assignee: Sony Corporation (Sony Group Corp.), Sony Mobile Communications Incorporated (Sony Group Corp.)
• Inventors: Ek, Martin, Landqvist, Per Magnus, Aronsson, Par-Anders, Thorn, Ola, Jendbro, Gert Magnus, Stenberg, Par
• Primary Examiner(s): FISCHER, MARK L
• Application Number:
  US13/823,879
• Publication Number:
  US 20140010391A1
• Time in Patent Office: 1,838 Days
• Field of Search: None
• US Class Current: 1/1
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/012 Head tracking input arrangements
• CPC Class Codes: G06F 3/165 Management of the audio stream, e.g. setting of volume, audio stream path
• CPC Class Codes: H04S 7/304 For headphones

Abstract

One or more devices may receive, from a media presentation device worn by a user, sensory data comprising at least information describing orientation of the media presentation device. The one or more devices may also determine an approximate direction of a gaze of the user in a three dimensional audio space based on the information describing the orientation of the media presentation device, identify a sound source at which the user gazes based on the approximate direction of the gaze of the user in the three dimensional audio space, obtain audio data associated with the sound source based on an identity of the identified sound source, determine an action to perform on the audio data associated with the sound source, and perform the determined action.

Claims

1. A mobile communication device comprising:

one or more communication interfaces configured to;

communicate with a media presentation device; and

one or more processors configured to;

receive, via the one or more communication interfaces, from the media presentation device, sensory data comprising information describing orientation of the media presentation device, and scene information representative of an image of a three dimensional audio space adjacent to a user,

receive a media request comprising selected information related to one or more objects,

determine an approximate direction of a gaze of the user in the three dimensional audio space based on the information describing the orientation of the media presentation device,

query a map database based on the selected information of the media request to identify a sound source in the three dimensional audio space at which the user gazes by;

identifying an object in the three dimensional audio space based on a location of the user, the approximate direction of the gaze, and information in the map database; and

determining whether the identified object is the sound source at which the user gazes by at least one of;

i) determining whether an image obtained from the scene information matches an image of the identified object obtained from the map database; and

/or ii) determining whether a distance from the identified object to the user is approximately equal to a distance from the sound source at which the user gazes to the user;

obtain audio data based on the identified sound source at which the user gazes,

determine an action to perform on the obtained audio data associated with the identified sound source at which the user gazes, and

perform the determined action.

2. The mobile communication device of claim 1, wherein when the one or more processors determine the action, the one or more processors are further configured to modify the obtained audio data such that a sound generated based on the obtained audio data is amplified.

3. The mobile communication device of claim 2, wherein when the one or more processors determine the action, the one or more processors are further configured to receive additional sensory data that describes one of:

a change in facial feature of the user;

or

movement of a head of the user.

4. The mobile communication device of claim 1, wherein the mobile communication device comprises one of:

a cellular telephone;

a laptop computer;

or

a tablet computer.

5. The mobile communication device of claim 1, wherein the sensory data further includes at least one of:

images of the eyes of the user;

or

information describing a location of the media presentation device.

6. The mobile communication device of claim 1, wherein the media presentation device includes one of:

a pair of ear buds;

a pair of virtual reality glasses;

or

a pair of augmented reality glasses.

7. A device comprising:

sensors configured to;

collect sensory data comprising images of eyes of a user and scene information representative of an image of an audio space adjacent to the user; and

one or more processors configured to;

receive the sensory data from the sensors,

receive a media request comprising selected information related to one or more images of sound sources,

determine an approximate direction of a gaze of the user based on the images, included in the sensory data, of the eyes of the user,

query a map database based on the selected information of the media request to identify a sound source at which the user gazes by;

identifying an object in the audio space based on a location of the user, the approximate direction of the gaze, and information in the map database; and

determining whether the identified object is the sound source at which the user gazes by at least one of;

i) determining whether an image obtained from the scene information matches an image of the identified object obtained from the map database; and

/or ii) determining whether a distance from the identified object to the user is approximately equal to a distance from the sound source at which the user gazes to the user;

obtain audio data associated with the identified sound source at which the user gazes based on an identity of the identified sound source at which the user gazes,

determine an action to perform on the obtained audio data associated with the identified sound source at which the user gazes, and

perform the determined action.

8. The device of claim 7, wherein when the one or more processors perform the determined action, the one or more processors are configured to:

apply a transformation to the obtained audio data to amplify an audio signal corresponding to the obtained audio data; and

send the amplified audio signal to a remote media presentation device.

9. The device of claim 7, wherein when the one or more processors perform the determined action, the one or more processors are further configured to:

apply a head-related transform to the obtained audio data to generate a stereo sound that is perceived by the user as emanating from a specific location of an image of the identified sound source at which the user gazes in a three dimensional audio space.

10. A method comprising:

receiving, from a media presentation device worn by a user, sensory data comprising i) information describing an orientation of the media presentation device, and ii) scene information representative of an image of a three dimensional audio space adjacent to the media presentation device;

receiving a media request comprising selected information related to one or more images of sound sources;

determining an approximate direction of a gaze of the user in the three dimensional audio space based on the information describing the orientation of the media presentation device;

querying a map database based on the selected information of the media request to identify a sound source at which the user gazes by;

identifying an object in the three dimensional audio space based on a location of the user, the approximate direction of the gaze, and information in the map database; and

determining whether the identified object is the sound source at which the user gazes by at least one of;

i) determining whether an image obtained from the scene information matches an image of the identified object obtained from the map database; and

/or ii) determining whether a distance from the identified object to the user is approximately equal to a distance from the sound source at which the user gazes to the user;

obtaining audio data associated with the identified sound source at which the user gazes based on an identity of the identified sound source at which the user gazes;

determining an action to perform on the obtained audio data associated with the identified sound source at which the user gazes; and

performing the determined action.

11. The method of claim 10, wherein the sensory data further comprises images of the user'"'"'s eyes.

12. The method of claim 11, wherein determining the approximate direction of the gaze comprises determining a direction of the gaze based on the information describing the orientation of the media presentation device and a direction of the user'"'"'s eyes,

wherein the direction of the user'"'"'s eyes is determined based on the images of the user'"'"'s eyes.

13. The method of claim 10, wherein performing the action comprises one of:

amplifying the obtained audio data;

or

ending amplification of the obtained audio data.

14. The method of claim 10, further comprising:

performing image recognition on the scene information received as part of the sensory data when the identified object is the sound source at which the user gazes;

identifying a set of images corresponding to a set of objects based on the image recognition;

selecting, among the identified set of images, a selected image based on the approximate direction of the gaze; and

identifying the sound source at which the user gazes as the identified sound source based on the selected image and information associated with the selected image.

15. The method of claim 10, wherein identifying the sound source at which the user gazes comprises:

determining a point, in a field of view of the user in the three dimensional audio space, at which the user gazes based on the approximate direction of the gaze; and

identifying the object as an object whose corresponding image, in the field of view, includes the point.

16. The method of claim 15, wherein performing the determined action comprises:

locking the sound source at which the user gazes in a particular location within the three dimensional audio space.

17. The method of claim 16, wherein the three dimensional audio space includes a virtual three dimensional audio space.

18. The method of claim 10, wherein identifying the sound source at which the user gazes comprises:

sending information describing the location of the user and the approximate direction of the gaze to a remote device over an associated network,

wherein the remote device is configured to identify the sound source at which the user gazes and retrieve the audio data from an audio database based on the identity of the identified sound source at which the user gazes; and

wherein the obtaining the audio data comprises receiving the audio data from the remote device via the associated network.