Three-dimensional audio sweet spot feedback

• US 9,522,330 B2
• Filed: 12/21/2012
• Issued: 12/20/2016
• Est. Priority Date: 10/13/2010
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Pile, Shawn, Vincent, Jon, Henson, Scott, Flaks, Jason, Bar-Zeev, Avi, Tardif, John
• Primary Examiner(s): Kim, Paul S
• Assistant Examiner(s): Faley, Katherine
• Application Number:
  US13/725,572
• Publication Number:
  US 20130208898A1
• Time in Patent Office: 1,460 Days
• Field of Search: 381/17
• US Class Current: 1/1
• CPC Class Codes: A63F 13/10 Control of the course of the game, e.g. start, progess, end
• CPC Class Codes: A63F 13/213 comprising photodetecting means, e.g. cameras, photodiodes or infrared cells A63F13/219, A63F13/655 take precedence
• CPC Class Codes: A63F 13/428 involving motion or position input signals, e.g. signals representing the rotation of an input controller or a player's arm motions sensed by accelerometers or gyroscopes
• CPC Class Codes: A63F 13/45 Controlling the progress of the video game
• CPC Class Codes: A63F 13/54 involving acoustic signals, e.g. for simulating revolutions per minute [RPM] dependent engine sounds in a driving game or reverberation against a virtual wall
• CPC Class Codes: A63F 2300/1012 involving biosensors worn by the player, e.g. for measuring heart beat, limb activity
• CPC Class Codes: A63F 2300/6081 generating an output signal, e.g. under timing constraints, for spatialization
• CPC Class Codes: H04R 5/02 Spatial or constructional arrangements of loudspeakers
• CPC Class Codes: H04S 2400/11 Positioning of individual sound objects, e.g. moving airplane, within a sound field H04S2420/13 takes precedence
• CPC Class Codes: H04S 2420/01 Enhancing the perception of the sound image or of the spatial distribution using head related transfer functions [HRTF's] or equivalents thereof, e.g. interaural time difference [ITD] or interaural level difference [ILD]
• CPC Class Codes: H04S 7/303 Tracking of listener position or orientation

Abstract

A method for providing three-dimensional audio is provided. The method includes receiving a depth map imaging a scene from a depth camera and recognizing a human subject present in the scene. The human subject is modeled with a virtual skeleton comprising a plurality of joints defined with a three-dimensional position. A world space ear position of the human subject is determined based on the virtual skeleton. Furthermore, a target world space ear position of the human subject is determined. The target world space ear position is the world space position where a desired audio effect can be produced via an acoustic transducer array. The method further includes outputting a notification representing a spatial relationship between the world space ear position and the target world space ear position.

Claims

1. A method for providing three-dimensional audio, comprising:

receiving a depth map imaging a scene from a depth camera;

recognizing a human subject present in the scene;

modeling the human subject with a virtual skeleton comprising a plurality of joints defined with a three-dimensional position;

determining, based on the virtual skeleton, a world space ear position of the human subject;

determining a target world space ear position of the human subject where a desired audio effect is produced via an acoustic transducer array; and

visually indicating a three-dimensional spatial relationship between the world space ear position and the target world space ear position.

2. The method of claim 1, wherein the visual indication directs the human subject to the target world space ear position if the world space ear position is not proximate to the target world space ear position.

3. The method of claim 1, wherein the visual indication alerts the human subject that the world space ear position is proximate to the target world space ear position.

4. The method of claim 1, wherein visually indicating the three-dimensional spatial relationship comprises displaying a representation of the target world space ear position via a display device.

5. The method of claim 4, further comprising:

displaying a representation of the scene via the display device; and

concurrently displaying the visual indication in spatial registration with a target virtual space ear position in virtual space corresponding to the target world space ear position in world space.

6. The method of claim 5, wherein the representation of the scene includes a color model of the scene based on color information received from one or more color image sensors.

7. The method of claim 5, wherein the representation of the scene includes a geometric model of the scene based on the depth map received from the depth camera.

8. The method of claim 5, wherein the representation of the visual indication comprises an overlay.

9. The method of claim 5, wherein the visual indication comprises one or more directional indicators based on the spatial relationship between the human subject and the target world space ear position.

10. The method of claim 1, further comprising outputting an audio notification of the spatial relationship to the acoustic transducer array.

11. The method of claim 1, further comprising, when the human subject is proximate the target world space ear position:

recognizing one or more joints of the virtual skeleton;

recognizing depth information in the depth map that corresponds to a head of the human subject based on the one or more joints; and

calculating a head-related transfer function (HRTF) based on the depth information; and

wherein determining the target world space ear position of the human subject includes determining the target world space ear position based on the HRTF.

12. The method of claim 1, further comprising:

determining a world space transducer position of the acoustic transducer array; and

determining a crosstalk cancellation transformation based on a spatial relationship between the world space transducer position and the world space ear position.

13. The method of claim 12, wherein determining the world space transducer position comprises:

providing calibration audio to the acoustic transducer array;

receiving acoustic sensor information from one or more acoustic sensors during output of the calibration audio by the acoustic transducer array; and

identifying the world space transducer position based on the output calibration audio and the acoustic sensor information.

14. The method of claim 12, wherein determining the world space transducer position comprises recognizing the acoustic transducer array via one or more of the depth map and color information imaging the scene.

15. A three-dimensional audio system, comprising:

a depth camera input to receive a depth map imaging a scene from one or more depth cameras;

an audio input to receive audio input information;

an audio output to provide three-dimensional audio output information to an acoustic transducer array comprising one or more acoustic transducers;

a logic subsystem; and

a storage subsystem storing instructions that are executable by the logic subsystem to;

receive the depth map;

recognize a human subject present in the scene;

model the human subject with a virtual skeleton comprising a plurality of joints defined with a three-dimensional position;

determine, based on the virtual skeleton, a world space ear position of the human subject;

determine a target world space ear position of the human subject where a desired audio effect is produced via the acoustic transducer array; and

output a notification representing a three-dimensional spatial relationship between the world space ear position and the target world space ear position.

16. The three-dimensional audio system of claim 15, wherein determining the target world space ear position of the human subject comprises:

recognizing one or more joints of the virtual skeleton;

recognizing depth information in the depth map that corresponds to a head of the human subject based on the one or more joints;

determining one or more audio-output transformations based on the depth information; and

determining the target world space ear position based on the one or more audio-output transformations.

17. The three-dimensional audio system of claim 16, wherein the one or more audio-output transformations include a crosstalk cancellation transformation.

18. The three-dimensional audio system of claim 16, wherein the instructions are further executable to receive color information imaging the scene from one or more color image sensors, wherein the one or more audio-output transformations are further based on the color information.

19. A method for providing three-dimensional audio, comprising:

receiving a depth map imaging a scene from a depth camera;

recognizing a human subject present in the scene;

modeling the human subject with a virtual skeleton comprising a plurality of joints defined with a three-dimensional position;

determining, based on the virtual skeleton, a world space ear position of the human subject;

determine, based on the world space ear position, one or more audio-output transformations configured to produce the desired audio effect from audio input information, the one or more audio-output transformations comprising a head-related transfer function (HRTF) and a crosstalk cancellation transformation;

determining a target world space ear position of the human subject where a desired audio effect is produced via an acoustic transducer array; and

visually indicating a three-dimensional spatial relationship between the world space ear position and the target world space ear position.

20. The method of claim 19, wherein visually indicating comprises:

displaying a visual notification via a display device,

responsive to the world space ear position of the human subject moving to the target world space ear position, determining, based on the moved world space ear position, one or more additional audio-output transformations configured to produce the desired audio effect from audio input information, and

providing the desired audio effect to the human subject via three-dimensional audio output from the acoustic transducer array.