Sound localization for user in motion

• US 9,736,613 B2
• Filed: 05/07/2015
• Issued: 08/15/2017
• Est. Priority Date: 10/27/2008
• Status: Active Grant

• Current Assignee: Sony Interactive Entertainment Inc. (Sony Group Corp.)
• Original Assignee: Sony Interactive Entertainment Inc. (Sony Group Corp.)
• Inventors: Osman, Steven
• Primary Examiner(s): Jackson, Jakieda
• Application Number:
  US14/706,575
• Publication Number:
  US 20150245159A1
• Time in Patent Office: 831 Days
• Field of Search: 704270
• CPC Class Codes: G06F 3/012 Head tracking input arrangements
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: H04S 2400/11 Positioning of individual sound objects, e.g. moving airplane, within a sound field H04S2420/13 takes precedence
• CPC Class Codes: H04S 2420/01 Enhancing the perception of the sound image or of the spatial distribution using head related transfer functions [HRTF's] or equivalents thereof, e.g. interaural time difference [ITD] or interaural level difference [ILD]
• CPC Class Codes: H04S 7/302 Electronic adaptation of stereophonic sound system to listener position or orientation H04S7/301 takes precedence
• CPC Class Codes: H04S 7/304 For headphones

Abstract

Methods, apparatus, and computer programs for simulating the source of sound are provided. One method includes operations for determining a location in space of the head of a user utilizing face recognition of images of the user. Further, the method includes an operation for determining a sound for two speakers, and an operation for determining an emanating location in space for the sound, each speaker being associated with one ear of the user. The acoustic signals for each speaker are established based on the location in space of the head, the sound, the emanating location in space, and the auditory characteristics of the user. In addition, the acoustic signals are transmitted to the two speakers. When the acoustic signals are played by the two speakers, the acoustic signals simulate that the sound originated at the emanating location in space.

Claims

1. A method for simulating a source of sound, the method comprising:

determining, by a computer, a location in physical space of a head of a user, wherein the location is determined by capturing images by a camera of the physical space in which a user is located and identifying light from at least one light emitting diode (LED) disposed on the head of the user;

determining, by the computer, a sound for delivery to two speakers worn by the user;

determining, by the computer, an emanating location in the physical space for the sound;

establishing, by the computer, acoustic signals for each speaker based on the location in the physical space of the head, the sound, the emanating location in the physical space, and auditory characteristics of the user, wherein the auditory characteristics of the user are identified based on a calibration process, the calibration process is configured to generate one or more sounds in the physical space and a user is requested to respond by moving a controller having a light proximate to where the user perceives the sound to be emanating, the camera identifying positions of the controller when moved by the user, the response of moving the controller and the generated one more sounds used to identify the auditory characteristics of the user, and wherein the acoustic signals are established to simulate for the user that the sound originates at the emanating location in the physical space when the acoustic signals are played by the two speakers; and

transmitting, by the computer, the acoustic signals to the two speakers.

2. The method as recited in claim 1, further including:

calculating new acoustic signals for transmission to the two speakers based on updates to any of the location in the physical space of the head, the sound for delivery to the two speakers, or the emanating location.

3. The method as recited in claim 1, wherein the sound is generated by a virtual object in an augmented reality environment, the augmented reality environment including one or more virtual objects imposed on a real-world view having real objects, wherein the emanating location is a simulated location in the physical space of the virtual object.

4. The method as recited in claim 3, further including:

displaying a portion of the augmented reality on a screen, wherein the acoustic signals played by the two speakers provide an acoustic clue to the user regarding the simulated location in the physical space of the virtual object.

5. The method as recited in claim 4, wherein establishing acoustic signals further includes:

providing a higher volume to the acoustic signals when the virtual object is displayed on the screen than when the virtual object is not displayed on the screen.

6. The method as recited in claim 1, wherein the auditory characteristics of the user is selected as a best match of a head-related transfer function (HRTF) for the user based on the calibration process, the HRTF of a listener defining how sound arriving at an eardrum of the listener is modified based on an anatomy of the listener, wherein the established acoustic signals for each speaker are based on the HRTF of the user.

7. A method for simulating a source of sound in a virtual space, the method comprising:

tracking, by a computer, a location in physical space of a head of a user, wherein the location is determined by capturing images by a camera, controlled by the computer, of the physical space in which a user is located and identifying light from at least one light emitting diode (LED) disposed on the head of the user for facilitating said tracking;

determining, by the computer, a sound for delivery to headphones worn by the user;

determining, by the computer, an emanating location in virtual space rendered on a screen for the sound;

establishing, by the computer, acoustic signals for the headphones based on the location in the physical space of the head, the emanating location in the virtual space, and auditory characteristics of the user, wherein the auditory characteristics of the user are determined based on a calibration process for the user, the calibration process is configured to generate one or more audio cues in the virtual space and a user is requested to respond by moving a controller having a light proximate to where the user perceives the sound to be emanating, the camera identifying positions of the controller when moved by the user and positions of the head of the user, and wherein the acoustic signals are established to simulate for the user that the sound originates at the emanating location in the virtual space when the acoustic signals are output by the headphones; and

transmitting, by the computer, the acoustic signals to the headphones.