AUGMENTED REALITY (AR) AUDIO WITH POSITION AND ACTION TRIGGERED VIRTUAL SOUND EFFECTS

• US 20130236040A1
• Filed: 03/08/2012
• Published: 09/12/2013
• Est. Priority Date: 03/08/2012
• Status: Active Grant

• Current Assignee: Disney Enterprises Incorporated (The Walt Disney Company)
• Original Assignee: Disney Enterprises Incorporated (The Walt Disney Company)
• Inventors: Crawford, David W., Kadar, Istvan B., Samdahl, Amber, Voris, Jeffrey, Mitchell, Kenny
• Granted Patent: US 8,831,255 B2
• Time in Patent Office: Days
• US Class Current: 381/310
• CPC Class Codes: H04R 3/005 for combining the signals of two or more microphones specially adapted for hearing aids H04R25/407
• CPC Class Codes: H04R 5/033 Headphones for stereophonic communication details thereof, e.g. relating to batteries, cables or control elements H04R1/10
• CPC Class Codes: H04S 2400/01 Multi-channel, i.e. more than two input channels, sound reproduction with two speakers wherein the multi-channel information is substantially preserved
• CPC Class Codes: H04S 2400/11 Positioning of individual sound objects, e.g. moving airplane, within a sound field H04S2420/13 takes precedence
• CPC Class Codes: H04S 2400/15 Aspects of sound capture and related signal processing for recording or reproduction
• CPC Class Codes: H04S 2420/01 Enhancing the perception of the sound image or of the spatial distribution using head related transfer functions [HRTF's] or equivalents thereof, e.g. interaural time difference [ITD] or interaural level difference [ILD]
• CPC Class Codes: H04S 7/304 For headphones
• CPC Class Codes: H04S 7/306 For headphones

Abstract

An augmented reality (AR) audio system for augmenting environment or ambient sound with sounds from a virtual speaker or sound source positioned at a location in the space surrounding an AR participant. The sound from the virtual speaker may be triggered by an action of the listener and/or by the location or relative orientation of the listener. The AR audio system includes stereo earphones receiving an augmented audio track from a control unit, and binaural microphones are provided to capture ambient sounds. The control unit operates to process trigger signals and retrieve one or more augmentation sounds. The control unit uses an AR audio mixer to combine the ambient sound from the microphones with the augmentation sounds to generate left and right ear augmented audio or binaural audio, which may be modified for acoustic effects of the environment including virtual objects in the environment or virtual characteristics of real objects.

Claims

1. A method for providing augmented audio to a listener wearing a headset including right and left ear speakers, comprising:

with binaural microphones on the headset, capturing ambient sound in an environment about the headset;

from a sensor array worn or carried by the listener, receiving a trigger signal;

with a track selection module, selecting an augmentation audio track in response to the trigger signal;

with a processor running an augmented reality (AR) audio mixer, combining the captured ambient sound with the selected augmentation audio track to generate an AR audio output track; and

playing the AR audio output track with the right and left ear speakers of the headset, wherein the selected augmentation audio track has binaural characteristics associated with a virtual speaker located relative to the listener'"'"'s headset in the environment.

2. The method of claim 1, further including isolating the listener from the ambient sound during the playing of the AR audio output track.

3. The method of claim 1, wherein the sensor array comprises an infrared (IR) receiver outputting the trigger signal in response to receiving an IR signal from an IR transmitter on a user input device actuated by the listener.

4. The method of claim 3, further wherein the IR receiver includes a left IR sensor and a right IR sensor positioned within the headset proximate to the left and right ear speakers, respectively, and wherein the virtual speaker is positioned relative to the listener'"'"'s headset based on processing of the trigger signal.

5. The method of claim 3, further wherein a second IR signal is received as a reflected IR signal from an object in the environment of an IR signal output from the IR transmitter of the user input device and wherein the virtual speaker is co-located with the object in the environment.

6. The method of claim 1, wherein the sensor array further comprises at least one head tracking sensor operating to transmit signals corresponding to a location of the headset in the environment and wherein the processor operates to set a location of the virtual speaker relative to the location of the headset determined based on the head tracking sensor signals.

7. The method of claim 1, wherein at least one of the selected augmentation audio track and the captured ambient sound are modified during the combining step based on an acoustic signature of the environment.

8. The method of claim 7, wherein the acoustic signature defines effects corresponding to at least one of attenuation, reflectance, absorption, scattering, transmission, occlusion, diffraction, and Doppler shift.

9. The method of claim 7, wherein the environment includes at least one virtual object or parameter, whereby the acoustic signature of the environment includes at least one virtual acoustic effect.

10. The method of claim 9, wherein the virtual parameter is a material of a physical object in the environment or is a virtual geometry differing from a physical geometry of a portion of the environment, whereby audio environmental effects including occlusion and reflectance differ from real audio effects in the environment.

11. An augmented audio apparatus, comprising:

a binaural audio headphone assembly including right and left earphones providing right and left speakers, respectively, wherein the headphone assembly further includes a left microphone on the left earphone and a right microphone on the right earphone; and

a control pack communicatively linked with the headphone assembly, the control pack including media storage storing a plurality of augmentation audio tracks and further including a binaural transfer function module generating an augmented audio output track for playing on the right and left speakers, the augmented audio output track combining ambient sound captured by the left and right microphones with at least one of the augmentation audio tracks output from a virtual sound source positioned at a physical location relative to the headphone assembly.

12. The apparatus of claim 11, wherein a space including the physical location of the virtual sound source includes physical objects defining a set of acoustic effects for a sound emitted from the virtual sound source and wherein the binaural transfer function modifies the at least one of the augmentation audio tracks based on at least one of the acoustic effects.

13. The apparatus of claim 12, wherein the at least one of the acoustic effects is chosen from the group consisting of attenuation, reflectance, absorption, scattering, transmission, occlusion, diffraction, and Doppler shift.

14. The apparatus of claim 12, wherein the binaural transfer function further modifies the at least one of the augmentation audio tracks or the captured ambient sound to apply an acoustic effect caused by a virtual object positioned in the space or an acoustic effect differing from a real acoustic effect of one of the physical objects in the space.

15. The apparatus of claim 11, wherein the headphone assembly further comprises right and left sensors receiving signals from an emitter and responding by transmitting a trigger signal to the control pack and wherein the binaural transfer function module selects the at least one of the augmentation audio tracks based on the trigger signals and defines the physical location based on the trigger signals.

16. The apparatus of claim 11, wherein the headphone assembly is adapted to isolate the microphones from the speakers.

17. An augmented reality audio system, comprising:

a headset with a left speaker and a right speaker and with a left microphone mounted proximate to the left speaker and a right microphone mounted proximate to the right speaker; and

a control unit communicatively linked with the headset, the control unit including an AR audio mixer and media storage storing augmentation audio tracks, wherein the AR audio mixer selectively combines one of the augmentation audio tracks with environmental sound captured by the left and right speakers to generate an augmented audio output played on the left and right speakers and wherein the AR audio mixer modifies the one of the augmentation audio tracks based on an acoustic signature of an AR environment.

18. The system of claim 17, wherein the acoustic signature defines effects of physical objects in the AR environment due to at least one of attenuation, reflectance, absorption, scattering, transmission, occlusion, diffraction, and Doppler shift.

19. The system of claim 17, wherein the acoustic signature defines effects of one or more virtual object or parameter in the AR environment due to at least one of attenuation, reflectance, absorption, scattering, transmission, occlusion, diffraction, and Doppler shift.

20. The system of claim 17, wherein the headset includes a sensor array detecting an event in the AR environment and a relative location of the headset in the AR environment, wherein the control unit further includes a module for selecting the augmentation audio track to combine with the environmental sound based on the detected event, and wherein the AR audio mixer modifies the selected augmentation audio track based on the relative location of the headset and a location of a virtual speaker provided in the AR environment to output the selected augmentation audio track.