Spatial audio signal processing for objects with associated audio content

• US 9,787,846 B2
• Filed: 01/21/2015
• Issued: 10/10/2017
• Est. Priority Date: 01/21/2015
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Srinivasan, Sriram, Menezes, Ryan S., Wright, Shawn Crispin, Huai, Hongwu
• Primary Examiner(s): Chin, Vivian
• Assistant Examiner(s): Kurr, Jason R
• Application Number:
  US14/602,246
• Publication Number:
  US 20160212272A1
• Time in Patent Office: 993 Days
• Field of Search: 381 2, 381306, 381310
• CPC Class Codes: G06F 3/165 Management of the audio stream, e.g. setting of volume, audio stream path
• CPC Class Codes: G06F 3/167 Audio in a user interface, e.g. using voice commands for navigating, audio feedback
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: H04M 3/567 Multimedia conference systems
• CPC Class Codes: H04M 3/568 audio processing specific to telephonic conferencing, e.g. spatial distribution, mixing of participants echo suppression in two-way loud-speaking telephone systems H04M9/02; sound field processing per se H04S7/30
• CPC Class Codes: H04N 21/41407 embedded in a portable device, e.g. video client on a mobile phone, PDA, laptop constructional details of equipment or arrangements specially adapted for portable computer application G06F1/1626; arrangements specially adapted for mobile receivers in broadcast systems H04H20/57
• CPC Class Codes: H04N 21/42202 environmental sensors, e.g. for detecting temperature, luminosity, pressure, earthquakes
• CPC Class Codes: H04N 21/439 Processing of audio elementary streams
• CPC Class Codes: H04N 7/147 Communication arrangements, e.g. identifying the communication as a video-communication, intermediate storage of the signals selecting H04Q
• CPC Class Codes: H04S 1/00 Two-channel systems H04S5/00, H04S7/00 take precedence
• CPC Class Codes: H04S 2400/11 Positioning of individual sound objects, e.g. moving airplane, within a sound field H04S2420/13 takes precedence
• CPC Class Codes: H04S 2420/01 Enhancing the perception of the sound image or of the spatial distribution using head related transfer functions [HRTF's] or equivalents thereof, e.g. interaural time difference [ITD] or interaural level difference [ILD]
• CPC Class Codes: H04S 7/303 Tracking of listener position or orientation
• CPC Class Codes: H04S 7/304 For headphones

Abstract

A user device for generating a scene, the user device comprising: an object determiner configured to determine an object for the scene, the object being associated with at least one audio signal; a relative position/orientation determiner configured to determine a relative position/orientation between the user device'"'"'s user and the object; an audio position processor configured to spatially audio signal process the at least one audio signal to generate at least two channel audio signal based on the relative position/orientation.

Claims

1. A user device for generating a scene, the user device comprising:

at least one image capturing device configured to capture video of an area within a field of view of a user wearing the user device, the user device being a head-mountable device;

an object determiner configured to determine a virtual object for the scene, the virtual object being associated with at least one audio signal, the scene being a shared mixed-reality scene environment representing the captured video of the area within the field of view of the user wearing the user device combined with the virtual object;

one or more components configured to communicate the shared mixed-reality scene environment to a device of at least one additional user over a communications network, the shared mixed-reality scene environment enabling one or more of a functional attribute or a positional attribute of the virtual object to be controlled by the user and the at least one additional user;

a relative position/orientation determiner configured to;

determine a relative object position and orientation of the virtual object relative to a device position and orientation of the user device;

determine over a determined time period a change in the relative object position and orientation;

determine that the virtual object has moved outside the field of view based on the change in the relative object position and orientation being greater than a determined threshold value; and

generate a smoothed relative object position and orientation outside the field of view as the relative object position and orientation based on a determination that the virtual object has moved outside the field of view; and

an audio position processor configured to spatially audio signal process the at least one audio signal to generate at least a two channel audio signal based on the relative object position and orientation.

2. The user device of claim 1, wherein the audio position processor is configured to process the at least one audio signal with a head related transfer function with the relative object position and orientation as a parameter input for the head related transfer function.

3. The user device of claim 1, wherein:

the virtual object is associated with the at least one additional user device; and

the at least one audio signal is a transmitted audio signal from the at least one additional user device.

4. The user device of claim 1, wherein the virtual object is further associated with an image or additional video, and the user device further comprises an object renderer configured to display the image or additional video at the relative object position and orientation.

5. The user device of claim 1, wherein:

the relative position/orientation determiner is configured to determine smoothed transition values based on the relative object position and orientation and a default relative object position and orientation; and

the audio position processor is configured to spatially audio signal process the at least one audio signal to generate at least a two channel audio signal based on the smoothed transition values.

6. The user device of claim 1, wherein the change in the relative object position and orientation is a result of the user device being physically moved to cause the at least one image capturing device to turn away from the virtual object.

7. A method implemented at a user device for generating a scene, the method comprising:

capturing video of an area within a field of view of a user wearing the user device, the user device being a head-mountable device;

determining a virtual object for the scene, the virtual object being associated with at least one audio signal, the scene being a shared mixed-reality scene environment representing the captured video of the area within the field of view of the user wearing the user device combined with the virtual object;

communicating the shared mixed-reality scene environment with a device of at least one additional user over a communications network, the shared mixed-reality scene environment enabling one or more of a functional attribute or a positional attribute of the virtual object to be controlled by the user and the at least one additional user;

determining a spatial relationship between a user of the user device and the virtual object;

spatially audio signal processing the at least one audio signal to generate at least a two channel audio signal based on the spatial relationship;

determining over a determined time period a change in the spatial relationship;

determining that the virtual object has moved outside the field of view based on the change in the spatial relationship being greater than a threshold value; and

generating one or more smoothed relative values representing a relative position and orientation of the virtual object outside the field of view based on a determination that the virtual object has moved outside the field of view, the relative position and orientation usable for the spatially audio signal processing of the at least one audio signal.

8. The method of claim 7, wherein spatially audio signal processing the at least one audio signal comprises processing the at least one audio signal with a head related transfer function with the spatial relationship as a parameter input for the head related transfer function.

9. The method of claim 7, wherein:

the virtual object is associated with the at least one additional user device; and

the at least one audio signal is a transmitted audio signal from the at least one additional user device.

10. The method of claim 7, wherein the virtual object is further associated with an image or additional video, and the method further comprises displaying the image or additional video at a relative position and orientation of the virtual object within the field of view based on the spatial relationship.

11. The method of claim 7, further comprising:

determining smoothed transition values based on the relative position and orientation of the virtual object and a default relative position and orientation of the virtual object; and

spatially audio signal processing the at least one audio signal to generate at least two channel audio signal based on the smoothed transition values.

12. The user device of claim 7, wherein the spatial relationship is defined by a relative direction from the user device to the virtual object and a distance between the user device and the virtual object.

13. The method of claim 7, wherein the spatial relationship is based on the relative position and orientation of the virtual object relative to a device position and orientation of the user device and a determined position and orientation offset, the determined position and orientation offset representing a difference between the device position and orientation of the user device and a center of hearing location of the user wearing the user device.

14. The method of claim 7, wherein the change in the spatial relationship is caused by physical movement of the user device that turns the user device away from the virtual object.

15. The method of claim 7, wherein the spatially audio signal processing includes applying a head related transfer function to the at least one audio signal to produce a spatial output before the at least two channel audio signal is passed to a sound card for output.

16. A computer-readable hardware device comprising a memory storing instructions that are executable by one or more processors of a user device to cause the one or more processors to perform operations for generating a scene, the operations comprising:

capturing video of an area within a field of view of a user wearing the user device, the user device being a head-mountable device;

determining a virtual object for the scene, the virtual object being associated with at least one audio signal, the scene being a shared mixed-reality scene environment representing the captured video of the area within the field of view of the user wearing the user device combined with the virtual object;

determining a relative object position and orientation of the virtual object relative to a device position and orientation of the user device, the relative object position determined to be located within the field of view of the user;

spatially audio signal processing the at least one audio signal to generate at least a two channel audio signal based on the relative object position and orientation;

determining that the virtual object has moved outside the field of view based on a change in the relative object position and orientation being greater than a determined threshold value over a determined period of time; and

generating a smoothed relative object position and orientation outside the field of view as the relative object position and orientation based on a determination that the virtual object has moved outside the field of view, the smoothed relative object position and orientation usable in the spatially audio signal processing of the at least one audio signal.

17. A computer-readable hardware device as recited in claim 16, wherein the user device comprises a wearable device having at least one camera configured to capture the video of the area within the field of view of the user.

18. A computer-readable hardware device as recited in claim 16, wherein:

the shared mixed-reality scene environment is communicated to a device of at least one additional user over a communications network;

the scene is controllable by the user via the user device; and

the shared mixed-reality scene environment enables one or more of a functional attribute or a positional attribute of the virtual object to be controlled by both the user and the additional user.

19. A computer-readable hardware device as recited in claim 16, wherein the change in the relative object position and orientation is caused by the user device being physically moved effective to cause the at least one image capturing device to turn away from the virtual object.

20. A computer-readable hardware device as recited in claim 16, wherein spatially audio signal processing includes applying a head related transfer function to the at least one audio signal to produce a spatial output before the two channel audio signal is passed to a sound card for output.