Spatial audio with remote speakers

• US 9,584,915 B2
• Filed: 01/19/2015
• Issued: 02/28/2017
• Est. Priority Date: 01/19/2015
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Fullam, Scott Francis, Tcherkachine, Lev, Legg, Travis, Potts, Lori, Elias, Annika, Haskins, Craig
• Primary Examiner(s): Edun, Muhammad N
• Application Number:
  US14/599,957
• Publication Number:
  US 20160212538A1
• Time in Patent Office: 771 Days
• US Class Current: 1/1
• CPC Class Codes: G02B 2027/0132 comprising binocular systems
• CPC Class Codes: G02B 2027/0138 comprising image capture systems, e.g. camera
• CPC Class Codes: G02B 2027/014 comprising information/image processing systems
• CPC Class Codes: G02B 2027/0178 Eyeglass type eyeglass details G02C
• CPC Class Codes: G02B 27/017 Head mounted
• CPC Class Codes: G06F 2218/22 Source localisation; Inverse modelling
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G06V 20/20 in augmented reality scenes
• CPC Class Codes: H04R 2420/07 Applications of wireless loudspeakers or wireless microphones
• CPC Class Codes: H04R 29/001 for loudspeakers H04R29/007 takes precedence
• CPC Class Codes: H04R 5/02 Spatial or constructional arrangements of loudspeakers
• CPC Class Codes: H04S 2400/11 Positioning of individual sound objects, e.g. moving airplane, within a sound field H04S2420/13 takes precedence
• CPC Class Codes: H04S 3/008 in which the audio signals are in digital form, i.e. employing more than two discrete digital channels data reduction aspects thereof based on psychoacoustics G10L19/02
• CPC Class Codes: H04S 7/301 Automatic calibration of stereophonic sound system, e.g. with test microphone
• CPC Class Codes: H04S 7/303 Tracking of listener position or orientation
• CPC Class Codes: H04S 7/40 Visual indication of stereophonic sound image

Abstract

Embodiments related to outputting spatial audio via a portable computing device are presented. For example, one disclosed embodiment provides a portable computing device for wirelessly detecting a presence of a remote speaker external to the portable computing device, determining a location of the remote speaker, and establishing a wireless connection with the remote speaker. The embodiment further includes presenting visual content via a display of the portable computing device, the visual content comprising an object positionally associated with a location in a real-world environment around the portable computing device, generating an audio signal corresponding to the object, and sending the audio signal to the remote speaker.

Claims

1. On a portable computing device, a method comprising:

wirelessly detecting a presence of a remote speaker external to the portable computing device;

determining a location of the remote speaker;

establishing a wireless connection with the remote speaker;

presenting visual content via a display of the portable computing device, the visual content comprising an object positionally associated with a location in a real-world environment around the portable computing device;

generating an audio signal corresponding to the object; and

sending the audio signal to the remote speaker.

2. The method of claim 1, wherein sending the audio signal to the remote speaker comprises sending the audio signal to the remote speaker based on the location of the object relative to the portable device.

3. The method of claim 1, wherein the object is out of a field of view of a user viewing the presented visual content.

4. The method of claim 1, wherein the object is in a field of view of a user viewing the presented visual content.

5. The method of claim 1, wherein the remote speaker is one of a plurality of remote speakers, and further comprising selecting a remote speaker to which to send the audio signal based on a location of each remote speaker, the location of the object, and a location of the portable computing device.

6. The method of claim 1, wherein determining the location of the remote speaker comprises detecting, based on feedback from one or more image sensors, a gesture performed by a user indicating the location of the remote speaker.

7. The method of claim 1, wherein determining the location of the remote speaker comprises performing object identification on image data collected by one or more image sensors.

8. The method of claim 1, wherein determining the location of the remote speaker comprises sending a test audio signal to the remote speaker, receiving an output from the remote speaker via a plurality of microphones on the portable computing device, and determining the location of the remote speaker based on signals from the plurality of microphones.

9. The method of claim 8, wherein the remote speaker is one of a plurality of remote speakers, and wherein determining a location of each remote speaker of the plurality of remote speakers comprises:

sending a first channel test audio signal, receiving an output from a first remote speaker via the plurality of microphones, and determining a location of the first remote speaker based on signals from the plurality of microphones; and

sending a second channel test audio signal, receiving an output from a second remote speaker via the plurality of microphones, and determining a location of the second remote speaker based on signals from the plurality of microphones.

10. The method of claim 1, wherein the visual content comprises augmented reality content.

11. The method of claim 1, wherein sending the audio signal to the remote speaker comprises sending compressed or uncompressed rendered audio data.

12. The method of claim 1, wherein sending the audio signal to the remote speaker comprises sending content information for rendering audio data on the receiver.

13. A portable computing device, comprising:

one or more sensors;

a display;

a logic machine; and

a storage machine holding instructions executable by the logic machine to;

generate a model of a virtual environment including a virtual object and audio content associated with the virtual object;

present visual content on the display, the visual content including the virtual object;

determine a location of a plurality of remote speakers in a real-world environment around the portable computing device based on data from the one or more sensors;

identify an audio channel corresponding to each remote speaker; and

send the audio content to one or more of the remote speakers based on a location of the virtual object in the model relative to the location of the one or more remote speakers in the real-world environment.

14. The portable computing device of claim 13, wherein the one or more sensors comprise one or more of an image sensor and a microphone array.

15. The portable computing device of claim 14, wherein to determine the location of the plurality of remote speakers and identify the audio channel corresponding to each remote speaker, the instructions are executable to

send a first channel test audio signal;

receive an output from a first remote speaker via the microphone array;

determine a location of the first remote speaker based on signals from the microphone array;

map the first channel to the first remote speaker;

send a second channel test audio signal;

receive an output from a second remote speaker via the microphone array,

determine a location of the second remote speaker based on signals from the microphone array; and

map the second channel to the second remote speaker.

16. A wearable computing device, comprising:

one or more sensors;

a see-through display;

one or more speakers;

a logic machine; and

a storage machine holding instructions executable by the logic machine to

wirelessly detect a presence of a remote speaker external to the wearable computing device;

determine a location of the remote speaker based on image information collected by the one or more sensors;

establish a wireless connection with the remote speaker;

present augmented reality content via the see-through display, the augmented reality content comprising an object associated with a location in a real-world environment around the wearable computing device;

generate an audio signal corresponding to the object; and

send the audio signal to the remote speaker.

17. The wearable computing device of claim 16, wherein the instructions are executable to determine a latency in a connection between the receiver and the wearable computing device and adjust the audio signal sent to the remote speaker based on the latency.

18. The wearable computing device of claim 16, wherein the instructions are executable to determine a distance between the portable computing device and the remote speaker and adjust the audio signal based on the distance.

19. The wearable computing device of claim 16, wherein the instructions are executable to generate a second audio signal corresponding to a second object associated with a second location in a real-world environment around the wearable computing device and send the second audio signal to the one or more speakers of the wearable computing device.

20. The wearable computing device of claim 16, wherein the remote speaker is a first remote speaker, and wherein the instructions are executable to generate a second audio signal corresponding to a second object associated with a second location in a real-world environment around the wearable computing device and send the second audio signal to a second remote speaker.