Distributed audio capturing techniques for virtual reality (VR), augmented reality (AR), and mixed reality (MR) systems

• US 10,531,220 B2
• Filed: 11/14/2017
• Issued: 01/07/2020
• Est. Priority Date: 12/05/2016
• Status: Active Grant

• Current Assignee: Magic Leap, Inc.
• Original Assignee: Magic Leap, Inc.
• Inventors: Sanger, George A., Schmidt, Brian Lloyd, Tajik, Anastasia A., O'Gara, Terry Micheal, Shumway, David Matthew, Howarth, Alan Steven
• Primary Examiner(s): Lao, Lun-See
• Application Number:
  US15/813,020
• Publication Number:
  US 20180160251A1
• Time in Patent Office: 784 Days
• Field of Search: 381303, 381 55- 58, 700 94
• CPC Class Codes: H04R 2430/21 Direction finding using differential microphone array [DMA]
• CPC Class Codes: H04R 2460/07 Use of position data from wide-area or local-area positioning systems in hearing devices, e.g. program or information selection
• CPC Class Codes: H04R 3/005 for combining the signals of two or more microphones specially adapted for hearing aids H04R25/407
• CPC Class Codes: H04R 5/027 Spatial or constructional arrangements of microphones, e.g. in dummy heads
• CPC Class Codes: H04R 5/04 Circuit arrangements, e.g. for selective connection of amplifier inputs/outputs to loudspeakers, for loudspeaker detection, or for adaptation of settings to personal preferences or hearing impairments combinations of amplifiers H03F3/68; stereophonic systems H04S
• CPC Class Codes: H04S 2400/11 Positioning of individual sound objects, e.g. moving airplane, within a sound field H04S2420/13 takes precedence
• CPC Class Codes: H04S 2400/15 Aspects of sound capture and related signal processing for recording or reproduction
• CPC Class Codes: H04S 3/008 in which the audio signals are in digital form, i.e. employing more than two discrete digital channels data reduction aspects thereof based on psychoacoustics G10L19/02
• CPC Class Codes: H04S 7/303 Tracking of listener position or orientation

Abstract

Systems and methods for capturing audio which can be used in applications such as virtual reality, augmented reality, and mixed reality systems. Some systems may include a plurality of distributed monitoring devices in an environment, each having a microphone and a location tracking unit. The system can capture audio signals while also capturing location tracking signals which indicate the locations of the monitoring devices over time during capture of the audio signals. The system can generate a representation of at least a portion of a sound wave field in the environment based on the audio signals and the location tracking signals. The system may also determine one or more acoustic properties of the environment based on the audio signals and the location tracking signals.

Claims

1. A system comprising:

a plurality of distributed monitoring devices, each monitoring device comprising at least one microphone and a location tracking unit, wherein the monitoring devices are configured to capture a plurality of audio signals from a sound source and to capture a plurality of location tracking signals which respectively indicate the locations of the monitoring devices over time during capture of the plurality of audio signals, and wherein there is a non-predetermined relative spatial relationship between the plurality of distributed monitoring devices; and

a processor configured to receive the plurality of audio signals and the plurality of location tracking signals, the processor being further configured to generate a representation of at least a portion of a sound wave field created by the sound source, for a simulated specific spatial point of interest, based on the audio signals and the location tracking signals.

2. The system of claim 1, wherein the plurality of distributed monitoring devices are mobile.

3. The system of claim 1, wherein the location tracking unit comprises a Global Positioning System (GPS).

4. The system of claim 1, wherein the representation of the sound wave field comprises sound values at each of a plurality of spatial points on a grid for a plurality of times.

5. The system of claim 1, wherein the processor is further configured to determine the location of the sound source.

6. The system of claim 1, wherein the processor is further configured to map the sound wave field to a virtual, augmented, or mixed reality environment.

7. The system of claim 1, wherein, using the representation of the sound wave field, the processor is further configured to determine a virtual audio signal at a selected location within the sound wave field, the virtual audio signal estimating an audio signal which would have been detected by a microphone at the selected location.

8. The system of claim 7, wherein the location is selected based on the location of a user of a virtual, augmented, or mixed reality system within a virtual or augmented reality environment.

9. A device comprising:

a processor configured to carry out a method comprising receiving, from a plurality of distributed monitoring devices, a plurality of audio signals captured from a sound source;

receiving, from the plurality of monitoring devices, a plurality of location tracking signals, the plurality of location tracking signals respectively indicating the locations of the monitoring devices over time during capture of the plurality of audio signals, wherein there is a non-predetermined relative spatial relationship between the plurality of distributed monitoring devices;

generating a representation of at least a portion of a sound wave field created by the sound source, for a simulated specific spatial point of interest, based on the audio signals and the location tracking signals; and

a memory to store the audio signals and the location tracking signals.

10. The device of claim 9, wherein the plurality of distributed monitoring devices are mobile.

11. The device of claim 9, wherein the representation of the sound wave field comprises sound values at each of a plurality of spatial points on a grid for a plurality of times.

12. The device of claim 9, wherein the processor is further configured to determine the location of the sound source.

13. The device of claim 9, wherein the processor is further configured to map the sound wave field to a virtual, augmented, or mixed reality environment.

14. The device of claim 9, wherein, using the representation of the sound wave field, the processor is further configured to determine a virtual audio signal at a selected location within the sound wave field, the virtual audio signal estimating an audio signal which would have been detected by a microphone at the selected location.

15. The device of claim 14, wherein the location is selected based on the location of a user of a virtual, augmented, or mixed reality system within a virtual or augmented reality environment.

16. A method comprising:

receiving, from a plurality of distributed monitoring devices, a plurality of audio signals captured from a sound source;

receiving, from the plurality of monitoring devices, a plurality of location tracking signals, the plurality of location tracking signals respectively indicating the locations of the monitoring devices over time during capture of the plurality of audio signals, wherein there is a non-predetermined relative spatial relationship between the plurality of distributed monitoring devices;

generating a representation of at least a portion of a sound wave field created by the sound source, for a simulated specific spatial point of interest, based on the audio signals and the location tracking signals.

17. The method of claim 16, wherein the plurality of distributed monitoring devices are mobile.

18. The method of claim 16, wherein the representation of the sound wave field comprises sound values at each of a plurality of spatial points on a grid for a plurality of times.

19. The method of claim 16, further comprising determining the location of the sound source.

20. The method of claim 16, further comprising mapping the sound wave field to a virtual, augmented, or mixed reality environment.

21. The method of claim 16, further comprising, using the representation of the sound wave field, determining a virtual audio signal at a selected location within the sound wave field, the virtual audio signal estimating an audio signal which would have been detected by a microphone at the selected location.

22. The method of claim 21, wherein the location is selected based on the location of a user of a virtual, augmented, or mixed reality system within a virtual or augmented reality environment.