Microphone array for generating virtual sound field

• US 10,206,040 B2
• Filed: 07/13/2017
• Issued: 02/12/2019
• Est. Priority Date: 10/30/2015
• Status: Expired due to Fees

• Current Assignee: Essential Products, Inc.
• Original Assignee: Essential Products, Inc.
• Inventors: Kolb, Michael, Jiang, Xinrui, Rubin, Andrew E., Hershenson, Matthew, Miao, Xiaoyu, Tate, Joseph Anthony, Gagne-Keats, Jason Sean, Evans, V, David John, Zavin, Rebecca Schultz
• Primary Examiner(s): Aghevli, Reza
• Application Number:
  US15/649,640
• Publication Number:
  US 20170311080A1
• Time in Patent Office: 579 Days
• Field of Search: None
• CPC Class Codes: G06F 16/58 Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually
• CPC Class Codes: G06F 16/583 using metadata automatically derived from the content
• CPC Class Codes: G06F 16/5866 using information manually generated, e.g. tags, keywords, comments, manually generated location and time information
• CPC Class Codes: G06F 16/587 using geographical or spatial information, e.g. location
• CPC Class Codes: G06F 16/683 using metadata automatically derived from the content
• CPC Class Codes: G06F 21/602 Providing cryptographic facilities or services
• CPC Class Codes: G06T 11/60 Editing figures and text; Combining figures or text
• CPC Class Codes: G06T 2200/32 involving image mosaicing
• CPC Class Codes: G06T 2207/10012 Stereo images
• CPC Class Codes: G06T 2207/20221 Image fusion; Image merging
• CPC Class Codes: G06T 5/80 Geometric correction
• CPC Class Codes: G06T 7/33 using feature-based methods
• CPC Class Codes: G11B 20/00992 Circuits for stereophonic or quadraphonic recording or reproducing
• CPC Class Codes: H04N 23/45 for generating image signals from two or more image sensors being of different type or operating in different modes, e.g. with a CMOS sensor for moving images in combination with a charge-coupled device [CCD] for still images
• CPC Class Codes: H04N 23/55 Optical parts specially adapted for electronic image sensors; Mounting thereof
• CPC Class Codes: H04N 23/63 by using electronic viewfinders
• CPC Class Codes: H04N 23/698 for achieving an enlarged field of view, e.g. panoramic image capture
• CPC Class Codes: H04N 23/90 Arrangement of cameras or camera modules, e.g. multiple cameras in TV studios or sports stadiums
• CPC Class Codes: H04N 5/77 between a recording apparatus and a television camera
• CPC Class Codes: H04R 1/342 for microphones
• CPC Class Codes: H04R 1/406 : microphones
• CPC Class Codes: H04R 3/005 : for combining the signals of two or more microphones specially adapted for hearing aids H04R25/407
• CPC Class Codes: H04R 5/027 : Spatial or constructional arrangements of microphones, e.g. in dummy heads
• CPC Class Codes: H04S 2400/15 : Aspects of sound capture and related signal processing for recording or reproduction
• CPC Class Codes: H04S 2420/11 : Application of ambisonics in stereophonic audio systems
• CPC Class Codes: H04S 7/30 : Control circuits for electronic adaptation of the sound field

Abstract

Certain aspects of the technology disclosed herein include generating a virtual sound field based on data from an ambisonic recording device. The ambisonic device records sound of a surrounding environment using at least four microphones having a tetrahedral orientation. An omnidirectional microphone having an audio-isolated portion can be used to isolate sound from a particular direction. Sound received from the plurality of microphones can be used to generate a virtual sound field. The virtual sound field include a dataset indicating a pressure signal and a plurality of velocity vectors. The ambisonic recording device can include a wide angle camera and generate wide angle video corresponding to the virtual sound field.

Claims

1. A method comprising:

receiving independent recordings from a plurality of microphones disposed in a tetrahedral arrangement around a recording device;

generating a virtual sound field by mapping velocity vectors to a determined spatial orientation of the recording device, wherein the velocity vectors are generated by employing a transfer function accounting for an angular difference between each direction and the plurality of microphones disposed around the recording device;

merging the virtual sound field with an integrated image of a surrounding environment by mapping the virtual sound field to the integrated image; and

isolating a portion of the virtual sound field and a portion of the integrated image corresponding to a predicted spatial orientation of a user.

2. The method of claim 1, wherein the plurality of microphones include at least one omnidirectional microphone.

3. The method of claim 1, wherein a portion of at least one omnidirectional microphone is acoustically shielded by an acoustic insulator and/or an acoustic reflector.

4. The method of claim 1, wherein the determined spatial orientation of the recording device is determined based on motion sensor data.

5. The method of claim 4, wherein the motion sensor data is received from any of an accelerometer, a compass, and a gyroscope.

6. The method of claim 5, wherein the accelerometer comprises a piezoelectric component, a piezoresistive component, a capacitive component, or any combination thereof.

7. The method of claim 5, wherein the gyroscope comprises a mechanical gyroscope, a microelectromechanical system gyroscope, a solid-state ring laser, a fiber optic gyroscope, a quantum gyroscope, or any combination thereof.

8. The method of claim 1, wherein the integrated image of the surrounding environment is generated by:

determining a pixel correspondence between a first image among a first plurality of images and a second image among a second plurality of images, based on a corresponding overlap area associated with the first image and the second image; and

combining the first image and the second image having the pixel correspondence.

9. The method of claim 8, wherein the said determining the pixel correspondence comprises:

identifying a plurality of overlap areas associated with the first image among the first plurality of images and the second image among the second plurality of images;

calculating a plurality of first match scores corresponding to the plurality of overlap areas;

identifying an overlap area of the plurality of overlap areas having a highest first match score of the plurality of first match scores as the corresponding overlap area;

identifying one or more first set of pixels in the corresponding overlap area of the first image and one or more second set of pixels in the corresponding overlap area of the second image;

calculating a plurality of second match scores corresponding to the one or more first set of pixels and the one or more second set of pixels; and

identifying the one or more first set of pixels and the one or more second set of pixels having a highest second match score of the plurality of second match scores as corresponding pixels.

10. The method of claim 8, wherein said combining the first plurality of images and the second plurality of images comprises:

merging corresponding pixels in the corresponding overlap area, said merging comprising;

for each pixel in the first image associated with the corresponding overlap area, assigning a first weight ranging from 0 to 1 inclusive, and wherein the first weight is associated with a determined distortion of a pixel in the first image; and

for each corresponding pixel in the second image associated with the corresponding overlap area, weighing the pixel by a second weight, wherein the second weight increases as the first weight decreases;

wherein the first weight and the second weight dictate a proportional contribution of each pixel to values of a merged pixel; and

tiling the first image, the second image, and the corresponding overlap area to obtain a 360°

image.

11. The method of claim 1, further comprising:

a first camera and a second camera disposed on the recording device.

12. The method of claim 11, wherein at least one microphone among the plurality of microphones is disposed adjacent to each of the first camera and the second camera.

13. The method of claim 1, wherein the velocity vectors correspond to a plurality of directions relative to the recording device.

14. The method of claim 1, wherein a direction corresponding with each of the velocity vectors is determined based on the determined spatial orientation of the recording device.

15. The method of claim 1, further comprising:

receiving a play request associated with the virtual sound field; and

in response to the play request, identifying a portion of the virtual sound field corresponding to a current spatial orientation of the user.

16. The method of claim 1, further comprising:

predicting a position of the user based on a determined spatial orientation of one or more devices; and

iteratively updating a portion of the virtual sound field corresponding to a current spatial orientation of the user.

17. The method of claim 1, further comprising:

transmitting the portion of the virtual sound field to a speaker and the portion of the integrated image to a display to cause the speaker and the display to synchronize playback.