Method and apparatus for recovering audio signals from images

• US 10,129,658 B2
• Filed: 07/21/2014
• Issued: 11/13/2018
• Est. Priority Date: 07/22/2013
• Status: Active Grant

• Current Assignee: Massachusetts Institute of Technology
• Original Assignee: Massachusetts Institute of Technology
• Inventors: Rubinstein, Michael, Davis, Myers Abraham, Durand, Frederic, Freeman, William T., Wadhwa, Neal
• Primary Examiner(s): Flanders, Andrew C
• Application Number:
  US14/337,091
• Publication Number:
  US 20150319540A1
• Time in Patent Office: 1,576 Days
• CPC Class Codes: G01H 9/00 Measuring mechanical vibrations or ultrasonic, sonic or infrasonic waves by using radiation-sensitive means, e.g. optical means
• CPC Class Codes: G01H 9/002 for representing acoustic field distribution sonar systems for imaging G01S7/56, G01S15/89; acoustic holography G03H3/00
• CPC Class Codes: H04R 23/008 using optical signals for detecting or generating sound
• CPC Class Codes: H04R 3/00 Circuits for transducers , loudspeakers or microphones

Abstract

A method of recovering audio signals and corresponding apparatus according to an embodiment of the present invention using video or other sequence of images enables recovery of sound that causes vibrations of a surface. An embodiment method includes combining representations of local motions of a surface to produce a global motion signal of the surface. The local motions are captured in a series of images of features of the surface, and the global motion signal represents a sound within an environment in which the surface is located. Some embodiments compare representations of local motions of a surface to determine which motions are in-phase or out-of-phase with each other, enabling visualization of surface vibrational modes. Embodiments are passive, as compared to other forms of remote audio recovery that employ active sensing, such as laser microphone systems. Example applications for the embodiments include espionage and surveillance.

Claims

1. A method of recovering audio signals, the method comprising:

deriving representations of local motions of different portions of a surface from a series of optical images of features of the different portions of the surface, the series of optical images having been captured with the surface in a presence of a sound, and deriving the representations of local motions including decomposing each image of the series of optical images into multiple dimensions using a complex steerable pyramid structure;

combining the representations of local motions of different portions of the surface to produce a global motion signal of the surface; and

producing an audio signal of the sound based on the global motion signal of the surface having been located in the presence of the sound.

2. The method of claim 1, wherein combining the representations includes combining the representations of the local motions over rows or batches of rows of pixels in at least a subset of the images.

3. The method of claim 1, wherein combining the representations includes combining the representations over one or more entire images of the series of images.

4. The method of claim 1, wherein combining the representations includes combining the representations over a segmented region or unmasked region of the series of images.

5. The method of claim 1, further comprising filtering frequencies in the global motion signal to recover an improved-audibility audio signal of the sound.

6. The method of claim 1, further comprising removing noise from the global motion signal.

7. The method of claim 1, wherein combining the representations of local motions to produce the global motion signal of the surface includes using a transfer function to produce the global motion signal of the surface, the transfer function representing the global motion signal as a function of arbitrary incident sounds within an environment in which the surface is located.

8. The method of claim 7, further including imparting a known sound to the environment in which the surface is located to calculate the transfer function.

9. The method of claim 1, further comprising capturing the series of images using an imaging subsystem viewing the surface through an optically transparent sound barrier.

10. The method of claim 1, wherein combining the representations of the local motions includes aligning scale and orientation for each pixel in each image, the method further comprising aligning pixels temporally across a plurality of images in the series of images.

11. The method of claim 1, wherein combining the representations of local motions includes calculating, by a processor, an average or weighted average of the representations.

12. The method of claim 1, wherein combining the representations of local motions of the surface includes combining the representations over a region of one or more of the images corresponding to a region of the surface smaller in size than a wavelength of the sound causing a motion of the surface.

13. The method of claim 1, wherein the surface substantially fills an entirety of pixels of the series of images.

14. The method of claim 1, wherein combining the representations of the local motions includes combining the representations with an effective sampling frequency greater than a frame rate with which the series of images is captured.

30. The method of claim 1, wherein combining the representations of local motions captured in the series of optical images includes using a series of at least one of infrared or visible images.

31. The method of claim 1, wherein combining the representations of local motions includes using representations of local motions of the surface corresponding to sub-pixel displacements in the series of optical images.

32. The method of claim 31, wherein combining the representations of local motions further includes using representations of local motions of the surface corresponding to displacements on the order of one hundredth or one thousandth of a pixel in the series of optical images.

33. The method of claim 1, wherein combining the representations of local motions includes combining local phase variations of the representations of local motions of different portions of the surface.

34. The method of claim 1, wherein combining the representations includes combining fluctuations in value of one or more pixels of the series of optical images over time.

35. The method of claim 1, wherein combining the representations includes combining representations of local motions having been captured in the series of optical images at a frame rate on the order of 2,000 frames per second (fps) or on the order of 20,000 fps.

36. The method of claim 1, wherein combining the representations includes combining representations of local motions having been captured in the series of optical images at a frame rate on the order of 30 fps or on the order of 60 fps.

37. The method of claim 1, wherein combining the representations includes combining representations of local motions having been captured in the series of optical images with a rolling-shutter camera.

38. The method of claim 1, wherein producing the audio signal includes producing a compressed audio signal.

15. An audio signal recovery apparatus comprising:

memory configured to store representations of local motions of different portions of a surface, the local motions having been captured in a series of optical images of features of the different portions of the surface with the surface located in a presence of a sound; and

a processor configured to derive the representations of local motions by decomposing each image of the series of optical images into multiple dimensions using a complex steerable pyramid structure,

the processor further configured to combine the representations of local motions of the surface to produce a global motion signal of the surface,

the processor further configured to produce an audio signal of the sound based on the global motion signal of the surface having been located in the presence of the sound.

16. The apparatus of claim 15, wherein the processor is further configured to combine the representations of the local motions over rows or batches of rows of pixels in at least a subset of the images.

17. The apparatus of claim 15, wherein the processor is further configured to combine the representations over one or more entire images of the series of images.

18. The apparatus of claim 15, wherein the processor is further configured to combine the representations over a segmented region or unmasked region of one or more images of the series of images.

19. The apparatus of claim 15, wherein the processor is further configured to filter frequencies in the global motion signal to recover an improved-audibility audio signal of the sound.

20. The apparatus of claim 15, wherein the processor is further configured to remove noise from the global motion signal.

21. The apparatus of claim 15, wherein the processor is further configured to use a transfer function to produce the global motion signal of the surface, the transfer function representing the global motion signal as a function of arbitrary incident sounds within the environment in which the surface is located.

22. The apparatus of claim 21, further comprising a sound transducer configured to impart a known sound to the environment in which the surface is located to calculate the transfer function.

23. The apparatus of claim 15, further comprising an imaging subsystem configured to capture the images of the surface through an optically transparent sound barrier.

24. The apparatus of claim 15, wherein the processor is further configured to align scale and orientation for each pixel in each image and to align pixels temporally across a plurality of images in the series of images.

25. The apparatus of claim 15, wherein the processor is further configured to produce an average or weighted average of the representations of local motions of the surface.

26. The apparatus of claim 15, wherein the processor is configured to combine the representations over a region of one or more of the images corresponding to a region of the surface smaller in size than a wavelength of the sound causing a motion of the surface.

27. The apparatus of claim 15, wherein the surface substantially fills an entirety of pixels of the series of images.

28. The apparatus of claim 15, wherein the processor is further configured to combine the representations with an effective sampling frequency greater than a frame rate with which the series of images is captured.

39. The apparatus of claim 15, wherein the processor is further configured to produce the audio signal of the sound as a compressed audio signal.

29. A method comprising:

deriving representations of local motions of different portions of a surface from a series of optical images of features of the different portions of the surface, the series of optical images having been captured with the surface in a presence of a sound, and deriving the representations of local motions including decomposing each image of the series of optical images into multiple dimensions using a complex steerable pyramid structure;

comparing the representations of local motions of different portions of the surface to make a determination of which local motions are in-phase or out-of-phase with each other; and

recovering one or more vibration mode shapes of the surface based on the determination of which local motions are in-phase or out-of-phase with each other.