Riesz Pyramids For Fast Phase-Based Video Magnification

• US 20150195430A1
• Filed: 01/08/2015
• Published: 07/09/2015
• Est. Priority Date: 01/09/2014
• Status: Active Grant

• Current Assignee: Massachusetts Institute of Technology
• Original Assignee: Massachusetts Institute of Technology
• Inventors: Wadhwa, Neal, Durand, Frederic, Freeman, William T., Rubinstein, Michael
• Granted Patent: US 9,338,331 B2
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: G06T 3/40 Scaling of whole images or parts thereof, e.g. expanding or contracting
• CPC Class Codes: G09G 2340/045 Zooming at least part of an image, i.e. enlarging it or shrinking it
• CPC Class Codes: H04N 5/144 Movement detection for video coding H04N19/503; analysis of motion in general G06T7/20
• CPC Class Codes: H04N 5/20 Circuitry for controlling amplitude response
• CPC Class Codes: H04N 5/213 Circuitry for suppressing or minimising impulsive noise for suppressing or minimising disturbance in image signal generation H04N23/81

Abstract

Some embodiments are directed to a method, corresponding system, and corresponding apparatus for rendering a video and/or image display to amplify small motions through video magnification. Some embodiments include a new compact image pyramid representation, the Riesz pyramid, that may be used for real-time, high-quality phase-based video magnification. Some embodiments are less overcomplete than even the smallest two orientation, octave-bandwidth complex steerable pyramid. Some embodiments are implemented using compact, efficient linear filters in the spatial domain. Some embodiments produce motion magnified videos that are of comparable quality to those using the complex steerable pyramid. In some embodiments, the Riesz pyramid is used with phase-based video magnification. The Riesz pyramid may phase-shift image features along their dominant orientation, rather than along every orientation like the complex steerable pyramid.

Claims

1. A computer-implemented method of rendering a video display, comprising the computer-implemented steps, by a processor and associated memory:

receiving a video stream formed of a sequence of one or more frames, each frame having image content;

for each given frame of the sequence of frames;

generating a first set of images, each image of the first set corresponding to the given frame, and each image of the first set including a unique spatial frequency band;

downsampling at least one image of the first set one or more times;

applying a Riesz transform to at least one of the downsampled images;

generating an image representation, based upon the application of the Riesz transform, the image representation including a second set of images, at least one image of the second set being horizontally phase shifted with respect to a given corresponding image of the first set, the image representation including a third set of images, at least one image of the third set being vertically phase shifted with respect to a given corresponding image of the first set;

computing a local phase for at least one image of the image representation;

generating a temporally filtered local phase based upon applying one or more temporal band pass filters to the local phase and based upon one or more previous local phases;

generating an updated image representation based upon the temporally filtered local phase and the at least one image of the image representation;

amplifying one or more images of the updated image representation;

constructing a given replacement image based upon the amplified image representation, the replacement image corresponding to the given frame and the replacement image exhibiting amplified motions as compared with the given frame; and

displaying the given replacement image.

2. The computer-implemented method of claim 1, further comprising:

selecting, by a user, one or more preferred temporal frequencies of the image representation, and applying the selected temporal frequencies to the one or more temporal band pass filters.

3. The computer-implemented method of claim 1, wherein the one or more temporal band pass filters include at least one noise filter.

4. The computer-implemented method of claim 1, further comprising:

applying one or more noise filters to the temporally filtered image representation, at least one of the one or more noise filters performing spatial blurring of the temporally filtered image representation.

5. The computer-implemented method of claim 1, wherein generating the first set of images and downsampling the first set of images are performed based on a Laplacian Pyramid.

6. The computer-implemented method of claim 1, wherein the Riesz transform includes two three-tap linear filters.

7. The computer-implemented method of claim 1, wherein the video display includes a real-time video display.

8. The computer-implemented method of claim 1, wherein the video display includes a two-dimensional (2D) video display.

9. The computer-implemented method of claim 1, wherein the video display includes an image display and the video stream includes an image stream.

10. The computer-implemented method of claim 1, further comprising:

for each given frame of the sequence of frames;

generating an output video stream, the output video stream including each given replacement image that corresponds to each given frame of the sequence of frames.

11. The computer-implemented method of claim 1, wherein the computer-implemented steps are performed at a speed that is at least a factor of 3.5 compared with, and with equivalent visible quality to, a method for motion amplification that uses a complex steerable pyramid.

12. The computer-implemented method of claim 1, further comprising

generating the temporally filtered local phase based upon applying the one or more temporal band pass filters to the local phase and based upon one or more previous local phases that are associated with one or more previous image representations.

13. A computer-implemented system for rendering a video display, comprising:

a receiving module configured to receive a video stream formed of a sequence of one or more frames, each frame having image content;

a computing module, for each given frame of the sequence of frames, the computing module being configured to;

generate a first set of images, each image of the first set corresponding to the given frame, and each image of the first set including a unique spatial frequency band;

downsample at least one image of the first set one or more times;

apply a Riesz transform to at least one of the downsampled images;

generate an image representation, based upon the application of the Riesz transform, the image representation including a second set of images, at least one image of the second set being horizontally phase shifted with respect to a given corresponding image of the first set, the image representation including a third set of images, at least one image of the third set being vertically phase shifted with respect to a given corresponding image of the first set;

compute a local phase for at least one image of the image representation;

generate a temporally filtered local phase based upon applying one or more temporal band pass filters to the local phase;

generate an updated image representation based upon the temporally filtered local phase and the at least one image of the image representation;

amplify one or more images of the updated image representation;

construct a given replacement image based upon the amplified image representation, the replacement image corresponding to the given frame and the replacement image exhibiting amplified motions as compared with the given frame; and

a display module configured to display the given replacement image.

14. The computer-implemented system of claim 13, wherein the computing module is further configured to select, by a user, one or more preferred temporal frequencies of the image representation, and apply the selected temporal frequencies to the one or more temporal band pass filters.

15. The computer-implemented system of claim 13, wherein the one or more temporal band pass filters include at least one noise filter.

16. The computer-implemented system of claim 13, wherein the computing module is further configured to apply one or more noise filters to the temporally filtered image representation, at least one of the one or more noise filters configured to perform spatial blurring of the temporally filtered image representation.

17. The computer-implemented system of claim 13, wherein the computing module is further configured to generate the first set of images and downsample the first set of images based on a Laplacian Pyramid.

18. The computer-implemented system of claim 13, wherein the Riesz transform includes two three-tap linear filters.

19. The computer-implemented system of claim 13, wherein the video display includes a real-time video display.

20. The computer-implemented system of claim 13, wherein the video display includes a two-dimensional (2D) video display.

21. The computer-implemented system of claim 13, wherein the video display includes an image display and the video stream includes an image stream.

22. The computer-implemented system of claim 13, wherein the computing module is further configured to generate, for each given frame of the sequence of frames, an output video stream, the output video stream including each given replacement image that corresponds to each given frame of the sequence of frames.

23. A non-transitory computer readable medium having stored thereon a sequence of instructions which, when loaded and executed by a processor coupled to an apparatus, causes the apparatus to:

receive a video stream formed of a sequence of one or more frames, each frame having image content;

for each given frame of the sequence of frames;

generate a first set of images, each image of the first set corresponding to the given frame, and each image of the first set including a unique spatial frequency band;

downsample at least one image of the first set one or more times;

apply a Riesz transform to at least one of the downsampled images;

generate an image representation, based upon the application of the Riesz transform, the image representation including a second set of images, at least one image of the second set being horizontally phase shifted with respect to a given corresponding image of the first set, the image representation including a third set of images, at least one image of the third set being vertically phase shifted with respect to a given corresponding image of the first set;

compute a local phase for at least one image of the image representation;

generate a temporally filtered local phase based upon applying one or more temporal band pass filters to the local phase;

generate an updated image representation based upon the temporally filtered local phase and the at least one image of the image representation;

amplify one or more images of the updated image representation;

construct a given replacement image based upon the amplified image representation, the replacement image corresponding to the given frame and the replacement image exhibiting amplified motions as compared with the given frame; and

display the given replacement image.