Riesz pyramids for fast phase-based video magnification
First Claim
1. A computer-implemented method of rendering a video display, comprising the computer-implemented steps, by a processor and associated memory:
- receiving a video stream formed of a sequence of one or more frames, each frame having image content;
for each given frame of the sequence of frames;
generating a first set of images, each image of the first set corresponding to the given frame, and each image of the first set including a unique spatial frequency band;
downsampling at least one image of the first set one or more times;
applying a Riesz transform to at least one of the downsampled images;
generating an image representation, based upon the application of the Riesz transform, the image representation including a second set of images, at least one image of the second set being horizontally phase shifted with respect to a given corresponding image of the first set, the image representation including a third set of images, at least one image of the third set being vertically phase shifted with respect to a given corresponding image of the first set;
computing a local phase for at least one image of the image representation;
generating a temporally filtered local phase based upon applying one or more temporal band pass filters to the local phase and based upon one or more previous local phases;
generating an updated image representation based upon the temporally filtered local phase and the at least one image of the image representation;
amplifying one or more images of the updated image representation;
constructing a given replacement image based upon the amplified image representation, the replacement image corresponding to the given frame and the replacement image exhibiting amplified motions as compared with the given frame; and
displaying the given replacement image.
2 Assignments
0 Petitions
Accused Products
Abstract
Some embodiments are directed to a method, corresponding system, and corresponding apparatus for rendering a video and/or image display to amplify small motions through video magnification. Some embodiments include a new compact image pyramid representation, the Riesz pyramid, that may be used for real-time, high-quality phase-based video magnification. Some embodiments are less overcomplete than even the smallest two orientation, octave-bandwidth complex steerable pyramid. Some embodiments are implemented using compact, efficient linear filters in the spatial domain. Some embodiments produce motion magnified videos that are of comparable quality to those using the complex steerable pyramid. In some embodiments, the Riesz pyramid is used with phase-based video magnification. The Riesz pyramid may phase-shift image features along their dominant orientation, rather than along every orientation like the complex steerable pyramid.
-
Citations
23 Claims
-
1. A computer-implemented method of rendering a video display, comprising the computer-implemented steps, by a processor and associated memory:
-
receiving a video stream formed of a sequence of one or more frames, each frame having image content; for each given frame of the sequence of frames; generating a first set of images, each image of the first set corresponding to the given frame, and each image of the first set including a unique spatial frequency band; downsampling at least one image of the first set one or more times; applying a Riesz transform to at least one of the downsampled images; generating an image representation, based upon the application of the Riesz transform, the image representation including a second set of images, at least one image of the second set being horizontally phase shifted with respect to a given corresponding image of the first set, the image representation including a third set of images, at least one image of the third set being vertically phase shifted with respect to a given corresponding image of the first set; computing a local phase for at least one image of the image representation; generating a temporally filtered local phase based upon applying one or more temporal band pass filters to the local phase and based upon one or more previous local phases; generating an updated image representation based upon the temporally filtered local phase and the at least one image of the image representation; amplifying one or more images of the updated image representation; constructing a given replacement image based upon the amplified image representation, the replacement image corresponding to the given frame and the replacement image exhibiting amplified motions as compared with the given frame; and displaying the given replacement image. - View Dependent Claims (2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12)
-
-
13. A computer-implemented system for rendering a video display, comprising:
-
a receiving module configured to receive a video stream formed of a sequence of one or more frames, each frame having image content; a computing module, for each given frame of the sequence of frames, the computing module being configured to; generate a first set of images, each image of the first set corresponding to the given frame, and each image of the first set including a unique spatial frequency band; downsample at least one image of the first set one or more times; apply a Riesz transform to at least one of the downsampled images; generate an image representation, based upon the application of the Riesz transform, the image representation including a second set of images, at least one image of the second set being horizontally phase shifted with respect to a given corresponding image of the first set, the image representation including a third set of images, at least one image of the third set being vertically phase shifted with respect to a given corresponding image of the first set; compute a local phase for at least one image of the image representation; generate a temporally filtered local phase based upon applying one or more temporal band pass filters to the local phase; generate an updated image representation based upon the temporally filtered local phase and the at least one image of the image representation; amplify one or more images of the updated image representation; construct a given replacement image based upon the amplified image representation, the replacement image corresponding to the given frame and the replacement image exhibiting amplified motions as compared with the given frame; and a display module configured to display the given replacement image. - View Dependent Claims (14, 15, 16, 17, 18, 19, 20, 21, 22)
-
-
23. A non-transitory computer readable medium having stored thereon a sequence of instructions which, when loaded and executed by a processor coupled to an apparatus, causes the apparatus to:
-
receive a video stream formed of a sequence of one or more frames, each frame having image content; for each given frame of the sequence of frames; generate a first set of images, each image of the first set corresponding to the given frame, and each image of the first set including a unique spatial frequency band; downsample at least one image of the first set one or more times; apply a Riesz transform to at least one of the downsampled images; generate an image representation, based upon the application of the Riesz transform, the image representation including a second set of images, at least one image of the second set being horizontally phase shifted with respect to a given corresponding image of the first set, the image representation including a third set of images, at least one image of the third set being vertically phase shifted with respect to a given corresponding image of the first set; compute a local phase for at least one image of the image representation; generate a temporally filtered local phase based upon applying one or more temporal band pass filters to the local phase; generate an updated image representation based upon the temporally filtered local phase and the at least one image of the image representation; amplify one or more images of the updated image representation; construct a given replacement image based upon the amplified image representation, the replacement image corresponding to the given frame and the replacement image exhibiting amplified motions as compared with the given frame; and display the given replacement image.
-
Specification