Linear-based Eulerian motion modulation

• US 10,217,218 B2
• Filed: 05/17/2018
• Issued: 02/26/2019
• Est. Priority Date: 09/07/2012
• Status: Active Grant

• Current Assignee: Massachusetts Institute of Technology, Quanta Computer
• Original Assignee: Massachusetts Institute of Technology, Quanta Computer
• Inventors: Wu, Hao-yu, Rubinstein, Michael, Shih, Eugene Inghaw, Guttag, John V., Durand, Frederic, Freeman, William T., Wadhwa, Neal
• Primary Examiner(s): Conner, Sean M
• Application Number:
  US15/982,765
• Publication Number:
  US 20180268543A1
• Time in Patent Office: 285 Days
• Field of Search: None
• CPC Class Codes: G06T 2207/20016 Hierarchical, coarse-to-fine, multiscale or multiresolution image processing; Pyramid transform
• CPC Class Codes: G06T 2207/30076 Plethysmography
• CPC Class Codes: G06T 2207/30088 Skin; Dermal
• CPC Class Codes: G06T 2207/30104 Vascular flow; Blood flow; Perfusion
• CPC Class Codes: G06T 2207/30201 Face
• CPC Class Codes: G06T 7/0012 Biomedical image inspection
• CPC Class Codes: G06T 7/0016 involving temporal comparison
• CPC Class Codes: G06T 7/254 involving subtraction of images
• CPC Class Codes: G06T 7/262 using transform domain methods, e.g. Fourier domain methods

Abstract

In an embodiment, a method converts two images to a transform representation in a transform domain. For each spatial position, the method examines coefficients representing a neighborhood of the spatial position that is spatially the same across each of the two images. The method calculates a first vector in the transform domain based on first coefficients representing the spatial position, the first vector representing change from a first to second image of the two images describing deformation. The method modifies the first vector to create a second vector in the transform domain representing amplified movement at the spatial position between the first and second images. The method calculates second coefficients based on the second vector of the transform domain. From the second coefficients, the method generates an output image showing motion amplified according to the second vector for each spatial position between the first and second images.

Claims

1. A method of amplifying temporal variation from at least two images, the method comprising:

converting at least two images to a transform representation in a transform domain;

for each particular spatial position within the at least two images, examining a plurality of coefficient values representing a neighborhood of the spatial position, the neighborhood of the spatial position being spatially the same across each of the at least two images;

calculating a first vector in the transform domain based on the plurality of coefficient values representing the particular spatial position, the first vector representing change from a first image to a second image of the at least two images describing deformation;

modifying the first vector to create a second vector in the transform domain representing amplified movement at the particular spatial position between the first and second images;

calculating a second plurality of coefficients based on the second vector of the transform domain; and

from the second plurality of coefficients, generating an output signal based on motion amplified according to the second vector for each particular spatial position between the first and second images.

2. The method of claim 1, wherein the transform representation is a complex steerable pyramid.

3. The method of claim 1, wherein the first and second pluralities of coefficient values include real and imaginary parts of the coefficients.

4. The method of claim 1, wherein the first vector is a scalar, and wherein modifying the first vector to create the second vector multiplies the scalar by a constant.

5. The method of claim 1, wherein the first vector represents a change in phase from a first image of the at least two images and the second image of the at least two images.

6. The method of claim 1, further comprising temporally bandpassing the second vector resulting in amplifying color variation between images.

7. The method of claim 1 further comprising extracting and visualizing a human pulse signal from the images.

8. A system for amplifying temporal variation from at least two images, the system comprising:

a transform module configured to convert at least two images to a transform representation in a transform domain;

a coefficient examination module configured to, for each particular spatial position within the at least two images, examine plurality of coefficient values representing a neighborhood of the spatial position, the neighborhood of the spatial position being spatially the same across each of the at least two images;

a vector calculation module configured to calculate a first vector in the transform domain based on the plurality of coefficient values representing the particular spatial position, the first vector representing change from a first image to a second image of the at least two images describing deformation;

a vector modification module configured to modify the first vector to create a second vector in the transform domain representing amplified movement at the particular spatial position between the first and second images;

a calculation module configured to calculate a second plurality of coefficients based on the second vector in the transform domain; and

a modified signal generation module configured to generate at least one signal based on the second plurality of coefficients, the generated at least one signal based on amplified motion according to the second vector for each particular spatial position between the first and second images.

9. The system of claim 8, wherein the transform representation is a complex steerable pyramid.

10. The system of claim 8, wherein the first and second pluralities of coefficient values include real and imaginary parts of the coefficients.

11. The system of claim 8, wherein the first vector is a scalar, and wherein the vector modification module is further configured to modify the first vector to create the second vector by multiplying the scalar by a constant.

12. The system of claim 8, further comprising a bandpass module configured to temporally bandpass the second vector resulting in amplifying color variations between images.

13. The system of claim 8, wherein the bandpass module is further configured to extract and visualize human pulse signals from the images.

14. A method of amplifying temporal variation from at least two images, the method comprising:

for each particular pixel of the at least two images, detecting a temporal variation of the particular pixel of the at least two images by applying a bandpass filter, the resulting detected temporal variation of the particular pixel between the two images being below a particular threshold;

amplifying the detected temporal variation of each particular pixel; and

providing an output signal by adding the amplified temporal variation to the at least two images by synthesizing the modulated images, for an amplification factor α

, according to ƒ

(x+(1+α

) δ

(t)), δ

(t) being a displacement function.

15. The method of claim 14, further comprising visualizing a pattern of flow of blood in a body shown in the at least two images.

16. The method of claim 14, wherein applying the bandpass filter includes, for each pixel value, analyzing a one-dimensional (1D) image intensity based on a function ƒ

(x+δ

(t)), where δ

(t) is a displacement function.

17. A system for amplifying temporal variation from at least two images, the system comprising:

a pixel examination module configured to, for each particular pixel of the at least two images, detecting a temporal variation of the particular pixel of the at least two images by applying a bandpass filter, the resulting detected temporal variation between the two images being below a particular threshold; and

a signal processing module configured to;

amplify the detected temporal variation of each particular pixel, and

provide an output signal by adding the amplified temporal variation to the at least two images by synthesizing the modulated images for an amplification factor α

, according to ƒ

(x+(1+α

) δ

(t)), δ

(t) being a displacement function.

18. The system of claim 17, further comprising a blood flow visualization module configured to visualize a pattern of flow of blood in a body shown in the at least two images.

19. The system of claim 17, wherein the pixel examination module is further configured to, for each pixel value, analyze a one-dimensional (1D) image intensity profile based on a function ƒ

(x+δ

(t)), where δ

(t) is a displacement function.