Method for dynamic video magnification

• US 10,217,187 B2
• Filed: 06/03/2016
• Issued: 02/26/2019
• Est. Priority Date: 06/05/2015
• Status: Expired due to Fees

• Current Assignee: Massachusetts Institute of Technology, Qatar Foundation For Education, Science And Community Development
• Original Assignee: Massachusetts Institute of Technology, Qatar Foundation For Education, Science And Community Development
• Inventors: Elgharib, Mohamed Abdelaziz A. Mohamed, Hefeeda, Mohamed M., Freeman, William T., Durand, Frederic
• Primary Examiner(s): Wu, Xiao M
• Assistant Examiner(s): Elbinger, Steven Z
• Application Number:
  US15/579,924
• Publication Number:
  US 20180225803A1
• Time in Patent Office: 998 Days
• Field of Search: None
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/20104 Interactive definition of region of interest [ROI]
• CPC Class Codes: G06T 3/40 Scaling of whole images or parts thereof, e.g. expanding or contracting
• CPC Class Codes: G06T 5/77 Retouching; Inpainting; Scratch removal
• CPC Class Codes: G06T 7/33 using feature-based methods
• CPC Class Codes: H04N 23/68 for stable pick-up of the scene, e.g. compensating for camera body vibrations
• CPC Class Codes: H04N 5/2628 Alteration of picture size, shape, position or orientation, e.g. zooming, rotation, rolling, perspective, translation

Abstract

The method for dynamic video magnification magnifies small motions occurring simultaneously within large motions. The method involves selecting a region of interest from a video for magnification. The region of interest is warped to obtain a stabilized sequence of frames that discounts large motions. Each frame of the stabilized sequence is decomposed to a foreground layer, a background layer, and an alpha matte layer, and each of the foreground and alpha matte layers is magnified. Then a magnified sequence is generated from the magnified layers using matte inversion. Any image holes in the magnified sequence are filled in using texture synthesis. Finally, the magnified sequence is de-warped to the original space-time coordinates.

Claims

1. A method for dynamic video magnification of small motions occurring simultaneously within large motions, comprising the steps of:

selecting the small motions as a region of interest (ROI) from a video for magnification;

warping the region of interest to discount the large motions, thereby obtaining a stabilized sequence of the small motion input video frames, wherein the warping technique is selected from the group consisting of a Kanade-Lucas-Tomasi (KLT) feature tracker, an optical flow, and parametric modeling;

for each of the input video frames in the stabilized sequence;

decomposing the frame into a foreground layer, a background layer, and an alpha matte layer;

magnifying each of the foreground and alpha matte layers, wherein the magnification includes applying phase-based magnification;

generating a magnified sequence of the magnified foreground and alpha matte layers by matte inversion, wherein the generated magnified sequence includes superimposing the magnified foreground layer over the background layer to reconstruct remaining unmagnified sites;

filling in holes in the magnified sequence by texture synthesis; and

de-warping the magnified sequence to original space-time coordinates after filling in the image holes to obtain dynamic video magnification of the region of interest.