Hyper-lapse video through time-lapse and stabilization

• US 10,002,640 B2
• Filed: 06/30/2014
• Issued: 06/19/2018
• Est. Priority Date: 02/28/2014
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Kopf, Johannes Peter, Cohen, Michael F., Szeliski, Richard
• Primary Examiner(s): Selby, Gevell
• Application Number:
  US14/320,411
• Publication Number:
  US 20150248916A1
• Time in Patent Office: 1,450 Days
• Field of Search: 3482081
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/20182 Noise reduction or smoothing in the temporal domain; Spatio-temporal filtering
• CPC Class Codes: G06T 2207/20221 Image fusion; Image merging
• CPC Class Codes: G06T 2207/30241 Trajectory
• CPC Class Codes: G06T 5/50 using two or more images, e.g. averaging or subtraction
• CPC Class Codes: G06T 5/70 Denoising; Smoothing
• CPC Class Codes: G06T 7/30 Determination of transform parameters for the alignment of images, i.e. image registration
• CPC Class Codes: G06T 7/557 from light fields, e.g. from plenoptic cameras
• CPC Class Codes: G06T 7/579 from motion
• CPC Class Codes: G11B 27/02 Editing, e.g. varying the order of information signals recorded on, or reproduced from, record carriers
• CPC Class Codes: G11B 27/034 on discs G11B27/036, G11B27/038 take precedence
• CPC Class Codes: H04N 23/683 performed by a processor, e.g. controlling the readout of an image memory
• CPC Class Codes: H04N 5/262 Studio circuits, e.g. for mixing, switching-over, change of character of image, other special effects ; Cameras specially adapted for the electronic generation of special effects
• CPC Class Codes: H04N 5/7605 on discs or drums H04N5/781, H04N5/805, H04N5/83, H04N5/85 take precedence

Abstract

Various technologies described herein pertain to generation of an output hyper-lapse video from an input video. A smoothed camera path can be computed based upon the input video. Further, output camera poses can be selected from the smoothed camera path for output frames of the output hyper-lapse video. One or more selected input frames from the input video can be chosen for an output frame. The selected input frames can be chosen based at least in part upon an output camera pose for the output frame. Moreover, the selected input frames can be combined to render the output frame. Choosing selected input frames from the input video and combining the selected input frames can be performed for each of the output frames of the output hyper-lapse video.

Claims

1. A computing system, comprising:

at least one processor; and

memory that comprises computer-executable instructions that, when executed by the at least one processor, cause the at least one processor to perform acts including;

computing a smoothed camera path based upon an input video that captures a period of time during which a camera moves through a space;

selecting output camera poses from the smoothed camera path for output frames of an output hyper-lapse video, the output camera poses for the output frames of the output hyper-lapse video being selected to speed up the period of time during which the camera moves through the space in the output hyper-lapse video relative to the input video, the output camera poses comprise output camera positions and output camera orientations for the output frames of the output-hyper lapse video, the output camera positions being evenly spaced along the smoothed camera path in arc-length; and

for each of the output frames of the output hyper-lapse video;

choosing one or more selected input frames from the input video for an output frame, the selected input frames chosen based at least in part upon an output camera position and an output camera orientation for the output frame; and

combining the selected input frames to render the output frame.

2. The computing system of claim 1, the memory further comprises computer-executable instructions that, when executed by the at least one processor, cause the at least one processor to perform acts including:

constructing a model of a scene and input camera poses for input frames of the input video, wherein the smoothed camera path is computed based upon the model of the scene and the input camera poses for the input frames of the input video.

3. The computing system of claim 1, the smoothed camera path being computed as a function of:

a length of the smoothed camera path;

position smoothness of the smoothed camera path;

orientation smoothness of the smoothed camera path;

proximity of the smoothed camera path to input camera positions for the input video; and

a view quality metric indicative of image-based rendering quality from a model of a scene along the smoothed camera path.

4. The computing system of claim 3, wherein:

the smoothed camera path comprises an output camera position curve and an output camera orientation curve; and

the memory further comprises computer-executable instructions that, when executed by the at least one processor, cause the at least one processor to perform acts including;

computing the output camera position curve as a function of the length of the smoothed camera path, the position smoothness of the smoothed camera path, and the proximity of the smoothed camera path to the input camera positions for the input video; and

computing the output camera orientation curve as a function of the orientation smoothness of the smoothed camera path and the view quality metric, wherein the output camera orientation curve is computed with the output camera position curve remaining fixed as previously computed.

5. The computing system of claim 4, the memory further comprises computer-executable instructions that, when executed by the at least one processor, cause the at least one processor to perform acts including:

pre-computing the view quality metric.

6. The computing system of claim 1, the memory further comprises computer-executable instructions that, when executed by the at least one processor, cause the at least one processor to perform acts including:

computing a view quality metric indicative of image-based rendering quality from a model of a scene along the smoothed camera path, the smoothed camera path being computed based upon the view quality metric, the view quality metric being a view-independent measure of an amount of foreshortening induced by texture mapping an original image onto a proxy geometry.

7. The computing system of claim 1, the memory further comprises computer-executable instructions that, when executed by the at least one processor, cause the at least one processor to perform acts including:

identifying candidate input frames for the output frame, the candidate input frames identified from the input frames of the input video;

generating respective weight maps for the candidate input frames; and

choosing the selected input frames for the output frame from the candidate input frames based upon the weight maps.

8. The computing system of claim 7, the memory further comprises computer-executable instructions that, when executed by the at least one processor, cause the at least one processor to perform acts including:

adjusting the respective weight maps for the candidate input frames based upon a per-pixel blur measure and pixel occlusion.

9. The computing system of claim 1, the memory further comprises computer-executable instructions that, when executed by the at least one processor, cause the at least one processor to perform acts including:

combining the selected input frames for the output frame and selected input frames chosen for output frames that surround the output frame in a frame window to render the output frame.

10. The computing system of claim 1, the memory further comprises computer-executable instructions that, when executed by the at least one processor, cause the at least one processor to perform acts including:

stitching and blending the selected input frames to render the output frame for each of the output frames, the selected input frames being stitched and blended using a temporal and spatial seam selection algorithm and Poisson blending.

11. The computing system of claim 1, the memory further comprises computer-executable instructions that, when executed by the at least one processor, cause the at least one processor to perform acts including:

receiving the input video from a client computing device by way of a network.

12. The computing system of claim 1, further comprising:

a data store;

an image sensor; and

the memory further comprises computer-executable instructions that, when executed by the at least one processor, cause the at least one processor to perform acts including;

storing output data from the image sensor in the data store as the input video.

13. A method of generating an output hyper-lapse video from an input video, comprising:

constructing a model of a scene and input camera poses for input frames of the input video;

computing a smoothed camera path based upon the model of the scene and the input camera poses for the input frames of the input video;

selecting output camera poses from the smoothed camera path for output frames of the output hyper-lapse video, the output camera poses comprise output camera positions and output camera orientations for the output frames of the output hyper-lapse video, the output camera positions being evenly spaced along the smoothed camera path in arc-length; and

for each of the output frames of the output hyper-lapse video;

choosing selected input frames from the input video for an output frame, the selected input frames chosen based at least in part upon an output camera position and an output camera orientation for the output frame; and

combining the selected input frames to render the output frame.

14. The method of claim 13, the smoothed camera path being computed as a function of:

a length of the smoothed camera path;

position smoothness of the smoothed camera path;

orientation smoothness of the smoothed camera path;

proximity of the smoothed camera path to input camera positions for the input video, the input camera poses respectively comprise the input camera positions; and

a view quality metric indicative of image-based rendering quality from the model of the scene along the smoothed camera path.

15. The method of claim 13, computing the smoothed camera path further comprises:

computing a view quality metric indicative of image-based rendering quality from the model of the scene along the smoothed camera path, the view quality metric being a view-independent measure of an amount of foreshortening induced by texture mapping an original image onto a proxy geometry; and

computing the smoothed camera path based upon the view quality metric.

16. The method of claim 13, the smoothed camera path comprises an output camera position curve and an output camera orientation curve, and computing the smoothed camera path further comprises:

computing the output camera position curve as a function of position based objectives; and

subsequent to computation of the output camera position curve, computing the output camera orientation curve as a function of orientation based objectives, the output camera orientation curve being computed with the output camera position curve fixed as previously computed.

17. The method of claim 13, choosing the selected input frames from the input video for the output frame further comprises:

identifying candidate input frames for the output frame, the candidate input frames identified from the input frames of the input video;

generating respective weight maps for the candidate input frames; and

choosing, using a greedy algorithm, the selected input frames for the output frame from the candidate input frames based upon the weight maps.

18. A computing system, comprising:

at least one processor; and

memory that comprises computer-executable instructions that, when executed by the at least one processor, cause the at least one processor to perform acts including;

computing a smoothed camera path based upon an input video that captures a period of time during which a camera moves through a space;

selecting output camera poses from the smoothed camera path for output frames of an output hyper-lapse video, the output camera poses for the output frames of the output hyper-lapse video being selected to speed up the period of time during which the camera moves through the space in the output hyper-lapse video relative to the input video, the output camera poses comprise output camera positions and output camera orientations for the output frames of the output-hyper lapse video, the output camera positions being spaced based upon input camera velocities from the input video; and

for each of the output frames of the output hyper-lapse video;

choosing one or more selected input frames from the input video for an output frame, the selected input frames chosen based at least in part upon an output camera position and an output camera orientation for the output frame; and

combining the selected input frames to render the output frame.

19. The computing system of claim 18, the memory further comprises computer-executable instructions that, when executed by the at least one processor, cause the at least one processor to perform acts including:

constructing a model of a scene and input camera poses for input frames of the input video, wherein the smoothed camera path is computed based upon the model of the scene and the input camera poses for the input frames of the input video.

20. The computing system of claim 18, the smoothed camera path being computed as a function of:

a length of the smoothed camera path;

position smoothness of the smoothed camera path;

orientation smoothness of the smoothed camera path;

proximity of the smoothed camera path to input camera positions for the input video; and

a view quality metric indicative of image-based rendering quality from a model of a scene along the smoothed camera path.

21. The computing system of claim 20, wherein:

the smoothed camera path comprises an output camera position curve and an output camera orientation curve; and

the memory further comprises computer-executable instructions that, when executed by the at least one processor, cause the at least one processor to perform acts including;

computing the output camera position curve as a function of the length of the smoothed camera path, the position smoothness of the smoothed camera path, and the proximity of the smoothed camera path to the input camera positions for the input video; and

computing the output camera orientation curve as a function of the orientation smoothness of the smoothed camera path and the view quality metric, wherein the output camera orientation curve is computed with the output camera position curve remaining fixed as previously computed.

22. The computing system of claim 21, the memory further comprises computer-executable instructions that, when executed by the at least one processor, cause the at least one processor to perform acts including:

pre-computing the view quality metric.

23. The computing system of claim 18, the memory further comprises computer-executable instructions that, when executed by the at least one processor, cause the at least one processor to perform acts including:

computing a view quality metric indicative of image-based rendering quality from a model of a scene along the smoothed camera path, the smoothed camera path being computed based upon the view quality metric, the view quality metric being a view-independent measure of an amount of foreshortening induced by texture mapping an original image onto a proxy geometry.

24. The computing system of claim 18, the memory further comprises computer-executable instructions that, when executed by the at least one processor, cause the at least one processor to perform acts including:

identifying candidate input frames for the output frame, the candidate input frames identified from the input frames of the input video;

generating respective weight maps for the candidate input frames; and

choosing the selected input frames for the output frame from the candidate input frames based upon the weight maps.

25. The computing system of claim 24, the memory further comprises computer-executable instructions that, when executed by the at least one processor, cause the at least one processor to perform acts including:

adjusting the respective weight maps for the candidate input frames based upon a per-pixel blur measure and pixel occlusion.

26. The computing system of claim 18, the memory further comprises computer-executable instructions that, when executed by the at least one processor, cause the at least one processor to perform acts including:

combining the selected input frames for the output frame and selected input frames chosen for output frames that surround the output frame in a frame window to render the output frame.

27. The computing system of claim 18, the memory further comprises computer-executable instructions that, when executed by the at least one processor, cause the at least one processor to perform acts including:

stitching and blending the selected input frames to render the output frame for each of the output frames, the selected input frames being stitched and blended using a temporal and spatial seam selection algorithm and Poisson blending.

28. The computing system of claim 18, the memory further comprises computer-executable instructions that, when executed by the at least one processor, cause the at least one processor to perform acts including:

receiving the input video from a client computing device by way of a network.

29. The computing system of claim 18, further comprising:

a data store;

an image sensor; and

the memory further comprises computer-executable instructions that, when executed by the at least one processor, cause the at least one processor to perform acts including;

storing output data from the image sensor in the data store as the input video.

30. A method of generating an output hyper-lapse video from an input video, comprising:

constructing a model of a scene and input camera poses for input frames of the input video;

computing a smoothed camera path based upon the model of the scene and the input camera poses for the input frames of the input video;

selecting output camera poses from the smoothed camera path for output frames of the output hyper-lapse video, the output camera poses comprise output camera positions and output camera orientations for the output frames of the output hyper-lapse video, the output camera positions being spaced based upon input camera velocities from the input video; and

for each of the output frames of the output hyper-lapse video;

choosing selected input frames from the input video for an output frame, the selected input frames chosen based at least in part upon an output camera position and an output camera orientation for the output frame; and

combining the selected input frames to render the output frame.

31. The method of claim 30, the smoothed camera path being computed as a function of:

a length of the smoothed camera path;

position smoothness of the smoothed camera path;

orientation smoothness of the smoothed camera path;

proximity of the smoothed camera path to input camera positions for the input video, the input camera poses respectively comprise the input camera positions; and

a view quality metric indicative of image-based rendering quality from the model of the scene along the smoothed camera path.

32. The method of claim 30, computing the smoothed camera path further comprises:

computing a view quality metric indicative of image-based rendering quality from the model of the scene along the smoothed camera path, the view quality metric being a view-independent measure of an amount of foreshortening induced by texture mapping an original image onto a proxy geometry; and

computing the smoothed camera path based upon the view quality metric.

33. The method of claim 30, the smoothed camera path comprises an output camera position curve and an output camera orientation curve, and computing the smoothed camera path further comprises:

computing the output camera position curve as a function of position based objectives; and

subsequent to computation of the output camera position curve, computing the output camera orientation curve as a function of orientation based objectives, the output camera orientation curve being computed with the output camera position curve fixed as previously computed.

34. The method of claim 30, choosing the selected input frames from the input video for the output frame further comprises:

identifying candidate input frames for the output frame, the candidate input frames identified from the input frames of the input video;

generating respective weight maps for the candidate input frames; and

choosing, using a greedy algorithm, the selected input frames for the output frame from the candidate input frames based upon the weight maps.