Virtual lens simulation for video and photo cropping

• US 10,186,012 B2
• Filed: 05/17/2016
• Issued: 01/22/2019
• Est. Priority Date: 05/20/2015
• Status: Active Grant

• Current Assignee: GoPro, Inc.
• Original Assignee: GoPro, Inc.
• Inventors: Newman, David A., Bodinet, Joshua Edward, Sievert, Otto Kenneth, Macmillan, Timothy
• Primary Examiner(s): Amin, Jwalant
• Application Number:
  US15/157,207
• Publication Number:
  US 20160343107A1
• Time in Patent Office: 980 Days
• CPC Class Codes: G06T 2207/10004 Still image; Photographic image
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/20021 Dividing image into blocks, subimages or windows
• CPC Class Codes: G06T 2210/22 Cropping
• CPC Class Codes: G06T 3/00 Geometric image transformations in the plane of the image
• CPC Class Codes: G06T 3/047 Fisheye or wide-angle transformations
• CPC Class Codes: G06T 3/40 Scaling of whole images or parts thereof, e.g. expanding or contracting
• CPC Class Codes: G06T 5/80 Geometric correction

Abstract

In a video capture system, a virtual lens is simulated when applying a crop or zoom effect to an input video. An input video frame is received from the input video that has a first field of view and an input lens distortion caused by a lens used to capture the input video frame. A selection of a sub-frame representing a portion of the input video frame is obtained that has a second field of view smaller than the first field of view. The sub-frame is processed to remap the input lens distortion to a desired lens distortion in the sub-frame. The processed sub-frame is the outputted.

Claims

1. A method for simulating a virtual lens when applying a crop or zoom effect to an input image, the method comprising:

receiving, by a processor, the input image, the input image having a first field of view of a scene, the input image depicting the scene with an input lens distortion centered in the first field of view, the input lens distortion caused by lens characteristics of a lens used to capture the input image, the lens centered at the first field of view during capture of the input image;

identifying, by the processor, the input lens distortion within the input image;

obtaining, by the processor, a selection of a sub-frame representing a portion of the input image, the sub-frame having a second field of view of the scene smaller than the first field of view;

determining, by the processor, an input lens distortion effect present in the sub-frame based on the input lens distortion centered in the first field of view, a location of the sub-frame within the first field of view, and a size of the second field of view; and

generating, by the processor, an output image based on the input lens distortion and the input lens distortion effect present in the sub-frame, the output image including the sub-frame remapped from the input lens distortion centered in the first field of view to the input lens distortion centered in the second field of view such that a portion of the scene depicted in the sub-frame appears to have been captured using the lens centered at the second field of view.

2. The method of claim 1, wherein the input lens distortion caused by the lens characteristics causes straight lines in the scene to appear as curved lines in at least a portion of the input image.

3. The method of claim 1, wherein the input lens distortion comprises a distortion produced by a conventional lens, a wide angle lens, a fisheye lens, a hemispherical lens, a zoom lens, or a flat lens.

4. The method of claim 1, wherein obtaining the selection of the sub-frame representing the portion of the input image comprises:

automatically identifying the sub-frame based on metadata associated with the input image or image characteristics of the input image;

automatically obtaining a sub-frame center location, a sub-frame size, and a scaling factor for transforming the input image; and

applying the crop or zoom effect applied to the input image based on the sub-frame center location, the sub-frame size, and the scaling factor to generate the sub-frame.

5. The method of claim 1, wherein obtaining the selection of the sub-frame representing the portion of the input image further comprises:

receiving a manual selection of the sub-frame from post-processing tools.

6. The method of claim 1, wherein generating the output image comprises:

obtaining metadata associated with the input image, the metadata specifying the lens characteristics of the lens used to capture the input image; and

processing the sub-frame using the lens characteristics specified in the metadata.

7. The method of claim 1, wherein generating the output image comprises:

applying an inverse input lens distortion function to the sub-frame to remove the input lens distortion effect present in the sub-frame to generate a rectilinear image, wherein straight lines in the scene within the sub-frame appear as straight lines in the rectilinear image; and

applying an input lens distortion function to apply the input lens distortion to the rectilinear image.

8. The method of claim 1, wherein the sub-frame is remapped from the input lens distortion centered in the first field of view to the input lens distortion centered in the second field of view by applying a direct function to remap the sub-frame without an intermediate step of removing the input lens distortion effect present in the sub-frame.

9. A non-transitory computer-readable storage medium storing instructions for simulating a virtual lens when applying a crop or zoom effect to an input image, the instructions when executed by one or more processors causing the one or more processors to perform steps including:

receiving the input image, the input image having a first field of view of a scene, the input image depicting the scene with an input lens distortion centered in the first field of view, the input lens distortion caused by lens characteristics of a lens used to capture the input image, the lens centered at the first field of view during capture of the input image;

identifying the input lens distortion within the input image;

obtaining a selection of a sub-frame representing a portion of the input image, the sub-frame having a second field of view of the scene smaller than the first field of view;

determining an input lens distortion effect present in the sub-frame based on the input lens distortion centered in the first field of view, a location of the sub-frame within the first field of view, and a size of the second field of view; and

generating an output image based on the input lens distortion and the input lens distortion effect present in the sub-frame, the output image including the sub-frame remapped from the input lens distortion centered in the first field of view to the input lens distortion centered in the second field of view such that a portion of the scene depicted in the sub-frame appears to have been captured using the lens centered at the second field of view.

10. The non-transitory computer-readable storage medium of claim 9, wherein the input lens distortion caused by the lens characteristics causes straight lines in the scene to appear as curved lines in at least a portion of the input image.

11. The non-transitory computer-readable storage medium of claim 9, wherein the input lens distortion comprises a distortion produced by a conventional lens, a wide angle lens, a fisheye lens, a hemispherical lens, a zoom lens, or a flat lens.

12. The non-transitory computer-readable storage medium of claim 9, wherein obtaining the selection of the sub-frame representing the portion of the input image comprises:

automatically identifying the sub-frame based on metadata associated with the input image or image characteristics of the input image;

automatically obtaining a sub-frame center location, a sub-frame size, and a scaling factor for transforming the input image; and

applying the crop or zoom effect applied to the input image based on the sub-frame center location, the sub-frame size, and the scaling factor to generate the sub-frame.

13. The non-transitory computer-readable storage medium of claim 9, wherein obtaining the selection of the sub-frame representing the portion of the input image further comprises:

receiving a manual selection of the sub-frame from post-processing tools.

14. The non-transitory computer-readable storage medium of claim 9, wherein generating the output image comprises:

obtaining metadata associated with the input image, the metadata specifying the lens characteristics of the lens used to capture the input image; and

processing the sub-frame using the lens characteristics specified in the metadata.

15. The non-transitory computer-readable storage medium of claim 9, wherein generating the output image comprises:

applying an inverse input lens distortion function to the sub-frame to remove the input lens distortion effect present in the sub-frame to generate a rectilinear image, wherein straight lines in the scene within the sub-frame appear as straight lines in the rectilinear image; and

applying an input lens distortion function to apply the input lens distortion to the rectilinear image.

16. The non-transitory computer-readable storage medium of claim 9, wherein the sub-frame is remapped from the input lens distortion centered in the first field of view to the input lens distortion centered in the second field of view by applying a direct function to remap the sub-frame without an intermediate step of removing the input lens distortion effect present in the sub-frame.

17. A system for simulating a virtual lens when applying a crop or zoom effect to an input image, the system comprising:

one or more processors; and

a non-transitory computer-readable storage medium storing instructions for simulating the virtual lens when applying a crop or zoom effect to the input image, the instructions when executed by one or more processors causing the one or more processors to perform steps including;

receiving the input image, the input image having a first field of view of a scene, the input image depicting the scene with an input lens distortion centered in the first field of view, the input lens distortion caused by lens characteristics of a lens used to capture the input image, the lens centered at the first field of view during capture of the input image;

identifying the input lens distortion within the input image;

obtaining a selection of a sub-frame representing a portion of the input image, the sub-frame having a second field of view of the scene smaller than the first field of view;

determining an input lens distortion effect present in the sub-frame based on the input lens distortion centered in the first field of view, a location of the sub-frame within the first field of view, and a size of the second field of view; and

generating an output image based on the input lens distortion and the input lens distortion effect present in the sub-frame, the output image including the sub-frame remapped from the input lens distortion centered in the first field of view to the input lens distortion centered in the second field of view such that a portion of the scene depicted in the sub-frame appears to have been captured using the lens centered at the second field of view.

18. The system of claim 17, wherein the input lens distortion caused by the lens characteristics causes straight lines in the scene to appear as curved lines in at least a portion of the input image.

19. The system of claim 17, wherein the input lens distortion comprises a distortion produced by a conventional lens, a wide angle lens, a fisheye lens, a hemispherical lens, a zoom lens, or a flat lens.

20. The system of claim 17, wherein obtaining the selection of the sub-frame representing the portion of the input image comprises:

automatically identifying the sub-frame based on metadata associated with the input image or image characteristics of the input image;

automatically obtaining a sub-frame center location, a sub-frame size, and a scaling factor for transforming the input image; and

applying the crop or zoom effect applied to the input image based on the sub-frame center location, the sub-frame size, and the scaling factor to generate the sub-frame.