Virtual Lens Simulation for Video and Photo Cropping

• US 20160343107A1
• Filed: 05/17/2016
• Published: 11/24/2016
• Est. Priority Date: 05/20/2015
• Status: Active Grant

• Current Assignee: GoPro, Inc.
• Original Assignee: GoPro, Inc.
• Inventors: Newman, David A., Bodinet, Joshua Edward, Sievert, Otto Kenneth, MacMillan, Timothy
• Granted Patent: US 10,186,012 B2
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: G06T 2207/10004 Still image; Photographic image
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/20021 Dividing image into blocks, subimages or windows
• CPC Class Codes: G06T 2210/22 Cropping
• CPC Class Codes: G06T 3/00 Geometric image transformations in the plane of the image
• CPC Class Codes: G06T 3/047 Fisheye or wide-angle transformations
• CPC Class Codes: G06T 3/40 Scaling of whole images or parts thereof, e.g. expanding or contracting
• CPC Class Codes: G06T 5/80 Geometric correction

Abstract

In a video capture system, a virtual lens is simulated when applying a crop or zoom effect to an input video. An input video frame is received from the input video that has a first field of view and an input lens distortion caused by a lens used to capture the input video frame. A selection of a sub-frame representing a portion of the input video frame is obtained that has a second field of view smaller than the first field of view. The sub-frame is processed to remap the input lens distortion to a desired lens distortion in the sub-frame. The processed sub-frame is the outputted.

Claims

1. A method for simulating a virtual lens when applying a crop or zoom effect to an input image, the method comprising:

receiving, by a processor, the input image, the input image having a first field of view of a scene, the input image depicting the scene with an input lens distortion centered in the first field of view, the input lens distortion caused by lens characteristics of a lens used to capture the input image;

obtaining, by the processor, a selection of a sub-frame representing a portion of the input image, the sub-frame having a second field of view of the scene smaller than the first field of view;

processing, by the processor, the sub-frame to remap from the input lens distortion centered in the first field of view to a desired lens distortion in the sub-frame centered in the second field of view; and

outputting the processed sub-frame.

2. The method of claim 1, wherein the input lens distortion caused by the lens characteristics causes straight lines in the scene to appear as curved lines in at least a portion of the input image.

3. The method of claim 1, wherein the input lens distortion comprises a distortion produced by a conventional lens, a wide angle lens, a fisheye lens, a hemispherical lens, a zoom lens, or a flat lens.

4. The method of claim 1, wherein obtaining the selection of the sub-frame representing the portion of the input image comprises:

automatically identifying the sub-frame based on metadata associated with the input image or image characteristics of the input image;

automatically obtaining a sub-frame center location, a sub-frame size, and a scaling factor for transforming the input image; and

applying the crop or zoom effect applied to the input image based on the sub-frame center location, the sub-frame size, and the scaling factor to generate the sub-frame.

5. The method of claim 1, wherein obtaining the selection of the sub-frame representing the portion of the input image further comprises:

receiving a manual selection of the sub-frame from post-processing tools.

6. The method of claim 1, wherein processing the sub-frame comprises:

obtaining metadata associated with the input image, the metadata specifying the lens characteristics of the lens used to capture the input image; and

processing the sub-frame using the lens characteristics specified in the metadata.

7. The method of claim 1, wherein processing the sub-frame to remap from the input lens distortion to the desired lens distortion comprises:

applying an inverse input lens distortion function to remove the input lens distortion of the selected sub-frame to generate a rectilinear image, wherein straight lines in the scene appear as straight lines in the rectilinear image; and

applying a desired lens distortion function to apply the desired lens distortion to the rectilinear image.

8. The method of claim 1, wherein processing the sub-frame to remap from the input lens distortion to the desired lens distortion to the sub-frame is performed by applying a direct function to map the input lens distortion to the desired lens distortion in the sub-frame without an intermediate step of removing the input lens distortion of the selected sub-frame.

9. A non-transitory computer-readable storage medium storing instructions for simulating a virtual lens when applying a crop or zoom effect to an input image, the instructions when executed by one or more processors causing the one or more processors to perform steps including:

receiving the input image, the input image having a first field of view of a scene, the input image depicting the scene with an input lens distortion centered in the first field of view, the input lens distortion caused by lens characteristics of a lens used to capture the input image;

obtaining a selection of a sub-frame representing a portion of the input image, the sub-frame having a second field of view of the scene smaller than the first field of view;

processing the sub-frame to remap from the input lens distortion centered in the first field of view to a desired lens distortion in the sub-frame centered in the second field of view; and

outputting the processed sub-frame.

10. The non-transitory computer-readable storage medium of claim 9, wherein the input lens distortion caused by the lens characteristics causes straight lines in the scene to appear as curved lines in at least a portion of the input image.

11. The non-transitory computer-readable storage medium of claim 9, wherein the input lens distortion comprises a distortion produced by a conventional lens, a wide angle lens, a fisheye lens, a hemispherical lens, a zoom lens, or a flat lens.

12. The non-transitory computer-readable storage medium of claim 9, wherein obtaining the selection of the sub-frame representing the portion of the input image comprises:

automatically identifying the sub-frame based on metadata associated with the input image or image characteristics of the input image;

automatically obtaining a sub-frame center location, a sub-frame size, and a scaling factor for transforming the input image; and

applying the crop or zoom effect applied to the input image based on the sub-frame center location, the sub-frame size, and the scaling factor to generate the sub-frame.

13. The non-transitory computer-readable storage medium of claim 9, wherein obtaining the selection of the sub-frame representing the portion of the input image further comprises:

receiving a manual selection of the sub-frame from post-processing tools.

14. The non-transitory computer-readable storage medium of claim 9, wherein processing the sub-frame comprises:

obtaining metadata associated with the input image, the metadata specifying the lens characteristics of the lens used to capture the input image; and

processing the sub-frame using the lens characteristics specified in the metadata.

15. The non-transitory computer-readable storage medium of claim 9, wherein processing the sub-frame to remap from the input lens distortion to the desired lens distortion comprises:

applying an inverse input lens distortion function to remove the input lens distortion of the selected sub-frame to generate a rectilinear image, wherein straight lines in the scene appear as straight lines in the rectilinear image; and

applying a desired lens distortion function to apply the desired lens distortion to the rectilinear image.

16. The non-transitory computer-readable storage medium of claim 9, wherein processing the sub-frame to remap from the input lens distortion to the desired lens distortion in the sub-frame is performed by applying a direct function to map the input lens distortion to the desired lens distortion in the sub-frame without an intermediate step of removing the input lens distortion of the selected sub-frame.

17. A media server for simulating a virtual lens when applying a crop or zoom effect to an input image, the media server comprising:

one or more processors; and

a non-transitory computer-readable storage medium storing instructions for simulating a virtual lens when applying a crop or zoom effect to the input image, the instructions when executed by one or more processors causing the one or more processors to perform steps including;

receiving the input image, the input image having a first field of view of a scene, the input image depicting the scene with an input lens distortion centered in the first field of view, the input lens distortion caused by lens characteristics of a lens used to capture the input image;

obtaining a selection of a sub-frame representing a portion of the input image, the sub-frame having a second field of view of the scene smaller than the first field of view;

processing the sub-frame to remap from the input lens distortion centered in the first field of view to a desired lens distortion in the sub-frame centered in the second field of view; and

outputting the processed sub-frame.

18. The media server of claim 17, wherein the input lens distortion caused by the lens characteristics causes straight lines in the scene to appear as curved lines in at least a portion of the input image.

19. The media server of claim 17, wherein the input lens distortion comprises a distortion produced by a conventional lens, a wide angle lens, a fisheye lens, a hemispherical lens, a zoom lens, or a flat lens.

20. The media server of claim 17, wherein obtaining the selection of the sub-frame representing the portion of the input image comprises:

automatically identifying the sub-frame based on metadata associated with the input image or image characteristics of the input image;

automatically obtaining a sub-frame center location, a sub-frame size, and a scaling factor for transforming the input image; and

applying the crop or zoom effect applied to the input image based on the sub-frame center location, the sub-frame size, and the scaling factor to generate the sub-frame.