Virtual lens simulation for video and photo cropping

• US 10,679,323 B2
• Filed: 12/13/2019
• Issued: 06/09/2020
• Est. Priority Date: 05/20/2015
• Status: Active Grant

• Current Assignee: GoPro, Inc.
• Original Assignee: GoPro, Inc.
• Inventors: Newman, David A., Bodinet, Joshua Edward, Sievert, Otto Kenneth, MacMillan, Timothy
• Primary Examiner(s): Amin, Jwalant
• Application Number:
  US16/713,839
• Publication Number:
  US 20200118240A1
• Time in Patent Office: 179 Days
• Field of Search: None
• CPC Class Codes: G06T 2207/10004 Still image; Photographic image
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/20021 Dividing image into blocks, subimages or windows
• CPC Class Codes: G06T 2210/22 Cropping
• CPC Class Codes: G06T 3/00 Geometric image transformations in the plane of the image
• CPC Class Codes: G06T 3/047 Fisheye or wide-angle transformations
• CPC Class Codes: G06T 3/40 Scaling of whole images or parts thereof, e.g. expanding or contracting
• CPC Class Codes: G06T 5/80 Geometric correction

Abstract

In a video capture system, a virtual lens is simulated when applying a crop or zoom effect to an input video. An input video frame is received from the input video that has a first field of view and an input lens distortion caused by a lens used to capture the input video frame. A selection of a sub-frame representing a portion of the input video frame is obtained that has a second field of view smaller than the first field of view. The sub-frame is processed to remap the input lens distortion to a desired lens distortion in the sub-frame. The processed sub-frame is the outputted.

Claims

1. A system that simulates image distortion of a virtual lens in a video, the system comprising:

one or more processors; and

a non-transitory computer-readable storage medium storing instructions that when executed cause the one or more processors to perform steps including;

accessing input images, the input images including fields of view of a scene captured through a lens of a camera, the input images depicting the scene with an input lens distortion centered in the fields of view, the input lens distortion caused by lens characteristics of the lens of the camera;

selecting sub-frames representing portions of the input images, the sub-frames including reduced fields of view of the scene smaller than the fields of view of the input images, the sub-frames including lens distortion effects as a function of the input lens distortion present in the input images, positions of areas of the input images included within the reduced fields of view, and a size of the areas of the input images included within the reduced fields of view; and

generating output images based on the lens distortion effects in the sub-frames and a desired lens distortion, the desired lens distortion being consistent with the lens characteristics of the lens of the camera, wherein the output images include the sub-frames remapped from the input lens distortion centered in the fields of view of the input images to the desired lens distortion centered in the reduced fields of view to transform the lens distortion effects present in the sub-frames to the desired lens distortion such that portions of the scene depicted in the sub-frames appear to have been captured using the reduced fields of view.

2. The system of claim 1, wherein image stabilization is applied to the input images.

3. The system of claim 1, wherein the sub-frames are selected based on metadata associated with the input images.

4. The system of claim 3, wherein the metadata indicates motion of the camera that captured the input images.

5. The system of claim 3, wherein the metadata indicates orientation of the camera that captured the input images.

6. The system of claim 1, wherein the output images are used as video frames of an output video.

7. The system of claim 6, wherein remapping of the sub-frames from the input lens distortion centered in the fields of view of the input images to the desired lens distortion centered in the reduced fields of view results in the output video having consistent frame-to-frame lens characteristics.

8. The system of claim 7, wherein the positions of the areas of the input images included within the reduced fields of view simulate virtual re-pointing of the camera using the reduced fields of view.

9. The system of claim 8, wherein the sub-frames are selected in post-processing after capture of the input images, the selection of the sub-frames in the post-processing performed using a post-processing tool.

10. The system of claim 9, wherein the post-processing tool includes an image editing tool or a video editing tool.

11. A method for simulating image distortion of a virtual lens in a video, the method comprising:

accessing input images, the input images including fields of view of a scene captured through a lens of a camera, the input images depicting the scene with an input lens distortion centered in the fields of view, the input lens distortion caused by lens characteristics of the lens of the camera;

selecting sub-frames representing portions of the input images, the sub-frames including reduced fields of view of the scene smaller than the fields of view of the input images, the sub-frames including lens distortion effects as a function of the input lens distortion present in the input images, positions of areas of the input images included within the reduced fields of view, and a size of the areas of the input images included within the reduced fields of view; and

generating output images based on the lens distortion effects in the sub-frames and a desired lens distortion, the desired lens distortion being consistent with the lens characteristics of the lens of the camera, wherein the output images include the sub-frames remapped from the input lens distortion centered in the fields of view of the input images to the desired lens distortion centered in the reduced fields of view to transform the lens distortion effects present in the sub-frames to the desired lens distortion such that portions of the scene depicted in the sub-frames appear to have been captured using the reduced fields of view.

12. The method of claim 11, wherein image stabilization is applied to the input images.

13. The method of claim 11, wherein the sub-frames are selected based on metadata associated with the input images.

14. The method of claim 13, wherein the metadata indicates motion of the camera that captured the input images.

15. The method of claim 13, wherein the metadata indicates orientation of the camera that captured the input images.

16. The method of claim 11, wherein the output images are used as video frames of an output video.

17. The method of claim 16, wherein remapping of the sub-frames from the input lens distortion centered in the fields of view of the input images to the desired lens distortion centered in the reduced fields of view results in the output video having consistent frame-to-frame lens characteristics.

18. The method of claim 17, wherein the positions of the areas of the input images included within the reduced fields of view simulate virtual re-pointing of the camera using the reduced fields of view.

19. The method of claim 18, wherein the sub-frames are selected in post-processing after capture of the input images, the selection of the sub-frames in the post-processing performed using a post-processing tool.

20. The method of claim 19, wherein the post-processing tool includes an image editing tool or a video editing tool.