Lens distortion method for broadcast video

• US 9,667,887 B2
• Filed: 11/19/2010
• Issued: 05/30/2017
• Est. Priority Date: 11/21/2009
• Status: Active Grant

• Current Assignee: Disney Enterprises Incorporated (The Walt Disney Company)
• Original Assignee: Disney Enterprises Incorporated (The Walt Disney Company)
• Inventors: Katz, Sheldon, House, Gregory
• Primary Examiner(s): Ren, Zhubing
• Application Number:
  US12/950,525
• Publication Number:
  US 20110128377A1
• Time in Patent Office: 2,384 Days
• CPC Class Codes: H04N 23/81 for suppressing or minimising disturbance in the image signal generation
• CPC Class Codes: H04N 25/61 the noise originating only from the lens unit, e.g. flare, shading, vignetting or "cos4"
• CPC Class Codes: H04N 5/272 Means for inserting a foreground image in a background image, i.e. inlay, outlay
• CPC Class Codes: H04N 5/2723 Insertion of virtual advertisement; Replacing advertisements physical present in the scene by virtual advertisement data processing systems or methods specially adapted for marketing G06Q30/02

Abstract

A method, apparatus, and computer program product are described to improve a lens distortion curve which roughly approximates distortion caused by a camera lens to capture an event onto video. The present invention selects a generic lens distortion curve that roughly approximates the distortion caused by the camera lens while capturing the event onto the video. The video as well as information from the generic lens distortion curve is used to generate a camera model. This camera model is used to integrate virtual insertions into the video. If the camera model is sufficiently accurate to present a realistic appearance of the virtual insertions to the remote viewer, this camera model is then used to integrate more virtual insertions into the video. However, if the camera model is not sufficiently accurate, an iterative process is employed to refine this camera model.

Claims

1. A method of modeling lens distortion for a camera lens used to capture video of an event, comprising:

(a) selecting a lens distortion curve, the lens distortion curve including a plurality of camera lens distortion parameters, wherein calibration of the camera lens using calibration targets is not performed;

(b) generating a camera model based upon the lens distortion curve;

(c) integrating one or more first virtual insertions onto the video using the camera model; and

(d) applying a correction factor including scale, offset and curve gain to the one or more camera lens distortion parameters to adjust the lens distortion curve, wherein the correction factor is determined based on a realistic appearance of the virtual insertion with respect to the video, wherein the realistic appearance is when the virtual insertion appears to be rationally integrated onto the video to a remote viewer.

2. The method of claim 1, wherein step (a) comprises:

(a)(i) selecting a further lens distortion curve corresponding to a further camera lens as the lens distortion curve, the further camera lens being different from the camera lens.

3. The method of claim 2, wherein the camera lens and the further camera lens share common characteristic.

4. The method of claim 1, wherein the camera lens is from among a family of camera lenses, and wherein step (a) comprises:

(a)(i) selecting a second lens distortion curve corresponding to the family of camera lenses as the lens distortion curve.

5. The method of claim 1, wherein the camera lens was previously used to capture a prior event, and where step (a) comprises:

(a)(i) selecting a second lens distortion curve used during the prior event as the lens distortion curve.

6. The method of claim 1, wherein step (a) comprises:

(a)(i) selecting the lens distortion curve right before the event is to begin.

7. The method of claim 1, wherein step (b) comprises:

(b)(i) generating the camera model based upon the lens distortion curve and an automated visual analysis of the video.

8. The method of claim 1, wherein step (b) comprises:

(b)(i) determining one or more camera lens distortion parameters using one or more camera parameters and the lens distortion curve;

(b)(ii) determining one or more undistorted image positions of one or more landmarks in the video; and

(b)(iii) updating the one or more camera parameters based on the undistorted image positions and one or more physical positions of the one or more landmarks.

9. The method of claim 8, wherein step (b)(ii) comprises:

(b)(ii)(A) determining one or more distorted image positions of the one or more landmarks; and

(b)(ii)(B) determining the one or more undistorted image positions from the one or more distorted image positions by adjusting the one or more distorted image positions using the one or more camera lens distortion parameters.

10. The method of claim 1, wherein step (c) comprises:

(c)(i) integrating the one or more first virtual insertions onto the video at a location that is remote to a venue hosting the event.

11. The method of claim 1, wherein step (d) comprises:

(d)(i) performing a visual analysis of the video to determine whether the camera model is sufficiently accurate to present a realistic appearance of the one or more first virtual insertions to a remote viewer; and

(d)(ii) adjusting the lens distortion curve when the camera model is not sufficiently accurate.

12. The method of claim 11, wherein step (d)(i) comprises:

(d)(i)(A) designating the camera model as sufficiently accurate when the one or more first virtual insertions appear within their intended location.

13. The method of claim 11, wherein step (d)(i) comprises:

(d)(i)(A) designating the camera model as sufficiently accurate when the one or more first virtual insertions appear integral to a scene depicted by the video.

14. The method of claim 11, further comprising:

(e) repeating steps (b) through (d) until the camera model is sufficiently accurate to present the realistic appearance.

15. A non-transitory electronic-readable medium having embodied thereon a program, the program being executable by a computer system to perform instructions for modeling lens distortion for a camera lens used to capture video of an event, the instructions comprising:

instructions for selecting a lens distortion curve, the lens distortion curve including a plurality of camera lens distortion parameters;

instructions for generating a camera model based upon the lens distortion curve;

instructions for integrating one or more first virtual insertions onto the video using the camera model; and

instructions for applying a correction factor including scale, offset and curve to the one or more camera lens distortion parameters to adjust the lens distortion curve, wherein the correction factor is determined based on a realistic appearance of the virtual insertion with respect to the video, wherein the realistic appearance is when the virtual insertion appears to be rationally integrated onto the video to a remote viewer.

16. The non-transitory electronic-readable medium of claim 15, wherein the instructions for selecting comprise:

instructions for selecting a further lens distortion curve corresponding to a further camera lens as the lens distortion curve, the further camera lens being different from the camera lens.

17. The non-transitory electronic-readable medium of claim 16, wherein the camera lens and the further camera lens share common characteristic.

18. The non-transitory electronic-readable medium of claim 15, wherein the camera lens is from among a family of camera lenses, and wherein the instructions for selecting comprise:

instructions for selecting a second lens distortion curve corresponding to the family of camera lenses as the lens distortion curve.

19. The non-transitory electronic-readable medium of claim 15, wherein the camera lens was previously used to capture a prior event, and where instructions for selecting comprise:

instructions for selecting a second lens distortion curve used during the prior event as the lens distortion curve.

20. The non-transitory electronic-readable medium of claim 15, wherein instructions for selecting comprise:

instructions for selecting the lens distortion curve right before the event is to begin.

21. The non-transitory electronic-readable medium of claim 15, wherein instructions for generating comprise:

instructions for generating the camera model based upon the lens distortion curve and an automated visual analysis of the video.

22. The non-transitory electronic-readable medium of claim 15, wherein instructions for generating comprise:

instructions for determining one or more camera lens distortion parameters using one or more camera parameters and the lens distortion curve;

instructions for determining one or more undistorted image positions of one or more landmarks in the video; and

instructions for updating the one or more camera parameters based on the undistorted image positions and one or more physical positions of the one or more landmarks.

23. The non-transitory electronic-readable medium of claim 22, wherein the instructions for determining the one or more undistorted image positions comprise:

instructions for determining one or more distorted image positions of the one or more landmarks; and

instructions for determining the one or more undistorted image positions from the one or more distorted image positions by adjusting the one or more distorted image positions using the one or more camera lens distortion parameters.

24. The non-transitory electronic-readable medium of claim 15, wherein the instructions for integrating comprise:

instructions for integrating the one or more first virtual insertions onto the video at a location that is remote to a venue hosting the event.

25. The non-transitory electronic-readable medium of claim 15, further comprising:

instructions for performing a visual analysis of the video to determine whether the camera model is sufficiently accurate to present a realistic appearance of the one or more first virtual insertions to a remote viewer; and

instructions for adjusting the lens distortion curve when the camera model is not sufficiently accurate.

26. The non-transitory electronic-readable medium of claim 24, wherein instructions for performing comprise:

instructions for designating the camera model as sufficiently accurate when the one or more first virtual insertions appear within their intended location.

27. The non-transitory electronic-readable medium of claim 24, wherein the instructions for designating comprise:

instructions for designating the camera model as sufficiently accurate when the one or more first virtual insertions appear integral to a scene depicted by the video.

28. The non-transitory electronic-readable medium of claim 25, further comprising:

instructions for repeating the instructions for generating, the instructions for integrating, and the instructions for adjusting until the camera model is sufficiently accurate to present the realistic appearance.