Lens Distortion Method for Broadcast Video

• US 20110128377A1
• Filed: 11/19/2010
• Published: 06/02/2011
• Est. Priority Date: 11/21/2009
• Status: Active Grant

• Current Assignee: Disney Enterprises Incorporated (The Walt Disney Company)
• Original Assignee: PVI Virtual Media Services, LLC (Altice USA, Inc.)
• Inventors: Katz, Sheldon, House, Gregory
• Granted Patent: US 9,667,887 B2
• Time in Patent Office: Days
• US Class Current: 348/143
• CPC Class Codes: H04N 23/81 for suppressing or minimising disturbance in the image signal generation
• CPC Class Codes: H04N 25/61 the noise originating only from the lens unit, e.g. flare, shading, vignetting or "cos4"
• CPC Class Codes: H04N 5/272 Means for inserting a foreground image in a background image, i.e. inlay, outlay
• CPC Class Codes: H04N 5/2723 Insertion of virtual advertisement; Replacing advertisements physical present in the scene by virtual advertisement data processing systems or methods specially adapted for marketing G06Q30/02

Abstract

A method, apparatus, and computer program product are described to improve a lens distortion curve which roughly approximates distortion caused by a camera lens to capture an event onto video. The present invention selects a generic lens distortion curve that roughly approximates the distortion caused by the camera lens while capturing the event onto the video. The video as well as information from the generic lens distortion curve is used to generate a camera model. This camera model is used to integrate virtual insertions into the video. If the camera model is sufficiently accurate to present a realistic appearance of the virtual insertions to the remote viewer, this camera model is then used to integrate more virtual insertions into the video. However, if the camera model is not sufficiently accurate, an iterative process is employed to refine this camera model.

Claims

1. A method of modeling lens distortion for a camera lens used to capture video of an event, comprising:

(a) selecting a lens distortion curve;

(b) generating a camera model based upon the lens distortion curve;

(c) integrating one or more first virtual insertions onto the video using the camera model; and

(d) adjusting the lens distortion curve based on an appearance of the virtual insertion with respect to the video.

2. The method of claim 1, wherein step (a) comprises:

(a)(i) selecting a second lens distortion curve corresponding to the second camera lens as the lens distortion curve, the second camera lens being different from the first camera lens.

3. The method of claim 2, wherein the first camera lens and the second camera lens share common characteristic.

4. The method of claim 1, wherein the camera lens is from among a family of camera lenses, and wherein step (a) comprises:

(a)(i) selecting a second lens distortion curve corresponding to the family of camera lenses as the lens distortion curve.

5. The method of claim 1, wherein the camera lens was previously used to capture a prior event, and where step (a) comprises:

(a)(i) selecting a second lens distortion curve used during the prior event as the lens distortion curve.

6. The method of claim 1, wherein step (a) comprises:

(a)(i) selecting the lens distortion curve right before the event is to begin thereby forgoing calibration of the camera lens using calibration targets.

7. The method of claim 1, wherein step (b) comprises:

(b)(i) generating the camera model based upon the lens distortion curve and an automated visual analysis of the video.

8. The method of claim 1, wherein step (b) comprises:

(b)(i) determining one or more camera lens distortion parameters using one or more camera parameters and the lens distortion curve;

(b)(ii) determining one or more undistorted image positions of one or more landmarks in the video; and

(b)(iii) updating the one or more camera parameters based on the undistorted image positions and one or more physical positions of the one or more landmarks.

9. The method of claim 8, wherein step (b)(ii) comprises:

(b)(ii)(A) determining one or more distorted image positions of the one or more landmarks; and

(b)(ii)(B) determining the one or more undistorted image positions from the one or more distorted image positions by adjusting the one or more distorted image positions using the one or more camera lens distortion parameters.

10. The method of claim 1, wherein step (c) comprises:

(c)(i) integrating the one or more first virtual insertions onto the video at a location that is remote to a venue hosting the event.

11. The method of claim 1, wherein step (d) comprises:

(d)(i) performing a visual analysis of the video to determine whether the camera model is sufficiently accurate to present a realistic appearance of the one or more first virtual insertions to a remote viewer; and

(d)(ii) adjusting the lens distortion curve when the camera model is not sufficiently accurate.

12. The method of claim 11, wherein step (d)(i) comprises:

(d)(i)(A) designating the camera model as sufficiently accurate when the one or more first virtual insertions appear within their intended location.

13. The method of claim 11, wherein step (d)(i) comprises:

(d)(i)(A) designating the camera model as sufficiently accurate when the one or more first virtual insertions appear integral to a scene depicted by the video.

14. The method of claim 1, wherein step (d) comprises:

(d)(i) applying correction factors to the lens distortion curve to adjust the lens distortion curve.

15. The method of claim 11, further comprising:

(e) repeating steps (b) through (d) until the camera model is sufficiently accurate to present the realistic appearance.

16. A non-transitory electronic-readable medium having embodied thereon a program, the program being executable by a computer system to perform instructions for modeling lens distortion for a camera lens used to capture video of an event, the instructions comprising:

instructions for selecting a lens distortion curve;

instructions for generating a camera model based upon the lens distortion curve;

instructions for integrating one or more first virtual insertions onto the video using the camera model; and

instructions for adjusting the lens distortion curve based on an appearance of the virtual insertion with respect to the video.

17. The non-transitory electronic-readable medium of claim 16, wherein the instructions for selecting comprise:

instructions for selecting a second lens distortion curve corresponding to the second camera lens as the lens distortion curve, the second camera lens being different from the first camera lens.

18. The non-transitory electronic-readable medium of claim 17, wherein the first camera lens and the second camera lens share common characteristic.

19. The non-transitory electronic-readable medium of claim 16, wherein the camera lens is from among a family of camera lenses, and wherein the instructions for selecting comprise:

instructions for selecting a second lens distortion curve corresponding to the family of camera lenses as the lens distortion curve.

20. The non-transitory electronic-readable medium of claim 16, wherein the camera lens was previously used to capture a prior event, and where instructions for selecting comprise:

instructions for selecting a second lens distortion curve used during the prior event as the lens distortion curve.

21. The non-transitory electronic-readable medium of claim 16, wherein instructions for selecting comprise:

instructions for selecting the lens distortion curve right before the event is to begin thereby forgoing calibration of the camera lens using calibration targets.

22. The non-transitory electronic-readable medium of claim 16, wherein instructions for generating comprise:

instructions for generating the camera model based upon the lens distortion curve and an automated visual analysis of the video.

23. The non-transitory electronic-readable medium of claim 16, wherein instructions for generating comprise:

instructions for determining one or more camera lens distortion parameters using one or more camera parameters and the lens distortion curve;

instructions for determining one or more undistorted image positions of one or more landmarks in the video; and

instructions for updating the one or more camera parameters based on the undistorted image positions and one or more physical positions of the one or more landmarks.

24. The non-transitory electronic-readable medium of claim 23, wherein the instructions for determining the one or more undistorted image positions comprise:

instructions for determining one or more distorted image positions of the one or more landmarks; and

instructions for determining the one or more undistorted image positions from the one or more distorted image positions by adjusting the one or more distorted image positions using the one or more camera lens distortion parameters.

25. The non-transitory electronic-readable medium of claim 16, wherein the instructions for integrating comprise:

instructions for integrating the one or more first virtual insertions onto the video at a location that is remote to a venue hosting the event.

26. The non-transitory electronic-readable medium of claim 16, wherein step the instructions for adjusting comprise:

instructions for performing a visual analysis of the video to determine whether the camera model is sufficiently accurate to present a realistic appearance of the one or more first virtual insertions to a remote viewer; and

instructions for adjusting the lens distortion curve when the camera model is not sufficiently accurate.

27. The non-transitory electronic-readable medium of claim 25, wherein instructions for performing comprise:

instructions for designating the camera model as sufficiently accurate when the one or more first virtual insertions appear within their intended location.

28. The non-transitory electronic-readable medium of claim 25, wherein the instructions for designating comprise:

instructions for designating the camera model as sufficiently accurate when the one or more first virtual insertions appear integral to a scene depicted by the video.

29. The non-transitory electronic-readable medium of claim 16, wherein step the instructions adjusting the lens distortion curve comprise:

instructions for applying correction factors to the lens distortion curve to adjust the lens distortion curve.

30. The non-transitory electronic-readable medium of claim 26, further comprising:

instructions for repeating the instructions for generating, the instructions for integrating, and the instructions for adjusting until the camera model is sufficiently accurate to present the realistic appearance.