Multi view camera registration

• US 10,290,119 B2
• Filed: 09/15/2016
• Issued: 05/14/2019
• Est. Priority Date: 09/15/2016
• Status: Active Grant

• Current Assignee: Sportsmedia Technology Corporation
• Original Assignee: Sportsmedia Technology Corporation
• Inventors: White, Marvin S., Parker, Radford, Ramakrishnan, Divya, Gentry, Louis, Pendleton, Rand
• Primary Examiner(s): Philippe, Gims S
• Application Number:
  US15/266,541
• Publication Number:
  US 20180075592A1
• Time in Patent Office: 971 Days
• Field of Search: 345420, 382154, 348 50, 348169, 3482231
• CPC Class Codes: G06T 2207/30228 Playing field
• CPC Class Codes: G06T 7/30 Determination of transform parameters for the alignment of images, i.e. image registration
• CPC Class Codes: G06T 7/75 involving models
• CPC Class Codes: G06T 7/80 Analysis of captured images to determine intrinsic or extrinsic camera parameters, i.e. camera calibration
• CPC Class Codes: G06T 7/85 Stereo camera calibration

Abstract

A system for registering one or more cameras and/or creating an accurate three dimensional (3D) model of a world space environment includes back projecting at least one image from at least one of a plurality of camera views to the 3D model based on a set of existing camera parameters. The back projected image is added as a texture for the 3D model. This texture is automatically compared to one or more images from other camera views using a color space comparison of images to determine a set of differences or errors. The camera parameters and the 3D model are automatically adjusted to minimize the differences or errors. Over time, the parameters and the 3D model converge on a state that can be used to track moving objects, insert virtual graphics and/or perform other functions.

Claims

1. A method, comprising:

back projecting at least one image from at least one of a plurality of camera views to a 3D model of a world space environment based on a set of camera parameters;

wherein the back projecting includes adding one image from a first camera to the 3D model as a texture;

automatically comparing the back projected image to one or more images from other camera views to determine a set of error metrics;

wherein the automatically comparing comprises transforming the 3D model to a 2D image for a second camera and comparing the 2D image to a video image captured by the second camera; and

automatically adjusting the camera parameters and the 3D model to minimize the error metrics based on the automatically comparing.

2. The method of claim 1, further comprising:

adding an additional detail to the 3D model based on the automatically comparing.

3. The method of claim 1, wherein the back projecting of the at least one image includes adding the at least one image to the 3D model as a texture, the method further comprising:

automatically adjusting the texture of the 3D model based on the automatically comparing.

4. The method of claim 1, wherein the automatically comparing comprises:

performing a color space comparison of images including a pixel by pixel comparison of color values of the back projected image and the one or more images from other camera views.

5. The method of claim 1, wherein:

the back projecting of at least one image from at least one of the plurality of camera views to the 3D model includes back projecting images from multiple camera views to one or more instances of the 3D model to create textures for the 3D model; and

the automatically comparing comprises comparing textures for the 3D model that originated from different camera views.

6. The method of claim 1, further comprising:

adding a new feature to the 3D model and repeating the automatically comparing and the automatically adjusting.

7. The method of claim 1, further comprising:

using the 3D model and the set of camera parameters to track a moving object subsequent to the automatically adjusting.

8. The method of claim 1, further comprising:

using the 3D model and the set of camera parameters to insert an image into video from one of the camera views subsequent to the automatically adjusting.

9. An apparatus, comprising:

one or more processors; and

one or more processor readable storage mediums connected to the one or more processors and configured to store code for programming the one or more processors to add an image from at least one of a plurality of camera views to a model of an environment based on a set of camera parameters, wherein the model is a 3D model in 3D world coordinates, compare the added image to an image from a different camera view using a color space comparison of images to determine one or more differences between the added image and the image from the different camera view, and adjust the camera parameters and the 3D model to minimize the one or more differences between the added image and the image from the different camera view based on color space comparisons of images; and

wherein the one or more processors are configured to add the image to the model of the environment as a texture for the model by transforming the 3D model from 3D world coordinates to 2D camera space for a first camera using current camera parameters for the first camera, adding the one image to the transformed 3D model and transforming the 3D model with the one image from 2D camera space for the first camera to 3D world coordinates.

10. The apparatus of claim 9, wherein the one or more processors are configured to store code for programming the one or more processors to add an additional detail to the 3D model based on the color space comparison of images.

11. The apparatus of claim 9, wherein:

the one or more processors are configured to add the image to the model of the environment as a texture for the model, the one or more processors are configured to adjust one or more textures for the model based on color space comparisons of images.

12. The apparatus of claim 9, wherein:

color space comparison of images comprises the one or more processors performing a pixel by pixel comparison of color values of the added image and the one or more images from other camera views.

13. The apparatus of claim 9, wherein:

the one or more processors are configured to automatically add a new feature to the 3D model and repeat the comparing of the added image to an image from a different camera view using the color space comparison of images and the adjusting of the camera parameters and the 3D model to minimize the one or more differences.

14. A method, comprising:

back projecting at least one image from at least one of a plurality of camera views to a 3D model of a world space environment based on a set of camera parameters;

wherein the back projecting includes adding one image from a first camera to the 3D model as a texture by transforming the 3D model from 3D world coordinates to 2D camera space for the first camera using current camera parameters for the first camera, adding the one image to the transformed model and transforming the model with the one image from 2D camera space for the first camera to 3D world coordinates;

automatically comparing the back projected image to one or more images from other camera views to determine a set of error metrics; and

automatically adjusting the camera parameters and the 3D model to minimize the error metrics based on the automatically comparing.

15. The method of claim 14, further comprising adding an additional detail to the 3D model based on the automatically comparing.

16. The method of claim 14, wherein the back projecting of the at least one image includes adding the at least one image to the 3D model as a texture, the method further comprising:

automatically adjusting the texture of the 3D model based on the automatically comparing.

17. The method of claim 14, wherein the automatically comparing comprises:

performing a color space comparison of images including a pixel by pixel comparison of color values of the back projected image and the one or more images from other camera views.

18. The method of claim 14, wherein:

the back projecting of the at least one image from at least one of the plurality of camera views to the 3D model includes back projecting images from multiple camera views to one or more instances of the 3D model to create textures for the 3D model; and

the automatically comparing comprises comparing textures for the 3D model that originated from different camera views.

19. The method of claim 14, wherein:

the automatically comparing comprises transforming the 3D model to a 2D image for a second camera and comparing the 2D image to a video image captured by the second camera.

20. The method of claim 14, further comprising:

adding a new feature to the 3D model and repeating the automatically comparing and the automatically adjusting.

21. The method of claim 14, further comprising:

using the 3D model and the set of camera parameters to track a moving object subsequent to the automatically adjusting.

22. The method of claim 14, further comprising:

using the 3D model and the set of camera parameters to insert an image into video from one of the camera views subsequent to the automatically adjusting.