MULTI VIEW CAMERA REGISTRATION

• US 20180075592A1
• Filed: 09/15/2016
• Published: 03/15/2018
• Est. Priority Date: 09/15/2016
• Status: Active Grant

• Current Assignee: Sportsmedia Technology Corporation
• Original Assignee: Sportsmedia Technology Corporation
• Inventors: White, Marvin S., Parker, Radford, Ramakrishnan, Divya, Gentry, Louis, Pendleton, Rand
• Granted Patent: US 10,290,119 B2
• Time in Patent Office: Days
• CPC Class Codes: G06T 2207/30228 Playing field
• CPC Class Codes: G06T 7/30 Determination of transform parameters for the alignment of images, i.e. image registration
• CPC Class Codes: G06T 7/75 involving models
• CPC Class Codes: G06T 7/80 Analysis of captured images to determine intrinsic or extrinsic camera parameters, i.e. camera calibration
• CPC Class Codes: G06T 7/85 Stereo camera calibration

Abstract

A system for registering one or more cameras and/or creating an accurate three dimensional (3D) model of a world space environment includes back projecting at least one image from at least one of a plurality of camera views to the 3D model based on a set of existing camera parameters. The back projected image is added as a texture for the 3D model. This texture is automatically compared to one or more images from other camera views using a color space comparison of images to determine a set of differences or errors. The camera parameters and the 3D model are automatically adjusted to minimize the differences or errors. Over time, the parameters and the 3D model converge on a state that can be used to track moving objects, insert virtual graphics and/or perform other functions.

Claims

1. A method, comprising:

back projecting at least one image from at least one of a plurality of camera views to a 3D model of a world space environment based on a set of camera parameters;

automatically comparing the back projected image to one or more images from other camera views to determine a set of error metrics; and

automatically adjusting the camera parameters and the 3D model to minimize the error metrics based on the comparing.

2. The method of claim 1, further comprising:

adding an additional detail to the 3D model based on the comparing.

3. The method of claim 1, wherein the back projecting of at least one image includes adding the at least one image to the 3D model as a texture, the method further comprising:

automatically adjusting a texture of the 3D model based on the comparing.

4. The method of claim 1, wherein the automatically comparing comprises:

performing a color space comparison of images including a pixel by pixel comparison of color values of the back projected image and the one or more images from other camera views.

5. The method of claim 1, wherein:

the back projecting of at least one image from at least one of the plurality of camera views to the 3D model includes back projecting images from multiple camera views to one or more instances of the 3D model to create textures for the 3D model; and

the automatically comparing comprises comparing textures for the 3D model that originated from different camera views.

6. The method of claim 1, wherein:

the back projecting includes adding one image from a first camera to the 3D model as a texture; and

the automatically comparing comprises transforming the 3D model to a 2D image for a second camera and comparing the 2D image to a video image captured by the second camera.

7. The method of claim 1, wherein:

the back projecting includes adding one image from a first camera to the 3D model as a texture by transforming the 3D model from 3D world coordinates to 2D camera space for the first camera using current camera parameters for the first camera, adding the one image to the transformed model and transforming the model with the one image from 2D camera space for the first camera to 3D world coordinates.

8. The method of claim 1, further comprising:

adding a new feature to the 3D model and repeating the automatically comparing and the automatically adjusting.

9. The method of claim 1, further comprising:

using the 3D model and camera parameters to track a moving object subsequent to the automatically adjusting.

10. The method of claim 1, further comprising:

using the 3D model and camera parameters to insert an image into video from one of the camera views subsequent to the automatically adjusting.

11. An apparatus, comprising:

one or more processors; and

one or more processor readable storage mediums connected to the one or more processors and configured to store code for programming the one or more processors to add an image from at least one of a plurality of camera views to a model of an environment based on a set of camera parameters, compare the added image to an image from a different camera view using a color spaced comparison of images to determine one or more differences between the added image and the image from the different camera view, and adjust the camera parameters and the 3D model to minimize the one or more differences between the added image and the image from the different camera view based on color spaced comparisons of images.

12. The apparatus of claim 11, further comprising:

the one or more processors and configured to store code for programming the one or more processors to add an additional detail to the 3D model based on the color space comparison of images.

13. The apparatus of claim 11, wherein:

the one or more processors are configured to add the image to the model of the environment as a texture for the model, the one or more processors are configured to adjust one or more textures for the model based on color spaced comparisons of images.

14. The apparatus of claim 11, wherein:

the model is a 3D model in 3D world coordinates;

the one or more processors are configured to add the image to the model of the environment as a texture for the model by transforming the 3D model from 3D world coordinates to 2D camera space for a first camera using current camera parameters for the first camera, adding the one image to the transformed model and transforming the model with the one image from 2D camera space for the first camera to 3D world coordinates.

15. The apparatus of claim 11, wherein:

color spaced comparison of images comprises the one or more processors performing a pixel by pixel comparison of color values of the added image and the one or more images from other camera views.

16. The apparatus of claim 11, wherein:

the one or more processors are configured to automatically add a new feature to the 3D model and repeat the comparing of the added image to an image from a different camera view using the color space comparison of images and the adjusting of the camera parameters and the 3D model to minimize the one or more differences.

17. An apparatus, comprising:

one or more processors; and

one or more processor readable storage mediums connected to the one or more processors, the one or more processors are configured to access sets of camera parameters for different camera views and access images from the different camera views, the one or more processors are configured to attempt to align the images from the different camera views using the camera parameters, the one or more processors configured to perform a color based comparison of the attempted aligned images and determine differences between the attempted aligned images based on the color based comparison, the one or more processors are configured to adjust the camera parameters to minimize the differences between the attempted aligned images.

18. The apparatus of claim 17, wherein:

the one or more processors configured to attempt to align the images from the different camera views by transforming one or more of the images using the camera parameters and a model of an environment.

19. The apparatus of claim 18, wherein:

the one or more processors configured to adjust the model to minimize the differences between the attempted aligned images.

20. A method, comprising:

projecting an image from at least one of a plurality of camera views to a 3D model of a word space environment;

performing a color based comparison of the projected image to one or more images from other camera views; and

adding an additional detail to the 3D model based on the color based comparison.

21. The method of claim 20, wherein:

the projecting the image includes adding the image as a texture for the 3D model.

22. The method of claim 21, wherein:

the color based comparison includes performing a pixel by pixel comparison of color values of the projected image as the texture for the 3D model to the one or more images from other camera views.

23. An apparatus, comprising:

one or more processors; and

one or more processor readable storage mediums connected to the one or more processors, the one or more processors are configured to add an image from at least one of a plurality of camera views to a model of a word space environment, the one or more processors are configured to perform an images based comparison of the added image to one or more images from other camera views, the one or more processors are configured to add an additional detail to the model based on the imaged based comparison.

24. The apparatus of claim 23, wherein:

the adding the image includes adding the image as a texture for the model; and

the model is a 3D representation of a world space location.

25. The apparatus of claim 23, wherein:

the image based comparison includes performing a pixel by pixel comparison of color values of the added image to the one or more images from other camera views.