PROVIDING VIRTUAL INSERTS USING IMAGE TRACKING WITH CAMERA AND POSITION SENSORS

• US 20090028439A1
• Filed: 12/19/2007
• Published: 01/29/2009
• Est. Priority Date: 07/27/2007
• Status: Active Grant

• Current Assignee: Sportsmedia Technology Corporation
• Original Assignee: Sportvision, Inc. (Sportsmedia Technology Corporation)
• Inventors: White, Marvin S., Cavallaro, Richard H., Milnes, Kenneth A., Elangovan, Vidya
• Granted Patent: US 8,077,981 B2
• Time in Patent Office: Days
• US Class Current: 382/209
• CPC Class Codes: A63F 2300/663 for simulating liquid objects, e.g. water, gas, fog, snow, clouds
• CPC Class Codes: A63F 2300/6692 using special effects, generally involving post-processing, e.g. blooming
• CPC Class Codes: A63F 2300/69 Involving elements of the real world in the game world, e.g. measurement in live races, real video
• CPC Class Codes: A63F 2300/8017 Driving on land or water; Flying
• CPC Class Codes: G06T 2207/30221 Sports video; Sports image
• CPC Class Codes: G06T 2207/30236 Traffic on road, railway or crossing
• CPC Class Codes: G06T 7/248 involving reference images or patches
• CPC Class Codes: G06T 7/74 involving reference images or patches
• CPC Class Codes: G06V 10/751 Comparing pixel values or logical combinations thereof, or feature values having positional relevance, e.g. template matching
• CPC Class Codes: G06V 2201/08 Detecting or categorising vehicles
• CPC Class Codes: H04N 5/265 Mixing
• CPC Class Codes: H04N 5/272 Means for inserting a foreground image in a background image, i.e. inlay, outlay
• CPC Class Codes: H04N 5/2723 Insertion of virtual advertisement; Replacing advertisements physical present in the scene by virtual advertisement data processing systems or methods specially adapted for marketing G06Q30/02

Abstract

Camera registration and/or sensor data is updated during a live event by determining a difference between an estimated position of an object in an image and an actual position of the object in the image. The estimated position of the object in the image can be based on an estimated position of the object in the live event, e.g., based on GPS or other location data. This position is transformed to the image space using current camera registration and/or sensor data. The actual position of the object in the image can be determined by template matching which accounts for an orientation of the object, a shape of the object, an estimated size of the representation of the object in the image, and the estimated position of the object in the image. The updated camera registration/sensor data can be used in detecting an object in a subsequent image.

Claims

1. A method for processing image data, comprising:

obtaining an image of a live event which is captured by at least one camera, the live event including an object;

transforming an estimated position of the object in the live event to an estimated position in the image based on registration data of the at least one camera; and

detecting a representation of the object in the image based on data indicating an orientation of the object, a shape of the object, an estimated size of the representation of the object in the image, and the estimated position of the object in the image.

2. The method of claim 1, wherein:

the estimated size of the representation of the object in the image is based on a zoom setting of the at least one camera and a distance between the at least one camera and the object.

3. The method of claim 1, wherein:

the data indicating the orientation of the object indicates a heading which is based on at least one of;

a) position differentials of the object at different times and b) an inertial measurement unit carried by the object.

4. The method of claim 1, wherein:

the data indicating the orientation of the object indicates a roll which is based on at least one of;

a) a track normal to a track on which the object travels and b) an inertial measurement unit carried by the object.

5. The method of claim 1, wherein:

the object moves on a track and the data indicating the orientation of the object is based on an orientation of the track.

6. The method of claim 1, wherein:

a position of the representation of the object in the image is processed to ensure consistency with physical constraints of the object and/or the at least one camera.

7. The method of claim 1, wherein:

the representation of the first object in the image is detected based on a position of a representation of another object in the image.

8. The method of claim 1, further comprising:

updating registration data of the at least one camera based on a position of the representation of the object in the image.

9. The method of claim 1, further comprising:

updating sensor data of the at least one camera based on a position of the representation of the object in the image.

10. The method of claim 1, wherein:

the detecting the representation comprises correlating a template of the object with different candidate representations of the object in the image to select an optimum candidate representation of the object.

11. The method of claim 10, wherein:

the template comprises a model of the object, the model has an orientation which corresponds to the orientation of the object.

12. The method of claim 10, wherein:

the template comprises a model of the object, the model has a shape which corresponds to the shape of the object.

13. The method of claim 10, wherein:

the template has a size which corresponds to the estimated size of the representation of the object in the image.

14. The method of claim 10, wherein:

the template comprises monochrome data.

15. The method of claim 10, wherein:

the template comprises a texture which is based on an appearance of the object.

16. The method of claim 10, wherein:

the texture comprises a predetermined skin which is based on an appearance of the object.

17. The method of claim 10, wherein:

the template comprises pixel data of the object from at least one prior image of the object.

18. A system for processing image data, comprising:

at least one camera for capturing an image of a live event, the live event including an object; and

at least one processing facility which receives the image, the at least one processing facility;

a) transforms an estimated position of the object in the live event to an estimated position in the image based on registration data of the at least one camera, and b) detects a representation of the object in the image based on data indicating an orientation of the object, a shape of the object, an estimated size of the representation of the object in the image, and the estimated position of the object in the image.

19. The system of claim 18, wherein:

the at least one processing facility updates registration data of the at least one camera based on a position of the representation of the object in the image.

20. The system of claim 18, wherein:

the at least one processing facility updates sensor data of the at least one camera based on a position of the representation of the object in the image.

21. The system of claim 18, wherein:

the data indicating the orientation of the object indicates a heading which is based on at least one of;

a) position differentials of the object at different times and b) an inertial measurement unit carried by the object.

22. The system of claim 18, wherein:

the object moves on a track and the data indicating the orientation of the object is based on at least one of an orientation of the track and an inertial measurement unit carried by the object.

23. A method for processing image data, comprising:

obtaining successive images of a live event which are captured by at least one camera, the live event including an object;

for each image, transforming an estimated position of the object in the live event to an estimated position in the image based on registration data of the at least one camera;

for each image, detecting a representation of the object in the image and detecting an offset of a position of the representation from the estimated position in the image; and

for at least one of the images, filtering the associated offset based on at least one offset of a past and/or future image.

24. The method of claim 23, wherein:

for at least one of the images, filtering the associated offset based on offsets of past and future images.

25. The method of claim 23, wherein:

for each image, the representation of the object in the image is detected based on data indicating an orientation of the object, a shape of the object, an estimated size of the representation of the object in the image, and the estimated position of the object in the image.