Providing virtual inserts using image tracking with camera and position sensors

• US 8,077,981 B2
• Filed: 12/19/2007
• Issued: 12/13/2011
• Est. Priority Date: 07/27/2007
• Status: Active Grant

• Current Assignee: Sportsmedia Technology Corporation
• Original Assignee: Sportvision, Inc. (Sportsmedia Technology Corporation)
• Inventors: Elangovan, Vidya, Cavallaro, Richard H., White, Marvin S., Milnes, Kenneth A.
• Primary Examiner(s): Mariam, Daniel
• Assistant Examiner(s): RICE, ELISA M
• Application Number:
  US11/960,586
• Publication Number:
  US 20090028439A1
• Time in Patent Office: 1,455 Days
• Field of Search: 382/209
• US Class Current: 382/209
• CPC Class Codes: A63F 2300/663 for simulating liquid objects, e.g. water, gas, fog, snow, clouds
• CPC Class Codes: A63F 2300/6692 using special effects, generally involving post-processing, e.g. blooming
• CPC Class Codes: A63F 2300/69 Involving elements of the real world in the game world, e.g. measurement in live races, real video
• CPC Class Codes: A63F 2300/8017 Driving on land or water; Flying
• CPC Class Codes: G06T 2207/30221 Sports video; Sports image
• CPC Class Codes: G06T 2207/30236 Traffic on road, railway or crossing
• CPC Class Codes: G06T 7/248 involving reference images or patches
• CPC Class Codes: G06T 7/74 involving reference images or patches
• CPC Class Codes: G06V 10/751 Comparing pixel values or logical combinations thereof, or feature values having positional relevance, e.g. template matching
• CPC Class Codes: G06V 2201/08 Detecting or categorising vehicles
• CPC Class Codes: H04N 5/265 Mixing
• CPC Class Codes: H04N 5/272 Means for inserting a foreground image in a background image, i.e. inlay, outlay
• CPC Class Codes: H04N 5/2723 Insertion of virtual advertisement; Replacing advertisements physical present in the scene by virtual advertisement data processing systems or methods specially adapted for marketing G06Q30/02

Abstract

Camera registration and/or sensor data is updated during a live event by determining a difference between an estimated position of an object in an image and an actual position of the object in the image. The estimated position of the object in the image can be based on an estimated position of the object in the live event, e.g., based on GPS or other location data. This position is transformed to the image space using current camera registration and/or sensor data. The actual position of the object in the image can be determined by template matching which accounts for an orientation of the object, a shape of the object, an estimated size of the representation of the object in the image, and the estimated position of the object in the image. The updated camera registration/sensor data can be used in detecting an object in a subsequent image.

Claims

1. A processor-implemented method for processing image data, comprising the processor-implemented steps of:

obtaining an image of a live event, the image includes a pixel representation of an object which is moving in the live event, the object have an appearance, and the image is captured at a point in time in the live event by at least one camera;

determining an orientation of the object in the live event for the point in time;

determining a position of the object in the live event for the point in time;

based on the position, determining a distance between the at least one camera and the object for the point in time;

based on the distance and a zoom setting of the at least one camera, estimating a size of the pixel representation of the object in the image;

based on the position and registration data of the at least one camera, estimating a position of the pixel representation of the object in the image;

providing a template comprising a model which mimics the appearance of the object, the model has an orientation which corresponds to the determined orientation of the object, and the template has a size which corresponds to the estimated size of the pixel representation of the object in the image; and

detecting the pixel representation of the object in the image by correlating the template with different candidate pixel representations of the object in the image to select an optimum candidate pixel representation of the object in the image, the candidate pixel representations are based on the estimated position of the pixel representation of the object in the image.

2. The processor-implemented method of claim 1, wherein:

the object moves on a track in the live event;

the at least one camera has a fixed position;

different distances from the fixed position to the object are calculated for different positions of the track; and

the distance between the at least one camera and the object for the point in time is based on one of the calculated distances which corresponds to the determined positioned of the object in the live event for the point in time.

3. The processor-implemented method of claim 1, wherein:

the determined orientation of the object is based on a heading which is based on position differentials of the object at different times in the live event.

4. The processor-implemented method of claim 1, wherein:

the determined orientation of the object is based on a roll of the object, the roll is determined by computing a track normal to a track on which the object travels, the computing is based on the determined position of the object.

5. The processor-implemented method of claim 1, wherein:

the object moves on a track and the orientation of the object is determined based on an orientation of the track at the determined position of the object.

6. The processor-implemented method of claim 1, wherein:

a position of the pixel representation of the object in the image is processed to ensure consistency with physical constraints of the object and/or the at least one camera.

7. The processor-implemented method of claim 1, wherein:

the pixel representation of the object in the image is detected based on a position of a pixel representation in the image of another object in the live event.

8. The processor-implemented method of claim 1, further comprising:

updating the registration data of the at least one camera during the live event based on a position of the pixel representation of the object in the image.

9. The processor-implemented method of claim 1, further comprising:

updating sensor data of the at least one camera during the live event based on a position of the pixel representation of the object in the image.

10. The processor-implemented method of claim 1, wherein:

the object has a shape and a paint scheme, and the model mimics the shape and the paint scheme of the object.

11. The processor-implemented method of claim 1, wherein:

the template comprises monochrome data.

12. The processor-implemented method of claim 1, wherein:

the template comprises a texture which mimics the appearance of the object.

13. The processor-implemented method of claim 1, wherein:

the model mimics the appearance of the object by using a predetermined skin which is based on the appearance of the object.

14. The processor-implemented method of claim 1, wherein:

the template comprises pixel data of the object from at least one prior image of the object.

15. The processor-implemented method of claim 1, wherein:

the object comprises a vehicle;

the appearance of the object includes a shape and paint scheme of the vehicle; and

the model mimics the shape and paint scheme of the vehicle.

16. The processor-implemented method of claim 1, wherein:

the determined orientation of the object is based on a roll of the object; and

the roll is determined based on an inertial measurement unit carried by the object.

17. A system for processing image data, comprising:

at least one camera for capturing an image of a live event at a point in time, the live event including an object; and

at least one processing facility which receives the image, the at least one processing facility;

a) transforms an estimated position of the object in the live event at the point in time to an estimated position in the image based on registration data of the at least one camera, and b) detects a pixel representation of the object in the image based on data indicating an orientation of the object in the live event for the point in time, a shape of the object, an estimated size of the pixel representation of the object in the image, and the estimated position of the object in the image.

18. The system of claim 17, wherein:

the at least one processing facility updates registration data of the at least one camera in each image of successive images obtained from the at least one camera, based on positions of pixel representations of the object in the each image of the successive images.

19. The system of claim 17, wherein:

the at least one processing facility updates sensor data of the at least one camera during the live event based on a position of the pixel representation of the object in the image.

20. The system of claim 17, wherein:

the data indicating the orientation of the object indicates a heading which is based on position differentials of the object at different times in the live event.

21. The system of claim 17, wherein:

the object moves on a track and the data indicating the orientation of the object is based on an orientation of the track at the estimated position of the object in the live event at the point in time.

22. The system of claim 17, wherein:

the at least one processing facility detects the representation of the object in the image based on a paint scheme of the object.

23. A processor-implemented method for processing image data, comprising the processor-implemented steps of:

obtaining successive images of a live event which are captured by at least one camera, the live event including an object;

for each image, transforming an estimated position of the object in the live event to an estimated position in the image based on registration data of the at least one camera;

for each image, detecting a representation of the object in the image and detecting an offset in pixel space of a position of the representation of the object in the image from the estimated position of the object in the image; and

for at least one of the images, filtering the associated offset based on at least one offset of a past and/or future image.

24. The processor-implemented method of claim 23, wherein:

for at least one of the images, filtering the associated offset in pixel space based on offsets in pixel space of past and future images.

25. The processor-implemented method of claim 23, wherein:

for each image, the representation of the object in the image is detected based on data indicating an orientation of the object, a shape of the object, an estimated size of the representation of the object in the image, and the estimated position of the object in the image.