Method of tracking scene motion for live video insertion systems

• US 5,808,695 A
• Filed: 12/29/1995
• Issued: 09/15/1998
• Est. Priority Date: 06/16/1995
• Status: Expired due to Term

• Current Assignee: Disney Enterprises Incorporated (The Walt Disney Company)
• Original Assignee: Princeton Video Image, Inc. (Gabriel Technologies Corporation)
• Inventors: Tan, Yi, Das, Subhodev, Rosser, Roy J.
• Primary Examiner(s): Chin, Tommy P.
• Assistant Examiner(s): Le, Vu
• Application Number:
  US08/580,892
• Time in Patent Office: 991 Days
• Field of Search: 348/584, 348/586, 348/589, 348/580, 348/578, 348/135, 348/141, 348/169, 348/722
• US Class Current: 348/584
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/30228 Playing field
• CPC Class Codes: G06T 7/248 involving reference images or patches
• CPC Class Codes: H04N 5/145 Movement estimation for video coding H04N19/51
• CPC Class Codes: H04N 5/272 Means for inserting a foreground image in a background image, i.e. inlay, outlay
• CPC Class Codes: H04N 5/2723 Insertion of virtual advertisement; Replacing advertisements physical present in the scene by virtual advertisement data processing systems or methods specially adapted for marketing G06Q30/02

Abstract

A method for tracking motion from field to field in a sequence of related video broadcast images. The method uses template correlation to follow a set of predetermined landmarks within a scene in order to provide position information of objects in the current image. The current image object position information is compared to position information of the same objects within a reference array data table. The comparison is accomplished through the use of warp equations that map points in the current image to points in the reference array. Motion is tracked according to a velocity prediction scheme utilizing a weighted formula that emphasizes the weight of landmarks that are closer to their predicted position.

Claims

1. A method for tracking motion from field to field with sub-pixel precision in a sequence of related video images having odd and even interlaced fields, the method comprising the steps of:

a) establishing an array of idealized x and y coordinates representing a reference array having a plurality of landmarks where each landmark has unique x and y coordinates;

b) detecting the positions x and y of the visible sub-set of the reference array landmarks in the current image;

c) predicting the location, x'"'"' and y'"'"', of said visible sub-set of landmarks in the next video field with sub-pixel precision by using the detected positions of said landmark in the current and the two preceding video fields,

wherein prediction errors due to interlace differences in the video images are minimized without creating two separate streams of prediction data, by using the position of the landmark found in the previous video field and adding to that the difference vector equal to the measured difference in said landmark'"'"'s position in the current field and said landmark'"'"'s position in the video two fields ago; and

,

d. mapping x and y coordinates in a current image to said x and y coordinates in said reference array using a calculated difference between the predicted location of the visible sub-set of landmarks in the current field, which was predicted in step (c) from the previous video field, and the detected positions of corresponding landmarks of the visible sub-set in the current field to thereby weight each detected landmark'"'"'s contribution to the mapping function.

2. A system for tracking motion from field to field with sub-pixel precision in a sequence of related video images having odd and even interlaced fields comprising:

a) scanning means for establishing an array of idealized x and y coordinates representing a reference array having a plurality of landmarks where each landmark has unique x and y coordinates;

b) detecting means for detecting the positions x and y of the visible sub-set of the reference array landmarks in the current image;

c) predicting means for predicting the location, x'"'"' and y'"'"', of said visible sub-set of landmarks in the next video field with sub-pixel precision by using the detected positions of said landmark in the current and the two preceding video fields,

wherein prediction errors due to interlace differences in the video images are minimized without creating two separate streams of prediction data, by using the position of the landmark found in the previous video field and adding to that the difference vector equal to the measured difference in said landmark'"'"'s position in the current field and said landmark'"'"'s position in the video two fields ago; and

,

d. mapping means for mapping x and y coordinates in a current image to said x and y coordinates in said reference array using a calculated difference between the predicted location of the visible sub-set of landmarks in the current field, which was predicted in step (c) from the previous video field, and the detected positions of corresponding landmarks of the visible sub-set in the current field to thereby weight each detected landmark'"'"'s contribution to the mapping function.

3. A method for tracking motion from field to field in a sequence of related video images having odd and even interlaced fields, the method comprising the steps of:

a) establishing an array of idealized x and y coordinates representing a reference array having a plurality of landmarks where each landmark has unique x and y coordinates;

b) mapping x and y coordinates in a current image to said x and y coordinates in said reference array, where said mapping is achieved according to the following relationships;

space="preserve" listing-type="equation">x'"'"'=a+bx+cy

space="preserve" listing-type="equation">y'"'"'=d+ex+fy

where;

x is a horizontal coordinate in the reference array,

y is a vertical coordinate in the reference array,

x'"'"' is a horizontal coordinate in the current scene,

y'"'"' is a vertical coordinate in the current scene,

a is a warp parameter for horizontal translation of the object in the x direction,

b is a warp parameter for magnification between the reference array and the current image in the x direction,

c is a warp parameter for a combination of rotation and skew in the x direction,

d is a warp parameter for vertical translation of the object in the y direction,

e is a warp parameter for a combination of rotation and skew in the y direction, and

f is a warp parameter for magnification between the reference array and the current image in the y direction; and

,

c) predicting the future location of said landmark coordinates, x'"'"' and y'"'"', based on a detected change of position of said landmark from the previous field,

wherein prediction errors due to interlace differences in the video images are minimized by using the position of the landmark found in the previous video field adjusted by the difference in position of the same landmark in the two preceding fields.

4. The method of claim 3 wherein said video images are vertically interlaced where images from field to field alternate between even and odd fields;

in which two even fields are defined to be like fields, two odd fields are defined to be like fields, and an even field associated with an odd field are defined to be unlike fields.

5. The method of claim 4 wherein said predicting the future location of said landmark coordinates, x'"'"' and y'"'"', for said interlaced video images is based on a detected change of position of said landmark from the previous like field.

6. The method of claim 5 further comprising the steps of:

d) searching for one of said landmarks in said current image by means of correlation using a template where the search is conducted over a substantial region spanning the predicted location of said landmark;

e) multiplying the results of said correlation search in step (d) by a weighting function giving greater weight to correlations closer in distance to the predicted location of said landmark to yield a weighted correlation surface;

f) searching said weighted correlation surface for its peak value.

7. The method of claim 5 further comprising the steps of:

g) determining new warp parameters a,b,c,d,e and f for a current image based on said landmark'"'"'s current position in a current image weighted by said weighting function for that landmark,

wherein emphasis is given to landmarks which are closer to their predicted position.

8. The method of claim 7 wherein said weighting function comprises the following relationship:

##EQU4## where;

g,h,i,j,k, and l are numerical constants;

xp is the predicted x coordinate location of said landmark;

xm is the measured x coordinate position of said landmark;

yp is the predicted y coordinate location of said landmark; and

,

ym is the measured y coordinate position of said landmark.

9. The method of claim 8 further including the step of:

h) updating said landmark locations in said reference array according to the location of said landmarks in said current image,

wherein said updating is performed based upon well identified landmarks and according to said landmark weighting function.

10. The method of claim 9 further comprising the step of

i) establishing three types of reference arrays prior to broadcast including;

i) a code reference array having landmark coordinates equal to said reference landmark coordinates,

ii) a game reference array having landmark coordinates initially set equal to said code reference array coordinates, and,

iii) a tracking reference array having landmark coordinates initially set equal to said code reference array coordinates.

11. The method of claim 10 further comprising the steps of:

j) changing said tracking reference array of coordinates during a broadcast; and

,

k) resetting the tracking reference array of coordinates to said game reference array of coordinates after a scene cut.

12. The method of claim 11 wherein said video system is controlled by an operator and said method further comprises the step of:

l) selectively choosing to set said current tracking reference array of coordinates equal to said game reference array of coordinates or to set said game reference array of coordinates back to said code reference array of coordinates,

wherein said operator can update or override the game or tracking reference array of coordinates.

13. The method of claim 12 further comprising the steps of:

m) establishing a set of sensor points in a pattern around the location of each said landmark said sensor points being able to detect changes in color and illumination;

n) determining if said sensor points are different in color or illumination from the expected color or illumination; and

,

o) excluding said landmark from future calculations if said color or illumination is substantially different from what was expected,

wherein said landmark is deemed to be occluded if said color or illumination at said sensor points is substantially different from the expected color or illumination.

14. The method of claim 13 wherein said correlation template is a 15 by 15 pixel window.

15. The method of claim 7 wherein said weighting function comprises the following relationship:

##EQU5## where;

xp is the predicted x coordinate location of said landmark;

xm is the measured x coordinate position of said landmark;

yp is the predicted y coordinate location of said landmark; and

,

ym is the measured y coordinate position of said landmark.

16. The method of claim 5 further comprising the steps of:

p) searching for one of said landmarks in said current image by means of correlation using a template where the starting point of the search is substantially centered at the predicted location of said landmark;

q) performing said search beginning from said predicted location and proceeding outward looking for a match; and

,

r) discontinuing said search for said landmark when said match exceeds a threshold value.

17. A method for tracking motion from field to field in a sequence of related video images having odd and even interlaced fields, the method comprising the steps of:

a) establishing an array of idealized x and y coordinates representing a reference array having a plurality of landmarks where each landmark has unique x and y coordinates;

b) mapping x and y coordinates in a current image to said x and y coordinates in said reference array, where said mapping is achieved according to the following relationships;

space="preserve" listing-type="equation">x'"'"'=a+bx

space="preserve" listing-type="equation">y'"'"'=d+by

where;

x is a horizontal coordinate in the reference array,

y is a vertical coordinate in the reference array,

x'"'"' is a horizontal coordinate in the current scene,

y'"'"' is a vertical coordinate in the current scene,

a is a warp parameter for horizontal translation of the object in the x direction,

b is a warp parameter for magnification between the reference array and the current image, and,

d is a warp parameter for vertical translation of the object in the y direction; and

,

c) predicting the future location of said landmark coordinates, x'"'"' and y'"'"', based on a detected change of position of said landmark from the previous field,

wherein prediction errors due to interlace differences in the video images are minimized by using the position of the landmark found in the previous video field adjusted by the difference in position of the same landmark in the two preceding fields.

18. A system for tracking motion from field to field in a sequence of related video images having like and unlike interlaced fields comprising:

scanning means for establishing an array of idealized x and y coordinates representing a reference array having a plurality of landmarks where each landmark has unique x and y coordinates;

mapping means for mapping x and y coordinates in a current image to said x and y coordinates in said reference array; and

,

predicting means for predicting the future location of said landmark coordinates, x'"'"' and y'"'"', based on a detected change of position of said landmark from the previous field,

wherein prediction errors due to interlace differences in the video images are minimized by using the position of the landmark found in the previous video field adjusted by the difference in position of the same landmark in the two preceding fields; and

,

searching means for searching for one of said landmarks in said current image with a correlation template substantially centered at the predicted location of said landmark wherein said search is performed beginning from said predicted location and proceeding outward looking for a match and discontinued when said match exceeds a threshold value.

19. The system of claim 18 further comprising:

weighting means for assigning each landmark a weighting function whose value is inversely proportional to the distance away said landmark is from its predicted position.

20. The system of claim 19 further comprising:

updating means for updating said landmark locations in said reference array according to the location of said landmarks in said current image wherein said updating is performed based upon well identified landmarks and according to said landmark weighting function.

21. The system of claim 20 further comprising:

means for establishing three types of reference arrays prior to broadcast including;

i) a code reference array having landmark coordinates equal to said reference landmark coordinates,

ii) a game reference array having landmark coordinates initially set equal to said code reference array of coordinates, and,

iii) a tracking reference array having landmark coordinates initially set equal to said code reference array of coordinates.

22. The system of claim 21 further comprising:

operator input means for changing said tracking reference array of coordinates during a broadcast and resetting the tracking reference array of coordinates to said game reference array of coordinates after a scene cut.

23. The system of claim 22 further comprising:

sensor means for establishing a set of sensor points in a pattern around the location of each landmark wherein said sensor points are able to detect changes in color and illumination for determining if said sensor points are different in color or illumination from the expected color or illumination.

24. The system of claim 23 further comprising:

occlusion means for excluding a landmark from future calculations if said color or illumination around said landmark is substantially different from what was expected.