System and method of real time insertions into video using adaptive occlusion with a synthetic reference image

• US 5,953,076 A
• Filed: 06/12/1996
• Issued: 09/14/1999
• Est. Priority Date: 06/16/1995
• Status: Expired due to Term

• Current Assignee: Disney Enterprises Incorporated (The Walt Disney Company)
• Original Assignee: Princeton Video Image, Inc. (Gabriel Technologies Corporation)
• Inventors: Das, Subhodev, Astle, Brian
• Primary Examiner(s): Flynn, Nathan
• Assistant Examiner(s): Srivastava, Vivek
• Application Number:
  US08/662,089
• Time in Patent Office: 1,189 Days
• Field of Search: 348/580, 348/581, 348/582, 348/583, 348/584, 348/585, 348/586, 348/587, 348/588, 348/589, 348/590, 348/591, 348/592
• US Class Current: 348/584
• CPC Class Codes: G06T 7/246 using feature-based methods, e.g. the tracking of corners or segments
• CPC Class Codes: H04N 5/145 Movement estimation for video coding H04N19/51
• CPC Class Codes: H04N 5/272 Means for inserting a foreground image in a background image, i.e. inlay, outlay
• CPC Class Codes: H04N 5/2723 Insertion of virtual advertisement; Replacing advertisements physical present in the scene by virtual advertisement data processing systems or methods specially adapted for marketing G06Q30/02

Abstract

A system and method for realtime occlusion processing for seamlessly and realistically blending an inserted image such as an advertisement into a region of a live broadcast image without obscuring the action of the live image. The average color and intensity of a synthetic reference image containing at least some of the region to be replaced is compared to the average color and intensity of the current live broadcast image to determine the difference between the two images. The resulting difference image obtained from processing the current image and synthetic, reference image determines areas of the intended insertion region within the current image which are obscured by live action. The processor then generates an occlusion mask based on the difference image and only those pixels that are unoccluded within the intended insertion region are allowed to be inserted into the live broadcast.

Claims

1. A method of occlusion processing for merging an inserted image with a current image in a live broadcast of an event, the method comprising the steps of:

(a) obtaining a reference image of a scene, said scene including at least one intended insertion region;

(b) processing the reference image to obtain a synthetic reference image;

(c) warping the synthetic reference image into the current image using warp parameters which transform the synthetic reference image to the same size and perspective as the current image;

(d) updating the warped, synthetic reference image to reflect camera and lighting conditions in the current image;

(e) creating a difference image by subtracting the updated, warped, synthetic reference image from the current image;

(f) creating an occlusion mask from said difference image for deciding whether to broadcast the pixels of the current image, the pixels of the inserted image, or a mixture of each; and

(g) mixing the inserted image and the current image according to the occlusion mask to obtain a broadcast image which seamlessly blends the inserted image into a portion of the current image without obscuring the action of the event.

2. The method of claim 1 wherein said synthetic reference image and said current image include sensor points that provide color and luminance information for dynamically adjusting the color and intensity of the warped, synthetic reference image to reflect camera and lighting conditions in the current image.

3. The method of claim 2 wherein said sensor points comprise array sensor points and distributed sensor points, said array sensor points dispersed throughout the intended insertion region, and said distributed sensor points dispersed outside of the intended insertion region.

4. The method of claim 3 wherein said updating step (d) further comprises the steps of:

(h) averaging the values of the sensor points in the current image to obtain an average value;

(i) creating three additional average values of reference sensor points based on the average value of the sensor points of the current image, said three additional average values comprising a code reference average value, a game reference average value, and a tracking reference average value, wherein each additional average value is initially set equal to the average value of the warped, synthetic reference image sensor points;

(j) updating the game reference average value periodically to reflect any changes in the overall lighting conditions;

(k) updating the tracking reference average value on a field to field basis;

(l) comparing the updated tracking reference average value against each warped array sensor point value in the current image yielding a sensor point difference value;

(m) comparing the sensor point difference value against a threshold value to determine whether and how many current image array sensor points have been occluded by objects in the current image wherein,

if a sufficient percentage of the array sensor points remain unoccluded then using said unoccluded array sensor points for updating the warped, synthetic reference image for this field,

if an insufficient percentage of the array sensor points remain unoccluded then using the distributed sensor points for updating the warped, synthetic reference image for this field provided a sufficient percentage of the distributed sensor points remain unoccluded,

if an insufficient percentage of the array sensor points remain unoccluded and there is an insufficient percentage of unoccluded distributed sensor points then,

calculating the variances of up to three subsets of the occluded array sensor points,

comparing the variance values to a threshold value for the intended insertion region and if any of said variance values is within said threshold value then using the occluded array sensor points for updating the warped, synthetic reference image for this field,

otherwise no updating is done for this field;

(n) averaging the usable current image sensor points;

(o) subtracting the average value of the current image sensor points from the average value of the usable warped, synthetic reference image sensor points yielding an average sensor point difference value; and

(p) updating each pixel of the warped, synthetic reference image using the magnitude of the average sensor point difference value.

5. The method of claim 4 wherein said creating a difference image step (e) further comprises the steps of:

(q) obtaining difference images for each of the Y, U, and V color components by subtracting the Y color component of the updated, warped, synthetic reference image from the Y color component of the current image, subtracting the U color component of the updated, warped, synthetic reference image from the U color component of the current image, and subtracting the V color component of the updated, warped, synthetic reference image from the V color component of the current image; and

(r) forming a composite difference image from each of the Y, U, and V difference images.

6. The method of claim 5 wherein said creating an occlusion mask step (f) further comprises the steps of:

(s) clipping the composite difference image according to operator specified high and low clipping values;

(t) filtering, with a two dimensional low pass filter, the clipped composite difference image in order to reduce errors due to noise;

(u) mapping all values of the filtered clipped composite difference image less than the operator specified low clipping value to a maximum transparency value so that the inserted image pixel will show through, mapping all values greater than the operator specified high clipping value to a minimum transparency so that the inserted image pixel is occluded, and linearly mapping all values within the operator specified high and low clipping values to a corresponding transparency percentage to obtain the occlusion mask.

7. The method of claim 6 further comprising the step of low pass filtering the occlusion mask to smooth off edges to avoid introducing unwanted high frequency components into the broadcast.

8. The method of claim 7 further comprising the step of altering by erosion or expansion the occlusion mask to adjust for artifacts introduced by unwanted inherent video effects.

9. The method of claim 8 further comprising the step of filtering the occlusion mask in two dimensions in order to group occluded pixels so that each occluded pixel has one or more occluded neighbors.

10. The method of claim 9 further comprising the step of temporally filtering the occlusion mask over one or more fields so that changes in the mask exhibit continuity over time.

11. The method of claim 10 wherein mixing the inserted image and current image according to the occlusion mask to obtain a broadcast image which seamlessly blends the inserted image into a portion of the current image without obscuring the action of the event comprises the steps of:

(v) combining the occlusion mask with a warped logo key to obtain a conditional key; and

(w) creating a final key which is used to mix the warped inserted image and the current image by adjusting the transparency of the conditional key to create a more realistic broadcast image.

12. The method of claim 11 wherein said averaging step (n) further comprises the step of weighting each usable sensor point according to its distance from the updated tracking reference average value.

13. The method of claim 12 wherein said forming step (r) is accomplished according to the relationship:

space="preserve" listing-type="equation">S=(w.sub.y Δ

Y.sup.2 +w.sub.c (Δ

U.sup.2 +Δ

V.sup.2)).sup.1/2

where S is the composite difference image,

w_y and w_c are weighting factors, and

Δ

Y, Δ

U, and Δ

V are the difference images for the Y,U, and V color components.

14. The method of claim 12 wherein said forming step (r) is accomplished according to the relationship:

space="preserve" listing-type="equation">S=(w.sub.y Δ

Y.sup.2 w.sub.u Δ

U.sup.2 +w.sub.v Δ

V.sup.2).sup.1/2

where S is the composite difference image,

w_y, w_u and w_v are weighting factors, and

Δ

Y, Δ

U, and Δ

V are the difference images for the Y,U, and V color components.

15. The method of claim 12 wherein said obtaining step (q) further comprises the step of obtaining a hue difference image by subtracting the hue of the updated, warped, synthetic reference image from the hue of the of the current image wherein hue is equal to arctan(U/V).

16. The method of claim 15 wherein said hue difference image is used as the composite difference image.

17. The method of claim 15 wherein said hue difference image is used in addition to the Y, U, and V color channels in forming the composite difference image.

18. The method of claim 12 wherein said obtaining step (q) further comprises the step of obtaining a texture difference image.

19. The method of claim 18 wherein said texture difference image is obtained from a variance of each pixel with respect to its eight nearest neighbors.

20. The method of claim 18 wherein said texture difference image is obtained from a variance of each pixel with respect to an average value.

21. The method of claim 1 wherein said current image is a filtered synthetic, representation of the actual current image.

22. The method of claim 5 wherein multiple synthetic reference images are obtained for a multi-colored intended insertion region, each synthetic reference image being a representation of a different color of said multi-colored intended insertion region and said composite difference image being formed from the difference image of the updated, warped, synthetic reference image which yields the minimum average sensor point difference value with respect to the current image.

23. The method of claim 1 further comprising the step of horizontally and vertically unpeaking the current image, said unpeaking comprising the steps of:

(x) delaying the current image signal by a time t;

(y) inverting said delayed current image signal;

(z) amplifying said inverted, delayed current image signal by multiplying with a gain of 2;

(aa) adding said amplified, inverted, delayed current image signal to a double delayed current image signal and to the current image signal yielding an unpeaking current image signal;

(bb) thresholding the unpeaking signal by removing all signals below a low threshold amplitude and decreasing all signals above a high threshold value to the high threshold value;

(cc) changing the gain of the thresholded signal; and

(dd) adding the gain adjusted thresholded unpeaking signal to the delayed current image signal to produce an unpeaked current image signal, wherein said steps (x) through (dd) are executed succesively for horizontal and vertical unpeaking.

24. The method of claim 23 wherein t is equal to a single horizontal scan line for vertical unpeaking and less than a single horizontal scan line for horizontal unpeaking.

25. A system for merging an inserted image with a current image in a live broadcast of an event while compensating for occluding objects in the foreground of the current image, the system comprising:

camera means for obtaining a reference image of a scene, said scene including an intended insertion region;

processor means for processing the reference image to obtain a synthetic reference image;

warping means for warping the synthetic reference image into the current image using warp parameters which transform the synthetic reference image to the same size and perspective as the current image;

updating means for updating the color and intensity of the warped, synthetic reference image to reflect camera and lighting conditions in the current image;

comparator means for creating a difference image by subtracting the updated, warped, synthetic reference image from the current image;

masking means for creating an occlusion mask from said difference image for deciding whether to broadcast the pixels of the current image or the pixels of the inserted image; and

mixing means for mixing the inserted image and the current image according to the occlusion mask to obtain a broadcast image which seamlessly blends the inserted image into a portion of the current image without obscuring the action of the event.

26. The system of claim 25 wherein said synthetic reference image and said current image include sensor points that provide color and luminance information for dynamically adjusting the color and intensity of the warped, synthetic reference image to reflect camera and lighting conditions in the current image.

27. The method of claim 26 wherein said sensor points comprise array sensor points and distributed sensor points, said array sensor points dispersed throughout the intended insertion region, and said distributed sensor points dispersed outside of the intended insertion region.

28. The method of claim 27 wherein said updating means further comprises:

sensor point averaging means for averaging the sensor points of the warped, synthetic reference image to obtain an average value;

reference creation means for creating three additional average values of reference sensor points based on the average value of the sensor points of the warped, synthetic reference image, said three additional average values comprising a code reference average value, a game reference average value, and a tracking reference average value, wherein each additional average value is initially set equal to the average value of the warped, synthetic reference image sensor points;

game reference updating means for updating the game reference average value periodically to reflect any changes in the overall lighting conditions;

tracking reference updating means for updating the tracking reference average value on a field to field basis;

tracking reference comparing means for comparing the updated tracking reference average value against each warped, synthetic reference image array sensor point value yielding a sensor point difference value;

sensor point comparing means for comparing the sensor point difference value against a threshold value to determine whether and how many warped, synthetic reference image array sensor points have been occluded by objects in the current image wherein,

if a sufficient percentage of the array sensor points remain unoccluded then using said unoccluded array sensor points for updating the warped, synthetic reference image for this field,

if an insufficient percentage of the array sensor points remain unoccluded then using the distributed sensor points for updating the warped, synthetic reference image for this field provided a sufficient percentage of the distributed sensor points remain unoccluded,

if an insufficient percentage of the array sensor points remain unoccluded and there is an insufficient percentage of unoccluded distributed sensor points then,

calculating the variances of up to three subsets of the occluded array sensor points,

comparing the variance values to a threshold value for the intended insertion region and if any of said variance values is within said threshold value then using the occluded array sensor points for updating the warped, synthetic reference image for this field,

otherwise no updating is done for this field;

averaging means for averaging the usable warped, synthetic reference image sensor points;

subtraction means for subtracting the average value of the current image sensor points from the average value of the usable warped, synthetic reference image sensor points yielding an average sensor point difference value; and

pixel updating means for updating each pixel of the warped, synthetic reference image using the magnitude of the average sensor point difference value.

29. The method of claim 28 wherein said means for creating a difference image further comprises:

differencing means for obtaining difference images for each of the Y, U, and V color components by subtracting the Y color component of the updated, warped, synthetic reference image from the Y color component of the current image, subtracting the U color component of the updated, warped, synthetic reference image from the U color component of the current image, and subtracting the V color component of the updated, warped, synthetic reference image from the V color component of the current image; and

forming means for forming a composite difference image from each of the Y, U, and V difference images.