System and method for inserting static and dynamic images into a live video broadcast

• US 5,892,554 A
• Filed: 11/28/1995
• Issued: 04/06/1999
• Est. Priority Date: 11/28/1995
• Status: Expired due to Term

• Current Assignee: Disney Enterprises Incorporated (The Walt Disney Company)
• Original Assignee: Princeton Video Image, Inc. (Gabriel Technologies Corporation)
• Inventors: Fant, Karl, DiCicco, Darrell S.
• Primary Examiner(s): FLYNN, NATHAN J
• Application Number:
  US08/563,598
• Time in Patent Office: 1,225 Days
• Field of Search: 382/100, 382/173, 382/295, 382/103, 382/284, 382/282, 348/589, 348/590, 348/588, 348/593, 348/584, 345/118, 345/133, 345/135, 345/125
• US Class Current: 348/584
• CPC Class Codes: H04N 5/272 Means for inserting a foreground image in a background image, i.e. inlay, outlay
• CPC Class Codes: H04N 5/2723 Insertion of virtual advertisement; Replacing advertisements physical present in the scene by virtual advertisement data processing systems or methods specially adapted for marketing G06Q30/02

Abstract

A live video insertion system efficiently places static or dynamic images into a live broadcast. The system initially identifies natural landmarks within the video scene that can be automatically identified and tracked as the field of view of the camera pans and zooms across the scene. The locations of the landmarks are mathematically modeled and stored as a constellation of locations on a mathematical grid. An arbitrary reference point, preferably not necessarily coincident with a selected natural landmark, is located within or without of the grid and used as an origin for the purpose of inserting the static or dynamic image within the field of view of the camera. For static images, it is frequently desirable to place an edge or border of the image to be inserted on the reference point. For dynamic insertions, the distance between the reference point and the inserted dynamic image is automatically changed from frame to frame in order to make the image appear as though it'"'"'s translating or changing, or both.

Claims

1. A method of placing an insertable image into a live video broadcast scene containing natural landmarks, said method comprising the steps of:

a. creating a landmark model characteristic of a first set of natural landmarks in a given scene, said landmark model comprising a sub-pixel accurate, geometric relationship between landmarks;

b. superimposing a coordinate system on said landmark model capable of registering each landmark position at a sub-pixel level of accuracy to thereby create a reference array of landmark positions;

c. defining a required location of said insertable image with respect to said reference array;

d. scanning said scene with a video camera to create a current image;

e. identifying a second set of natural landmarks in said current image, corresponding to a sub-set of the original first set of natural landmarks, by using the landmark model;

f. comparing the sub-pixel location of multiple corresponding landmarks from the second set of natural landmarks located in said current image with their location in the reference array as defined by the landmark model and using that comparison to calculate the zoom and translation of the current scene with respect to the reference array of landmarks; and

,

g. inserting said insertable image into said current scene of said live video broadcast at a location in the current image corresponding to the defined required location, with sub-pixel accuracy,

wherein said insertable image appears natural in said live video broadcast.

3. The method of claim 1 wherein step (a) further comprises the steps of:

h. scanning the scene with a video camera to create a preliminary image of said scene; and

,

i. identifying said first set of natural landmarks in said preliminary scene; and

,

j. mathematically organizing said first set of natural landmarks into said landmark model.

4. The method of claim 3 above wherein said video camera has a variable field of view and said method further comprises the further step of:

k. continually searching the field of view of said video camera to identify landmarks in the current scene; and

,

l. tracking landmarks identified in step (k) above in order to keep the landmarks in the current scene aligned with the landmarks in the landmark model.

5. The method of claim 4 wherein said identification of said landmarks in step (e) further comprises the step of:

m. decimating the preliminary scene by incremental steps to at least 3 levels of decimation to determine what landmark features will survive searching and tracking in a current scene.

6. The method of claim 5 wherein the identification of said landmarks in step (i) above further comprises the step of:

n. decimating the current scene by incremental steps to at least three levels of decimation to determine what landmark feature will survive searching and tracking in a current scene.

7. The method of claim 6 wherein said decimating steps (m) and (n) above further comprise performing a Burt Pyramid analysis.

8. The method of claim 7 wherein said insertable image is a static image.

9. The method of claim 8 further including the following step:

o. translating said static insertable image across the current scene by incrementally changing its coordinates from frame to frame with respect to a reference point.

10. The method of claim 7 wherein said insertable image is a dynamic image.

11. The method of claim 10 wherein said dynamic image changes shape from frame to frame when inserted into said current scene.

12. The method of claim 11 further including the following step:

p. translating said dynamic insertable image across the current scene by incrementally changing its coordinates from frame to frame with respect to a reference point.

13. The method of claim 12 above further comprising the step of:

q. automatically adjusting the zoom scale of the insertable image in response to changes of location of identified landmarks in the current scene with respect to each other.

14. The method of claim 13 wherein said coordinate system also includes a reference point and wherein said insertion step (g) further comprises the step of inserting said insertable image into said current scene at a location determined with respect to said reference point.

15. The method of claim 14 further comprising the step of:

r. artificially enhancing the visibility of natural landmarks so that they can be more readily identified by Burt Pyramid analysis.

16. The method of claim 15 wherein step (a) includes the step of creating a two-dimensional landmark model which is characteristic of said first set of landmarks in said given scene.

17. The method of claim 16 further comprising the step of:

s. adjusting said insertable image to match distortion in said two-dimensional landmark model.

18. The method of claim 17 wherein step (a) includes the step of creating a three-dimensional landmark model which is characteristic of said first set of landmarks in said given scene.

19. The method of claim 18 further comprising the step of:

t. adjusting said insertable image to match distortion in said three-dimensional landmark model.

20. The method of claim 19 further comprising the step of:

u. preparing artwork for said insertable image and adjusting it for perspective and storing the same prior to insertion and adjustment for zoom.

36. The method of claim 1 wherein said comparing step (f) is performed by comparing the sub-pixel location of at least three corresponding landmarks.

37. The method of claim 36 wherein said calculation of the zoom and translation in step (f) is performed in a single computational pass.

38. The method of claim 37 further including the following step:

h. using said landmarks located in said current scene to modify the reference array of landmark positions for use in subsequent scenes.

39. The method of claim 38 wherein the location in the current image referenced in step (g) corresponds to a predefined path of locations in the reference coordinate system.

40. The method of claim 38 wherein the location in the current image referenced in step (g) corresponds to an operator controlled arbitrary location in the reference coordinate system, which location may vary from scene to scene.

2. A system for placing an insertable image into a live video broadcast scene containing natural landmarks, said system comprising the steps of:

a. creating means for creating a landmark model characteristic of a first set of natural landmarks in a given scene, said landmark model comprising a sub-pixel accurate, geometric relationship between landmarks;

b. superimposing means for superimposing a coordinate system on said landmark model capable of registering each landmark position at a sub-pixel level of accuracy to thereby create a reference array of landmark positions;

c. defining means for defining a required location of said insertable image with respect to said reference array;

d. scanning means for scanning said scene with a video camera to create a current image;

e. identifying means for identifying a second set of natural landmarks in said current image, corresponding to a sub-set of the original first set of natural landmarks, by using the landmark model;

f. comparing means for comparing the sub-pixel location of multiple corresponding landmarks from the second set of natural landmarks located in said current image with their location in the reference array as defined by the landmark model and using that comparison to calculate the zoom and translation of the current scene with respect to the reference array of landmarks; and

,

g. inserting means for inserting said insertable image into said current scene of said live video broadcast at a location in the current image corresponding to the defined required location c, with sub-pixel accuracy,

wherein said insertable image appears natural in said live video broadcast.

21. The system of claim 2 wherein said coordinate system is an X, Y grid coordinate system and further comprises:

a reference point means incorporated in said X, Y grid coordinate system for determining the distance between a specific point in said X, Y grid coordinate system and the point of insertion of said insertable image.

22. The system of claim 21 further comprising:

tracking means for keeping the landmarks identified in said current scene aligned with the landmarks in said landmark model.

23. The system of claim 22 wherein said identifying means includes a decimation means for decimating the current scene by incremental steps to at least three levels of decimation to determine what landmark features will survive searching and tracking in a current scene.

24. The system of claim 23 wherein said decimating means comprises a Burt Pyramid decimating means.

25. The system of claim 24 wherein said insertable image is a static image.

26. The system of claim 24 wherein said insertable image is a dynamic image.

27. The system of claim 26 wherein said dynamic image changes shape from frame to frame after it is inserted into said current scene.

28. The system of claim 27 further comprising:

image driving means for driving said insertable image across the current scene by incrementally changing its coordinates with respect to said coordinate system from frame to frame.

29. The system of claim 28 further comprising:

zoom adjustment means for automatically adjusting the scale of said insertable image in response to changes of location of identified landmarks in the current scene with respect to each other.

30. The system of claim 29 further comprising:

distortion matching means for adjusting said insertable image to match distortion in said X-Y grid.

31. The system of claim 30 further including:

landmark enhancement means for artificially enhancing the visibility of natural landmarks so they could be more readily identified by said identifying means.

32. The system of claim 31 wherein said landmark model is a two-dimensional landmark model.

33. The system of claim 32 wherein said landmark model is a three-dimensional landmark model.

34. The system of claim 33 wherein said scanning means comprising a video camera.

35. The system of claim 34 wherein said scanning means also scans a preliminary scene and wherein said identifying means also identifies landmarks in said preliminary scene scanned by said video camera and wherein said landmark model is a mathematical model formed by the landmarks in said preliminary scene identified by said identifying means.

41. The system of claim 2 wherein said comparing means (f) is performed by comparing the sub-pixel location of at least three corresponding landmarks.

42. The system of claim 42 wherein said calculation of the zoom and translation in comparing means (f) is performed in a single computational pass.

43. The system of claim 42 further comprising:

h. utilization step for utilizing said landmarks located in said current scene to modify the reference array of landmark positions for use in subsequent scenes.

44. The system of claim 43 wherein the location in the current image referenced in inserting means (g) corresponds to a predefined path of locations in the reference coordinate system.

45. The system of claim 43 wherein the location in the current image referenced in inserting means (g) corresponds to an operator controlled arbitrary location in the reference coordinate system, which location may vary from scene to scene.