Video merging employing pattern-key insertion

• US 5,923,791 A
• Filed: 07/29/1996
• Issued: 07/13/1999
• Est. Priority Date: 09/18/1991
• Status: Expired due to Term

• Current Assignee: Sarnoff Corporation (SRI International, Inc.)
• Original Assignee: Sarnoff Corporation (SRI International, Inc.)
• Inventors: Burt, Peter Jeffrey, Hanna, Keith James
• Primary Examiner(s): Shalwala, Bipin H.
• Application Number:
  US08/688,688
• Time in Patent Office: 1,079 Days
• Field of Search: 382/100, 382/107, 382/115, 382/284, 382/103, 382/199, 382/203, 382/282, 382/291, 382/295, 348/578, 348/587
• US Class Current: 382/295
• CPC Class Codes: H04N 5/272 Means for inserting a foreground image in a background image, i.e. inlay, outlay
• CPC Class Codes: H04N 5/2723 Insertion of virtual advertisement; Replacing advertisements physical present in the scene by virtual advertisement data processing systems or methods specially adapted for marketing G06Q30/02
• CPC Class Codes: H04N 5/275 Generation of keying signals

Abstract

Disclosed is a technique for deriving a composite video image by merging foreground and background video image data supplied from a plurality of separate video signal sources employing pattern-key insertion, rather than prior-art color-key insertion, for this purpose. Pattern-key insertion involves replacing a first pattern in a video scene with a second pattern. This is accomplished by first detecting the first pattern in the video scene and estimating the pose of this detected first pattern with respect to a reference object in the video scene. The second pattern is then geometrically transformed using the pose estimate of the detected first pattern. Finally, the detected first pattern is replaced with the geometrically-transformed second pattern.

Claims

1. Apparatus for replacing a first pattern in a sequence of successive 1-dimensional video image frames of a scene with a second pattern comprising:

first means for detecting relative locations of at least two landmarks related to said first pattern and for estimating, from the detected landmarks the pose of said first pattern with respect to each of said video image frames in said sequence;

second means for geometrically transforming said second pattern using the pose estimate of said detected first pattern; and

third means responsive to said detected first pattern for replacing said detected first pattern with said geometrically transformed second pattern.

2. Apparatus for processing a first video signal which includes a plurality of successive frames to locate a spatial region in each of the successive frames into which a second video image, independent of the first video image, is to be inserted, the apparatus comprising:

pattern recognition means for processing the first video signal to automatically identify images of a plurality of landmarks in each of the plurality of frames, and to determine a position for each of the plurality of landmarks in each of the plurality of frames, the pattern recognition means including;

means for determining a position of a first landmark in a frame;

means, responsive to the determined position of a first landmark of the plurality of landmark images in a frame, for identifying a target search region for a second landmark of the plurality of landmarks in the frame; and

means for determining a position of the second landmark in the frame by searching the target search region; and

means for identifying boundaries in the first video image which correspond to an area in the first video image into which the second video image is to be inserted, said boundaries being identified relative to the determined positions of the first and second landmarks.

3. Apparatus for replacing a first pattern in a sequence of successive video image frames of a scene with a second pattern comprising:

first means for locating the first pattern in the sequence of successive video image frames and for obtaining respective estimates of orientation and size for the first pattern in each of the image frames, including;

means for determining locations for a plurality of landmarks relative to an image represented by one image frame of the sequence of successive video image frames,

means, responsive to the determined locations of a subset of the plurality of landmarks, for determining the location of at least two of the plurality of landmarks in each of said video image frames in said sequence; and

means for estimating, from the determined relative location of at least two landmarks in each of said video image frames, an orientation and size for said first pattern with respect to each of said video image frames in said sequence;

second means for geometrically transforming said second pattern into a sequence of transformed second patterns responsive to the respective estimates of orientation and size of said first pattern; and

third means responsive to the estimates of the orientation and size of said first pattern for replacing occurrences of said first pattern in said video image frames with respective ones of said geometrically transformed second patterns.

4. The apparatus of claim 3, wherein:

said second means comprises means employing affine precise alignment to provide perspective transformation for estimating, from the determined location of the first landmark, the orientation and size of the first pattern in each frame of said sequence of successive video image frames.

5. The apparatus of claim 3, wherein:

said first means includes means, responsive to said sequence of successive video image frames derived from a camera view of the one of said plurality of landmarks, for employing coarse-to-fine search techniques for detecting said one of said plurality of landmarks in said scene.

6. The apparatus of claim 3, wherein:

said second pattern is a predetermined portion of a second image; and

said third means includes means, responsive to selected geometric patterns that define said portion of the second image, applied as an input thereto, for use in computing the geometric transformations of said second pattern using the estimated orientation and size of said first pattern in each respective frame of said sequence of successive video frames.

7. The apparatus of claim 3, wherein:

said second pattern is a moving pattern in a second video scene defined by a further sequence of successive video image frames, wherein a predetermined portion of said moving second pattern constitutes a reference pattern;

said apparatus further comprises fourth means for detecting said moving second pattern in said second scene and for estimating the orientation and size of said detected moving second pattern with respect to said reference-pattern portion thereof; and

said second means for geometrically transforming said second pattern uses the reference-pattern portion of said estimated orientation and size of said detected moving second pattern as an origin to provide a stabilized transformed image of said second pattern with respect to the reference-pattern portion thereof, and uses the estimated orientation and size of said detected first pattern to geometrically transform said stabilized transformed image of said second pattern.

8. The apparatus of claim 7 wherein:

said fourth means is responsive to said sequence of successive video image frames defining said second scene and employing affine precise alignment estimation for estimating the orientation and size of said detected moving second pattern with respect to said reference pattern portion in said second scene.

17. The apparatus of claim 3 wherein the respective locations of the plurality of landmarks and of said first pattern are stored in a world map, whereby said first pattern may be only partially included or entirely absent from one of the video image frames in said sequence;

and wherein;

the means for determining locations for the plurality of landmarks comprises;

means, responsive to the respective locations of the landmarks stored in said world map and to the determined location of the one of said plurality of landmarks in a video image frame, for inferring a relative location of a second one of said plurality of landmarks in the image represented by the one image frame; and

means, responsive to the inferred relative location of the second one of said plurality of landmarks to detect the second one of said plurality of landmarks; and

the means for estimating the orientation and size of the first pattern includes means responsive to the respective locations of the first and second ones of said plurality of landmarks for estimating the orientation and size of the first pattern in the video image frame.

18. The apparatus of claim 3, wherein each of the first and second ones of the plurality of landmarks is separated from the first pattern by a plurality of pixel positions.

19. The apparatus of claim 5, wherein:

said first means is responsive to the determined location of the one of the plurality of landmarks in a first video image frame of said sequence of successive video image frames and to a motion vector for inferring a position for the one of the plurality of landmarks in a subsequent video image frame of said sequence of successive video image frames; and

first means is responsive to the inferred position for the one of the plurality of landmarks for determining the location of the one of said plurality of landmarks in said subsequent video image frame;

said first means is responsive to the respective determined positions of the one of the plurality of landmarks in the first video image frame and in the subsequent video image frame to produce a motion vector for use by the first means in determining the location of the one of the plurality of landmarks in a further subsequent video image frame.

20. The apparatus of claim 3, wherein:

each video image frame in said sequence of video image frames includes an upper image field and a lower image field;

said first means includes first upper field processing means for locating the first pattern in the upper field portion of the sequence of successive video image frames and first lower field processing means for locating the first pattern in the lower field portion of the sequence of successive video image frames;

said second means includes second upper field processing means for geometrically transforming said second pattern into a sequence of upper field portions of the transformed second patterns and lower field processing means for geometrically transforming said second pattern into a sequence of lower field portions of the transformed second patterns; and

said third means includes third upper field processing means for replacing occurrences of the first pattern in each of the upper fields of the sequence of video image frames with a respective upper field portion of the transformed second pattern and a third lower field processing means for replacing occurrences of the first pattern in each of the lower fields of the sequence of video image frames with a respective lower field portion of the transformed second pattern.

21. The apparatus of claim 3, wherein:

said sequence of video image frames are color video image frames including a luminance signal component and a chrominance signal component;

the means for determining locations of the subset of the plurality of landmarks relative to the image represented by the one image frame is responsive to the luminance component of the video image frame to the relative exclusion of the chrominance component; and

the means for estimating an orientation and size for the first pattern with respect to each of said video image frames is responsive to the luminance component of the video image frames to the relative exclusion of the chrominance component.

9. A method for replacing a first pattern in a sequence of video image frames of a scene with a second pattern comprising the steps of:

a) locating the first pattern in the sequence of successive video image frames and obtaining respective estimates of orientation and size for the first pattern in each of the image frames including the steps of;

a1) determining locations for a plurality of landmarks relative to an image represented by one image frame of the sequence of successive video image frames;

a2) determining, responsive to the determined locations of the plurality of landmarks, the relative location of of at least two the plurality of landmarks in each of said video image frames in said sequence; and

a3) estimating an orientation and size for the first pattern with respect to the determined location of the one of the plurality of landmarks in each of said video image frames in said sequence;

b) geometrically transforming said second pattern into a sequence of second patterns using the respective estimates of orientation and size of said first pattern; and

c) replacing said detected first pattern with a respective one of said geometrically transformed second patterns in response to the estimates of orientation and size of said first pattern in each of said video image frames in said sequence.

10. The method of claim 9, wherein:

step (a3) employs affine precise alignment to provide perspective transformation for estimating the orientation and size of said first pattern with respect to the determined location of the one of the plurality of landmarks in each of said video image frames in said sequence.

11. The method of claim 9, wherein:

step (a2) is responsive to video image frames derived from a camera view of ones of said plurality of landmarks and employs coarse-to-fine search techniques for detecting said one of said plurality of landmarks in said scene.

12. The method of claim 9, wherein:

said second pattern is a fixed pattern; and

step (b) is responsive to selected geometric parameters that define said fixed pattern to compute the geometric transform of said second pattern using the estimated orientation and size of said first pattern.

13. The method of claim 9 wherein:

said second pattern is a moving pattern in a second scene defined by a further sequence of successive video image frames, wherein a predetermined portion of said moving second pattern constitutes a reference pattern;

the method further comprises the step of (d) detecting said moving second pattern in said second scene and estimating the orientation and size of said detected moving second pattern with respect to said reference pattern portion thereof; and

the step (b) geometrically transforms said second pattern using the estimated orientation and size of the moving second pattern to provide a stabilized transformed image of said second pattern, and uses the estimated orientation and size of the first pattern to geometrically transform said stabilized transformed image of said second pattern.

14. The method of claim 13, wherein:

step (d) is responsive to said sequence of successive video image frames defining said second scene and employs affine precise alignment estimation for estimating the orientation and size of said detected moving second pattern with respect to said reference-pattern portion in said second scene.

15. The method of claim 9, wherein:

the method further comprises the step of (d) generating said second pattern using a graphics generator; and

step (b) employs the graphics generator to geometrically transform the second pattern.

16. The method of claim 9 wherein the respective locations of the plurality of landmarks and of said first pattern are stored in a world map, whereby said first pattern may be only partially included or entirely absent from one of the video image frames in said sequence;

and wherein;

step (a1) comprises the steps of;

employing the respective locations of the landmarks stored in said world map together with the determined location of the one of said plurality of landmarks in a video image frame to infer a relative location of a further one of said plurality of landmarks in the image represented by the one image frame; and

employing the inferred relative location of the further one of said plurality of landmarks to determine the location the further one of said plurality of landmarks; and

step (a3) comprises the step of;

employing the respective locations of the one of said plurality of landmarks and the further one of said plurality of landmarks to estimate the orientation and size of the first pattern in the video image frame.