Video merging employing pattern-key insertion

• US 5,566,251 A
• Filed: 05/31/1995
• Issued: 10/15/1996
• Est. Priority Date: 09/18/1991
• Status: Expired due to Term

• Current Assignee: Sarnoff Corporation (SRI International, Inc.)
• Original Assignee: David Sarnoff Research Center, Inc.
• Inventors: Hanna, Keith J., Burt, Peter J.
• Primary Examiner(s): Boudreau, Leo
• Assistant Examiner(s): ANDERSON, DAVID
• Application Number:
  US08/456,020
• Time in Patent Office: 503 Days
• Field of Search: 382/100, 382/173, 382/103, 382/282, 382/295, 382/298, 382/284, 395/118, 395/133, 395/135, 395/127, 348/578, 348/580, 348/584, 348/588, 348/589, 348/590, 348/593
• US Class Current: 382/284
• CPC Class Codes: H04N 5/272 Means for inserting a foreground image in a background image, i.e. inlay, outlay
• CPC Class Codes: H04N 5/2723 Insertion of virtual advertisement; Replacing advertisements physical present in the scene by virtual advertisement data processing systems or methods specially adapted for marketing G06Q30/02
• CPC Class Codes: H04N 5/275 Generation of keying signals

Abstract

Disclosed is a technique for deriving a composite video image by merging foreground and background video image data supplied from a plurality of separate video signal sources employing pattern-key insertion, rather than prior-art color-key insertion, for this purpose. Pattern-key insertion involves replacing a first pattern in a video scene with a second pattern. This is accomplished by first detecting the first pattern in the video scene and estimating the pose of this detected first pattern with respect to a reference object in the video scene. The second pattern is then geometrically transformed using the pose estimate of the detected first pattern. Finally, the detected first pattern is replaced with the geometrically-transformed second pattern.

Claims

1. In a sequence of successive 1-dimensional video image frames of a scene defined by its pictorial content, wherein each of said successive video image frames of said sequence is composed of a set of digital pixels, and wherein said scene may or may not include respective first subsets of pixels in at least some of said successive video image frames of said sequence that define a first pattern that may move in its relative location and change in its orientation in each of said successive video image flames of said sequence;

the improvement in apparatus for replacing said respective first subsets of pixels defining said first pattern in video image frames of said sequence which include said first pattern with respective second sets of pixels defining a second pattern wherein none of said pictorial content of said scene is specified whereby said scene may be composed of any type of pictorial content rather than any part of said scene being unique;

wherein said first pattern is an uncoded first pattern and said second pattern is an uncoded second pattern, and wherein said improved apparatus comprises;

first means responsive to the respective sets of pixels defining the video images manifested by said successive frames of said sequence that employs dynamic pattern recognition for detecting the relative location of said uncoded first pattern in those video image frames that do include said first pattern and for estimating the pose of said detected uncoded first pattern in accordance with those pixels defining said respective first subsets of pixels of said successive video image frames of said sequence, said first means comprising means responsive to said sequence of successive 1-dimensional video image frames derived from a camera view of at least one object, said one object defining said uncoded first pattern, and employing affine precise alignment to provide perspective transformation for estimating the pose of said detected uncoded first pattern with respect to said camera'"'"'s view in each said sequence of successive 1-dimensional video image frames;

second means for geometrically transforming said respective second sets of pixels defining said uncoded second pattern using the pose estimate of said detected uncoded first pattern; and

third means responsive to said detected uncoded first pattern for replacing said detected uncoded first pattern with said geometrically transformed uncoded second pattern.

2. In a sequence of successive 1-dimensional video image frames of a scene defined by its pictorial content, wherein each of said successive video image frames of said sequence is composed of a set of digital pixels, and wherein said scene may or may not include respective first subsets of pixels in at least some of said successive video image frames of said sequence that define a first pattern that may move in its relative location and change in its orientation in each of said successive video image frames of said sequence;

the improvement in a method for replacing video image frames of said sequence which include said first pattern with respective second sets of pixels defining a second pattern wherein none of said pictorial content of said scene is specified, whereby said scene may be composed of any type of pictorial content rather than any part of said scene being unique;

wherein said first pattern is an uncoded first pattern and said second pattern is an uncoded second pattern, and wherein said improved method comprises the steps of;

a) in response to the respective sets of pixels defining the video images manifested by said successive frames of said sequence, employing dynamic pattern recognition for detecting the relative location of said uncoded first pattern in those video image frames that do include said first pattern and estimating the pose of said detected uncoded first pattern in accordance with those pixels defining said respective first subsets of pixels of said successive video image frames of said sequence wherein step (a) is responsive to said sequence of successive 1-dimensional video image frames derived from a camera view of at least one object, said object defining said uncoded first pattern, and employs affine precise alignment to provide perspective transformation for estimating the pose of said detected uncoded first pattern with respect to said camera'"'"'s view in each said sequence of successive 1-dimensional video image frames;

b) geometrically transforming said respective second sets of pixels defining said uncoded second pattern using the pose estimate of said detected uncoded first pattern; and

c) in response to said detected uncoded first pattern, replacing said detected uncoded first pattern with said geometrically transformed uncoded second pattern.

3. In a sequence of successive 1-dimensional video image frames of a scene defined by its pictorial content, wherein each of said successive video image frames of said sequence is composed of a set of digital pixels, and wherein said scene may or may not include respective first subsets of pixels in at least some of said successive video image frames of said sequence that define a first pattern that may move in its relative location and change in its orientation in each of said successive video image frames of said sequence;

the improvement in a method for replacing video image frames of said sequence which include said first pattern with respective second sets of pixels defining a second pattern wherein none of said pictorial content of said scene is specified, and said second pattern is a moving pattern and a predetermined portion of said moving second pattern constitutes a reference pattern whereby said scene may be composed of any type of pictorial content rather than any part of said scene being unique;

wherein said first pattern is an uncoded first pattern and said second moving pattern is an uncoded second moving pattern, and wherein said improved method comprises the steps of;

a) in response to the respective sets of pixels defining the video images manifested by said successive frames of said sequence, employing dynamic pattern recognition for detecting the relative location of said uncoded first pattern in those video image frames that do include said first pattern and estimating the pose of said detected uncoded first pattern in accordance with those pixels defining said respective first subsets of pixels of said successive video image frames of said sequence;

b) geometrically transforming said respective second sets of pixels defining said uncoded second moving pattern using the pose estimate of said detected uncoded first pattern; and

c) in response to said detected uncoded first pattern, replacing said detected uncoded first pattern with said geometrically transformed uncoded second moving pattern;

d) detecting said uncoded moving second pattern in said second scene and estimating the pose of said detected uncoded moving second pattern with respect to said reference pattern portion thereof, wherein step d) is responsive to said sequence of successive 1-dimensional video image frames defining said second scene and employs affine precise alignment estimation for estimating the pose of said detected uncoded moving second pattern with respect to said reference-pattern portion in said second scene; and

said step (b) geometrically transforms said uncoded second moving pattern using the reference-pattern portion of said pose estimate of said detected uncoded moving second pattern as an origin to provide a stabilized transformed image of said uncoded second moving pattern with respect to the reference-pattern portion thereof, and using the pose estimate of said detected uncoded first pattern to geometrically transform said stabilized transformed image of said uncoded second moving pattern.

4. Apparatus for merging a portion of a first video image derived from a first video signal into a section of a second video image, the second video image being defined by a plurality of successive frames of a second video signal representing a scene, wherein the section of the second video image changes in at least one of orientation, scale and perspective in the plurality of frames, the apparatus comprising:

pattern recognition means for processing the second video signal to automatically identify images of a plurality of landmarks in each of the plurality of frames, the plurality of landmarks having a fixed spatial relationship in the scene represented by the second video image;

alignment means for automatically determining respective boundaries for the section in each of the plurality of frames of the second video signal with reference to the identified landmark images, wherein at least one of the images of the plurality of landmarks is outside of the determined boundaries in each of the plurality of frames;

geometric transform means for processing signals representing the portion of the first video image to geometrically transform the first video image to conform to the boundaries of the section of the second video image in orientation, scale and perspective in each of the plurality of frames of the second video signal; and

switch means for substituting the processed first video signal representing the geometrically transformed portion of the first video image for a part of the second video signal defined by the determined boundaries to produce a signal representing the merged first and second video images.

5. A computer implemented method for merging a portion of a first video image derived from a first video signal into a section of a second video image, the second video image being defined by a plurality of successive frames of a second video signal representing a scene, wherein the section of the second video image changes in at least one of orientation, scale and perspective in the plurality of frames, the method comprising the steps of:

processing the second video signal to automatically identify images of a plurality of landmarks in each of the plurality of frames, the plurality of landmarks having a fixed spatial relationship in the scene represented by the second video image;

determining respective boundaries for the section in each of the plurality of frames of the second video signal with reference to the identified landmark images, wherein at least one of the images Of the plurality of landmarks is outside of the determined boundaries in each of the plurality of frames;

geometrically transforming signals representing the portion of the first video image in orientation, scale and perspective to conform to the boundaries of the section of the second video image in each of the plurality of frames of the second video signal; and

substituting the processed first video signal representing the geometrically transformed portion of the first video image for a part of the second video signal defined by the determined boundaries to produce a signal representing the merged first and second video images.