Motion tracking using image-texture templates

• US 6,741,725 B2
• Filed: 12/10/2002
• Issued: 05/25/2004
• Est. Priority Date: 05/26/1999
• Status: Expired due to Term

• Current Assignee: Disney Enterprises Incorporated (The Walt Disney Company)
• Original Assignee: Princeton Video Image, Inc. (Gabriel Technologies Corporation)
• Inventors: Astle, Brian
• Primary Examiner(s): Mehta, Bhavesh M.
• Assistant Examiner(s): CHAWAN, SHEELA C
• Application Number:
  US10/316,246
• Publication Number:
  US 20030081813A1
• Time in Patent Office: 532 Days
• Field of Search: 382/103, 382/107, 382/218, 382/284, 382/209, 382/217, 382/199, 382/118, 382/291, 382/190, 382/276, 348/584, 348/585-592, 348/169, 348/135, 395/118, 395/133, 395/135, 395/127
• US Class Current: 382/103
• CPC Class Codes: G01S 3/7864 T.V. type tracking systems
• CPC Class Codes: G06T 7/246 using feature-based methods, e.g. the tracking of corners or segments
• CPC Class Codes: H04N 5/272 Means for inserting a foreground image in a background image, i.e. inlay, outlay
• CPC Class Codes: H04N 5/2723 Insertion of virtual advertisement; Replacing advertisements physical present in the scene by virtual advertisement data processing systems or methods specially adapted for marketing G06Q30/02

Abstract

Image templates are extracted from video images in real-time and stored in memory. Templates are selected on the basis of their ability to provide useful positional data and compared with regions of subsequent images to find the position giving the best match. From the position data a transform model is calculated. The transform model tracks the background motion in the current image to accurately determine the motion and attitude of the camera recording the current image. The transform model is confirmed by examining pre-defined image templates. Transform model data and camera sensor data are then used to insert images into the live video broadcast at the desired location in the correct perspective. Stored templates are periodically updated to purge those that no longer give valid or significant positional data. New templates extracted from recent images are used to replace the discarded templates.

Claims

1. A method for tracking motion in a stream of images, comprising the steps of:

a) selecting one or more image templates from an original image, said image templates satisfying one or more template capture criteria;

b) storing said image templates;

c) using a first transform model to relate said image templates to a reference model, said reference model representing said original image in a coordinate system;

d) determining a current position of said image templates with respect to a current image;

e) calculating a current transform model using said current positions, said current transform model defining the way in which said reference model must be changed in order to correspond to said current image;

f) purging said image templates that fail one or more template retention criteria, including said image retention criteria of the current position of said image template being in agreement with said current transform; and

,

g) extracting new image templates from said current image, said new image templates satisfying one or more template capture criteria.

2. The method of claim 1 wherein each of said image templates includes an array of pixels substantially less than all the pixels in said current video image.

3. The method of claim 1 wherein said first transform model is derived when a target area is detected.

4. The method of claim 3 wherein said detection uses a pose of a camera.

5. The method of claim 1 wherein said current transform model includes one or more camera parameters.

6. The method of claim 1 wherein said image templates include one or more synthetic templates.

7. The method of claim 1 further including the steps of:

h) selecting one or more low resolution image templates from a low resolution original image, said low resolution image templates satisfying one or more template capture criteria; and

,

i) determining a low resolution current position of said low resolution image templates with respect to at least one low resolution current image, wherein said step d) of determining of a current position of said image templates further includes calculating a current low resolution transform model using said low resolution current positions and predicting a current position of said image templates using said current low resolution transform model, whereby a search range for finding said image templates is reduced.

8. The method of claim 1 wherein said template capturing criteria includes a color constraint.

9. The method of claim 1 wherein said template capturing criteria includes a texture constraint.

10. A method for tracking motion in a sequence of images comprising the steps of:

a) obtaining a set of image templates and original positions from an original image, said image templates satisfying one or more template capture criteria;

b) storing said image templates and original positions;

c) determining current positions of said image templates in a current image;

d) calculating a transform model based on said original positions and said current positions, said transform model being indicative of changes between said original image and said current image;

e) purging said image templates that fail one or more template retention criteria, including said image retention criteria of the current position of said image template being in agreement with said transform model; and

,

f) obtaining new image templates from said current images.

11. The method of claim 10 wherein said step f) of obtaining new image templates further includes obtaining new image templates from said original image.

12. The method of claim 10 wherein said transform model further includes one or more indicators of change within said current image.

13. The method of claim 10 wherein said transform model further includes one or more indicators of changes of camera parameters used to obtain said original and said current images.

14. A method for tracking motion in a sequence of images comprising the steps of:

a) obtaining a set of image templates and original positions from an original image, said image templates satisfying one or more template capture criteria;

b) storing said image templates and original positions;

c) determining current positions of said image templates in a current image;

d) calculating a transform model based on said original positions and said current positions, said transform model being indicative of changes between said original image and said current image;

e) purging said image templates that fail one or more template retention criteria;

f) obtaining new image templates from said current images;

g) archiving said purged image templates; and

,

h) reusing said archived image templates when they are predicted to appear in said current image.

15. The method of claim 14 further including the step of:

i) processing said archived image templates prior to said step c) of determining a current position.

16. The method of claim 14 further including the step of:

j) processing said stored image templates prior to said step c) of determining a current position.

17. The method of claim 14 further including the step of:

k) detecting image feature positions, and wherein said step d) of calculating a transform model is further based on said image feature positions.

18. The method of claim 17 where in said step d) of calculating a transform model further includes gradual modifications of said transform model to conform to said image feature positions.

19. An apparatus for tracking motion in a stream of video images, comprising:

means for deriving a first transform model;

means for selecting one or more image templates from an original image, said image templates satisfying one or more template capture criteria;

means for storing said image templates;

means for using said first transform model to relate said image templates to a reference model, said reference model representing said original image in a coordinate system;

means for determining a current position of said image templates with respect to a current image;

means for calculating a current transform model using said current positions, said current transform model defining the way in which said reference model must be changed to correspond to said current image;

means for purging said image templates that fail one or more template retention criteria, including said image retention criteria of the current position of said image template being in agreement with said current transform; and

,

means for extracting new image templates from said current image, said new image templates satisfying one or more template capture criteria.

20. An apparatus for tracking motion in a sequence of images, comprising:

means for obtaining a set of image templates and original positions from an original image, said image templates satisfying one or more template capture criteria;

means for storing said image templates and original positions;

means for determining current positions of said image templates in a current image;

means for calculating a transform model based on said original positions and said current positions, said transform model being indicative of changes between said original image and said current image;

means for purging said image templates that fail one or more template retention criteria, including said image retention criteria of the current position of said image template being in agreement with said current transform; and

,

means for obtaining new image templates from said current images.