VISUAL TRACKING FRAMEWORK

• US 20140219499A1
• Filed: 11/04/2013
• Published: 08/07/2014
• Est. Priority Date: 02/18/2009
• Status: Active Grant

• Current Assignee: Lucasfilm Entertainment Co Ltd (The Walt Disney Company)
• Original Assignee: Lucasfilm Entertainment Co Ltd (The Walt Disney Company)
• Inventors: Bhat, Kiran S., Allen, Brett A., Bregler, Christoph
• Granted Patent: US 9,002,064 B2
• Time in Patent Office: Days
• US Class Current: 382/103
• CPC Class Codes: G06T 2207/30201 Face
• CPC Class Codes: G06T 2207/30204 Marker
• CPC Class Codes: G06T 7/251 involving models
• CPC Class Codes: G06V 10/245 by locating a pattern; Special marks for positioning
• CPC Class Codes: G06V 10/248 by interactive preprocessing or interactive shape modelling, e.g. feature points assigned by a user
• CPC Class Codes: G06V 40/165 using facial parts and geometric relationships

Abstract

A computer program product tangibly embodied in a computer-readable storage medium includes instructions that when executed by a processor perform a method. The method includes identifying a frame of a video sequence, transforming a model into an initial guess for how the region appears in the frame, performing an exhaustive search of the frame, performing a plurality of optimization procedures, wherein at least one additional model parameter is taken into account as each subsequent optimization procedure is initiated. A system includes a computer readable storage medium, a graphical user interface, an input device, a model for texture and shape of the region, the model generated using the video sequence and stored in the computer readable storage medium, and a solver component.

Claims

1. A method of tracking a region in a sequence of video images, the method comprising:

identifying a frame of a video sequence on which to perform region tracking for a region defined using markers in another frame of the video sequence, wherein a model for the region has been generated using the video sequence, the model being based on user-selected positions for the markers in multiple reference frames of the video sequence, and wherein the model includes an average image and multiple component images, the average image representing an average appearance of the region in the reference frames, and each of the component images representing differences between the reference frames and the average image;

tracking the region in the frame of the video sequence using a multi-pass search that adapts the model to the frame, wherein the multi-pass search associates the model with a first location in the frame and performs a plurality of optimization procedures in an iterative manner seeking to improve a match between a model transformation and the frame to generate a revised model associated with a second location in the frame; and

recording the revised model transformation and the second location as an outcome of tracking the region in the frame.

2. The method set forth in claim 1 wherein the reference frames used to generate the model reflect the region to be tracked under varying conditions including one or more of lighting, states of the feature and motion blur.

3. The method set forth in claim 1 wherein each of the multiple component images are based on principal component analysis computations.

4. A method of tracking a region in a sequence of video frames, the method comprising:

identifying a frame of a video sequence on which to perform region tracking for a region defined using markers in another frame of the video sequence, wherein a model for the region has been generated using multiple reference frames of the video sequence selected by a user along with morph images between the reference frames, each morph image representing an intermediate appearance of the region generated at a location among the reference frames selected by;

(i) identifying a total number of the morph images to be generated;

(ii) computing, for every pair of the reference images, a distance value representing how much the pair of reference images differ with respect to shape change and gray level change;

(iii) adding the distance values to form a total distance value;

(iv) dividing the total distance value by the total number of the morph images to obtain a per- morph-image distance value; and

(v) distributing the morph images among the reference frames based on the distance values and the per-morph-image distance value;

tracking the region in the frame of the video sequence using a multi-pass search that adapts the model to the frame, wherein the multi-pass search associates the model with a first location in the frame and performs a plurality of optimization procedures in an iterative manner seeking to improve a match between a model transformation and the frame to generate a revised model associated with a second location in the frame; and

recording the revised model transformation and the second location as an outcome of tracking the region in the frame.

5. The method set forth in claim 4 wherein the reference frames used to generate the model reflect the region to be tracked under varying conditions including one or more of lighting, states of the feature and motion blur.

6. A method of tracking an image feature in a sequence of video frames, the method comprising:

receiving a video sequence comprising a plurality of video frames;

selecting a plurality of reference frames from the plurality of video frames, the plurality of reference frames reflecting the image feature to be tracked under varying conditions;

placing one or more markers on each reference frame to identify a plurality of reference images representing the image feature under the varying conditions;

generating a plurality of morph images from the plurality of reference images, each morph image representing an intermediate appearance of the image feature between consecutive reference images;

generating a model for texture and shape of the image feature from the reference images and the morph images;

tracking the feature in a frame of the video sequence using a multi-pass search that adapts the model to the frame.

7. The method set forth in claim 6 wherein the model includes a mean image and a plurality of component images, each component image representing a difference between an individual reference frame and the mean image.

8. The method set forth in claim 6 wherein generating the model aligns the reference images into a set of aligned images that have a common orientation.

9. The method set forth in claim 6 wherein generating the model warps the reference images to fit a canonical reference image.

10. The method set forth in claim 6 wherein at least one marker in the plurality of markers placed in each reference image is an anchor marker that cannot be modified by subsequent tracking.

11. The method set forth in claim 6 wherein the varying conditions include one or more of lighting conditions, states of the feature and motion blurr.

12. The method set forth in claim 6 further comprising, after generating the model, adding an additional reference frame to the plurality of reference frames, placing one or more markers on the additional reference frame to identify an additional reference image representing the image feature, generating at least one additional morph image based on the additional reference image, and generating a revised model for texture and shape of the image feature from the reference images, the additional reference image, the morph images and the at least one additional morph image.

13. The method set forth in claim 6 wherein the multi-pass search associates the model with a first location in the frame and performs a plurality of optimization procedures in an iterative manner seeking to improve a match between a model transformation and the frame and resulting in a revised model associated with a second location in the frame;

and wherein the method further comprises recording the revised model transformation and the second location as an outcome of tracking the region in the frame.

14. A computer program product embodied in a non-transitory computer-readable storage medium and comprising instructions that, when executed by a processor, perform a method comprising:

identifying a frame of a video sequence on which to perform region tracking for a region defined using markers in another frame of the video sequence, wherein a model for the region has been generated using the video sequence, the model being based on user-selected positions for the markers in multiple reference frames of the video sequence, and wherein the model includes an average image and multiple component images, the average image representing an average appearance of the region in the reference frames, and each of the component images representing differences between the reference frames and the average image;

tracking the region in the frame of the video sequence using a multi-pass search that adapts the model to the frame, wherein the multi-pass search associates the model with a first location in the frame and performs a plurality of optimization procedures in an iterative manner seeking to improve a match between a model transformation and the frame to generate a revised model associated with a second location in the frame; and

recording the revised model transformation and the second location as an outcome of tracking the region in the frame.

15. The computer program product of claim 14 wherein each of the multiple component images are based on principal component analysis computations.

16. A computer program product embodied in a non-transitory computer-readable storage medium and comprising instructions that, when executed by a processor, perform a method comprising:

identifying a frame of a video sequence on which to perform region tracking for a region defined using markers in another frame of the video sequence, wherein a model for the region has been generated using multiple reference frames of the video sequence selected by a user along with morph images between the reference frames, each morph image representing an intermediate appearance of the region generated at a location among the reference frames selected by;

(i) identifying a total number of the morph images to be generated;

(ii) computing, for every pair of the reference images, a distance value representing how much the pair of reference images differ with respect to shape change and gray level change;

(iii) adding the distance values to form a total distance value;

(iv) dividing the total distance value by the total number of the morph images to obtain a per-morph-image distance value; and

(v) distributing the morph images among the reference frames based on the distance values and the per-morph-image distance value;

tracking the region in the frame of the video sequence using a multi-pass search that adapts the model to the frame, wherein the multi-pass search associates the model with a first location in the frame and performs a plurality of optimization procedures in an iterative manner seeking to improve a match between a model transformation and the frame to generate a revised model associated with a second location in the frame; and

recording the revised model transformation and the second location as an outcome of tracking the region in the frame.

17. The computer program product of claim 16 wherein the reference frames used to generate the model reflect the region to be tracked under varying conditions including one or more of lighting, states of the feature and motion blur.

18. A computer program product embodied in a non-transitory computer-readable storage medium and comprising instructions that, when executed by a processor, perform a method comprising:

receiving a video sequence comprising a plurality of video frames;

selecting a plurality of reference frames from the plurality of video frames, the plurality of reference frames reflecting the image feature to be tracked under varying conditions;

placing one or more markers on each reference frame to identify a plurality of reference images representing the image feature under the varying conditions;

generating a plurality of morph images from the plurality of reference images, each morph image representing an intermediate appearance of the image feature between consecutive reference images;

generating a model for texture and shape of the image feature from the reference images and the morph images;

tracking the feature in a frame of the video sequence using a multi-pass search that adapts the model to the frame.

19. The computer program product of claim 18 wherein the model includes a mean image and a plurality of component images, each component image representing a difference between an individual reference frame and the mean image.

20. The computer program product of claim 18 wherein generating the model aligns the reference images into a set of aligned images that have a common orientation.

21. The computer program product of claim 18 wherein generating the model warps the reference images to fit a canonical reference image.

22. The computer program product of claim 18 wherein at least one marker in the plurality of markers placed in each reference image is an anchor marker that cannot be modified by subsequent tracking.

23. The computer program product of claim 18 wherein the varying conditions include one or more of lighting conditions, states of the feature and motion blurr.

24. The computer program product of claim 18 further comprising, after generating the model, adding an additional reference frame to the plurality of reference frames, placing one or more markers on the additional reference frame to identify an additional reference image representing the image feature, generating at least one additional morph image based on the additional reference image, and generating a revised model for texture and shape of the image feature from the reference images, the additional reference image, the morph images and the at least one additional morph image.

25. The computer program product of claim 18 wherein the multi-pass search associates the model with a first location in the frame and performs a plurality of optimization procedures in an iterative manner seeking to improve a match between a model transformation and the frame and resulting in a revised model associated with a second location in the frame;

and wherein the method further comprises recording the revised model transformation and the second location as an outcome of tracking the region in the frame.