Robust, on-line, view-based appearance models for visual motion analysis and visual tracking

• US 7,058,205 B2
• Filed: 12/07/2001
• Issued: 06/06/2006
• Est. Priority Date: 12/07/2001
• Status: Expired due to Fees

• Current Assignee: Xerox Corporation (Xerox Holdings Corp.)
• Original Assignee: Xerox Corporation (Xerox Holdings Corp.)
• Inventors: El-Maraghi, Thomas F., Jepson, Allan D., Fleet, David J.
• Primary Examiner(s): Wu, Jingge
• Assistant Examiner(s): Lu, Tom Y.
• Application Number:
  US10/016,659
• Publication Number:
  US 20030108220A1
• Time in Patent Office: 1,642 Days
• Field of Search: 382/103, 382/107, 382/156, 382/181, 382/260, 382/173, 382238-239, 348/169
• US Class Current: 382/103
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/30201 Face
• CPC Class Codes: G06T 7/251 involving models
• CPC Class Codes: G06T 7/277 involving stochastic approaches, e.g. using Kalman filters
• CPC Class Codes: G06V 40/161 Detection; Localisation; Normalisation

Abstract

A robust, adaptive, appearance model is disclosed that includes both a stable model component, learned over a long time course, and a transient component, learned over a relatively short time course (e.g., a 2-frame motion component and/or an outlier processing component). An on-line EM-algorithm is used to adapt the appearance model parameters over time. An implementation of this approach is developed for an appearance model based on the filter responses from a steerable pyramid. The appearance model is used in a motion-based tracking algorithm to provide robustness in the face of image outliers, such as those caused by occlusions. It is also provides the ability to adapt to natural changes in appearance, such as those due to facial expressions, or variations in 3D pose.

Claims

1. A method for generating an appearance model utilizing image data provided in a plurality of sequential image frames, the appearance model defined by a stable component including a first mixing probability and a first data parameter that is calculated using a plurality of image data values respectively provided in a relatively large number of said sequential image frames, the relatively large number being greater than three, the appearance model also including a transient component having a second mixing probability and second data parameter that is calculated using a plurality of image data values respectively provided in a relatively small number of said sequential image frames, wherein the method comprises:

receiving an image datum corresponding to a most current image frame of the plurality of sequential image frames;

determining a first likelihood value for the stable component and a second likelihood value for the transient component, the first likelihood value indicating a relative consistency between the image datum and the first data parameter, and the second likelihood value indicating a relative consistency between the image datum and the second data parameter; and

updating the first mixing probability of the stable component and the second mixing probability of the transient component using the first and second likelihood values, respectively.

2. The method according to claim 1, further comprising filtering the image datum before determining the first and second likelihood values.

3. The method according to claim 2, wherein the filtering is performed using a steerable pyramid.

4. The method according to claim 1, wherein determining the likelihood comprises comparing the first data parameter with the image datum.

5. The method according to claim 1, further comprising updating said first data parameter and said second data parameter after updating said first and second mixing probabilities.

6. The method of claim 5, further comprising resetting said first and second mixing probabilities when said first mixing probability falls below a preset minimum value.

7. The method of claim 6, further comprising resetting the first data parameter to the image datum value when the first mixing probability is reset.

8. A method for tracking a selected target object comprising:

receiving a current image frame including image datum associated with of the target object;

estimating a motion of the target object using an adaptive appearance model including a first image component having parameters that are calculated using a plurality of image data values respectively received over a relatively large number of image frames temporally preceding the current image frame, the relatively large number being greater than three, and a second image component having parameters that are calculated using a plurality of image data values respectively over the relatively small number of said sequential image frames temporally preceding the current image frame; and

updating the first and second image components,

wherein the parameters of the first component include a first data parameter and a first contribution parameter,

wherein the parameters of the second component include a second data parameter and a second contribution parameter, and

wherein updating the first and second components comprises;

comparing the image datum of the current image frame with the first data parameter of the first component, and

recalculating the first and second contribution parameters based upon a difference between the first data parameter and the image datum.

9. The method according to claim 8, wherein the first contribution parameter comprises a mean value and a variance value calculated from a plurality of image data received in said relatively large number of image frames temporally preceding the current image frame, and wherein comparing comprises determining a likelihood value determined by a difference between the image datum and the mean and variance values.

10. The method of claim 9, further comprising calculating a first ownership probability for the first component using the likelihood value.

11. The method according to claim 9, further comprising recalculating the mean and variance values using the likelihood value.

12. An adaptive appearance model implemented on a processor-controlled machine for identifying an object appearing in a plurality of sequential image frames, the adaptive appearance model comprising:

a first image component having parameters defined by image data that remains stable over a relatively large number of said sequential image frames, the relatively large number being greater than three, wherein the parameters of the first image component include a first parameter that is calculated using a plurality of image data values respectively provided in said relatively large number of sequential frames; and

a second image component having parameters defined by a relatively small number of said sequential image frames, and

means for updating said first and said image components after receiving a current image frame of the plurality of sequential image frames,

wherein the parameters of the first component include the first data parameter and a first contribution parameter,

wherein the parameters of the second component include a second data parameter and a second contribution parameter, and

wherein said means for updating comprises;

means for comparing the image datum of the current image frame with the first data parameter of the first component, and

means for recalculating the first and second contribution parameters based upon a difference between the first data parameter and the image datum.

13. The appearance model according to claim 12,

wherein the first contribution parameter comprises a mean value and a variance value calculated from a plurality of image data received in at least some of said plurality of sequential image frames temporally preceding the current image frame, and

wherein the appearance model further comprises means for determining a likelihood value determined by a difference between the image datum and the mean and variance values.

14. The appearance model according to claim 13, further comprising means for calculating a first ownership probability for the first component using the likelihood value.

15. The appearance model according to claim 13, further comprising means for recalculating the mean and variance values using the likelihood value.

16. An adaptive appearance model implemented on a processor-controlled machine for identifying an object appearing in a plurality of sequential image frames, the adaptive appearance model comprising:

a first image component including a first mixing probability having a value that is determined by a first parameter that is calculated using a plurality of image data values respectively provided in a relatively large number of said sequential image frames, the relatively large number being greater than three;

a second image component including a second mixing probability having a value determined by a relatively small number of said sequential image frames, and

an outlier component including a third mixing probability that is determined by the occurrence of outliers in the image data received in the plurality of image frames.

17. The adaptive appearance model according to claim 16, further comprising:

means for receiving a current image frame; and

means for updating said first, second, and third mixing probabilities in accordance with image datum received in the current image frame.