Flexible layer tracking with weak online appearance model

• US 20070086621A1
• Filed: 10/13/2005
• Published: 04/19/2007
• Est. Priority Date: 10/13/2004
• Status: Active Grant

• Current Assignee: Samoff Corporation, SRI International, Inc.
• Original Assignee: Sarnoff Corporation (SRI International, Inc.)
• Inventors: Samarasekera, Supun, Kumar, Rakesh, Sawhney, Harpreet, Aggarwal, Manoj
• Granted Patent: US 7,706,571 B2
• Time in Patent Office: Days
• US Class Current: 382/103
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/30196 Human being; Person
• CPC Class Codes: G06T 2207/30232 Surveillance
• CPC Class Codes: G06T 7/251 involving models
• CPC Class Codes: G06T 7/277 involving stochastic approaches, e.g. using Kalman filters
• CPC Class Codes: G06V 10/24 Aligning, centring, orientation detection or correction of the image
• CPC Class Codes: G06V 40/103 Static body considered as a whole, e.g. static pedestrian or occupant recognition

Abstract

Method for tracking an object recorded within a selected frame of a sequence of frames of video data, using a plurality of layers, where at least one object layer of the plurality of layers represents the object includes initializing layer ownership probabilities for pixels of the selected frame using a non-parametric motion model, estimating a set of motion parameters of the plurality of layers for the selected frame using a parametric maximization algorithm and tracking the object. The non-parametric motion model is optical flow and includes warping the mixing probabilities, the appearances of the plurality of layers, and the observed pixel data from the pixels of the preceding frame to the pixels of the selected frame to initialize the layer ownership probabilities for the pixels of the selected frame.

Claims

1. A method for tracking an object recorded within a selected frame of a sequence of frames of video data, using a plurality of layers, at least one object layer of the plurality of layers representing the object, the method comprising the steps of:

a) initializing layer ownership probabilities for pixels of the selected frame using a non-parametric motion model;

b) estimating a set of motion parameters of the plurality of layers for the selected frame using a parametric maximization algorithm; and

c) tracking the object.

2. The method of claim 1 wherein estimating the set of motion parameters is based upon:

appearances of the plurality of layers for a proceeding frame of the sequence of frames;

mixing probabilities and observed pixel data from pixels of the preceding frame;

observed pixel data and the layer ownership probabilities for the pixels of the selected frame; and

an initial estimated set of motion parameters of the plurality of layers for the selected frame.

3. The method of claim 1 wherein tracking the object is based upon:

the layer ownership probabilities for the pixels of the selected frame; and

the set of motion parameters of the at least one object layer estimated in step (b).

4. The method according to claim 1, wherein step (b) further includes the steps of:

b1) refining layer ownership probabilities for the plurality of layers in the selected frame using the expectation maximization algorithm based on;

the appearances of the plurality of layers for the preceding frame of the sequence of frames;

the mixing probabilities and the observed pixel data from the pixels of the preceding frame; and

the estimated set of motion parameters of the plurality of layers for the selected frame; and

b2) refining the estimated set of motion parameters of the plurality of layers for the selected frame using the estimation maximization algorithm based on;

the appearances of the plurality of layers for the preceding frame of the sequence of frames;

the mixing probabilities and the observed pixel data from the pixels of the preceding frame;

the observed pixel data and the updated layer ownership probabilities for the pixels of the selected frame; and

the estimated set of motion parameters of the plurality of layers for the selected frame.

5. The method according to claim 4, wherein step (b) further includes the step of:

b3) repeating steps (b1) and (b2) at least once.

6. The method according to claim 1, wherein:

the non-parametric motion model used in step (a) is optical flow; and

step (a) includes warping the mixing probabilities, the appearances of the plurality of layers, and the observed pixel data from the pixels of the preceding frame to the pixels of the selected frame to initialize the layer ownership probabilities for the pixels of the selected frame.

7. The method according to claim 1, wherein the non-parametric motion model used in step (a) includes matching of two dimensional or three dimensional histograms of the appearances of the plurality of layers of the preceding frame and the observed pixel data of blobs of pixels of the selected frame.

8. A method for instantiating one or more object layers in a frame of video data that includes a plurality of pre-existing layers, the plurality of pre-existing layers including a background layer and an outlier layer, the method comprising the steps of:

a) forming a change image of the frame of video data using a non-parametric motion model to compare the frame of video data to a preceding frame of video data;

b) initializing ownership probabilities for pixels of the frame by the plurality of pre-existing layers using the non-parametric motion model;

c) applying connected component analysis to the change image to extract a blob;

d) computing an ownership score for the blob extracted in step (c);

e) instantiating the blob extracted in step (c) as an object layer when the ownership score computed in step (d) is less than a threshold ownership score based on a size of the blob; and

f) repeating steps (c), (d), and (e) until all blobs in the change image have been extracted.

9. The method according to claim 8, wherein:

the non-parametric motion model is optical flow; and

step (b) includes warping mixing probabilities, appearances of the plurality of pre-existing layers, and observed pixel data from pixels of the preceding frame to the pixels of the frame to initialize the ownership probabilities for the pixels of the frame by the plurality of pre-existing layers.

10. The method according to claim 8, wherein the non-parametric motion model includes matching of two dimensional histograms of appearances of the plurality of preexisting layers of the preceding frame and observed pixel data of the blob of the frame extracted in step (c).

11. The method according to claim 8, wherein the non-parametric motion model includes a geometry-based constraint on the ownership probabilities.

12. The method according to claim 8, wherein the non-parametric motion model includes a three dimensional depth constraint on the ownership probabilities.

13. The method according to claim 8, wherein step (c) includes:

c1) using connected component analysis to form a set of initial blobs in the change image;

c2) selecting an initial blob of the set of initial blobs;

c3) identifying a portion of the initial blob selected in step (c2) that represents an object;

c4) extracting only the portion of the initial blob identified in step (c3); and

c5) returning the remaining portion of the selected initial blob to the set of initial blobs.

14. The method according to claim 13, wherein step (c3) includes removing a shadow portion from the selected initial blob.

15. The method according to claim 8, wherein:

the extracted blob includes a subset of the pixels of the frame; and

the ownership score for the extracted blob is computed in step (d) by summing the ownership probabilities initialized in step (b) over the subset of pixels in the extracted blob.

16. A method of identifying an object in a plurality of frames of video data recorded substantially simultaneously by a corresponding plurality of video cameras, each video camera of the plurality of video cameras having a viewpoint different than the other video cameras, each frame of substantially simultaneously recorded video data is represented by a corresponding set of layers, at least one of the sets of layers including one object layer representing the object in one frame of video data, the method comprising the steps of:

a) for each frame of video data, calculating ownership probabilities for pixels of the frame by the set of layers corresponding to the frame and an appearance of each layer of the set of layers corresponding to the frame;

b) selecting the one object layer representing the object in the one frame of video data;

c) mapping the one object layer selected in step (b) onto a corresponding section of another frame of video data using ground plane geometry constraints;

d) choosing a layer of the set of layers corresponding to the other frame that is substantially located in the corresponding section of the other frame;

e) matching the one object layer selected in step (b) to the layer chosen in step (d) using appearance constraints; and

f) identifying the layer chosen is step (d) as an object layer representing the object, if the layers matched in step (e) exceed predetermined matching criteria, otherwise temporarily removing the layer chosen in step (d) from the set of layers corresponding to other frame and repeating steps (d), (e), and (f).

17. The method according to claim 16, further comprising the step of:

g) repeating steps (c), (d), (e), and (f), until an attempt has been made to match the one object layer selected in step (b) to at least one layer in each frame other than the one frame.

18. The method according to claim 16, wherein step (a) includes the steps of:

a1) initializing the layer ownership probabilities for the pixels of each frame using a non-parametric motion model; and

a2) estimating the appearance of each layer for each frame using a parametric maximization algorithm.

19. The method according to claim 16, wherein step (d) includes the steps of:

d1) choosing an object layer of the set of layers corresponding to the other frame that was previously identified as representing the object, if the previously identified object layer is substantially located in the corresponding section of the other frame;

or

d2) choosing the layer of the set of layers corresponding to the other frame that has a highest average ownership probability for pixels in the corresponding section of the other frame, if there is no previously identified object layer substantially located in the corresponding section.