Tracker component for behavioral recognition system

• US 8,300,924 B2
• Filed: 09/11/2008
• Issued: 10/30/2012
• Est. Priority Date: 09/27/2007
• Status: Active Grant

• Current Assignee: Motorola Solutions, Inc.
• Original Assignee: Behavioral Recognition Systems Incorporated
• Inventors: Eaton, John Eric, Cobb, Wesley Kenneth, Gottumukkal, Rajkiran K., Saitwal, Kishor Adinath, Seow, Ming-Jung, Yang, Tao, Blythe, Bobby Ernest
• Primary Examiner(s): Wu, Jingge
• Assistant Examiner(s): Patel, Nirav G
• Application Number:
  US12/208,538
• Publication Number:
  US 20090087085A1
• Time in Patent Office: 1,510 Days
• Field of Search: None
• US Class Current: 382/159
• CPC Class Codes: G06F 15/16 Combinations of two or more digital computers each having at least an arithmetic unit, a program unit and a register, e.g. for a simultaneous processing of several programs coordinating program control therefor G06F9/52; in regulating and control system G05B

Abstract

A tracker component for a computer vision engine of a machine-learning based behavior-recognition system is disclosed. The behavior-recognition system may be configured to learn, identify, and recognize patterns of behavior by observing a video stream (i.e., a sequence of individual video frames). The tracker component may be configured to track objects depicted in the sequence of video frames and to generate, search, match, and update computational models of such objects.

Claims

1. A method for tracking one or more objects depicted in a sequence of video frames, comprising:

receiving a current video frame, of the sequence of video frames;

receiving a background model of a scene depicted by the sequence of video frames and one or more foreground patches, wherein each foreground patch includes a set of pixels covered by the foreground patch in the current video frame;

for each of the one or more tracked objects;

identifying, a predicted location of the tracked object,

identifying, at the predicted location, one or more foreground patches,

searching for the tracked object at a plurality of points in the scene using a model of the tracked object generated from one or more previous video frames,

selecting one of the searched points as a position of the tracked object in the current video frame, and

updating the model of the tracked object based on the selected position of the tracked object in the current video frame;

passing the updated models of the one or more tracked objects to a machine learning engine; and

generating, by the machine learning engine, from the passed models, one or more semantic representations of behavior engaged in by the tracked objects in the scene over the sequence of video frames, wherein the machine learning engine is configured to learn patterns of behavior engaged in by the tracked objects in the scene over the plurality of video frames and to identify occurrences of the patterns of behavior engaged in by the tracked objects.

2. The method of claim 1, further comprising:

identifying one or more foreground patches including a pixels not assigned to one of the tracked objects; and

seeding a new object model for tracking the object depicted in the pixels not assigned to one of the tracked objects in subsequent frames of video.

3. The method of claim 1, wherein searching at a plurality of points of the one or more foreground patches for the tracked object comprises performing a hierarchical block search.

4. The method of claim 1, wherein searching at a plurality of points of the one or more foreground patches for the tracked object comprises performing a particle filter search.

5. The method of claim 1, wherein the tracked object model for a given object comprises a feature covariance model that is estimated using a set of weighted/unweighted updates of covariance matrix of the given object depicted in a plurality of frames, of the sequence of video frames.

6. The method of claim 1, wherein the tracked object model comprises an appearance based model, and wherein each pixel of the appearance based model is updated with one or more corresponding appearance values of that pixel in the current frame of video.

7. The method of claim 1, further comprising,

classifying at least one of tracked objects using a trained classifier, wherein the tracked object is classified as being one of a vehicle, a person, an other, or unknown.

8. The method of claim 1, further comprising,

receiving a region ordering of the scene, wherein the region ordering specifies pairs of occluding and non-occluding regions of the scene;

determining that at least a portion of the tracked object is occluded by the one of the ordered regions, and wherein updating the model of the tracked object includes updating the position of the model and updating portions of the model not occluded by one of the ordered regions.

9. The method of claim 1, wherein the predicted location is identified based on a previous location of the tracked object and on kinematic data associated with the tracked object model.

10. A computer-readable storage medium containing a program which, when executed by a processor, performs an operation for tracking one or more objects depicted in a sequence of video frames, the operation comprising:

receiving a current video frame, of the sequence of video frames;

receiving a background model of a scene depicted by the sequence of video frames and one or more foreground patches, wherein each foreground patch includes a set of pixels covered by the foreground patch in the current video frame;

for each of the one or more tracked objects;

identifying, a predicted location of the tracked object,

identifying, at the predicted location, one or more foreground patches,

searching for the tracked object at a plurality of points in the scene using a model of the tracked object generated from one or more previous video frames,

selecting one of the searched points as a position of the tracked object in the current video frame, and

updating the model of the tracked object based on the selected position of the tracked object in the current video frame; and

passing the updated models of the one or more tracked objects to a machine learning engine; and

generating, by the machine learning engine, from the passed models, one or more semantic representations of behavior engaged in by the tracked objects in the scene over the sequence of video frames, wherein the machine learning engine is configured to learn patterns of behavior engaged in by the tracked objects in the scene over the plurality of video frames and to identify occurrences of the patterns of behavior engaged in by the tracked objects.

11. The computer-readable storage medium of claim 10, wherein the operation further comprises:

identifying one or more foreground patches including a pixels not assigned to one of the tracked objects; and

seeding a new object model for tracking the object depicted in the pixels not assigned to one of the tracked objects in subsequent frames of video.

12. The computer-readable storage medium of claim 10, wherein searching at a plurality of points of the one or more foreground patches for the tracked object comprises performing a hierarchical block search.

13. The computer-readable storage medium of claim 10, wherein searching at a plurality of points of the one or more foreground patches for the tracked object comprises performing a particle filter search.

14. The computer-readable storage medium of claim 10, wherein the tracked object model for a given object comprises a feature covariance model that is estimated using a set of weighted/unweighted updates of covariance matrix of the given object depicted in a plurality of frames, of the sequence of video frames.

15. The computer-readable storage medium of claim 10, wherein the tracked object model comprises an appearance based model, and wherein each pixel of the appearance based model is updated with one or more corresponding appearance values of that pixel in the current frame of video.

16. The computer-readable storage medium of claim 10, wherein the operation further comprises,

classifying at least one of tracked objects using a trained classifier, wherein the tracked object is classified as being one of a vehicle, a person, an other, or unknown.

17. The computer-readable storage medium of claim 10, wherein the operation further comprises:

receiving a region ordering of the scene, wherein the region ordering specifies pairs of occluding and non-occluding regions of the scene; and

determining that at least a portion of the tracked object is occluded by the one of the ordered regions, and wherein updating the model of the tracked object includes updating the position of the model and updating portions of the model not occluded by one of the ordered regions.

18. The computer-readable storage medium of claim 10, wherein the predicted location is identified based on a previous location of the tracked object and on kinematic data associated with the tracked object model.

19. A system comprising,

a video input source configured to provide a sequence of video frames, each depicting a scene;

a processor; and

a memory containing a computer vision engine which when executed by the processor is configured to perform an operation for tracking one or more objects depicted in a sequence of video frames, the operation comprising;

receiving a current video frame, of the sequence of video frames;

receiving a background model of a scene depicted by the sequence of video frames and one or more foreground patches, wherein each foreground patch includes a set of pixels covered by the foreground patch in the current video frame;

for each of the one or more tracked objects;

identifying, a predicted location of the tracked object,

identifying, at the predicted location, one or more foreground patches,

searching for the tracked object at a plurality of points in the scene using a model of the tracked object generated from one or more previous video frames, of the sequence of video frames,

selecting one of the searched points as a position of the tracked object in the current video frame, and

updating the model of the tracked object based on the selected position of the tracked object in the current video frame; and

passing the updated models of the one or more tracked objects to a machine learning engine; and

generating, by the machine learning engine, from the passed models, one or more semantic representations of behavior engaged in by the tracked objects in the scene over the sequence of video frames, wherein the machine learning engine is configured to learn patterns of behavior engaged in by the tracked objects in the scene over the plurality of video frames and to identify occurrences of the patterns of behavior engaged in by the tracked objects.

20. The system of claim 19, wherein the operation further comprises:

identifying one or more foreground patches including a pixels not assigned to one of the tracked objects; and

seeding a new object model for tracking the object depicted in the pixels not assigned to one of the tracked objects in subsequent frames of video.

21. The method of claim 19, wherein searching at a plurality of points of the one or more foreground patches for the tracked object comprises performing a hierarchical block search.

22. The system of claim 19, wherein searching at a plurality of points of the one or more foreground patches for the tracked object comprises performing a particle filter search.

23. The system of claim 19, wherein the tracked object model for a given object comprises a feature covariance model that is estimated using a set of weighted/unweighted updates of covariance matrix of the given object depicted in a plurality of frames, of the sequence of video frames.

24. The system of claim 19, wherein the tracked object model comprises an appearance based model, and wherein each pixel of the appearance based model is updated with one or more corresponding appearance values of that pixel in the current frame of video.

25. The system of claim 19, further comprising,

receiving a region ordering of the scene, and wherein region ordering specifies a depth ordering for one or more regions of the scene;

determining that at least a portion of the tracked object is occluded by the one of the ordered regions, and wherein updating the model of the tracked object includes updating the position of the model and updating portions of the model not occluded by one of the ordered regions.