Estimator identifier component for behavioral recognition system

• US 8,175,333 B2
• Filed: 09/11/2008
• Issued: 05/08/2012
• Est. Priority Date: 09/27/2007
• Status: Active Grant

• Current Assignee: Motorola Solutions, Inc.
• Original Assignee: Behavioral Recognition Systems Incorporated
• Inventors: Eaton, John Eric, Cobb, Wesley Kenneth, Gottumukkal, Rajkiran K., Seow, Ming-Jung, Yang, Tao, Saitwal, Kishor Adinath
• Primary Examiner(s): Werner, Brian
• Assistant Examiner(s): Patel, Nirav G
• Application Number:
  US12/208,526
• Publication Number:
  US 20090087027A1
• Time in Patent Office: 1,335 Days
• Field of Search: 382/103
• US Class Current: 382/103
• CPC Class Codes: G06F 15/16 Combinations of two or more digital computers each having at least an arithmetic unit, a program unit and a register, e.g. for a simultaneous processing of several programs coordinating program control therefor G06F9/52; in regulating and control system G05B

Abstract

An estimator/identifier component for a computer vision engine of a machine-learning based behavior-recognition system is disclosed. The estimator/identifier component may be configured to classify an object being one of two or more classification types, e.g., as being a vehicle or a person. Once classified, the estimator/identifier may evaluate the object to determine a set of kinematic data, static data, and a current pose of the object. The output of the estimator/identifier component may include the classifications assigned to a tracked object, as well as the derived information and object attributes.

Claims

1. A method for analyzing an object being tracked in a sequence of video frames, comprising:

receiving a representation of the tracked object, as depicted by a current video frame, of the sequence of video frames;

evaluating, by operation of one or more computer processors, the representation of the tracked object using at least a first classifier and a second classifier, wherein the first classifier is configured to determine a first classification score indicating whether the tracked object depicts an instance of a first classification type, and wherein the second classifier is configured to determine a second classification score indicating whether the tracked object depicts an instance of a second classification type;

adding the first classification score to a first rolling average, wherein the first rolling average provides an average of the first classification score determined for the tracked object for each of a specified number of previous video frames, of the plurality of;

adding the second classification score to a second rolling average, wherein the second rolling average provides an average of the second classification score determined for the tracked object for each of a specified number of previous video frames, of the plurality, wherein the final classification value is determined from the first rolling average and the second rolling average;

determining a final classification value for the tracked object in the current video frame, based on the first and second rolling averages; and

passing the final classification value for the tracked objects to a machine learning engine configured to identify patterns of behavior engaged in by the tracked object, based at least in part on the final classification value.

2. The method of claim 1, wherein the first classification type comprises a vehicle and the second classification type comprises a person.

3. The method of claim 1, further comprising, analyzing the tracked object to determine one or more instance-specific attributes of the object depicted in the sequence of video frames based on a final classification value indicating that the tracked object depicts one of vehicle or a person.

4. The method of claim 1, wherein the final classification value specifies that the tracked object depicts one of a car, a person, an unknown-object, or an other-object.

5. The method of claim 4, wherein the classification value of other-object indicates that the tracked object does not depict an instance of either the first classification type or the second classification type.

6. The method of claim 1, further comprising:

training the first classifier using a set of positive training examples depicting instances of the first classification type;

training the second classifier using a set positive training examples depicting instances of the second classification type;

providing a first set of negative training examples and a second set of negative training examples to the first and second classifiers, respectively, for classification;

for each negative training example incorrectly classified by the first classifier, training the first classifier on the incorrectly classified negative training examples; and

for each negative training example incorrectly classified by the second classifier, training the second classifier on the incorrectly classified negative training examples.

7. The method of claim 6, further comprising:

providing the set of positive training examples depicting instances of the first classification type to the first classifier for classification;

providing the set of positive training examples depicting instances of the second classification type to the second classifier for classification;

for each positive training example incorrectly classified by the first classifier, training the first classifier on the incorrectly classified positive training examples; and

for each positive training example incorrectly classified by the second classifier, training the second classifier on the incorrectly classified positive training examples.

8. The method of claim 1, wherein the representation of the tracked object includes a plurality of pixel position values and a plurality of pixel color values for pixels which depict the object in the current frame of video.

9. A non-transitory computer-readable storage medium containing a program which, when executed by a processor, performs an operation for analyzing an object being tracked in a sequence of video frames, the operation comprising:

receiving a representation of the tracked object, as depicted by a current video frame, of the sequence of video frames;

evaluating the representation of the tracked object using at least a first classifier and a second classifier, wherein the first classifier is configured to determine a first classification score indicating whether the tracked object depicts an instance of a first classification type, and wherein the second classifier is configured to determine a second classification score indicating whether the tracked object depicts an instance of a second classification type;

determining a final classification value for the tracked object, based on the first and second classification scores; and

passing the final classification value for the tracked objects to a machine learning engine configured to identify patterns of behavior engaged in by the tracked object, based at least in part on the final classification value.

10. The non-transitory computer-readable storage medium of claim 9, wherein the first classification type comprises a vehicle and the second classification type comprises a person.

11. The non-transitory computer-readable storage medium of claim 10, wherein the operation further comprises, analyzing the tracked object to determine one or more instance-specific attributes of the object depicted in the sequence of video frames based on a final classification value indicating that the tracked object depicts one of vehicle or a person.

12. The non-transitory computer-readable storage medium of claim 9, wherein the final classification value specifies that the tracked object depicts one of a car, a person, an unknown-object, or an other-object.

13. The non-transitory computer-readable storage medium of claim 12, wherein the classification value of other-object indicates that the tracked object does not depict an instance of either the first classification type or the second classification type.

14. The non-transitory computer-readable storage medium of claim 9, wherein the operation further comprises:

training the first classifier using a set of positive training examples depicting instances of the first classification type;

training the second classifier using a set positive training examples depicting instances of the second classification type;

providing a first set of negative training examples and a second set of negative training examples to the first and second classifiers, respectively, for classification;

for each negative training example incorrectly classified by the first classifier, training the first classifier on the incorrectly classified negative training examples;

for each negative training example incorrectly classified by the second classifier, training the second classifier on the incorrectly classified negative training examples.

15. The non-transitory computer-readable storage medium of claim 14, wherein the operation further comprises:

providing the set of positive training examples depicting instances of the first classification type to the first classifier for classification;

providing the set of positive training examples depicting instances of the second classification type to the second classifier for classification;

for each positive training example incorrectly classified by the first classifier, training the first classifier on the incorrectly classified positive training examples; and

for each positive training example incorrectly classified by the second classifier, training the second classifier on the incorrectly classified positive training examples.

16. The non-transitory computer-readable storage medium of claim 9, wherein the representation of the tracked object includes a plurality of pixel position values and a plurality of pixel color values for pixels which depict the object in the current frame of video.

17. A system comprising,

a video input source configured to provide a sequence of video frames, each depicting a scene;

a processor; and

a memory containing a computer vision engine, which when executed by the processor is configured to perform an operation for analyzing an object being tracked in a sequence of video frames, the operation comprising;

receiving a representation of the tracked object, as depicted by a current video frame, of the sequence of video frames,

evaluating the representation of the tracked object using at least a first classifier and a second classifier, wherein the first classifier is configured to determine a first classification score indicating whether the tracked object depicts an instance of a first classification type, and wherein the second classifier is configured to determine a second classification score indicating whether the tracked object depicts an instance of a second classification type,

adding the first classification score to a first rolling average, wherein the first rolling average provides an average of the first classification score determined for the tracked object for each of a specified number of previous video frames, of the plurality of video frames,

adding the second classification score to a second rolling average, wherein the second rolling average provides an average of the second classification score determined for the tracked object for each of a specified number of previous video frames, of the plurality of video frames, wherein the final classification value is determined from the first rolling average and the second rolling average,

determining a final classification value for the tracked object in the current video frame, based on the first and second rolling averages, and

passing the final classification value for the tracked objects to a machine learning engine configured to identify patterns of behavior engaged in by the tracked object, based at least in part on the final classification value.

18. The system of claim 17, wherein the first classification type comprises a vehicle and the second classification type comprises a person.

19. The system of claim 17, wherein the final classification value specifies that the tracked object depicts one of a car, a person, an unknown-object, or an other-object, and wherein the classification value of other-object indicates that the tracked object does not depict an instance of either the first classification type or the second classification type.

20. The system of claim 17, wherein the operation further comprises:

training the first classifier using a set of positive training examples depicting instances of the first classification type;

training the second classifier using a set positive training examples depicting instances of the second classification type;

providing a first set of negative training examples and a second set of negative training examples to the first and second classifiers, respectively, for classification;

for each negative training example incorrectly classified by the first classifier, training the first classifier on the incorrectly classified negative training examples;

for each negative training example incorrectly classified by the second classifier, training the second classifier on the incorrectly classified negative training examples.

21. The system of claim 20, wherein the operation further comprises:

providing the set of positive training examples depicting instances of the first classification type to the first classifier for classification;

providing the set of positive training examples depicting instances of the second classification type to the second classifier for classification;

for each positive training example incorrectly classified by the first classifier, training the first classifier on the incorrectly classified positive training examples; and

for each positive training example incorrectly classified by the second classifier, training the second classifier on the incorrectly classified positive training examples.

22. The system of claim 17, wherein the representation of the tracked object includes a plurality of pixel position values and a plurality of pixel color values for pixels which depict the object in the current frame of video.