Detecting pedestrians using patterns of motion and appearance in videos

• US 7,212,651 B2
• Filed: 06/17/2003
• Issued: 05/01/2007
• Est. Priority Date: 06/17/2003
• Status: Active Grant

• Current Assignee: Mitsubishi Electric Research Laboratories, Inc. (Mitsubishi Electric Corporation)
• Original Assignee: Mitsubishi Electric Research Laboratories, Inc. (Mitsubishi Electric Corporation)
• Inventors: Jones, Michael J., Viola, Paul A.
• Primary Examiner(s): Mancuso; Joseph
• Assistant Examiner(s): ALLISON, ANDRAE S
• Application Number:
  US10/463,800
• Publication Number:
  US 20040258307A1
• Time in Patent Office: 1,414 Days
• Field of Search: 382100-107, 382/190, 382/218, 382/224, 382/260, 345/155
• US Class Current: 382/103
• CPC Class Codes: G06F 18/214 Generating training patterns; Bootstrap methods, e.g. bagging or boosting
• CPC Class Codes: G06T 7/246 using feature-based methods, e.g. the tracking of corners or segments
• CPC Class Codes: G06V 10/446 using Haar-like filters, e.g. using integral image techniques
• CPC Class Codes: G06V 10/7515 Shifting the patterns to accommodate for positional errors
• CPC Class Codes: G06V 10/774 Generating sets of training patterns; Bootstrap methods, e.g. bagging or boosting
• CPC Class Codes: G06V 40/103 Static body considered as a whole, e.g. static pedestrian or occupant recognition

Abstract

A method detects a moving object in a temporal sequence of images. Images are selected from the temporally ordered sequence of images. A set of functions is applied to the selected images to generate a set of combined images. A linear combination of filters is applied to a detection window in the set of combined images to determine motion and appearance features of the detection window. The motion and appearance features are summed to determine a cumulative score, which enables a classification of the detection window as including the moving object.

Claims

1. A method for detecting a moving object in a temporal sequence of images of a video, comprising:

selecting images from the temporally ordered sequence of images;

applying a set of functions to the selected images to generate a set of combined images;

evaluating a linear combination of filters on a detection window in the set of combined images to determine motion and appearance features of the detection window, in which the filters are two-dimensional;

summing the motion and appearance features to determine a cumulative score; and

classifying the detection window as including the moving object when the cumulative score is greater than a predetermined threshold.

2. The method of claim 1 wherein the moving object is a pedestrian.

3. The method of claim 1 wherein the selected images are an adjacent pair of images.

4. The method of claim 1 wherein the selecting, applying, evaluating, summing and classifying is repeated for the entire video.

5. The method of claim 1 further comprising:

partitioning each combined image into a set of detection windows; and

repeating the evaluating, summing, and classifying for each detection window.

6. The method of claim 1 wherein internal components of a particular filter are rectangular and arranged on a diagonal.

7. The method of claim 6 wherein each rectangle filter is sized to match the associated detection window.

8. The method of claim 1 wherein a particular feature h_j is evaluated to a weight α

_j or β

_j according to

$h_j<br><br>\left(x\right)=\{\begin{array}{cc}{\mathrm{\alpha <br><br>}}_j& \mathrm{if}<br><br><br><br>f_j<br><br>\left(x\right)><br><br>{\mathrm{\theta <br><br>}}_j\\ {\mathrm{\beta <br><br>}}_j& \mathrm{otherwise}\end{array},$

where a particular filter fj(x) is a linear function of the detection window x, and θ

j is a predetermined filter threshold value.

9. The method of claim 8 wherein a particular cumulative sum C(x) is assigned a value 1 or 0 according to

$C$

( x ) = { 1 if ⁢

⁢

∑

j ⁢

h j ⁡

( x ) >

T 0 otherwise ,

where h_j are multiple evaluated features of the detection window x, and T is the predetermined classifier threshold.

10. The method of claim 1 wherein a particular combined image is a difference of the selected images.

11. The method of claim 1 wherein a particular combined image is a difference of shifted selected images.

12. The method of claim 11 wherein the shifting is linear.

13. The method of claim 11 wherein the shifting is rotational.

14. The method of claim 11 wherein the shifting is in scale.

15. The method of claim 1 wherein the set of functions are defined with respect to a scale of the detection window.

16. The method of claim 1 further comprising:

determining a direction of motion of the moving object.

17. The method of claim 1 further comprising:

determining a magnitude of motion of the moving object.

18. The method of claim 1 wherein the filter is a rectangular filter.

19. The method of claim 1 wherein the filter is non-linear.