Behavior and pattern analysis using multiple category learning

• US 8,103,090 B2
• Filed: 01/22/2007
• Issued: 01/24/2012
• Est. Priority Date: 01/22/2007
• Status: Active Grant

• Current Assignee: Honeywell International Inc.
• Original Assignee: Honeywell International Inc.
• Inventors: Ma, Yunqian, Cohen, Isaac
• Primary Examiner(s): DULANEY, KATHLEEN YUAN
• Application Number:
  US11/625,649
• Publication Number:
  US 20080175482A1
• Time in Patent Office: 1,828 Days
• Field of Search: 382/190, 382/103, 382/155, 348/155, 348/169, 340/540
• US Class Current: 382/155
• CPC Class Codes: G06F 18/2413 based on distances to training or reference patterns
• CPC Class Codes: G06V 10/764 using classification, e.g. of video objects
• CPC Class Codes: G06V 40/20 Movements or behaviour, e.g. gesture recognition recognition of facial expressions G06V40/16

Abstract

A video processing system is configured to receive training video samples from a plurality of video sensing devices. The training video samples are sets of pair video samples. These pair video samples can include both substantially similar subject matter and different subject matter. In the first step, there is a patch pool sampled from videos, and the system select patches with more saliency. The saliency is represented by the conditional probability density function of the similar subject and the conditional probability of the different subject. During the testing phase, the system applies the selected patches from the training phase, and returns the matched subjects.

Claims

1. A video processing system comprising:

one or more computer processors configured to;

receive first training video samples from a plurality of video sensing devices, the first training video samples comprising substantially similar subject matter;

generate a first training probability density function using features extracted from the first training video samples;

receive second training video samples from the plurality of video sensing devices, the second training video samples comprising insubstantially similar subject matter; and

generate a second training probability density function using features extracted from the second training video samples;

wherein the features extracted from the first training video samples are extracted from a plurality of sub-images in the first training video samples, wherein the features extracted from the second training video samples are extracted from a plurality of sub-images in the second training video samples, wherein a location in a video frame of each sub-image in the second video training samples corresponds to a substantially similar location in a video frame for each corresponding sub-image in the first training video samples, and wherein the sub-images in the first training video samples and the sub-images in the second video training samples are non-contiguous within a field of view of the plurality of video sensing devices.

2. The video processing system of claim 1, further comprising:

one or more computer processors configured to;

receive testing video samples from the plurality of video sensing devices;

extract features from the testing video samples;

calculate a distance between features from the testing video samples and stored video samples; and

apply the distance to the first training probability density function and the second probability density functions to determine whether the testing video samples correlate to one or more of the first training video samples and the second video training samples.

3. The video processing system of claim 2, further comprising one or more computer processors configured to apply a threshold to the first training probability density function, wherein the threshold provides an indication of whether the testing video samples correlate with the first training video samples.

4. The video processing system of claim 1, wherein the video processing system is further comprises one or more computer processors configured to identify the one or more sub-images by identifying points of interest in the first and second training video samples.

5. The video processing system of claim 1, wherein the features comprise one or more of a temporal feature, a spatial feature, and a spatial-temporal feature.

6. The video processing system of claim 5, wherein the spatial feature comprises a distance between a point within the sub-image and a point within an object in the sub-image.

7. The video processing system of claim 5, wherein the spatial features include directional and derived features, and the temporal features include optical flow components.

8. The video processing system of claim 7, wherein the directional features comprise one or more of a vertical feature, a horizontal feature, a forward diagonal feature, a backward diagonal feature, a diagonal energy feature, and a mean energy feature.

9. The video processing system of claim 1, wherein the first training probability density function is generated by one or more computer processors configured for:

estimating a distance between features on the similar subject matter; and

further wherein

the second training probability density function is generated by;

estimating the distance between features on the insubstantially similar subject matter.

10. The video processing system of claim 1, wherein the sub-images in the first training video samples and the sub-images in the second video training samples are positioned in the field of view of the plurality of video sensing devices as a function of an information gain provided by the sub-images in the first training video samples and the sub-images in the second video training samples.

11. The video processing system of claim 10, wherein the information gain of the sub-images images in the first training video samples and the sub-images in the second video training samples is a function of changes in video content of the sub-images in the first training video samples and the sub-images in the second video training samples.

12. A video processing system comprising:

one or more computer processors configured to;

receive first training video samples,

the first training video samples captured by a plurality of video sensing devices, each video sensing device representing a different view of a field of view,

each first training video sample comprising a first video sequence and a second video sequence, the first video sequence and the second video sequence comprising substantially similar subject matter captured by a single video sensing device of the plurality of video sensing devices;

identify a plurality of sub-images in each frame of the first video sequence and the second video sequence, each sub-image in the first video sequence having a corresponding sub-image in the second video sequence, and each sub-image in the first video sequence having a substantially similar location in a video frame as a location of a corresponding sub-image in a video frame of the second video sequence, wherein the sub-image in each frame of the first video sequence and the sub-images in each frame of the second video sequence are non-contiguous within a field of view of the plurality of video sensing devices;

extract features from each of the sub-images; and

generate a first training probability density function for each sub-image and corresponding sub-image as a function of the extracted features;

and wherein the video processing system further comprises one or more computer processors configured to;

receive second training video samples,

the second training video samples captured by the plurality of video sensing devices,

each second training video sample comprising a first video sequence and a second video sequence, the first video sequence and the second video sequence of the second training video sample comprising insubstantially similar subject matter captured by a single video sensing device of the plurality of video sensing devices;

identify a plurality of sub-images in each frame of the first and second video sequences of the second training video sample, each sub-image in the first video sequence of the second training video sample having a corresponding sub-image in the second video sequence of the second training video sample, wherein the plurality of sub-images are non-contiguous within a field of view of the plurality of video sensing devices;

extract features from each of the sub-images of the second training video sample; and

generate a second training probability density function for each sub-image and corresponding sub-image as a function of the extracted features of the second training video samples.

13. The video processing system of claim 12, further comprising:

one or more computer processors configured to;

receive testing video samples, the testing video samples captured by the plurality of video sensing devices;

identify a plurality of sub-images in each frame of the testing video samples, each sub-image in the testing video samples corresponding to a sub-image in one or more of the first training video samples and the second training video samples, and each sub-image comprising less than the entire frame of the testing video samples;

extract features from each of the sub-images of the testing video samples;

calculate a feature number for each of the sub-images of the testing video samples; and

apply the feature number to the first training probability density function and the second probability density function to determine whether the testing video samples correlate to the first training probability density function or the second training probability density function.

14. The video processing system of claim 12, wherein the video processing system is further comprises one or more computer processors configured to identify the plurality of sub-images in each of the first training video samples, the second training video samples, and the testing video samples by identifying points of interest in the frame of a video sequence.

15. The video processing system of claim 12, wherein the features comprise one or more of a spatial feature, a temporal feature, and a spatial-temporal feature.

16. The video processing system of claim 12, wherein the first training probability density function is generated by one or more computer processors configured for:

estimating a distance between features for each sub-image of the similar subject matter in the first training video sample;

and further wherein

the second training probability density function is generated by one or more computer processors configured for;

estimating the distance between features for each sub-image of the insubstantially similar subject in the second training video sample.

17. The video processing system of claim 12, wherein

each first training video sample comprises a single image frame;

each second training video sample comprises a single image frame; and

each testing video sample comprises a single image frame.

18. An image processing system comprising:

one or more computer processors configured to;

receive first training images from a plurality of video sensing devices, the first training images comprising substantially similar subject matter;

identify a plurality of sub-images in each first training image, wherein the sub-images in each first training image are non-contiguous within a field of view of the plurality of video sensing devices;

generate a first training probability density function using features extracted from the first training images;

receive second training images from the plurality of video sensing devices, the second training images comprising insubstantially similar subject matter;

identify a plurality of sub-images in each second training image, each sub-image in the second training image having a corresponding sub-image in the first training images and each sub-image in the first training images having a substantially similar location in a video frame as a location of each corresponding sub-image in a video frame of the second training images, wherein the sub-images in each second training image are non-contiguous within a field of view of the plurality of video sensing devices; and

generate a second training probability density function using features extracted from the second training images;

wherein the first probability density function is generated by one or more computer processors configured for;

estimating the distance between features for each sub-image of the similar subject matter;

wherein the second probability density function is generated by one or more computer processors configured for;

estimating with the computer processor the distance between features for each sub-image of the insubstantially similar subject matter.

19. The image processing system of claim 18, further comprising:

one or more computer processors configured to;

receive testing images from the plurality of video sensing devices;

identify a plurality of sub-images in each testing image, the plurality of sub-images in each testing image corresponding to a sub-image in one or more of a first training image and the second training image, wherein each sub-image in each testing image comprises less than an entire frame of the testing image;

extract features from each of the sub-images in the testing images;

combine the features from each of the sub-images of the testing images to form a feature number; and

apply the distance between features to the first training probability density function and the second training probability density function to determine whether the testing images correlate to the first training probability density function or the second training probability density function.

20. The image processing system of claim 19, wherein the image processing system is further comprises one or more computer processors configured to identify with the computer processor the plurality of sub-images in each of the first training images, the second training images, and the testing images by identifying points of interest in the frame of a video sequence.