Multispectral detection of personal attributes for video surveillance

• US 8,515,127 B2
• Filed: 07/28/2010
• Issued: 08/20/2013
• Est. Priority Date: 07/28/2010
• Status: Expired due to Fees

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: Brown, Lisa M., Feris, Rogerio S., Hampapur, Arun, Vaquero, Daniel A.
• Primary Examiner(s): Bitar, Nancy
• Application Number:
  US12/845,121
• Publication Number:
  US 20120027249A1
• Time in Patent Office: 1,119 Days
• Field of Search: None
• US Class Current: 382/103
• CPC Class Codes: G06F 18/214 Generating training patterns; Bootstrap methods, e.g. bagging or boosting
• CPC Class Codes: G06V 10/143 Sensing or illuminating at different wavelengths
• CPC Class Codes: G06V 10/446 using Haar-like filters, e.g. using integral image techniques
• CPC Class Codes: G06V 10/774 Generating sets of training patterns; Bootstrap methods, e.g. bagging or boosting
• CPC Class Codes: G06V 20/52 Surveillance or monitoring of activities, e.g. for recognising suspicious objects recognising microscopic objects G06V20/69

Abstract

Techniques for detecting an attribute in video surveillance include generating training sets of multispectral images, generating a group of multispectral box features comprising receiving input of a detector size of a width and height, a number of spectral bands in the multispectral images, and integer values representing a minimum and maximum width and height of multispectral box features, fixing a feature width and height, generating feature building blocks with the fixed width and height, placing a feature building block at a same location for each spectral band level, and enumerating combinations of the feature building blocks through each spectral level until all sizes within the integer values have been covered, and wherein each combination determines a multispectral box feature, using the training sets to select multispectral box features to generate a multispectral attribute detector, and using the multispectral attribute detector to identify a location of an attribute in video surveillance.

Claims

1. A method for detecting an attribute in video surveillance, the method comprising:

generating one or more training sets of one or more multispectral images;

generating a group of one or more multispectral box features, wherein generating a group of one or more multispectral box features comprises the steps of;

receiving input of a two-dimensional detector size of a width and height, a number of spectral bands in the one or more multispectral images, and integer values representing a minimum and maximum width and height of one or more multispectral box features;

fixing a feature width and height;

generating one or more feature building blocks with the fixed width and height;

for one or more locations in the two-dimensional detector size, placing a feature building block at a same location for each spectral band level; and

enumerating one or more combinations of the one or more feature building blocks through each spectral band level until all sizes within the integer values have been covered, and wherein each combination determines a multispectral box feature;

using the one or more training sets to select one or more of the one or more multispectral box features to generate a multispectral attribute detector; and

using the multispectral attribute detector to identify a location of an attribute in video surveillance, wherein using the multispectral attribute detector to identify a location of an attribute in video surveillance comprises, for one or more locations on each spectral band level of the multispectral image, applying the multispectral attribute detector and producing an output indicating attribute detection or an output indicating no attribute detection, and wherein the attribute corresponds to the multispectral attribute detector;

wherein generating one or more training sets of one or more multispectral images comprises generating a positive training set, wherein generating a positive training set comprises;

collecting one or more multispectral images of a given attribute for which a detector is to be trained;

selecting one or more rectangular regions in the one or more multispectral images that correspond to the attribute; and

rescaling the one or more selected regions to correspond to a desired detector size.

2. The method of claim 1, wherein the one or more multispectral images comprise at least one of a thermal infrared image and a visible image.

3. The method of claim 1, wherein an attribute comprises at least one of facial hair type, nose shape, head characteristic, presence of hat, absence of hat, hairstyle, scar, clothing color, clothing type, and eyewear type.

4. The method of claim 1, wherein generating one or more training sets of one or more multispectral images comprises, for a given attribute for which a detector is to be trained, using a desired detector size as input.

5. The method of claim 1, wherein collecting one or more multispectral images of a given attribute for which a detector is to be trained comprises simultaneously capturing multiple images of a scene from one point of view, wherein each image corresponds to a different portion of an electromagnetic spectrum.

6. The method of claim 1, wherein generating one or more training sets of one or more multispectral images comprises generating a negative training set, wherein generating a negative training set comprises:

collecting one or more multispectral images that do not contain a given attribute for which a detector is to be trained; and

randomly sampling portions of the one or more multispectral images of a size that corresponds to a desired detector size.

7. The method of claim 1, wherein a multispectral box feature comprises a sum of one or more pixel values along one or more regions in a three-dimensional space given by one or more captured images.

8. The method of claim 1, wherein using the one or more training sets to select one or more of the one or more multispectral box features to generate a multispectral attribute detector comprises using adaptive boosting learning.

9. The method of claim 1, wherein using the one or more training sets to select one or more of the one or more multispectral box features to generate a multispectral attribute detector comprises using a positive training set, a negative training set and the group of multispectral box features as input.

10. The method of claim 9, wherein using the one or more training sets to select one or more of the one or more multispectral box features to generate a multispectral attribute detector comprises using an algorithm to select one or more features for discrimination between items from the positive training set and negative training set.

11. The method of claim 1, wherein using the multispectral attribute detector to identify a location of an attribute in video surveillance comprises using a multispectral image to be classified and the multispectral attribute detector as input.

12. The method of claim 11, wherein for every location and scale of the multispectral image, applying the multispectral attribute detector to identify the attribute in each spectra of the multispectral image.

13. The method of claim 1, further comprising classifying one or more new multispectral images.

14. The method of claim 1, further comprising using adaptive boosting learning to assemble one or more weak classifiers into one single strong classifier.

15. The method of claim 14, wherein assembling one or more weak classifiers into one single strong classifier comprises initializing one or more sample weights and, for each cycle, identifying a classifier that performs on a weighted sample and increasing weights of one or more misclassified examples.

16. The method of claim 1, further comprising providing a system, wherein the system comprises one or more distinct software modules, each of the one or more distinct software modules being embodied on a tangible computer-readable recordable storage medium, and wherein the one or more distinct software modules comprise an examples collector module, a labeling interface module, a training set generator module, a feature enumerator module and a multispectral Adaboost learner module executing on a hardware processor.

17. A tangible computer device includes a computer useable program code for detecting an attribute in video surveillance, the computer device including:

computer useable program code for generating one or more training sets of one or more multispectral images;

computer useable program code for generating a group of one or more multispectral box features, wherein the computer useable program code for generating a group of one or more multispectral box features comprises;

computer useable program code for receiving input of a two-dimensional detector size of a width and height, a number of spectral bands in the one or more multispectral images, and integer values representing a minimum and maximum width and height of one or more multispectral box features;

computer useable program code for fixing a feature width and height;

computer useable program code for generating one or more feature building blocks with the fixed width and height;

computer useable program code for placing a feature building block at a same location for each spectral band level for one or more locations in the two-dimensional detector size; and

computer useable program code for enumerating one or more combinations of the one or more feature building blocks through each spectral band level until all sizes within the integer values have been covered, and wherein each combination determines a multispectral box feature;

computer useable program code for using the one or more training sets to select one or more of the one or more multispectral box features to generate a multispectral attribute detector; and

computer useable program code for using the multispectral attribute detector to identify a location of an attribute in video surveillance, wherein using the multispectral attribute detector to identify a location of an attribute in video surveillance comprises, for one or more locations on each spectral band level of the multispectral image, applying the multispectral attribute detector and producing an output indicating attribute detection or an output indicating no attribute detection, and wherein the attribute corresponds to the multispectral attribute detector;

wherein the computer useable program code for generating one or more training sets of one or more multispectral images comprises computer useable program code for generating a positive training set, and wherein the computer useable program code for generating a positive training set further comprises;

computer useable program code for collecting one or more multispectral images of a given attribute for which a detector is to be trained; and

selecting one or more rectangular regions in the one or more multispectral images that correspond to the attribute; and

resealing the one or more selected regions to correspond to a desired detector size.

18. The computer device of claim 17, wherein the computer useable program code for generating one or more training sets of one or more multispectral images comprises computer useable program code for generating a negative training set, and where the computer useable program code for generating a negative training set further comprises:

computer useable program code for collecting one or more multispectral images that do not contain a given attribute for which a detector is to be trained; and

computer useable program code for randomly sampling portions of the one or more multispectral images of a size that corresponds to a desired detector size.

19. The computer device of claim 17, wherein the computer useable program code for using the one or more training sets to select one or more of the one or more multispectral box features to generate a multispectral attribute detector comprises computer useable program code for using a positive training set, a negative training set and the group of multispectral box features as input.

20. A system for detecting an attribute in video surveillance, comprising:

a memory; and

at least one processor coupled to the memory and operative to;

generate one or more training sets of one or more multispectral images;

generate a group of one or more multispectral box features, wherein generating a group of one or more multispectral box features comprises;

receiving input of a two-dimensional detector size of a width and height, a number of spectral bands in the one or more multispectral images, and integer values representing a minimum and maximum width and height of one or more multispectral box features;

fixing a feature width and height;

generating one or more feature building blocks with the fixed width and height;

for one or more locations in the two-dimensional detector size, placing a feature building block at a same location for each spectral band level; and

enumerating one or more combinations of the one or more feature building blocks through each spectral band level until all sizes within the integer values have been covered, and wherein each combination determines a multispectral box feature;

use the one or more training sets to select one or more of the one or more multispectral box features to generate a multispectral attribute detector; and

use the multispectral attribute detector to identify a location of an attribute in video surveillance, wherein using the multispectral attribute detector to identify a location of an attribute in video surveillance comprises, for one or more locations on each spectral band level of the multispectral image, applying the multispectral attribute detector and producing an output indicating attribute detection or an output indicating no attribute detection, and wherein the attribute corresponds to the multispectral attribute detector;

wherein at least one processor coupled to the memory operative to generate the one or more training sets of one or more multispectral images is further operative to generate a positive training set, and collect one or more multispectral images of a given attribute for which a detector is to be trained;

select one or more rectangular regions in the one or more multispectral images that correspond to the attribute; and

rescale the one or more selected regions to correspond to a desired detector size.

21. The system of claim 20, wherein the at least one processor coupled to the memory operative to generate one or more training sets of one or more multispectral images is further operative to generate a negative training set, and wherein the at least one processor coupled to the memory operative to generate a negative training set is further operative to:

collect one or more multispectral images that do not contain a given attribute for which a detector is to be trained; and

randomly sample portions of the one or more multispectral images of a size that corresponds to a desired detector size.

22. An apparatus for detecting an attribute in video surveillance, the apparatus comprising:

means for generating one or more training sets of one or more multispectral images;

means for generating a group of one or more multispectral box features, wherein means generating a group of one or more multispectral box features comprises;

means for receiving input of a two-dimensional detector size of a width and height, a number of spectral bands in the one or more multispectral images, and integer values representing a minimum and maximum width and height of one or more multispectral box features;

means for fixing a feature width and height;

means for generating one or more feature building blocks with the fixed width and height;

means for placing a feature building block at a same location for each spectral band level for one or more locations in the two-dimensional detector size; and

means for enumerating one or more combinations of the one or more feature building blocks through each spectral band level until all sizes within the integer values have been covered, and wherein each combination determines a multispectral box feature;

means for using the one or more training sets to select one or more of the one or more multispectral box features to generate a multispectral attribute detector; and

means for using the multispectral attribute detector to identify a location of an attribute in video surveillance, wherein using the multispectral attribute detector to identify a location of an attribute in video surveillance comprises, for one or more locations on each spectral band level of the multispectral image, applying the multispectral attribute detector and producing an output indicating attribute detection or

an output indicating no attribute detection, and wherein the attribute corresponds to the multispectral attribute detector;

wherein generating one or more training sets of one or more multispectral images comprises generating a positive training set, wherein generating a positive training set comprises;

collecting one or more multispectral images of a given attribute for which a detector is to be trained;

selecting one or more rectangular regions in the one or more multispectral images that correspond to the attribute; and

rescaling the one or more selected regions to correspond to a desired detector size.