Pixel-level based micro-feature extraction

• US 9,633,275 B2
• Filed: 08/18/2009
• Issued: 04/25/2017
• Est. Priority Date: 09/11/2008
• Status: Active Grant

• Current Assignee: Intellective Ai, Inc.
• Original Assignee: Omni AI, Inc.
• Inventors: Graham, Jeff, Cobb, Wesley Kenneth, Gottumukkal, Rajkiran K., Saitwal, Kishor Adinath, Seow, Ming-Jung, Xu, Gang, Risinger, Lon W.
• Primary Examiner(s): Chu, Randolph I
• Application Number:
  US12/543,141
• Publication Number:
  US 20110044536A1
• Time in Patent Office: 2,807 Days
• Field of Search: None
• CPC Class Codes: G06V 10/25 Determination of region of interest [ROI] or a volume of interest [VOI]
• CPC Class Codes: G06V 10/40 Extraction of image or video features
• CPC Class Codes: G06V 10/46 Descriptors for shape, contour or point-related descriptors, e.g. scale invariant feature transform [SIFT] or bags of words [BoW]; Salient regional features colour feature extraction G06V10/56
• CPC Class Codes: G06V 10/473 using gradient analysis
• CPC Class Codes: G06V 10/507 Summing image-intensity values; Histogram projection analysis
• CPC Class Codes: H04N 7/18 Closed-circuit television [CCTV] systems, i.e. systems in which the video signal is not broadcast

Abstract

Techniques are disclosed for extracting micro-features at a pixel-level based on characteristics of one or more images. Importantly, the extraction is unsupervised, i.e., performed independent of any training data that defines particular objects, allowing a behavior-recognition system to forgo a training phase and for object classification to proceed without being constrained by specific object definitions. A micro-feature extractor that does not require training data is adaptive and self-trains while performing the extraction. The extracted micro-features are represented as a micro-feature vector that may be input to a micro-classifier which groups objects into object type clusters based on the micro-feature vectors.

Claims

1. A computer-implemented method for extracting pixel-level micro-features from image data captured by a video camera, the method comprising:

receiving the image data;

identifying a set of pixels in the image data associated with a foreground patch that depicts a foreground object;

evaluating appearance values of the pixels included in the set of pixels to compute a plurality of micro-feature values representing the foreground object, each based on at least one pixel-level characteristic of the foreground patch, wherein the micro-feature values are computed independent of training data that defines a plurality of object types;

generating a micro-feature vector that includes the plurality of micro-feature values;

classifying the foreground object as depicting an object type as based on the micro-feature vector, wherein the object type is determined by mapping the micro-feature vector to a cluster in a self-organizing map (SOM) adaptive resonance theory (ART) network generated from a plurality of micro-feature vectors; and

updating one or more cluster properties associated with the cluster based on the plurality of micro-feature values in the generated micro-feature vector.

2. The computer-implemented method of claim 1, wherein a bounding box minimally encloses the foreground patch and one of the computed micro-feature values is a normalized pixel-level area of the foreground patch.

3. The computer-implemented method of claim 1, wherein a bounding box minimally encloses the foreground patch and one of the computed micro-feature values is an aspect ratio of the bounding box.

4. The computer-implemented method of claim 1, wherein one of the computed micro-feature values is set to predetermined value indicating that the computed micro-feature value is not valid when a threshold value is not met.

5. The computer-implemented method of claim 1, wherein one of the computed micro-feature values is a hue entropy value and the pixel-level characteristic is hue values of the foreground patch.

6. The computer-implemented method of claim 1, one of the computed micro-feature values is a normalized magnitude-saturation ratio value and the pixel-level characteristic is magnitude and saturation values of the foreground patch.

7. The computer-implemented method of claim 1, wherein one of the computed micro-feature values is an absolute value of a cosine of an angle of alignment between a major axis and an orientation line of the foreground patch and the pixel-level characteristic is the orientation line of the foreground patch.

8. The computer-implemented method of claim 1, wherein one of the computed micro-feature values is a groupiness value and the pixel-level characteristic is a horizontal distribution and vertical distribution of the foreground patch.

9. The computer-implemented method of claim 1, wherein one of the computed micro-feature values is a legged-ness value based on a sum of angles between neighboring segments of a start skeleton of the foreground patch and the pixel-level characteristic is the star skeleton of the foreground patch.

10. The computer-implemented method of claim 1, wherein one of the computed micro-feature values is a verticality value and the pixel-level characteristic is instantaneous per-pixel gradients of the foreground patch.

11. The computer-implemented method of claim 1, wherein one of the computed micro-feature values is an animateness value and the pixel-level characteristic is a covariance matrix of the foreground patch in a first frame and a second frame of the image data.

12. The computer-implemented method of claim 1, wherein one of the computed micro-feature values is a periodicity value and the pixel-level characteristic is an auto-correlation of a self-similarity matrix computed based on a sequence of frames including the foreground patch.

13. A non-transitory computer-readable storage medium containing a program which, when executed by a processor, performs an operation for extracting pixel-level micro-features from image data captured by a video camera, the operation comprising:

receiving the image data;

identifying a set of pixels in the image data associated with a foreground patch that depicts a foreground object;

evaluating appearance values of the pixels included in the set of pixels to compute a plurality of micro-feature values representing the foreground object, each based on at least one pixel-level characteristic of the foreground patch, wherein the micro-feature values are computed independent of training data that defines a plurality of object types;

generating a micro-feature vector that includes the micro-feature value; and

classifying the foreground object as depicting an object type as based on the micro-feature vector, wherein the object type is determined by mapping the micro-feature vector to a cluster in a self-organizing map (SOM) adaptive resonance theory (ART) network generated from a plurality of micro-feature vectors; and

updating one or more cluster properties associated with the cluster based on the plurality of micro-feature values in the generated micro-feature vector.

14. The computer-readable storage medium of claim 13, wherein one of the computed micro-feature value is a hue entropy value and the pixel-level characteristic is hue values of the foreground patch.

15. The computer-readable storage medium of claim 13, wherein one of the computed micro-feature value is a normalized magnitude-saturation ratio value and the pixel-level characteristic is magnitude and saturation values of the foreground patch.

16. The computer-readable storage medium of claim 13, wherein one of the computed micro-feature value is an absolute value of a cosine of an angle of alignment between a major axis and an orientation line of the foreground patch and the pixel-level characteristic is the orientation line of the foreground patch.

17. The computer-readable storage medium of claim 13, wherein one of the computed micro-feature value is a groupiness value and the pixel-level characteristic is a horizontal distribution and vertical distribution of the foreground patch.

18. A system, comprising:

a video input source configured to provide image data;

a processor; and

a memory containing a program, which, when executed on the processor is configured to perform an operation for extracting pixel-level micro-features from the image data captured by the video input source, the operation comprising;

receiving the image data;

identifying a set of pixels in the image data associated with a foreground patch that depicts a foreground object;

evaluating appearance values of the pixels included in the set of pixels to compute a plurality of micro-feature values representing the foreground object, each based on at least one pixel-level characteristic of the foreground patch, wherein the micro-feature values are computed independent of training data that defines a plurality of object types;

generating a micro-feature vector that includes the micro-feature value;

classifying the foreground object as depicting an object type as based on the micro-feature vector, wherein the object type is determined by mapping the micro-feature vector to a cluster in a self-organizing map (SOM) adaptive resonance theory (ART) network generated from a plurality of micro-feature vectors; and

updating one or more cluster properties associated with the cluster based on the plurality of micro-feature values in the generated micro-feature vector.

19. The system of claim 18, wherein a bounding box minimally encloses the foreground patch and one of the computed micro-feature values is a normalized pixel-level area of the foreground patch or an aspect ratio of the bounding box.

20. The system of claim 18, wherein one of the computed micro-feature values is an absolute value of a cosine of an angle of alignment between a major axis and an orientation line of the foreground patch and the pixel-level characteristic is the orientation line of the foreground patch.

21. The system of claim 18, wherein one of the computed micro-feature values is a groupiness value and the pixel-level characteristic is a horizontal distribution and vertical distribution of the foreground patch.

22. The system of claim 18, wherein one of the computed micro-feature values is a legged-ness value based on a sum of angles between neighboring segments of a start skeleton of the foreground patch and the pixel-level characteristic is the star skeleton of the foreground patch.

23. The system of claim 18, wherein one of the computed micro-feature values is a verticality value and the pixel-level characteristic is instantaneous per-pixel gradients of the foreground patch.

24. The system of claim 18, wherein one of the computed micro-feature values is an animateness value and the pixel-level characteristic is a covariance matrix of the foreground patch in a first frame and a second frame of the image data.

25. The system of claim 18, wherein one of the computed micro-feature values is a periodicity value and the pixel-level characteristic is an auto-correlation of a self-similarity matrix computed based on a sequence of frames including the foreground patch.