PIXEL-LEVEL BASED MICRO-FEATURE EXTRACTION

• US 20110044536A1
• Filed: 08/18/2009
• Published: 02/24/2011
• Est. Priority Date: 09/11/2008
• Status: Active Grant

• Current Assignee: Intellective Ai, Inc.
• Original Assignee: Omni AI, Inc.
• Inventors: Seow, Min-Jung, Graham, Jeff, Cobb, Wesley Kenneth, Saitwal, Kishor Adinath, Xu, Gang, Gottumukkal, Rajkiran Kumar, Risinger, Lon William
• Granted Patent: US 9,633,275 B2
• Time in Patent Office: Days
• US Class Current: 382/165
• CPC Class Codes: G06V 10/25 Determination of region of interest [ROI] or a volume of interest [VOI]
• CPC Class Codes: G06V 10/40 Extraction of image or video features
• CPC Class Codes: G06V 10/46 Descriptors for shape, contour or point-related descriptors, e.g. scale invariant feature transform [SIFT] or bags of words [BoW]; Salient regional features colour feature extraction G06V10/56
• CPC Class Codes: G06V 10/473 using gradient analysis
• CPC Class Codes: G06V 10/507 Summing image-intensity values; Histogram projection analysis
• CPC Class Codes: H04N 7/18 Closed-circuit television [CCTV] systems, i.e. systems in which the video signal is not broadcast

Abstract

Techniques are disclosed for extracting micro-features at a pixel-level based on characteristics of one or more images. Importantly, the extraction is unsupervised, i.e., performed independent of any training data that defines particular objects, allowing a behavior-recognition system to forgo a training phase and for object classification to proceed without being constrained by specific object definitions. A micro-feature extractor that does not require training data is adaptive and self-trains while performing the extraction. The extracted micro-features are represented as a micro-feature vector that may be input to a micro-classifier which groups objects into object type clusters based on the micro-feature vectors.

Claims

1. A computer-implemented method for extracting pixel-level micro-features from image data captured by a video camera, the method comprising:

receiving the image data;

identifying a foreground patch that depicts a foreground object;

processing the foreground patch to compute a plurality of micro-feature values based on at least one pixel-level characteristic of the foreground patch, wherein the micro-feature value is computed independent of training data that defines a plurality of object types;

generating a micro-feature vector that includes the plurality of micro-feature values; and

classifying the foreground object as depicting an object type as based on the micro-feature vector.

2. The computer-implemented method of claim 1, wherein a bounding box minimally encloses the foreground patch and the computed micro-feature value is a normalized pixel-level area of the foreground patch.

3. The computer-implemented method of claim 1, wherein a bounding box minimally encloses the foreground patch and the computed micro-feature value is an aspect ratio of the bounding box.

4. The computer-implemented method of claim 1, wherein the computed micro-feature value is set to predetermined value indicating that the computed micro-feature value is not valid when a threshold value is not met.

5. The computer-implemented method of claim 1, wherein the computed micro-feature value is a hue entropy value and the pixel-level characteristic is hue values of the foreground patch.

6. The computer-implemented method of claim 1, wherein the computed micro-feature value is a normalized magnitude-saturation ratio value and the pixel-level characteristic is magnitude and saturation values of the foreground patch.

7. The computer-implemented method of claim 1, wherein the computed micro-feature value is an absolute value of a cosine of an angle of alignment between a major axis and an orientation line of the foreground patch and the pixel-level characteristic is the orientation line of the foreground patch.

8. The computer-implemented method of claim 1, wherein the computed micro-feature value is a groupiness value and the pixel-level characteristic is a horizontal distribution and vertical distribution of the foreground patch.

9. The computer-implemented method of claim 1, wherein the computed micro-feature value is a legged-ness value based on a sum of angles between neighboring segments of a start skeleton of the foreground patch and the pixel-level characteristic is the star skeleton of the foreground patch

10. The computer-implemented method of claim 1, wherein the computed micro-feature value is a verticality value and the pixel-level characteristic is instantaneous per-pixel gradients of the foreground patch.

11. The computer-implemented method of claim 1, wherein the computed micro-feature value is an animateness value and the pixel-level characteristic is a covariance matrix of the foreground patch in a first frame and a second frame of the image data.

12. The computer-implemented method of claim 1, wherein the computed micro-feature value is a periodicity value and the pixel-level characteristic is an auto-correlation of a self-similarity matrix computed based on a sequence of frames including the foreground patch.

13. A computer-readable storage medium containing a program which, when executed by a processor, performs an operation for extracting pixel-level micro-features from image data captured by a video camera, the operation comprising:

receiving the image data;

identifying a foreground patch that depicts a foreground object;

processing the foreground patch to compute a micro-feature value based on at least one pixel-level characteristic of the foreground patch, wherein the micro-feature value is computed independent of training data that defines a plurality of object types;

generating a micro-feature vector that includes the micro-feature value; and

classifying the foreground object as depicting an object type as based on the micro-feature vector.

14. The computer-readable storage medium of claim 13, wherein the computed micro-feature value is a hue entropy value and the pixel-level characteristic is hue values of the foreground patch.

15. The computer-readable storage medium of claim 13, wherein the computed micro-feature value is a normalized magnitude-saturation ratio value and the pixel-level characteristic is magnitude and saturation values of the foreground patch.

16. The computer-readable storage medium of claim 13, wherein the computed micro-feature value is an absolute value of a cosine of an angle of alignment between a major axis and an orientation line of the foreground patch and the pixel-level characteristic is the orientation line of the foreground patch.

17. The computer-readable storage medium of claim 13, wherein the computed micro-feature value is a groupiness value and the pixel-level characteristic is a horizontal distribution and vertical distribution of the foreground patch.

18. A system, comprising:

a video input source configured to provide image data;

a processor; and

a memory containing a program, which, when executed on the processor is configured to perform an operation for extracting pixel-level micro-features from the image data captured by the video input source, the operation comprising;

receiving the image data;

identifying a foreground patch that depicts a foreground object;

processing the foreground patch to compute a micro-feature value based on at least one pixel-level characteristic of the foreground patch, wherein the micro-feature value is computed independent of training data that defines a plurality of object types;

generating a micro-feature vector that includes the micro-feature value; and

classifying the foreground object as depicting an object type as based on the micro-feature vector.

19. The system of claim 18, wherein a bounding box minimally encloses the foreground patch and the computed micro-feature value is a normalized pixel-level area of the foreground patch or an aspect ratio of the bounding box.

20. The system of claim 18, wherein the computed micro-feature value is an absolute value of a cosine of an angle of alignment between a major axis and an orientation line of the foreground patch and the pixel-level characteristic is the orientation line of the foreground patch.

21. The system of claim 18, wherein the computed micro-feature value is a groupiness value and the pixel-level characteristic is a horizontal distribution and vertical distribution of the foreground patch.

22. The system of claim 18, wherein the computed micro-feature value is a legged-ness value based on a sum of angles between neighboring segments of a start skeleton of the foreground patch and the pixel-level characteristic is the star skeleton of the foreground patch

23. The system of claim 18, wherein the computed micro-feature value is a verticality value and the pixel-level characteristic is instantaneous per-pixel gradients of the foreground patch.

24. The system of claim 18, wherein the computed micro-feature value is an animateness value and the pixel-level characteristic is a covariance matrix of the foreground patch in a first frame and a second frame of the image data.

25. The system of claim 18, wherein the computed micro-feature value is a periodicity value and the pixel-level characteristic is an auto-correlation of a self-similarity matrix computed based on a sequence of frames including the foreground patch.

26. A computer-implemented method for analyzing a sequence of video frames depicting a scene captured by a video camera, the method comprising:

identifying a plurality of foreground objects depicted in the sequence of video frames;

for each foreground object, deriving feature data for the foreground object from each frame of video depicting the foreground object;

generating, from the derived feature data of at least a first foreground object, an object type model;

correlating the derived feature data for at least a second foreground object with the object type model; and

assigning an object type identifier to the second foreground indicating that the second foreground object is an instance of an object type associated with the object type model.

27. The method of claim 26, wherein the object type model comprises an adaptive resonance theory (ART) network.

28. The method of claim 27, wherein generating the object type model comprises generating a cluster in the adaptive resonance theory (ART) network from the derived feature data of at least the first foreground object.

29. The method of claim 28, wherein correlating the derived feature data the second foreground object with the object type model comprises, mapping the derived feature data to the cluster the adaptive resonance theory network.

30. The method of claim 26, further comprising, updating the object type model based on the derived feature data for the second foreground object.

31. The method of claim 26, wherein the feature data derived for the foreground object includes one or more elements of static data characterizing the foreground object in one or more of the video frames.

32. The method of claim 26, wherein the feature data derived for the foreground object includes one or more elements of kinematic data characterizing the foreground object in one or more of the video frames.