Object detection approach using generative sparse, hierarchical networks with top-down and lateral connections for combining texture/color detection and shape/contour detection

• US 9,477,901 B2
• Filed: 07/22/2015
• Issued: 10/25/2016
• Est. Priority Date: 09/13/2012
• Status: Active Grant

• Current Assignee: TRIAD National Security, LLC
• Original Assignee: Los Alamos National Security LLC (Government of the United States of America)
• Inventors: Paiton, Dylan M., Kenyon, Garrett T., Brumby, Steven P., Schultz, Peter F., George, John S.
• Primary Examiner(s): PATEL, KANJIBHAI B
• Application Number:
  US14/805,540
• Publication Number:
  US 20150325007A1
• Time in Patent Office: 461 Days
• Field of Search: 382/113, 382/155, 382/159, 382/162
• US Class Current: 1/1
• CPC Class Codes: G06F 18/253 of extracted features
• CPC Class Codes: G06T 2207/10024 Color image
• CPC Class Codes: G06T 7/20 Analysis of motion motion estimation for coding, decoding, compressing or decompressing digital video signals H04N19/43, H04N19/51
• CPC Class Codes: G06T 7/40 Analysis of texture depth or shape recovery from texture G06T7/529
• CPC Class Codes: G06V 10/449 Biologically inspired filters, e.g. difference of Gaussians [DoG] or Gabor filters
• CPC Class Codes: G06V 10/56 relating to colour
• CPC Class Codes: G06V 10/806 of extracted features

Abstract

An approach to detecting objects in an image dataset may combine texture/color detection, shape/contour detection, and/or motion detection using sparse, generative, hierarchical models with lateral and top-down connections. A first independent representation of objects in an image dataset may be produced using a color/texture detection algorithm. A second independent representation of objects in the image dataset may be produced using a shape/contour detection algorithm. A third independent representation of objects in the image dataset may be produced using a motion detection algorithm. The first, second, and third independent representations may then be combined into a single coherent output using a combinatorial algorithm.

Claims

1. A computer-implemented method, comprising:

combining, by a computing system, pixel-based and feature-based dictionaries to reduce effective dimensionality of feature-based inputs, augmenting learning of the feature-based dictionaries;

producing, by the computing system, a hierarchical network of a plurality of computational layers for color/texture analysis of an image or video;

creating vertical competition, by the computing system, between the plurality of computational layers by including top-down feedback from subsequent layers to previous layers; and

generating cortical representations, by the computing system, at each hierarchical layer incorporating the top-down feedback to reduce redundancy and increase sparseness of the generated cortical representations.

2. The computer-implemented method of claim 1, further comprising:

employing pixel-based dictionaries, by the computing system, to guide acquisition of feature-based dictionaries by training a first layer of dictionary elements on image patches and training subsequent layers on a combination of direct feature-based inputs and down-sampled, pixel-based inputs arising from images generated by sparsely activated features of a previous layer.

3. The computer-implemented method of claim 1, further comprising:

subdividing the plurality of computational layers, by the computing system, into a first stage of selective feature detectors and a second stage of invariant feature detectors that are implemented by combining local and lateral dictionaries, wherein

the selective feature detectors form a first stage of processing within each layer of the plurality of computational layers, and

the invariant feature detectors form a second stage of processing within each layer of the plurality of computational layers.

4. The computer-implemented method of claim 3, further comprising:

creating lateral competition, by the computing system, among invariant feature detectors in a given layer by receiving and processing spatially convergent lateral input from a surrounding neighborhood of the selective feature detectors in the layer, resulting in interlayer invariance to linear and non-linear transformations while maintaining scalability.

5. The computer-implemented method of claim 1, wherein the entire hierarchical network is learned simultaneously.

6. The computer-implemented method of claim 1, further comprising:

training, by the computing system, a target class kernel and a distractor class kernel using ground truth bounding boxes;

creating, by the computing system, an object-distractor difference kernel that represents a normalized difference between the target class kernel and the distractor class kernel; and

processing each layer, by the computing system, using an ODD kernel, wherein

the processing is performed in real time.

7. The computer-implemented method of claim 1, further comprising:

outputting rasterized images, by the computing system, for color/texture analysis, shape/contour analysis, and/or motion analysis, wherein pixel values for the one or more rasterized images represent different categories or levels of activation; and

combining the rasterized images, by the computing system, into a coherent and intelligible output utilizing a spatially aware combination method or representing each rasterized image as a neuronal activity file that can be combined within the hierarchical network itself.

8. The computer-implemented method of claim 1, further comprising:

representing each classifier of a plurality of classifiers, by the computing system, as a binary detector comprising logical binary images.

9. The computer-implemented method of claim 8, further comprising:

deriving hit rate and false alarm values, by the computing system, on a pixel level, wherein

pixels inside a ground-truth box are considered to be an object and pixels outside the ground-truth box are considered not to be the object.

10. The computer-implemented method of claim 9, further comprising:

computing Boolean combination rules, by the computing system, and determining an optimal Boolean combination rule;

combining outputs of the classifiers, by the computing system, based on the computed optimal Boolean combination rule;

clustering hits, by the computing system, using a spatial clustering algorithm; and

outputting, by the computing system, bounding box locations, labels, and label confidences based on the clustered hits.

11. The computer-implemented method of claim 1, further comprising:

learning pixel-based dictionaries, by the computing system, from reconstructed images generated by a sparse set of features that are active in the previous layer in response to a given image or video frame.

12. The computer-implemented method of claim 1, wherein the image or video comprises satellite image or video, one or more regional-scale satellite imagery collections for monitoring change over time of geographic or man-made features, multi-spectral satellite imagery, or any combination thereof.

13. The computer-implemented method of claim 12, further comprising:

down-sampling a reconstructed image generated by the sparse set of features with non-zero activity, by the computing system, so that a required size of the pixel-based dictionaries remains constant at each layer of the plurality of computational layers in the hierarchical network.

14. A computer program embodied on a non-transitory computer-readable medium, the program configured to cause at least one processor to:

produce a hierarchical network of a plurality of computational layers for color/texture analysis of an image or video, shape/contour analysis of the image or video, motion analysis of the image or video, or any combination thereof, the image or video comprising satellite image or video, one or more regional-scale satellite imagery collections for monitoring change over time of geographic or man-made features, multi-spectral satellite imagery, or any combination thereof;

create vertical competition between the plurality of computational layers by including top-down feedback from subsequent layers to previous layers;

create lateral competition among invariant feature detectors in each computational layer by receiving and processing spatially convergent lateral input from a surrounding neighborhood of selective feature detectors in the respective layer; and

generate cortical representations at each hierarchical layer incorporating the top-down feedback to reduce redundancy and increase sparseness of the generated cortical representations.

15. The computer program of claim 14, the program configured to cause the at least one processor to:

employ pixel-based dictionaries, by the computing system, to guide acquisition of the feature-based dictionaries by training a first layer of dictionary elements on image patches and training subsequent layers on a combination of direct feature-based inputs and down-sampled, pixel-based inputs arising from images generated by sparsely activated features of a previous layer; and

combine the pixel-based and feature-based dictionaries to reduce effective dimensionality of feature-based inputs, augmenting learning of the feature-based dictionaries.

16. The computer program of claim 14, the program further configured to cause the at least one processor to:

subdivide the plurality of computational layers into a first stage of selective feature detectors and a second stage of invariant feature detectors that are implemented by combining local and lateral dictionaries; and

create lateral competition among invariant feature detectors in a given layer by receiving and processing spatially convergent lateral input from a surrounding neighborhood of the selective feature detectors in the layer, resulting in interlayer invariance to linear and non-linear transformations while maintaining scalability, wherein

the selective feature detectors form a first stage of processing within each layer of the plurality of computational layers, and

the invariant feature detectors form a second stage of processing within each layer of the plurality of computational layers.

17. The computer program of claim 14, the program further configured to cause the at least one processor to:

represent each classifier of a plurality of classifiers as a binary detector comprising logical binary images;

derive hit rate and false alarm values on a pixel level, wherein pixels inside a ground-truth box are considered to be an object and pixels outside the ground-truth box are considered not to be the object;

compute Boolean combination rules and determine an optimal Boolean combination rule;

combine outputs of the classifiers based on the computed optimal Boolean combination rule;

cluster hits using a spatial clustering algorithm; and

output bounding box locations, labels, and label confidences based on the clustered hits.

18. An apparatus, comprising:

memory storing computer program instructions; and

at least one processor configured to execute the computer program instructions, the at least one processor configured to;

create vertical competition between a plurality of computational layers of a hierarchical network by including top-down feedback from subsequent layers to previous layers,

train a target class kernel and a distractor class kernel using ground truth bounding boxes,

create an object-distractor difference kernel that represents a normalized difference between the target class kernel and the distractor class kernel, and

process each layer using an ODD kernel, wherein

the processing is performed in real time.

19. The apparatus of claim 18, wherein the at least one processor is further configured to:

employ pixel-based dictionaries, by the computing system, to guide acquisition of the feature-based dictionaries by training a first layer of dictionary elements on image patches and training subsequent layers on a combination of direct feature-based inputs and down-sampled, pixel-based inputs arising from images generated by sparsely activated features of a previous layer;

combine the pixel-based and feature-based dictionaries to reduce effective dimensionality of feature-based inputs, augmenting learning of the feature-based dictionaries;

subdivide the plurality of computational layers into a first stage of selective feature detectors and a second stage of invariant feature detectors that are implemented by combining local and lateral dictionaries; and

create lateral competition among invariant feature detectors in a given layer by receiving and processing spatially convergent lateral input from a surrounding neighborhood of the selective feature detectors in the layer, resulting in interlayer invariance to linear and non-linear transformations while maintaining scalability.