Visual attention and object recognition system

• US 8,165,407 B1
• Filed: 10/04/2007
• Issued: 04/24/2012
• Est. Priority Date: 10/06/2006
• Status: Active Grant

• Current Assignee: HRL Laboratories LLC (The Boeing Co.)
• Original Assignee: HRL Laboratories LLC (The Boeing Co.)
• Inventors: Khosla, Deepak, Srinivasa, Narayan, Kanan, Christopher, Huber, David, Chelian, Suhas
• Primary Examiner(s): Mehta, Bhavesh
• Assistant Examiner(s): DRENNAN, BARRY T
• Application Number:
  US11/973,161
• Time in Patent Office: 1,664 Days
• Field of Search: None
• US Class Current: 382/224
• CPC Class Codes: G06V 10/255 Detecting or recognising potential candidate objects based on visual cues, e.g. shapes
• CPC Class Codes: G06V 10/451 with interaction between the filter responses, e.g. cortical complex cells

Abstract

Described is a bio-inspired vision system for object recognition. The system comprises an attention module, an object recognition module, and an online labeling module. The attention module is configured to receive an image representing a scene and find and extract an object from the image. The attention module is also configured to generate feature vectors corresponding to color, intensity, and orientation information within the extracted object. The object recognition module is configured to receive the extracted object and the feature vectors and associate a label with the extracted object. Finally, the online labeling module is configured to alert a user if the extracted object is an unknown object so that it can be labeled.

Claims

1. A vision system for object recognition, comprising:

one or more processors and a memory, the memory having instructions encoded thereon to include;

an attention module configured to receive an image representing a scene with an object in the scene and find and extract the object from the image as an extracted object, the attention module also being configured to generate feature vectors corresponding to color, intensity, and orientation information within the extracted object; and

an object recognition module configured to receive the extracted object and the feature vectors and associate a label with the extracted object to classify the object, whereby a user can use the vision system to classify an object in a scene; and

wherein the attention module is further configured to;

receive an image that includes a representation of an object in a scene, the image having color features;

determine light and dark intensity channels from the color features;

create four fully-saturated color channels from the color features;

compute feature opponency maps from the light and dark intensity channels and the four fully-saturated color channels;

compute an edge map for each opponency map;

segment the scene into a series of “

proto-objects”

based on the edge maps, where boundaries of the proto-objects are defined by common features between immediate regions within the image;

compute a saliency of a given proto-object using color and intensity information contained within the image;

rank the proto-objects according to saliency;

designate the proto-object with the highest saliency as the object to be extracted from the image; and

extract the object from the image.

2. A vision system as set forth in claim 1, wherein the object recognition module is further configured to:

reformat the object to an invariant representation;

extract simple shape features from the image;

extract high-level features from the simple shape features;

perform a coarse classification;

perform a fine classification to generate an object label; and

output the object label.

3. A vision system for object recognition, comprising:

one or more processors and a memory, the memory having instructions encoded thereon to include;

an attention module configured to receive an image representing a scene with an object in the scene and find and extract the object from the image as an extracted object, the attention module also being configured to generate feature vectors corresponding to color, intensity, and orientation information within the extracted object; and

an object recognition module configured to receive the extracted object and the feature vectors and associate a label with the extracted object to classify the object, whereby a user can use the vision system to classify an object in a scene;

wherein the object recognition module is further configured to;

rotate and rescale the object to an invariant representation utilizing a filter;

extract simple shape features from the image utilizing a Log-Gabor filter;

extract high-level features from the simple shape features utilizing a spatial pyramid matching technique;

perform a coarse classification utilizing a k-Nearest Neighbor technique;

perform a fine classification to generate an object label utilizing a Support Vector Machine; and

output the object label.

4. A computer program product for recognizing an object, the computer program product comprising computer-readable instruction means stored on a non-transitory computer-readable medium that are executable by a computer for causing the computer to:

receive an image representing a scene with an object in the scene;

find and extract the object from the image as an extracted object;

generate feature vectors corresponding to color intensity, and orientation information within the extracted object;

associate a label with the extracted object to classify the object, whereby a user can use the computer to classify an object in a scene;

receive an image that includes a representation of an object in a scene, the image having color features;

determine light and dark intensity channels from the color features;

create four fully-saturated color channels from the color features;

compute feature opponency maps from the light and dark intensity channels and the four fully-saturated color channels;

compute an edge map for each opponency map;

segment the scene into a series of “

proto-objects”

based on the edge maps, where boundaries of the prow-objects are defined by common features between immediate regions within the image;

compute a saliency of a given proto-object using color and intensity information contained within the image;

rank the proto-objects according to saliency;

designate the proto-object with the highest saliency as the object to be extracted from the image; and

extract the object from the image.

5. A computer program product as set forth in claim 4, further comprising instruction means for causing the computer to:

reformat the object to an invariant representation;

extract simple shape features from the image;

extract high-level features from the simple shape features;

perform a coarse classification;

perform a fine classification to generate an object label; and

output the object label.

6. A computer program product for recognizing an object, the computer program product comprising computer-readable instruction means stored on a non-transitory computer-readable medium that are executable by a computer for causing the computer to:

receive an image representing a scene with an object in the scene;

find and extract the object from the image as an extracted object;

generate feature vectors corresponding to color, intensity, and orientation information within the extracted object;

associate a label with the extracted object to classify the object, whereby a user can use the computer to classify an object in a scene;

rotate and rescale the object to an invariant representation utilizing a filter;

extract simple shape features from the image utilizing a Log-Gabor filter;

extract high-level features from the simple shape features utilizing a spatial pyramid matching technique;

perform a coarse classification utilizing a k-Nearest Neighbor technique;

perform a fine classification to generate an object label utilizing a Support Vector Machine; and

output the object label.

7. A method for recognizing an object, the method comprising acts of:

receiving an image representing a scene with an object in the scene;

finding and extracting the object from the image as an extracted object;

generating feature vectors corresponding to color, intensity, and orientation information within the extracted object; and

associating a label with the extracted object to classify the object, whereby a user can use the computer to classify an object in a scene;

receiving an image that includes a representation of an object in a scene, the image having color features;

determining light and dark intensity channels from the color features;

creating four fully-saturated color channels from the color features;

computing feature opponency maps from the light and dark intensity channels and the four fully-saturated color channels;

computing an edge map for each opponency map;

segmenting the scene into a series of “

proto-objects”

based on the edge maps, where boundaries of the proto-objects are defined by common features between immediate regions within the image;

computing a saliency of a given proto-object using color and intensity information contained within the image;

ranking the proto-objects according to saliency;

designating the proto-object with the highest saliency as the object to be extracted from the image; and

extracting the object from the image.

8. A method as set forth in claim 7, further comprising acts of

reformatting the object to an invariant representation;

extracting simple shape features from the image;

extracting high-level features from the simple shape features;

performing a coarse classification;

performing a fine classification to generate an object label; and

outputting the object label.

9. A method for recognizing an object, the method comprising acts of:

receiving an image representing a scene with an object in the scene;

finding and extracting the object from the image as an extracted object;

generating feature vectors corresponding to color, intensity, and orientation information within the extracted object; and

associating a label with the extracted object to classify the object, whereby a user can use the computer to classify an object in a scene;

rotating and resealing the object to an invariant representation utilizing a filter;

extracting simple shape features from the image utilizing a Log-Gabor filter;

extracting high-level features from the simple shape features utilizing a spatial pyramid matching technique;

performing a coarse classification utilizing a k-Nearest Neighbor technique;

performing a fine classification to generate an object label utilizing a Support Vector Machine; and

outputting the object label.