Visual attention and segmentation system

• US 8,363,939 B1
• Filed: 06/16/2008
• Issued: 01/29/2013
• Est. Priority Date: 10/06/2006
• Status: Expired due to Fees

• Current Assignee: HRL Laboratories LLC (The Boeing Co.)
• Original Assignee: HRL Laboratories LLC (The Boeing Co.)
• Inventors: Huber, David, Khosla, Deepak
• Primary Examiner(s): Drennan, Barry
• Application Number:
  US12/214,259
• Time in Patent Office: 1,688 Days
• Field of Search: None
• US Class Current: 382/173
• CPC Class Codes: G06V 10/255 Detecting or recognising potential candidate objects based on visual cues, e.g. shapes
• CPC Class Codes: G06V 10/451 with interaction between the filter responses, e.g. cortical complex cells

Abstract

Described is a bio-inspired vision system for attention and object segmentation capable of computing attention for a natural scene, attending to regions in a scene in their rank of saliency, and extracting the boundary of an attended proto-object based on feature contours to segment the attended object. The attention module can work in both a bottom-up and a top-down mode, the latter allowing for directed searches for specific targets. The region growing module allows for object segmentation that has been shown to work under a variety of natural scenes that would be problematic for traditional object segmentation algorithms. The system can perform at multiple scales of object extraction and possesses a high degree of automation. Lastly, the system can be used by itself for stand-alone searching for salient objects in a scene, or as the front-end of an object recognition and online labeling system.

Claims

1. A vision system for object-based attention and segmentation, comprising a memory and one or more processors, the memory encoded with instructions to cause the system to include:

a feature extraction module configured to receive an image representing a scene with an object in the scene and generate a feature map corresponding to receptive fields in the retina for red-center/green-surround, green-center/red-surround, blue-center/yellow-surround, bright-center/dark-surround, and dark-center/bright-surround;

a feature edge detection module configured to extract edge information from the feature map and generate an edge map, the edge map including incomplete boundaries and unclosed contours;

a watershed module configured to generate proto-objects based on the edge map;

a saliency module configured to compute a saliency of each proto-object and rank the proto-objects relative to other proto-objects in terms of their saliency; and

a region growing module configured to compute a feature boundary for the proto-object with the highest saliency and a list of pixels in the image that constitutes the proto-object feature boundary, and further configured to output the pixels confined by the feature boundary as an extracted object.

2. A vision system as set forth in claim 1, further comprising a top-down biasing module configured to assign a given weight to each feature of the object, the weight being used by the saliency module in computing the saliency.

3. A vision system as set forth in claim 2, further comprising an object recognition module configured to receive the extracted object and associate a label with the extracted object to classify the object, whereby a user can use the vision system to classify an object.

4. A vision system as set forth in claim 3, further comprising an online labeling module, the online labeling module configured to alert a user if the extracted object is an unknown object and prompt the user to input a label for the unknown object.

5. A vision system as set forth in claim 4, wherein the edge-detection module further comprises an edge threshold, whereby changing the edge threshold varies a number of watershed-regions being generated by the watershed module.

6. A vision system as set forth in claim 1, further comprising an object recognition module configured to receive the extracted object and associate a label with the extracted object to classify the object, whereby a user can use the vision system to classify an object.

7. A vision system as set forth in claim 1, further comprising an online labeling module, the online labeling module configured to alert a user if the extracted object is an unknown object and prompt the user to input a label for the unknown object.

8. A vision system as set forth in claim 1, wherein the edge-detection module further comprises an edge threshold, whereby changing the edge threshold varies a number of watershed-regions being generated by the watershed module.

9. A vision system as set forth in claim 1, wherein:

in receiving an image and generating a feature map, the feature extraction module is configured to receive an image frame representing a scene with an object in the scene and generate a series of feature maps for each image frame, the series of feature maps corresponding to specific visual features in the scene such that a single feature map is generated for each visual feature processed;

in extracting edge information, the feature edge detection module is configured to receive as an input actual feature maps and extract edge information from the feature maps to generate an edge map, the edge map includes incomplete boundaries and unclosed contours; and

in generating proto-objects based on the edge map, the proto-objects are generated by completing boundaries and closing contours.

10. A computer program product for visual object-based attention and segmentation, the computer program product comprising computer-readable instruction means stored on a non-transitory computer-readable medium that are executable by a computer for causing the computer to:

receive an image representing a scene and generate feature maps corresponding to receptive fields in the retina for red-center/green-surround, green-center/red-surround, blue-center/yellow-surround, bright-center/dark-surround, and dark-center/bright-surround;

receive the feature maps and generate a master edge map using the feature maps;

receive the master edge map and generate a plurality of watershed-regions, each watershed-region having a plurality of features;

receive the plurality of watershed-regions and compute a plurality of saliencies corresponding to each watershed-region;

select among the plurality of watershed-regions a most-salient-region; and

receive the watershed-regions and grow the most-salient-region by joining neighboring watershed-regions to form a proto-object, the proto-object having a feature boundary;

generate a list of pixels in the image that constitutes the proto-object feature boundary; and

output the pixels confined by the feature boundary as an extracted object.

11. A computer program product as set forth in claim 10, further comprising instruction means for causing a computer to assign a given weight to each feature of the object, the weight being used by the saliency module in computing the saliency.

12. A computer program product as set forth in claim 11, further comprising instruction means for causing a computer to receive the extracted object and associate a label with the extracted object to classify the object, whereby a user can use the vision system to classify an object.

13. A computer program product as set forth in claim 12, further comprising instruction means for causing a computer to alert a user if the extracted object is an unknown object and prompt the user to input a label for the unknown object.

14. A computer program product as set forth in claim 13, further comprising instruction means for causing a computer to operate with an edge threshold, such that changing the edge threshold varies a number of watershed-regions being generated by the watershed module.

15. A computer program product as set forth in claim 10, further comprising instruction means for causing a computer to receive the extracted object and associate a label with the extracted object to classify the object, whereby a user can use the vision system to classify an object.

16. A computer program product as set forth in claim 10, further comprising instruction means for causing a computer to alert a user if the extracted object is an unknown object and prompt the user to input a label for the unknown object.

17. A computer program product as set forth in claim 10, further comprising instruction means for causing a computer to operate with an edge threshold, such that changing the edge threshold varies a number of watershed-regions being generated by the watershed module.

18. A computer program product as set forth in claim 10, wherein:

in receiving an image and generating a feature map, an image frame representing a scene is received and a series of feature maps are generated, such that a single feature map is generated for each of the receptive fields;

in receiving the feature maps and generating a master edge map corresponding to the feature maps, the edge map includes incomplete boundaries and unclosed contours; and

in receiving the watershed-regions and growing the most-salient-region by joining neighboring watershed-regions to form a proto-object, the proto-object is formed through completing boundaries and closing contours.

19. A method for bio-inspired visual object-based attention and segmentation comprising acts of:

receiving an image representing a scene and generating feature maps corresponding to receptive fields in the retina for red-center/green-surround, green-center/red-surround, blue-center/yellow-surround, bright-center/dark-surround, and dark-center/bright-surround;

receiving the feature maps and generating a master edge map using to the feature maps;

receiving the master edge map and generating a plurality of watershed-regions, each watershed-region having a plurality of features;

receiving the plurality of watershed-regions and computing a plurality of saliencies corresponding to each watershed-region;

selecting among the plurality of watershed-regions a most-salient-region; and

receiving the watershed-regions and growing the most-salient-region by joining neighboring watershed-regions to form a proto-object, the proto-object having a feature boundary;

generating a list of pixels in the image that constitutes the proto-object feature boundary; and

outputting the pixels confined by the feature boundary as an extracted object.

20. A method as set forth in claim 19, further comprising an act of assigning a given weight to each feature of the object, the weight being used by the saliency module in computing the saliency.

21. A method as set forth in claim 20, further comprising an act of associating a label with the extracted object to classify the object, whereby a user can use the vision system to classify an object.

22. A method as set forth in claim 21, further comprising acts of alerting a user if the extracted object is an unknown object and prompting the user to input a label for the unknown object.

23. A method as set forth in claim 22, further comprising an act of operating with an edge threshold, such that changing the edge threshold varies a number of watershed-regions being generated by the watershed module.

24. A method as set forth in claim 19, further comprising an act of associating a label with the extracted object to classify the object, whereby a user can use the vision system to classify an object.

25. A method as set forth in claim 19, further comprising acts of alerting a user if the extracted object is an unknown object and prompting the user to input a label for the unknown object.

26. A method as set forth in claim 19, wherein:

in receiving an image and generating a feature map, an image frame representing a scene is received and a series of feature maps are generated, such that a single feature map is generated for each of the receptive fields;

in receiving the feature maps and generating a master edge map corresponding to the feature maps, the edge map includes incomplete boundaries and unclosed contours; and

in receiving the watershed-regions and growing the most-salient-region by joining neighboring watershed-regions to form a proto-object, the proto-object is formed through completing boundaries and closing contours.

27. A method as set forth in claim 26, further comprising an act of operating with an edge threshold, such that changing the edge threshold varies a number of watershed-regions being generated by the watershed module.