Object recognition using a cognitive swarm vision framework with attention mechanisms

• US 7,599,894 B2
• Filed: 03/04/2006
• Issued: 10/06/2009
• Est. Priority Date: 03/04/2005
• Status: Expired due to Fees

• Current Assignee: HRL Laboratories LLC (The Boeing Co.)
• Original Assignee: HRL Laboratories LLC (The Boeing Co.)
• Inventors: Owechko, Yuri, Medasani, Swarup
• Primary Examiner(s): Vincent; David R
• Assistant Examiner(s): Rifkin; Ben M
• Application Number:
  US11/367,755
• Publication Number:
  US 20070019865A1
• Time in Patent Office: 1,312 Days
• Field of Search: 706/14, 382/104, 382/113, 382/224
• US Class Current: 706/14
• CPC Class Codes: G06F 18/2111 by using evolutionary computational techniques, e.g. genetic algorithms
• CPC Class Codes: G06V 10/25 Determination of region of interest [ROI] or a volume of interest [VOI]
• CPC Class Codes: G06V 10/771 Feature selection, e.g. selecting representative features from a multi-dimensional feature space
• CPC Class Codes: G06V 40/103 Static body considered as a whole, e.g. static pedestrian or occupant recognition

Abstract

An object recognition system is described that incorporates swarming classifiers with attention mechanisms. The object recognition system includes a cognitive map having a one-to-one relationship with an input image domain. The cognitive map records information that software agents utilize to focus a cooperative swarm'"'"'s attention on regions likely to contain objects of interest. Multiple agents operate as a cooperative swarm to classify an object in the domain. Each agent is a classifier and is assigned a velocity vector to explore a solution space for object solutions. Each agent records its coordinates in multi-dimensional space that are an observed best solution that the agent has identified, and a global best solution that is used to store the best location among all agents. Each velocity vector thereafter changes to allow the swarm to concentrate on the vicinity of the object and classify the object when a classification level exceeds a preset threshold.

Claims

1. An object recognition system incorporating swarming domain classifiers, comprising:

at least one cognitive map stored in memory having a one-to-one relationship with an input image domain, the cognitive map being capable of recording information in the memory that software agents utilize to focus a cooperative swarm'"'"'s attention on regions in the domain most likely to contain objects of interest;

a plurality of software agents executing on a processor configured to operate as a cooperative swarm to classify an object in the domain, where each agent is a complete classifier and is assigned an initial velocity vector to explore a solution space for object solutions, where each agent is configured to perform at least one iteration as influenced by the recorded information of the cognitive map, the iteration being a search in the solution space for a potential solution optima where each agent keeps track of its coordinates in multi-dimensional space that are associated with an observed best solution (pbest) that the agent has identified, and a global best solution (gbest), where the gbest is used to store the best location among all agents, with each velocity vector thereafter changing towards pbest and gbest, allowing the cooperative swarm to concentrate on the vicinity of the object and classify the object when a classification level exceeds a preset threshold;

the cognitive map is a map selected from a group consisting of a ground surface map, an interest map, an object map, and a saliency map; and

the interest map stored in the memory is configured to run on the processor and maintain a sorted list for gbest and pbest, along with the associated F_A values, where F_A is an objective function and is calculated according to the following;

F_A=μ

(Q₊−

Q₋

)+(1−

μ

)F_C,

where Q₊ denotes an attracting pheromone and Q₋

denotes a repelling pheromone, and where m is a nonnegative weighting factor, and F_C is an object classifier confidence value; and

the interest map is updated in the memory at each iteration of the swarm and F_A is updated for each entry in the sorted list, whereby the swarm is modified by the interest map in such a way as to focus attention on regions of increased saliency.

2. An object recognition system as set forth in claim 1, wherein using the ground surface map, the system is configured to:

store expected object height in pixels at each image location;

constrain an analysis window to an appropriate size at each domain location; and

implement space-variant initialization of the cooperative swarm.

3. An object recognition system as set forth in claim 2, wherein using the ground surface map, the system is further configured to calculate a vertical coordinate of y of an imaged object using a pinhole imaging model, the vertical coordinate of y is calculated according to the following:

$y=-f<br><br><br><br>\tan <br><br><br><br>\mathrm{\alpha <br><br>}-\frac{f<br><br>\left(z-z_c\right)<br><br>\left({\tan }^2<br><br>\mathrm{\alpha <br><br>}+1\right)}{Y-\left(z-z_c\right)<br><br>\tan <br><br><br><br>\mathrm{\alpha <br><br>}},$

where z is the height of the object, z_c is the camera height, Y is the distance of the object from the camera, α

is the camera tilt angle, and ƒ

is the camera focal length.

4. An object recognition system as set forth in claim 3, wherein the interest map is configured to store swarm attracting/repelling pheromones at each domain location, with attracting pheromones having positive values and being stored to attract swarms to high saliency regions and to regions more likely to contain objects based on previous detection results or external inputs, and with repelling pheromones having negative values and being stored to repel swarms away from regions that do not contain objects or which have already been explored, thereby preventing clustering of agents on low confidence regions.

5. An object recognition system as set forth in claim 4, wherein using the object map, the system is configured to perform at least one of the operations of:

storing information at each domain location on previously detected objects;

preventing unnecessary classifier evaluations;

initializing swarms only in regions where objects have not yet been detected;

initializing swarms in regions more likely to contain objects based on previous detection results; and

recognizing object groups and behaviors.

6. An object recognition system as set forth in claim 5, wherein the system is further configured to track an object in multiple input images by performing operations of:

receiving a first current input image;

initializing a global swarm to search for objects within the input image;

assigning local swarms to objects identified by the global swarm;

receiving a next input image, where the next input image is deemed the current input image and a previous current input image is deemed the previous input image;

initializing the local swarms to search for and identify objects in the current input image;

deleting local swarms that lost their identified objects between the current and previous input images;

initializing the global swarm to search for new objects in the current input image;

assigning local swarms to new objects identified in the current input image; and

repeating the acts of receiving a next input image, initializing, deleting, initializing, and assigning for subsequent next images.

7. An object recognition system as set forth in claim 1, wherein using the ground surface map, the system is further configured to calculate a vertical coordinate of y of an imaged object using a pinhole imaging model, the vertical coordinate of y is calculated according to the following:

$y=-f<br><br><br><br>\tan <br><br><br><br>\mathrm{\alpha <br><br>}-\frac{f<br><br>\left(z-z_c\right)<br><br>\left({\tan }^2<br><br>\mathrm{\alpha <br><br>}+1\right)}{Y-\left(z-z_c\right)<br><br>\tan <br><br><br><br>\mathrm{\alpha <br><br>}},$

where z is the height of the object, z_c is the camera height, Y is the distance of the object from the camera, α

is the camera tilt angle, and ƒ

is the camera focal length.

8. An object recognition system as set forth in claim 1, wherein the interest map is configured to store swarm attracting/repelling pheromones at each domain location, with attracting pheromones having positive values and being stored to attract swarms to high saliency regions and to regions more likely to contain objects based on previous detection results or external inputs, and with repelling pheromones having negative values and being stored to repel swarms away from regions that do not contain objects or which have already been explored, thereby preventing clustering of agents on low confidence regions.

9. An object recognition system as set forth in claim 1, wherein using the object map, the system is configured to perform at least one of the operations of:

storing information at each domain location on previously detected objects;

preventing unnecessary classifier evaluations;

initializing swarms only in regions where objects have not yet been detected;

initializing swarms in regions more likely to contain objects based on previous detection results; and

recognizing object groups and behaviors.

10. An object recognition system as set forth in claim 1, wherein the system is further configured to track an object in multiple input images by performing operations of:

receiving a first current input image;

initializing a global swarm to search for objects within the input image;

assigning local swarms to objects identified by the global swarm;

receiving a next input image, where the next input image is deemed the current input image and a previous current input image is deemed the previous input image;

initializing the local swarms to search for and identify objects in the current input image;

deleting local swarms that lost their identified objects between the current and previous input images;

initializing the global swarm to search for new objects in the current input image;

assigning local swarms to new objects identified in the current input image; and

repeating the acts of receiving a next input image, initializing, deleting, initializing, and assigning for subsequent next images.

11. A computer program product for object recognition, the computer program product comprising computer-readable instruction means encoded on a computer-readable medium for causing a computer to, when executed, perform the operations of:

configuring at least one cognitive map to have a one-to-one relationship with an input image domain, the cognitive map being capable of recording information that software agents utilize to focus a cooperative swarm'"'"'s attention on regions in the domain most likely to contain objects of interest;

configuring a plurality of software agents to operate as a cooperative swarm to classify an object in the domain, where each agent is a complete classifier and is assigned an initial velocity vector to explore a solution space for object solutions, where each agent is configured to perform at least one iteration as influenced by the recorded information of the cognitive map, the iteration being a search in the solution space for a potential solution optima where each agent keeps track of its coordinates in multi-dimensional space that are associated with an observed best solution (pbest) that the agent has identified, and a global best solution (gbest) where the gbest is used to store the best location among all agents, with each velocity vector thereafter changing towards pbest and gbest, allowing the cooperative swarm to concentrate on the vicinity of the object and classify the object when a classification level exceeds a preset threshold; and

further comprising instruction means for causing a computer to;

running and maintaining a sorted list for gbest and pbest, along with the associated F_A values, where F_A is an objective function and is calculated according to the following;

F_A=μ

(Q₊−

Q₋

)+(1−

μ

)F_C,

where Q₊ denotes an attracting pheromone and Q₋

denotes a repelling pheromone, and where m is a nonnegative weighting factor, and F_C is an object classifier confidence value; and

updating the cognitive map at each iteration of the swarm and update F_A for each entry in the sorted list, whereby the swarm is modified by the cognitive map in such a way as to focus attention on regions of increased saliency.

12. A computer program product as set forth in claim 11, wherein the cognitive map is a map selected from a group consisting of a ground surface map, an interest map, an object map, and a saliency map.

13. A computer program product as set forth in claim 12, further comprising instruction means for causing a computer to:

store expected object height in pixels at each image location;

constrain an analysis window to an appropriate size at each domain location; and

implement space-variant initialization of the cooperative swarm.

14. A computer program product as set forth in claim 13, further comprising instruction means for causing a computer to calculate a vertical coordinate of y of an imaged object using a pinhole imaging model, the vertical coordinate of y is calculated according to the following:

$y=-f<br><br><br><br>\tan <br><br><br><br>\mathrm{\alpha <br><br>}-\frac{f<br><br>\left(z-z_c\right)<br><br>\left({\tan }^2<br><br>\mathrm{\alpha <br><br>}+1\right)}{Y-\left(z-z_c\right)<br><br>\tan <br><br><br><br>\mathrm{\alpha <br><br>}},$

where z is the height of the object, z_c is the camera height, Y is the distance of the object from the camera, α

is the camera tilt angle, and ƒ

is the camera focal length.

15. A computer program product as set forth in claim 14, further comprising instruction means for causing a computer to store swarm attracting/repelling pheromones at each domain location, with attracting pheromones having positive values and being stored to attract swarms to high saliency regions and to regions more likely to contain objects based on previous detection results or external inputs, and with repelling pheromones having negative values and being stored to repel swarms away from regions that do not contain objects or which have already been explored, thereby preventing clustering of agents on low confidence regions.

16. A computer program product as set forth in claim 15, further comprising instruction means for causing a computer to perform at least one of the operations of:

storing information at each domain location on previously detected objects;

preventing unnecessary classifier evaluations;

initializing swarms only in regions where objects have not yet been detected;

initializing swarms in regions more likely to contain objects based on previous detection results; and

recognizing object groups and behaviors.

17. A computer program product as set forth in claim 16, further comprising instruction means for causing a computer to track an object in multiple input images by performing operations of:

receive a first current input image;

initialize a global swarm to search for objects within the input image;

assign local swarms to objects identified by the global swarm;

receive a next input image, where the next input image is deemed the current input image and a previous current input image is deemed the previous input image;

initialize the local swarms to search for and identify objects in the current input image;

delete local swarms that lost their identified objects between the current and previous input images;

initialize the global swarm to search for new objects in the current input image;

assign local swarms to new objects identified in the current input image; and

repeat the operations of receiving a next input image, initializing, deleting, initializing, and assigning for subsequent next images.

18. A computer program product as set forth in claim 11, further comprising instruction means for causing a computer to:

store expected object height in pixels at each image location;

constrain an analysis window to an appropriate size at each domain location; and

implement space-variant initialization of the cooperative swarm.

19. A computer program product as set forth in claim 11, further comprising instruction means for causing a computer to calculate a vertical coordinate of y of an imaged object using a pinhole imaging model, the vertical coordinate of y is calculated according to the following:

$y=-f<br><br><br><br>\tan <br><br><br><br>\mathrm{\alpha <br><br>}-\frac{f<br><br>\left(z-z_c\right)<br><br>\left({\tan }^2<br><br>\mathrm{\alpha <br><br>}+1\right)}{Y-\left(z-z_c\right)<br><br>\tan <br><br><br><br>\mathrm{\alpha <br><br>}},$

where z is the height of the object, z_c is the camera height, Y is the distance of the object from the camera, α

is the camera tilt angle, and ƒ

is the camera focal length.

20. A computer program product as set forth in claim 11, further comprising instruction means for causing a computer to store swarm attracting/repelling pheromones at each domain location, with attracting pheromones having positive values and being stored to attract swarms to high saliency regions and to regions more likely to contain objects based on previous detection results or external inputs, and with repelling pheromones having negative values and being stored to repel swarms away from regions that do not contain objects or which have already been explored, thereby preventing clustering of agents on low confidence regions.

21. A computer program product as set forth in claim 11, further comprising instruction means for causing a computer to perform at least one of the operations of:

storing information at each domain location on previously detected objects;

preventing unnecessary classifier evaluations;

initializing swarms only in regions where objects have not yet been detected;

initializing swarms in regions more likely to contain objects based on previous detection results; and

recognizing object groups and behaviors.

22. A computer program product as set forth in claim 11, further comprising instruction means for causing a computer to track an object in multiple input images by performing operations of:

receive a first current input image;

initialize a global swarm to search for objects within the input image;

assign local swarms to objects identified by the global swarm;

receive a next input image, where the next input image is deemed the current input image and a previous current input image is deemed the previous input image;

initialize the local swarms to search for and identify objects in the current input image;

delete local swarms that lost their identified objects between the current and previous input images;

initialize the global swarm to search for new objects in the current input image;

assign local swarms to new objects identified in the current input image; and

repeat the operations of receiving a next input image, initializing, deleting, initializing, and assigning for subsequent next images.

23. A computer implemented method for object recognition using swarming domain classifiers, when executed on the computer comprises the acts of:

configuring at least one cognitive map to have a one-to-one relationship with an input image domain, the cognitive map being capable of recording information that software agents utilize to focus a cooperative swarm'"'"'s attention on regions in the domain most likely to contain objects of interest;

configuring a plurality of software agents to operate as a cooperative swarm to classify an object in the domain, where each agent is a complete classifier and is assigned an initial velocity vector to explore a solution space for object solutions, where each agent is configured to perform at least one iteration as influenced by the recorded information of the cognitive map, the iteration being a search in the solution space for a potential solution optima where each agent keeps track of its coordinates in multi-dimensional space that are associated with an observed best solution (pbest) that the agent has identified, and a global best solution (gbest), where the gbest is used to store the best location among all agents, with each velocity vector thereafter changing towards pbest and gbest, allowing the cooperative swarm to concentrate on the vicinity of the object and classify the object when a classification level exceeds a preset threshold; and

running and maintaining a sorted list for gbest and pbest, along with the associated F_A values, where F_A is an objective function and is calculated according to the following;

F_A=μ

(Q₊−

Q₋

)+(1−

μ

)F_C,

where Q₊ denotes an attracting pheromone and Q₋

denotes a repelling pheromone, and where m is a nonnegative weighting factor, and F_C is an object classifier confidence value; and

updating the cognitive map at each iteration of the swarm and updating F_A for each entry in the sorted list, whereby the swarm is modified by the cognitive map in such a way as to focus attention on regions of increased saliency.

24. A method as set forth in claim 23, wherein the cognitive map is a map selected from a group consisting of a ground surface map, an interest map, an object map, and a saliency map.

25. A method as set forth in claim 23, further comprising acts of:

storing expected object height in pixels at each image location;

constraining an analysis window to an appropriate size at each domain location; and

implementing space-variant initialization of the cooperative swarm.

26. A method as set forth in claim 23, further comprising an act of calculating a vertical coordinate of y of an imaged object using a pinhole imaging model, the vertical coordinate of y is calculated according to the following:

$y=-f<br><br><br><br>\tan <br><br><br><br>\mathrm{\alpha <br><br>}-\frac{f<br><br>\left(z-z_c\right)<br><br>\left({\tan }^2<br><br>\mathrm{\alpha <br><br>}+1\right)}{Y-\left(z-z_c\right)<br><br>\tan <br><br><br><br>\mathrm{\alpha <br><br>}},$

where z is the height of the object, z_c is the camera height, Y is the distance of the object from the camera, α

is the camera tilt angle, and ƒ

is the camera focal length.

27. A method as set forth in claim 23, further comprising an act of storing swarm attracting/repelling pheromones at each domain location, with attracting pheromones having positive values and being stored to attract swarms to high saliency regions and to regions more likely to contain objects based on previous detection results or external inputs, and with repelling pheromones having negative values and being stored to repel swarms away from regions that do not contain objects or which have already been explored, thereby preventing clustering of agents on low confidence regions.

28. A method as set forth in claim 23, further comprising an act of performing at least one of the operations of:

storing information at each domain location on previously detected objects;

preventing unnecessary classifier evaluations;

initializing swarms only in regions where objects have not yet been detected;

initializing swarms in regions more likely to contain objects based on previous detection results; and

recognizing object groups and behaviors.

29. A method as set forth in claim 23, further comprising an act of tracking an object in multiple input images by performing operations of:

receiving a first current input image;

initializing a global swarm to search for objects within the input image;

assigning local swarms to objects identified by the global swarm;

receiving a next input image, where the next input image is deemed the current input image and a previous current input image is deemed the previous input image;

initializing the local swarms to search for and identify objects in the current input image;

deleting local swarms that lost their identified objects between the current and previous input images;

initializing the global swarm to search for new objects in the current input image;

assigning local swarms to new objects identified in the current input image; and

repeating the acts of receiving a next input image, initializing, deleting, initializing, and assigning for subsequent next images.

30. A computer program product for object recognition, the computer program product comprising computer-readable instruction means encoded on a computer-readable medium for causing a computer to, when executed, perform the operations of:

receiving a first current input image;

initializing a global swarm to search for objects within the input image;

assigning local swarms to objects identified by the global swarm;

receiving a next input image, where the next input image is deemed the current input image and a previous current input image is deemed the previous input image;

initializing the local swarms to search for and identify objects in the current input image;

deleting local swarms that lost their identified objects between the current and previous input images;

initializing the global swarm to search for new objects in the current input image;

assigning local swarms to new objects identified in the current input image; and

repeating the operations of receiving a next input image, initializing, deleting, initializing, and assigning for subsequent next images; and

further camp rising instruction means for causing a computer to;

running and maintaining a sorted list for gbest and pbest, along with the associated F_A values, where F_A is an objective function and is calculated according to the following;

F_A=μ

(Q₊−

Q₋

)+(1−

μ

)F_C,

where Q₊ denotes an attracting pheromone and Q₋

denotes a repelling pheromone, and where m is a nonnegative weighting factor, and F_C is an object classifier confidence value; and

updating a cognitive map at each iteration of the swarm and update F_A for each entry in the sorted list, whereby the swarm is modified by the cognitive map in such a way as to focus attention on regions of increased saliency.