Multi-view cognitive swarm for object recognition and 3D tracking

• US 20070183669A1
• Filed: 03/20/2006
• Published: 08/09/2007
• Est. Priority Date: 08/14/2004
• Status: Active Grant

• Current Assignee: HRL Laboratories LLC (The Boeing Co.)
• Original Assignee: HRL Laboratories LLC (The Boeing Co.)
• Inventors: Saisan, Payam, Owechko, Yuri, Medasani, Swarup
• Granted Patent: US 7,558,762 B2
• Time in Patent Office: Days
• US Class Current: 382/224
• CPC Class Codes: G06F 18/2111 by using evolutionary computational techniques, e.g. genetic algorithms
• CPC Class Codes: G06F 18/254 of classification results, e.g. of results related to same input data
• CPC Class Codes: G06V 10/771 Feature selection, e.g. selecting representative features from a multi-dimensional feature space
• CPC Class Codes: G06V 10/809 of classification results, e.g. where the classifiers operate on the same input data
• CPC Class Codes: G06V 40/103 Static body considered as a whole, e.g. static pedestrian or occupant recognition

Abstract

An object recognition system is described that incorporates swarming classifiers. The swarming classifiers comprise a plurality of software agents configured to operate as a cooperative swarm to classify an object in a domain as seen from multiple view points. Each agent is a complete classifier and is assigned an initial velocity vector to explore a solution space for object solutions. Each agent is configured to perform an iteration, the iteration being a search in the solution space for a potential solution optima where each agent keeps track of its coordinates in multi-dimensional space that are associated with an observed best solution (pbest) that the agent has identified, and a global best solution (gbest) where the gbest is used to store the best location among all agents. Each velocity vector changes towards pbest and gbest, allowing the cooperative swarm to concentrate on the vicinity of the object and classify the object.

Claims

1. A multi-view object recognition system incorporating swarming domain classifiers, comprising:

a processor having a plurality of software agents configured to operate as a cooperative swarm to classify an object in a domain as seen from multiple view points, where each agent is a complete classifier and is assigned an initial velocity vector to explore a solution space for object solutions, where each agent is configured to perform at least one iteration, the iteration being a search in the solution space for a potential solution optima where each agent keeps track of its coordinates in multi-dimensional space that are associated with an observed best solution (pbest) that the agent has identified, and a global best solution (gbest) where the gbest is used to store the best location among all agents, with each velocity vector thereafter changing towards pbest and gbest, allowing the cooperative swarm to concentrate on the vicinity of the object and classify the object when a classification level exceeds a preset threshold.

2. A multi-view object recognition system as set forth in claim 1, wherein the agents are configured to search for the object in three-dimensional (3D) spatial coordinates, such that the object is a 3D object and the 3D object has distinct appearances from each view point in the multiple view points.

3. A multi-view object recognition system as set forth in claim 2, wherein the distinct appearances of the 3D object from the multiple view points are linked by agents searching for the 3D object in the spatial coordinates, such that each agent has an associated 3D location X and an object height h, and wherein each of the multiple view points is provided as a 2D image from a calibrated camera having a given geometry, such that given the known geometry of the calibrated cameras, a 2D location, [x,y]^T=π

(X), of an agent'"'"'s projection in each view (2D image) is calculated and used to select an image window that is sent to a classifier having a classifier output that corresponds to the classifier'"'"'s confidence that the image window contains the object, where superscript T denotes transpose.

4. A multi-view object recognition system as set forth in claim 3, wherein each agent has a search trajectory that is guided by a cost function, such that the cost function is formed by combining the classifier outputs evaluated at the agent'"'"'s projection points in each of the views, wherein the projection points are points in an image corresponding to the 3D object.

5. A multi-view object recognition system as set forth in claim 4, wherein for an agent at 3D location X=[x,y,z]^T, a value of the cost function is calculated according to the following:

ƒ

(X,h)=w₁*classifier(image₁,π

₁(X),Π

₁(X,h))+w₂*classifier(image₂,π

₂(X),Π

₂(X,h)),

where, π

;

³→

² is a projection operator such that π

_n maps the object'"'"'s 3D location X into a 2D location [x,y] in image n, and w₁+w₂=1, where w₁ and w₂ are positive weighting factors and normally w₁=w₂=0.5, and where Π

is a projection operator for object height, h, such that the projection operator Π

_n maps the 3D object height to its corresponding projection size in image n, and where classifer denotes a confidence output of the object classifier operating on the window in image n with location π

_n(X) and window size Π

_n(X,h), and where * denotes multiplication.

6. A multi-view object recognition system as set forth in claim 1, wherein each of the multiple view points is provided as a 2D image from calibrated stereo cameras, and wherein the multiple view points include at least two 2D views, view 1 and view 2, and wherein the agents are configured to move within each view independently to localize the object in each view independently, and wherein each view has 2D spatial coordinates.

7. A multi-view object recognition system as set forth in claim 6, wherein the object is a 3D object having 3D spatial coordinates, and wherein the 3D object has a 2D projection in each view in the multiple view points such that the object has a distinct appearance in each view, and wherein the agents are configured to search for the object in the 2D spatial coordinates.

8. A multi-view object recognition system as set forth in claim 7, wherein each of the 2D views is connected through geometric constraints of the calibrated stereo cameras, and wherein the agents are further configured to operate as two distinct sets of agents such that each set searches for the object in a view independently to locate a 2D location [x,y] and a 2D image window height ĥ

of the object in each view, and wherein using triangulation, the 2D locations from each view are combined to estimate the object'"'"'s 3D spatial coordinates from the 2D projections.

9. A multi-view object recognition system as set forth in claim 8, wherein the system is further configured to recognize multiple objects in the domain, such that when there is more than one object in the domain, the system is further configured to establish a correspondence between the 2D locations found in each 2D view to identify inter-view pairs.

10. A multi-view object recognition system as set forth in claim 9, wherein when establishing a correspondence between the 2D locations found in each 2D view, the system is further configured to form a cost/distance matrix for all possible inter-view pairs of identified object locations, the cost/distance matrix is a pair-wise cost (Cost_ij) function, calculated as follows:

$\begin{array}{c}\mathrm{Cos}<br><br>\text{\hspace{1em}<br><br>}<br><br>t<br><br>\left(i,j\right)=<br><br>{\mathrm{\lambda <br><br>}}_1<br><br><br><br>x_2^T<br><br>{\mathrm{Fx}}_1<br><br>+{\mathrm{\lambda <br><br>}}_2<br><br>\left(\frac{<br><br>{\hat{h}}_1-h_1<br><br>}{h_1}+\frac{<br><br>{\hat{h}}_2-h_2<br><br>}{h_2}\right)+\\ <br><br>{\mathrm{\lambda <br><br>}}_3\left(\underset{w}{\sum <br><br>}<br><br>{<br><br>I_1<br><br>\left(x,y\right)-I_2<br><br>\left(x,y\right)<br><br>}^2\right)\end{array}$

where Cost denotes a cost function, minimization of which ensures a consistent localization of an object in the 3D spatial coordinates;

i and j denote point i in view 1 and point j in view 2 that correspond to detected objects in the two views;

λ

₁ denotes weighting factor for an epipolar constraint portion of the cost function;

x₂ denotes a coordinate vector for an object in view 2;

F denotes a fundamental matrix that determines epipolar lines in one view corresponding to points in the other view;

x₁ denotes a coordinate vector for an object in view 1;

superscript T denotes a transpose of the vector x₂;

wherein x₁ is a column vector and F is a matrix, so Fx₁ is also a column vector;

λ

₂ denotes a weighting factor for a window size consistency portion of the cost function;

h₁ denotes a size of the object in view 1 determined from the 2D projection of the object in 3D spatial coordinates to 2D view 1;

ĥ

denotes the size of the object in view 1 or 2 as determined from the object classifier outputs;

h₂ denotes the size of an object in view 2 determined from the 2D projection of the object in 3D spatial coordinates to the 2D view 2;

λ

₃ denotes a weighting factor for a window appearance similarity portion of the cost function;

w denotes window index;

∥

denotes a magnitude operator;

I₁ denotes an intensity distribution of the window in view 1;

x denotes an x coordinate in either view;

y denotes a y coordinate in either view;

I₂ denotes an intensity distribution of the window in view 2; and

‡

′

^;

denotes summation.

11. A multi-view object recognition system as set forth in claim 10, wherein the system is further configured to optimize pairing between the inter-view points (point i in view 1 and point j in view 2) using a bipartite weighted matching problem, and further comprising a smoothing filter for optimal 3D trajectory estimation.

12. A multi-view object recognition system as set forth in claim 11, wherein the object is further configured to track multiple objects.

13. A method for multi-view object recognition using swarming domain classifiers, the method comprising acts of:

configuring a plurality of software agents (i.e., particles) to operate as a cooperative swarm to classify an object in a domain as seen from multiple view points, where each agent is a complete classifier and is assigned an initial velocity vector to explore a solution space for object solutions;

configuring each agent to perform at least one iteration, the iteration being a search in the solution space for a potential solution optimum where each agent keeps track of its coordinates in multi-dimensional space that are associated with an observed best solution (pbest) that the agent has identified, and a global best solution (gbest) where the gbest is used to store the best location among all agents, with each velocity vector thereafter changing towards pbest and gbest, allowing the cooperative swarm to concentrate on the vicinity of the object and classify the object when a classification level exceeds a preset threshold.

14. A method as set forth in claim 13, further comprising an act of configuring the agents to search for the object in three-dimensional (3D) spatial coordinates, such that the object is a 3D object and the 3D object has distinct appearances from each view point in the multiple view points.

15. A method as set forth in claim 14, further comprising an act of linking the agents such that the distinct appearances of the 3D object from the multiple view points are linked by agents searching for the 3D object in the spatial coordinates, such that each agent has an associated 3D location X and an object height h, and wherein each of the multiple view points is provided as a 2D image from a calibrated camera having a given geometry, such that given the known geometry of the calibrated cameras, a 2D location, [x,y]^T=π

(X), of an agent'"'"'s projection in each view (2D image) is calculated and used to select an image window that is sent to a classifier having a classifier output that corresponds to the classifier'"'"'s confidence that the image window contains the object, where superscript T denotes transpose.

16. A method as set forth in claim 15, further comprising an act of configuring each agent to have a search trajectory that is guided by a cost function, such that the cost function is formed by combining the classifier outputs evaluated at the agent'"'"'s projection points in each of the views, wherein the projection points are points in an image corresponding to the 3D object.

17. A method as set forth in claim 16, further comprising an act of calculating a cost function for an agent at 3D location X=[x,y,z]^T, according to the following:

ƒ

(X,h)=w₁*classifier(image₁,π

₁(X),Π

₁(X,h))+w₂*classifier(image₂,π

₂(X),Π

₂(X,h)),

where, π

;

³→

² is a projection operator such that π

_n maps the object'"'"'s 3D location X into a 2D location [x,y] in image n, and w₁+w₂=1, where w₁ and w₂ are positive weighting factors and normally w₁=w₂=0.5, and where Π

is a projection operator for object height, h, such that the projection operator Π

_n maps the 3D object height to its corresponding projection size in image n, and where classifer denotes a confidence output of the object classifier operating on the window in image n with location π

_n(X) and window size Π

_n(X,h), and where * denotes multiplication.

18. A method as set forth in claim 13, further comprising acts of:

receiving each of the multiple view points as a 2D image from calibrated stereo cameras, and wherein the multiple view points include at least two 2D views, view 1 and view 2, wherein each view has 2D spatial coordinates; and

configuring the agents to move within each view independently to localize the object in each view independently.

19. A method as set forth in claim 18, further comprising acts of:

forming a 2D projection of a 3D object having 3D spatial coordinates, such that the 3D object has a 2D projection in each view in the multiple view points such that the object has a distinct appearance in each view; and

configuring the agents to search for the object in the 2D spatial coordinates.

20. A method as set forth in claim 19, further comprising acts of:

configuring the agents to operate as two distinct sets of agents such that each set searches for the object in a view independently to locate a 2D location [x,y] and a 2D image window height ĥ

of the object in each view; and

using triangulation to combine the 2D locations from each view to estimate the object'"'"'s 3D spatial coordinates from the 2D projections.

21. A method as set forth in claim 20, further comprising an act of establishing a correspondence between the 2D locations found in each 2D view to identify inter-view pairs.

22. A method as set forth in claim 21, wherein the act of establishing a correspondence, further comprises an act of forming a cost/distance matrix for all possible inter-view pairs of identified object locations, the cost/distance matrix being a pair-wise cost (Cost_ij) function, calculated as follows:

$\begin{array}{c}\mathrm{Cos}<br><br>\text{\hspace{1em}<br><br>}<br><br>t<br><br>\left(i,j\right)=<br><br>{\mathrm{\lambda <br><br>}}_1<br><br><br><br>x_2^T<br><br>{\mathrm{Fx}}_1<br><br>+{\mathrm{\lambda <br><br>}}_2<br><br>\left(\frac{<br><br>{\hat{h}}_1-h_1<br><br>}{h_1}+\frac{<br><br>{\hat{h}}_2-h_2<br><br>}{h_2}\right)+\\ <br><br>{\mathrm{\lambda <br><br>}}_3\left(\underset{w}{\sum <br><br>}<br><br>{<br><br>I_1<br><br>\left(x,y\right)-I_2<br><br>\left(x,y\right)<br><br>}^2\right)\end{array}$

where Cost denotes a cost function, minimization of which ensures a consistent localization of an object in the 3D spatial coordinates;

i and j denote point i in view 1 and point j in view 2 that correspond to detected objects in the two views;

λ

₁ denotes weighting factor for an epipolar constraint portion of the cost function;

x₂ denotes a coordinate vector for an object in view 2;

F denotes a fundamental matrix that determines epipolar lines in one view corresponding to points in the other view;

x₁ denotes a coordinate vector for an object in view 1;

superscript T denotes a transpose of the vector x₂;

wherein x₁ is a column vector and F is a matrix, so Fx₁ is also a column vector;

λ

₂ denotes a weighting factor for a window size consistency portion of the cost function;

h₁ denotes a size of the object in view 1 determined from the 2D projection of the object in 3D spatial coordinates to 2D view 1;

ĥ

denotes the size of the object in view 1 or 2 as determined from the object classifier outputs;

h₂ denotes the size of an object in view 2 determined from the 2D projection of the object in 3D spatial coordinates to the 2D view 2;

λ

₃ denotes a weighting factor for a window appearance similarity portion of the cost function;

w denotes window index;

∥

denotes a magnitude operator;

I₁ denotes an intensity distribution of the window in view 1;

x denotes an x coordinate in either view;

y denotes a y coordinate in either view;

I₂ denotes an intensity distribution of the window in view 2; and

‡

′

^;

denotes summation.

23. A method as set forth in claim 22, further comprising acts of:

optimizing pairing between the inter-view points (point i in view 1 and point j in view 2) using a bipartite weighted matching problem; and

estimating a 3D trajectory of at least one 3D object.

24. A method as set forth in claim 23, further comprising an act of tracking multiple objects.

25. A computer program product for object recognition, the computer program product comprising computer-readable instruction means encoded on a computer-readable medium and executable by a computer for causing a computer to:

configure a plurality of software agents (i.e., particles) to operate as a cooperative swarm to classify an object in a domain as seen from multiple view points, where each agent is a complete classifier and is assigned an initial velocity vector to explore a solution space for object solutions, where each agent is configured to perform at least one iteration, the iteration being a search in the solution space for a potential solution optima where each agent keeps track of its coordinates in multi-dimensional space that are associated with an observed best solution (pbest) that the agent has identified, and a global best solution (gbest) where the gbest is used to store the best location among all agents, with each velocity vector thereafter changing towards pbest and gbest, allowing the cooperative swarm to concentrate on the vicinity of the object and classify the object when a classification level exceeds a preset threshold.

26. A computer program product as set forth in claim 25, further comprising instruction means to cause a computer to perform an operation of utilizing the agents to search for the object in three-dimensional (3D) spatial coordinates, such that the object is a 3D object and the 3D object has distinct appearances from each view point in the multiple view points.

27. A computer program product as set forth in claim 26, further comprising instruction means to cause a computer to perform an operation of linking the agents such that the distinct appearances of the 3D object from the multiple view points are linked by agents searching for the 3D object in the spatial coordinates, such that each agent has an associated 3D location X and an object height h, and wherein each of the multiple view points is provided as a 2D image from a calibrated camera having a given geometry, such that given the known geometry of the calibrated cameras, a 2D location, [x,y]^T=π

(X), of an agent'"'"'s projection in each view (2D image) is calculated and used to select an image window that is sent to a classifier having a classifier output that corresponds to the classifier'"'"'s confidence that the image window contains the object, where superscript T denotes transpose.

28. A computer program product as set forth in claim 27, further comprising instruction means to cause a computer to perform an operation of configuring each agent to have a search trajectory that is guided by a cost function, such that the cost function is formed by combining the classifier outputs evaluated at the agent'"'"'s projection points in each of the views, wherein the projection points are points in an image corresponding to the 3D object.

29. A computer program product as set forth in claim 28, further comprising instruction means to cause a computer to perform an operation of calculating a cost function for an agent at 3D location X=[x,y,z]^T, according to the following:

ƒ

(X,h)=w₁*classifier(image₁,π

₁(X),Π

₁(X,h))+w₂*classifier(image₂,π

₂(X),Π

₂(X,h)),

where, π

;

³→

² is a projection operator such that π

_n maps the object'"'"'s 3D location X into a 2D location [x,y] in image n, and w₁+w₂=1, where w₁ and w₂ are positive weighting factors and normally w₁=w₂=0.5, and where Π

is a projection operator for object height, h, such that the projection operator Π

_n maps the 3D object height to its corresponding projection size in image n, and where classifer denotes a confidence output of the object classifier operating on the window in image n with location π

_n(X) and window size Π

_n(X,h), and where * denotes multiplication.

30. A computer program product as set forth in claim 25, further comprising instruction means for causing a computer to perform operations of:

receive each of the multiple view points as a 2D image from calibrated stereo cameras, and wherein the multiple view points include at least two 2D views, view 1 and view 2, wherein each view has 2D spatial coordinates; and

configure the agents to move within each view independently to localize the object in each view independently.

31. A computer program product as set forth in claim 30, further comprising instruction means to cause a computer to perform operations of:

forming a 2D projection of a 3D object having 3D spatial coordinates, such that the 3D object has a 2D projection in each view in the multiple view points such that the object has a distinct appearance in each view; and

configuring the agents to search for the object in the 2D spatial coordinates.

32. A computer program product as set forth in claim 31, further comprising instruction means to cause a computer to perform operations of:

configuring the agents to operate as two distinct sets of agents such that each set searches for the object in a view independently to locate a 2D location [x,y] and a 2D image window height ĥ

of the object in each view; and

using triangulation to combine the 2D locations from each view to estimate the object'"'"'s 3D spatial coordinates from the 2D projections.

33. A computer program product as set forth in claim 32, further comprising instruction means to cause a computer to perform an operation of establishing a correspondence between the 2D locations found in each 2D view to identify inter-view pairs.

34. A computer program product as set forth in claim 33, further comprising instruction means to cause a computer to perform an operation of forming a cost/distance matrix for all possible inter-view pairs of identified object locations, the cost/distance matrix being a pair-wise cost (Cost_ij) function, calculated as follows:

$\begin{array}{c}\mathrm{Cos}<br><br>\text{\hspace{1em}<br><br>}<br><br>t<br><br>\left(i,j\right)=<br><br>{\mathrm{\lambda <br><br>}}_1<br><br><br><br>x_2^T<br><br>{\mathrm{Fx}}_1<br><br>+{\mathrm{\lambda <br><br>}}_2<br><br>\left(\frac{<br><br>{\hat{h}}_1-h_1<br><br>}{h_1}+\frac{<br><br>{\hat{h}}_2-h_2<br><br>}{h_2}\right)+\\ <br><br>{\mathrm{\lambda <br><br>}}_3\left(\underset{w}{\sum <br><br>}<br><br>{<br><br>I_1<br><br>\left(x,y\right)-I_2<br><br>\left(x,y\right)<br><br>}^2\right)\end{array}$

where Cost denotes a cost function, minimization of which ensures a consistent localization of an object in the 3D spatial coordinates;

i and j denote point i in view 1 and point j in view 2 that correspond to detected objects in the two views;

λ

₁ denotes weighting factor for an epipolar constraint portion of the cost function;

x₂ denotes a coordinate vector for an object in view 2;

F denotes a fundamental matrix that determines epipolar lines in one view corresponding to points in the other view;

x₁ denotes a coordinate vector for an object in view 1;

superscript T denotes a transpose of the vector x₂;

wherein x₁ is a column vector and F is a matrix, so Fx₁ is also a column vector;

λ

₂ denotes a weighting factor for a window size consistency portion of the cost function;

h₁ denotes a size of the object in view 1 determined from the 2D projection of the object in 3D spatial coordinates to 2D view 1;

ĥ

denotes the size of the object in view 1 or 2 as determined from the object classifier outputs;

h₂ denotes the size of an object in view 2 determined from the 2D projection of the object in 3D spatial coordinates to the 2D view 2;

λ

₃ denotes a weighting factor for a window appearance similarity portion of the cost function;

w denotes window index;

∥

denotes a magnitude operator;

I₁ denotes an intensity distribution of the window in view 1;

x denotes an x coordinate in either view;

y denotes a y coordinate in either view;

I₂ denotes an intensity distribution of the window in view 2; and

‡

′

^;

denotes summation.

35. A computer program product as set forth in claim 34, further instruction means to cause a computer to perform operations:

optimizing pairing between the inter-view points (point i in view 1 and point j in view 2) using a bipartite weighted matching problem; and

estimating a 3D trajectory of at least one 3D object.

36. A computer program product as set forth in claim 35, further comprising instruction means to cause a computer to perform an operation of tracking multiple objects.