Active coordinated tracking for multi-camera systems

• US 8,180,107 B2
• Filed: 02/11/2010
• Issued: 05/15/2012
• Est. Priority Date: 02/13/2009
• Status: Active Grant

• Current Assignee: SRI International, Inc.
• Original Assignee: SRI International, Inc.
• Inventors: Broaddus, Christopher P., Germano, Thomas, Vandervalk, Nicholas, Wu, Shunguang, Divakaran, Ajay, Sawhney, Harpreet Singh
• Primary Examiner(s): Seth, Manav
• Application Number:
  US12/704,189
• Publication Number:
  US 20100208941A1
• Time in Patent Office: 824 Days
• Field of Search: 382100-322
• US Class Current: 382/103
• CPC Class Codes: G01S 13/726 Multiple target tracking
• CPC Class Codes: G01S 13/867 Combination of radar systems with cameras
• CPC Class Codes: G01S 3/7864 T.V. type tracking systems
• CPC Class Codes: G06T 2207/20076 Probabilistic image processing
• CPC Class Codes: G06T 2207/30232 Surveillance
• CPC Class Codes: G06T 7/194 involving foreground-background segmentation
• CPC Class Codes: G06T 7/292 Multi-camera tracking
• CPC Class Codes: G08B 13/19608 Tracking movement of a target, e.g. by detecting an object predefined as a target, using target direction and or velocity to predict its new position
• CPC Class Codes: G08B 13/19645 Multiple cameras, each having view on one of a plurality of scenes, e.g. multiple cameras for multi-room surveillance or for tracking an object by view hand-over
• CPC Class Codes: G08B 13/19697 Arrangements wherein non-video detectors generate an alarm themselves

Abstract

A method and system for coordinated tracking of objects is disclosed. A plurality of images is received from a plurality of nodes, each node comprising at least one image capturing device. At least one target in the plurality of images is identified to produce at least one local track corresponding to each of the plurality of nodes having the at least one target in its field of view. The at least one local track corresponding to each of the plurality of nodes is fused according to a multi-hypothesis tracking method to produce at least one fused track corresponding to the at least one target. At least one of the plurality of nodes is assigned to track the at least one target based on minimizing at least one cost function comprising a cost matrix using the k-best algorithm for tracking at least one target for each of the plurality of nodes. The at least one fused track is sent to the at least one of the plurality of nodes assigned to track the at least one target based on the at least one fused track.

Claims

1. A computer-implemented method for coordinated tracking of objects, the method being executed by at least one processor, comprising the steps of:

receiving a plurality of images from a plurality of nodes;

identifying at least one target in the plurality of images to produce at least one local track corresponding to each node of the plurality of nodes having the identified at least one target in a field of view of each node;

fusing the at least one local track corresponding to each node of the plurality of nodes according to a multi-hypothesis tracking method to produce at least one fused track corresponding to the at least one target;

assigning at least one node of the plurality of nodes to track the at least one target based on minimizing at least one cost function for tracking at least one target for each node of the plurality of nodes; and

sending the at least one fused track to the at least one node of the plurality of nodes to assigned track the at least one target based on the at least one fused track.

2. The method of claim 1, wherein at least one sensor is assigned per node, wherein minimizing the at least one cost function corresponds to minimizing an N×

M cost matrix C, wherein M and N are the number of sensors and the number of targets, respectively, and wherein the cost matrix is minimized using the k-best algorithm to find the best sensor to task assignment, wherein a task represents one target.

3. The method of claim 2, wherein each element of the cost matrix is C_ij=V_ij+D_ij+O_ij+W_ij,

wherein V_ij is a measure between [0,1] indicating whether an associated geographic location of a task j is visible to a sensor i,

wherein j is greater than or equal to 1 and i is greater than or equal to 2,

wherein D_ij is a measure between [0,1] indicating the cost for the task i to follow a location of the sensor j based on pixels on a target,

wherein O_ij is a measure between [0,1] indicating whether moving objects associated with task j are being observed by sensor i, and

wherein W_ij is measure between [0,1] indicating the cost for sensor i to follow task j.

4. The method of claim 3, wherein minimizing the cost matrix is subject to a constraint of an N×

M forbidden matrix, F, wherein the forbidden matrix is based on a sensor state, the sensor state being a binary measure (0 or

     1) indicating whether sensor i is available for assignment to follow a task j, wherein the assignment is forbidden.

5. The method of claim 2, further comprising employing a plurality of sensors to observe the same target using a k-redundancy method, wherein k is the maximum number of sensors that are assigned to a task, wherein the k-redundancy method comprises the steps of, for each of the k sensors,

generating the cost matrix for all untasked sensors in an untasked pool;

solving the cost matrix using the k-best algorithm; and

assigning each sensor to its best task and remove the sensors from the untasked pool,

wherein a task represents one target, and

wherein number of rows in the cost matrix is reduced for each step less than or equal to the number of tasks.

6. The method of claim 1, wherein the at least one node comprises at least one image capturing device.

7. The method of claim 6, wherein the at least one image capturing device comprises at least one of a pan-tilt-zoom (PTZ) camera, a fixed camera, a GPS device, an AIS device, and a radar-based device.

8. The method of claim 7, wherein the PTZ camera is configured to track more than one target per task.

9. The method of claim 8, wherein the more than one target are grouped into clusters according to the K-means clustering algorithm.

10. The method of claim 9, wherein each of the clusters is labeled as tasks, and wherein the at least one cost function is minimized with clusters substituted for tasks.

11. The method of claim 1, wherein the multi-hypothesis tracking method is the simplified joint probabilistic data association method.

25. The computer readable medium of claim 4, wherein minimizing the cost matrix is subject to a constraint of an N×

M forbidden matrix, F, wherein the forbidden matrix is based on a sensor state, the sensor state being a binary measure (0 or

     1) indicating whether sensor i is available for assignment to follow a task j, wherein the assignment is forbidden.

12. A system for coordinated tracking of objects, comprising:

a plurality of nodes each comprising at least one image capturing device for receiving a plurality of images; and

at least one processor for;

receiving the plurality of images from the plurality of nodes;

identifying at least one target in the plurality of images using to produce at least one local track corresponding to each node of the plurality of nodes having the at least one target in a field of view of each node;

fusing the at least one local track corresponding to each node of the plurality of nodes according to a multi-hypothesis tracking method to produce at least one fused track corresponding to the at least one target;

assigning at least one node of the plurality of nodes to track the at least one target based on minimizing at least one cost function for tracking at least one target for each node of the plurality of nodes; and

sending the at least one fused track to the at least one node of the plurality of nodes assigned to track the at least one target based on the at least one fused track.

13. The system of claim 12, wherein at least one sensor is assigned per node, wherein minimizing the at least one cost function corresponds to minimizing an N×

M cost matrix C, wherein M and N are the number of sensors and the number of targets, respectively, and wherein the cost matrix is minimized using the k-best algorithm to find the best sensor to task assignment, wherein a task represents one target.

14. The system of claim 13, wherein each element of the cost matrix is C_ij=V_ij+D_ij+O_ij+W_ij,

wherein V_ij is a measure between [0,1] indicating whether an associated geographic location of a task j is visible to a sensor i,

wherein j is greater than or equal to 1 and i is greater than or equal to 2,

wherein D_ij is a measure between [0,1] indicating the cost for the task i to follow a location of the sensor j based on pixels on a target,

wherein O_ij is a measure between [0,1] indicating whether moving objects associated with task j are being observed by sensor i, and

wherein W_ij is measure between [0,1] indicating the cost for sensor i to follow task j.

15. The system of claim 14, wherein minimizing the cost matrix is subject to a constraint of an N×

M forbidden matrix, F, wherein the forbidden matrix is based on a sensor state, the sensor state being a binary measure (0 or

     1) indicating whether node i is available for assignment to follow a task j, wherein the assignment is forbidden.

16. The system of claim 13, further comprising employing a plurality of sensors to observe the same target using a k-redundancy method, wherein k is the maximum number of sensors that are assigned to a task, wherein the k-redundancy method comprises the steps of, for each of the k sensors,

generating the cost matrix for all untasked sensors in an untasked pool;

solving the cost matrix using the k-best algorithm; and

assigning each sensors to its best task and remove the sensors from the untasked pool,

wherein a task represents one target, and

wherein number of rows in the cost matrix is reduced for each step less than or equal to the number of tasks.

17. The system of claim 13, wherein the at least one image capturing device comprises at least one of a pan-tilt-zoom (PTZ) camera, a fixed camera, a GPS device, an AIS device, and a radar-based device.

18. The system of claim 17, wherein the PTZ camera is configured to track more than one target per task.

19. The system of claim 18, wherein the more than one target are grouped into clusters according to the K-means clustering algorithm.

20. The system of claim 19, wherein each of the clusters is labeled as tasks, and wherein the at least one cost function is minimized with clusters substituted for tasks.

21. The system of claim 13, wherein the multi-hypothesis tracking method is the simplified joint probabilistic data association method.

22. A computer-readable medium storing computer code for coordinated tracking of objects, the code being executed by at least one processor, wherein the computer code comprises code for:

receiving a plurality of images from a plurality of nodes;

identifying at least one target in the plurality of images to produce at least one local track corresponding to each node of the plurality of nodes having the at least one target in a field of view of each node;

fusing the at least one local track corresponding to each node of the plurality of nodes according to a multi-hypothesis tracking method to produce at least one fused track corresponding to the at least one target;

assigning at least one node of the plurality of nodes to track the at least one target based on minimizing at least one cost function for tracking at least one target for each node of the plurality of nodes; and

sending the at least one fused track to the at least one of the plurality of nodes to assigned track the at least one target based on the at least one fused track.

23. The computer readable medium of claim 22, wherein code for minimizing the at least one cost function corresponds to code for minimizing an N×

M cost matrix C, wherein M and N are the number of sensors and the number of targets, respectively, and wherein the cost matrix is minimized using the k-best algorithm to find the best sensor to task assignment, wherein a task represents one target.

24. The computer readable medium of claim 23, wherein each element of the cost matrix C_ij=V_ij+D_ij+O_ij+W_ij,

wherein V_ij is a measure between [0,1] indicating whether an associated geographic location of a task j is visible to a sensor i,

wherein j is greater than or equal to 1 and i is greater than or equal to 2,

wherein D_ij is a measure between [0,1] indicating the cost for the task i to follow a location of the sensor j based on pixels on a target,

wherein O_ij is a measure between [0,1] indicating whether moving objects associated with task j are being observed by sensor i, and

wherein W_ij is measure between [0,1] indicating the cost for sensor i to follow task j.