System and method for real-time 3-D object tracking and alerting via networked sensors

• US 9,520,040 B2
• Filed: 02/26/2009
• Issued: 12/13/2016
• Est. Priority Date: 11/21/2008
• Status: Active Grant

• Current Assignee: Raytheon Company (Rtx Corporation)
• Original Assignee: Raytheon Company (Rtx Corporation)
• Inventors: Mavromatis, Theofanis
• Primary Examiner(s): Nguyen, Phuoc
• Assistant Examiner(s): Belani, Kishin G
• Application Number:
  US12/394,029
• Publication Number:
  US 20100128110A1
• Time in Patent Office: 2,847 Days
• Field of Search: 348/143, 348/47
• US Class Current: 1/1
• CPC Class Codes: G06T 2207/30232 Surveillance
• CPC Class Codes: G06T 2207/30241 Trajectory
• CPC Class Codes: G06T 7/292 Multi-camera tracking
• CPC Class Codes: G08B 13/19641 Multiple cameras having overlapping views on a single scene
• CPC Class Codes: H04N 13/243 using three or more 2D image sensors

Abstract

The invention is an integrated system consisting of a network of two-dimensional sensors (such as cameras) and processors along with target detection, data fusion, tracking, and alerting algorithms to provide three-dimensional, real-time, high accuracy target tracks and alerts in a wide area of interest. The system uses both target kinematics (motion) and features (e.g., size, shape, color, behavior, etc.) to detect, track, display, and alert users to potential objects of interest.

Claims

1. An object detection and tracking system of a three-dimensional (3-D) space, the system comprising:

a network for interconnecting components of the system;

a common reference clock connected to the network and configured to provide time stamps to the components of the system;

two or more spatially separated two-dimensional (2-D) sensors connected to the network and having overlapping fields of view of the 3-D space for detecting and 3-D tracking of one or more moving objects in the 3-D space through concurrent time-stamped 2-D observations, each point in the 3-D space being observable by at least two of the 2-D sensors;

at least one image processor connected to the network and for converting each 2-D sensor'"'"'s time-stamped 2-D observations into time-stamped 2-D spatial characteristics and for extracting 2-D feature data of the one or more moving objects;

at least one tracking processor connected to the network and for combining the time-stamped 2-D spatial characteristics and the 2-D feature data from respective said 2-D sensors into time-stamped 3-D spatial characteristics through triangulation and then fusing the time-stamped 3-D spatial characteristics and the 2-D feature data from the respective 2-D sensors into 3-D tracks comprising 3-D kinematic data, and into 3-D feature data of the one or more moving objects; and

at least one behavior analysis and alert processor connected to the network and for analyzing the 3-D kinematic and feature data and generating appropriate alerts corresponding to 3-D behaviors of interest.

2. The system of claim 1 further comprising at least one display connected to the network and for presenting the 3-D kinematic and feature data.

3. The system of claim 2, wherein the at least one display is further configured to present alert data of the appropriate alerts.

4. The system of claim 1 wherein the 2-D sensors are passive electro-optical/infrared 2-D sensors.

5. The system of claim 1 further comprising one or more pan-tilt-zoom (PTZ) cameras connected to the network and for object identification of the one or more moving objects, wherein the one or more PTZ cameras can be activated automatically or from operator input to track a particular one of the one or more moving objects.

6. The system of claim 1 further comprising at least one spatially oriented 3-D sensor connected to the network and for detecting the one or more moving objects through time-stamped 3-D observations to produce time-stamped 3-D observation data, wherein the time-stamped 3-D observation data is fed directly to the at least one tracking processor.

7. The system of claim 1 wherein the 2-D sensors are calibrated with respect to their corresponding said fields of view and spatially oriented with respect to their observation points of the 3-D space.

8. The system of claim 1 wherein the system processes the data in real time.

9. An object detection and tracking method of a three-dimensional (3-D) space, the method comprising:

spatially separating two or more two-dimensional (2-D) sensors having overlapping fields of view of the 3-D space, each point in the 3-D space being observable by at least two of the 2-D sensors;

interconnecting the 2-D sensors, a common reference clock, and at least one processor to a network, the common reference clock being configured to provide time stamps to the 2-D sensors; and

detecting and 3-D tracking, by the at least one processor, of one or more moving objects in the 3-D space, comprising;

collecting concurrent time-stamped 2-D observations from the 2-D sensors;

converting each sensor'"'"'s time-stamped 2-D observations into time-stamped 2-D spatial characteristics and extracting 2-D feature data of the one or more moving objects;

combining the time-stamped 2-D spatial characteristics and the 2-D feature data from respective said 2-D sensors into time-stamped 3-D spatial characteristics through triangulation;

fusing the time-stamped 3-D spatial characteristics and the 2-D feature data from the respective 2-D sensors into 3-D tracks comprising 3-D kinematic data, and into 3-D feature data of the one or more moving objects; and

analyzing the 3-D kinematic and feature data, and generating appropriate alerts corresponding to 3-D behaviors of interest.

10. The method of claim 9 further comprising:

connecting a display to the network; and

displaying the 3-D kinematic and feature data on the display.

11. The method of claim 10 further comprising displaying alert data of the appropriate alerts on the display.

12. The method of claim 9 wherein the 3-D tracking further comprises:

creating the 3-D tracks directly from new time-stamped 2-D observations and existing ones of the 3-D tracks;

evaluating the 3-D tracks for likelihood based on consistent said 3-D kinematic and feature data; and

pruning least likely ones of the 3-D tracks.

13. The method of claim 12 wherein the 3-D tracking further comprises:

clustering ones of the 3-D tracks that share ones of the time-stamped 2-D observations and

identifying incompatible ones of the 3-D tracks.

14. The method of claim 13 wherein the 3-D tracking further comprises multiple hypothesis testing, the multiple hypothesis testing comprising:

creating hypotheses from combinations of compatible ones of the 3-D tracks;

evaluating the hypotheses for likelihood based on likelihood of corresponding said combinations of the compatible ones of the 3-D tracks; and

pruning unlikely ones of the hypotheses.

15. The method of claim 14 wherein the 3-D tracking further comprises:

reevaluating the 3-D tracks for likelihood based on likelihood of corresponding said hypotheses and

pruning unlikely ones of the 3-D tracks.

16. The method of claim 15 wherein the 3-D tracking further comprises 3-D track filtering, the 3-D track filtering comprising:

identifying similar ones of the 3-D tracks likely to represent a same object of the one or more objects; and

merging the similar ones of the 3-D tracks into a single 3-D track that represents the same object.

17. The method of claim 9 further comprising processing the data in real-time.

18. The method of claim 9 further comprising:

connecting one or more pan-tilt-zoom (PTZ) cameras to the network;

selecting one or more of the moving objects; and

tracking, by the at least one processor, the selected one or more of the moving objects on the one or more PTZ cameras.

19. The method of claim 9 further comprising:

connecting at least one 3-D sensor to the network;

spatially orienting the at least one 3-D sensor with respect to a corresponding at least one observation point of the 3-D space;

collecting time-stamped 3-D observations from the at least one 3-D sensor; and

fusing the time stamped 3-D observations into the 3-D kinematic and feature data.

20. The method of claim 9 further comprising:

calibrating the 2-D sensors with respect to their corresponding said fields of view; and

spatially orienting the 2-D sensors with respect to their observation points of the 3-D space.