Method and apparatus for total situational awareness and monitoring

• US 20060279630A1
• Filed: 07/28/2005
• Published: 12/14/2006
• Est. Priority Date: 07/28/2004
• Status: Active Grant

• Current Assignee: SRI International, Inc.
• Original Assignee: SRI International, Inc.
• Inventors: Samarasekera, Supun, Arpa, Aydin, Kumar, Rakesh, Paragano, Vincent, Sawhney, Harpreet, Hanna, Keith, Zhao, Tao, Aggarwal, Manoj, Kalokitis, David, Patterson, David R., Germano, Thomas
• Granted Patent: US 8,289,390 B2
• Time in Patent Office: Days
• US Class Current: 348/143
• CPC Class Codes: G01S 3/7864 T.V. type tracking systems
• CPC Class Codes: G06F 18/251 of input or preprocessed data
• CPC Class Codes: G06T 2207/30232 Surveillance
• CPC Class Codes: G06T 2207/30241 Trajectory
• CPC Class Codes: G06T 7/246 using feature-based methods, e.g. the tracking of corners or segments
• CPC Class Codes: G06T 7/285 using a sequence of stereo image pairs
• CPC Class Codes: G06T 7/292 Multi-camera tracking
• CPC Class Codes: G06T 7/70 Determining position or orientation of objects or cameras camera calibration G06T7/80
• CPC Class Codes: G06V 10/24 Aligning, centring, orientation detection or correction of the image
• CPC Class Codes: G06V 10/803 of input or preprocessed data
• CPC Class Codes: G06V 20/52 Surveillance or monitoring of activities, e.g. for recognising suspicious objects recognising microscopic objects G06V20/69
• CPC Class Codes: G08B 13/19608 Tracking movement of a target, e.g. by detecting an object predefined as a target, using target direction and or velocity to predict its new position
• CPC Class Codes: G08B 13/19641 Multiple cameras having overlapping views on a single scene
• CPC Class Codes: G08B 13/19645 Multiple cameras, each having view on one of a plurality of scenes, e.g. multiple cameras for multi-room surveillance or for tracking an object by view hand-over
• CPC Class Codes: G08B 13/19682 Graphic User Interface [GUI] presenting system data to the user, e.g. information on a screen helping a user interacting with an alarm system
• CPC Class Codes: H04N 13/204 using stereoscopic image cameras stereoscopic photography G03B35/00
• CPC Class Codes: H04N 13/239 using two 2D image sensors having a relative position equal to or related to the interocular distance H04N13/243 takes precedence
• CPC Class Codes: H04N 13/243 using three or more 2D image sensors
• CPC Class Codes: H04N 2013/0081 Depth or disparity estimation from stereoscopic image signals
• CPC Class Codes: H04N 2013/0085 Motion estimation from stereoscopic image signals
• CPC Class Codes: H04N 5/2224 : related to virtual studio applications
• CPC Class Codes: H04N 5/76 : Television signal recording

Abstract

A sentient system combines detection, tracking, and immersive visualization of a cluttered and crowded environment, such as an office building, terminal, or other enclosed site using a network of stereo cameras. A guard monitors the site using a live 3D model, which is updated from different directions using the multiple video streams. As a person moves within the view of a camera, the system detects its motion and tracks the person'"'"'s path, it hands off the track to the next camera when the person goes out of that camera'"'"'s view. Multiple people can be tracked simultaneously both within and across cameras, with each track shown on a map display. The track system includes a track map browser that displays the tracks of all moving objects as well as a history of recent tracks and a video flashlight viewer that displays live immersive video of any person that is being tracked.

Claims

1. A method for situational awareness and monitoring, comprising:

receiving a plurality of video streams from a network of monocular and stereo video cameras;

receiving a plurality of identities from a plurality of sensors;

detecting a plurality of objects, events, and identities, the events and identities being associated with particular objects;

fusing a plurality of tracks for each object from the video cameras and sensors into a global track for each object; and

providing a display of a 3D model and a map display overlaid with at least a portion of the global tracks, the 3D model being of a site overlaid with a plurality of video streams.

2. The method of claim 1, wherein each sensor is associated with a radio frequency (RF) or biometric identification system.

3. The method of claim 1, further comprising:

segmenting each object from the rest of the objects; and

tracking each object.

4. The method of claim 1, further comprising:

matching each object with a learned shape, appearance, and depth.

5. The method of claim 1, further comprising:

carrying forward the identity of each object from one video camera to the next as the object moves, the identity of each object being associated with the global track of that object.

6. The method of claim 1, further comprising:

measuring a location, velocity, and height of each object.

7. The method of claim 1, further comprising:

storing the global tracks in a database; and

providing a tool for reviewing stored global tracks in the display.

8. A system for situational awareness and monitoring, comprising:

a plurality of cameras and sensors to provide video streams and sensor information, the sensor information including identification information;

a tracker pipeline to perform stereo computation and to perform object tracking;

a global tracker resolver to receive the sensor information and track information from the tracker pipeline, to perform track fusion of video and sensor information, and to provide a plurality of global tracks, each global track being for a particular object and having an associated identity; and

a visualizer and integrated graphical user interface (GUI) to provide a 3D model of a site overlaid with a plurality of video streams and a map overlaid with at least a portion of the global tracks.

9. The system of claim 8, wherein each sensor is associated with a radio frequency (RF) or biometric identification system.

10. The system of claim 8, further comprising:

an event recognizer to retrieve the stored global tracks and to provide events to be associated with the global tracks by the global tracker resolver and to be displayed by the visualizer and integrated GUI.

11. The system of claim 8, further comprising:

a database to store the global tracks; and

a database query engine to retrieve the stored global tracks and to provide query results to be displayed by the visualizer and integrated GUI.

12. The system of claim 8, wherein the visualizer and integrated GUI provides an option to display a recent history associated with a selected global track.

13. The system of claim 8, wherein the visualizer and integrated GUI provides an option to display a live video associated with a selected global track.

14. The system of claim 8, wherein the visualizer and integrated GUI provides a timeline of at least a portion of the global tracks.

15. A computer readable medium having instructions stored thereon to perform a method for situational awareness and monitoring, the method comprising:

receiving a plurality of video streams from a network of monocular and stereo video cameras;

receiving a plurality of identities from a plurality of sensors;

detecting a plurality of objects, events, and identities, the events and identities being associated with particular objects;

fusing a plurality of tracks for each object from the video cameras and sensors into a global track for each object; and

providing a display of a 3D model and a map display overlaid with at least a portion of the global tracks, the 3D model being of a site overlaid with a plurality of video streams.