Method and apparatus for total situational awareness and monitoring

• US 8,289,390 B2
• Filed: 07/28/2005
• Issued: 10/16/2012
• Est. Priority Date: 07/28/2004
• Status: Active Grant

• Current Assignee: SRI International, Inc.
• Original Assignee: SRI International, Inc.
• Inventors: Aggarwal, Manoj, Hanna, Keith, Sawhney, Harpreet, Paragano, Vincent, Kumar, Rakesh, Samarasekera, Supun, Arpa, Aydin, Germano, Thomas, Zhao, Tao, Kalokitis, David, Patterson, David R.
• Primary Examiner(s): Perungavoor, Sath V
• Assistant Examiner(s): PONTIUS, JAMES M
• Application Number:
  US11/192,486
• Publication Number:
  US 20060279630A1
• Time in Patent Office: 2,637 Days
• Field of Search: 348/143, 348/47, 348/155, 382/130, 382/154, 340/522, 386/326
• US Class Current: 348/143
• CPC Class Codes: G01S 3/7864 T.V. type tracking systems
• CPC Class Codes: G06F 18/251 of input or preprocessed data
• CPC Class Codes: G06T 2207/30232 Surveillance
• CPC Class Codes: G06T 2207/30241 Trajectory
• CPC Class Codes: G06T 7/246 using feature-based methods, e.g. the tracking of corners or segments
• CPC Class Codes: G06T 7/285 using a sequence of stereo image pairs
• CPC Class Codes: G06T 7/292 Multi-camera tracking
• CPC Class Codes: G06T 7/70 Determining position or orientation of objects or cameras camera calibration G06T7/80
• CPC Class Codes: G06V 10/24 Aligning, centring, orientation detection or correction of the image
• CPC Class Codes: G06V 10/803 of input or preprocessed data
• CPC Class Codes: G06V 20/52 Surveillance or monitoring of activities, e.g. for recognising suspicious objects recognising microscopic objects G06V20/69
• CPC Class Codes: G08B 13/19608 Tracking movement of a target, e.g. by detecting an object predefined as a target, using target direction and or velocity to predict its new position
• CPC Class Codes: G08B 13/19641 Multiple cameras having overlapping views on a single scene
• CPC Class Codes: G08B 13/19645 Multiple cameras, each having view on one of a plurality of scenes, e.g. multiple cameras for multi-room surveillance or for tracking an object by view hand-over
• CPC Class Codes: G08B 13/19682 Graphic User Interface [GUI] presenting system data to the user, e.g. information on a screen helping a user interacting with an alarm system
• CPC Class Codes: H04N 13/204 using stereoscopic image cameras stereoscopic photography G03B35/00
• CPC Class Codes: H04N 13/239 using two 2D image sensors having a relative position equal to or related to the interocular distance H04N13/243 takes precedence
• CPC Class Codes: H04N 13/243 using three or more 2D image sensors
• CPC Class Codes: H04N 2013/0081 Depth or disparity estimation from stereoscopic image signals
• CPC Class Codes: H04N 2013/0085 Motion estimation from stereoscopic image signals
• CPC Class Codes: H04N 5/2224 : related to virtual studio applications
• CPC Class Codes: H04N 5/76 : Television signal recording

Abstract

A sentient system combines detection, tracking, and immersive visualization of a cluttered and crowded environment, such as an office building, terminal, or other enclosed site using a network of stereo cameras. A guard monitors the site using a live 3D model, which is updated from different directions using the multiple video streams. As a person moves within the view of a camera, the system detects its motion and tracks the person'"'"'s path, it hands off the track to the next camera when the person goes out of that camera'"'"'s view. Multiple people can be tracked simultaneously both within and across cameras, with each track shown on a map display. The track system includes a track map browser that displays the tracks of all moving objects as well as a history of recent tracks and a video flashlight viewer that displays live immersive video of any person that is being tracked.

Claims

1. A computer implemented method for situational awareness and monitoring, comprising:

using at least one computer for;

receiving a plurality of video streams from a network of monocular and stereo video cameras;

receiving position information and identifying information from a plurality of non-vision sensors;

detecting a plurality of objects and corresponding the objects with the identifying information;

tracking each object using the plurality of video streams, position information and the identifying information to form a plurality of tracks for each object;

fusing the plurality of tracks for each object into a global track for each object by associating identifying information in the plurality of tracks for each object to form a plurality of global tracks;

providing a first display of a 3D model and a map display overlaid with at least a portion of the plurality of global tracks, the 3D model being of a site overlaid with a plurality of video streams; and

providing a second display, proximate to the first display, of a list of recently fused tracks.

2. The method of claim 1, wherein each non-vision sensor is associated with a radio frequency (RF) or biometric identification system.

3. The method of claim 1, further comprising:

segmenting each object from within the plurality of objects.

4. The method of claim 1, further comprising:

matching each object with a learned shape, appearance, and depth.

5. The method of claim 1, further comprising:

carrying forward an identity of each object from one video camera to a next video camera as the object moves, the identity of each object being associated with the global track of that object.

6. The method of claim 1, further comprising:

measuring a location, velocity, and height of each object.

7. The method of claim 1, further comprising:

storing the global tracks in a database; and

providing a tool for reviewing stored global tracks in the display.

8. A system for situational awareness and monitoring, comprising:

a plurality of cameras and non-vision sensors configured to provide video streams and non-vision sensor information for detecting a plurality of objects, the non-vision sensor information including position information and identifying information corresponding to the detected objects;

a tracker pipeline configured to perform stereo computation and to perform object tracking to form a plurality of tracks for each object;

a global tracker resolver configured to receive the non-vision sensor position information, the identifying information and the plurality of tracks for each object from the tracker pipeline, to perform track fusion of video and non-vision sensor position information and identifying information into a global track for each object by associating identifying information in the plurality of tracks for each object to form a plurality of global tracks, each global track being for a particular object; and

a visualizer and integrated graphical user interface (GUI) configured to provide a 3D model of a site overlaid with a plurality of video streams and a map overlaid with at least a portion of the plurality of global tracks and to provide a proximate display of a list of recently fused tracks.

9. The system of claim 8, wherein each non-vision sensor is associated with a radio frequency (RF) or biometric identification system.

10. The system of claim 8, further comprising:

a database to store the global tracks; and

a database query engine to retrieve the stored global tracks and to provide query results to be displayed by the visualizer and integrated GUI.

11. The system of claim 10, further comprising:

an event recognizer to retrieve the stored global tracks and to provide events to be associated with the global tracks by the global tracker resolver and to be displayed by the visualizer and integrated GUI.

12. The system of claim 8, wherein the visualizer and integrated GUI provides an option to display a recent history associated with a selected global track.

13. The system of claim 8, wherein the visualizer and integrated GUI provides an option to display a live video associated with a selected global track.

14. The system of claim 8, wherein the visualizer and integrated GUI provides a timeline of at least a portion of the global tracks.

15. A non-transitory computer readable storage medium having stored thereon a computer program encoded with instructions, causing at least one computer to execute code for:

receiving a plurality of video streams from a network of monocular and stereo video cameras;

receiving position information and identifying information from a plurality of non-vision sensors;

detecting a plurality of objects and corresponding the objects with the identifying information;

tracking each object to form a plurality of tracks for each object using the plurality of video streams, position information and the identifying information;

fusing the plurality of tracks for each object into a global track for each object by associating identifying information across the plurality of tracks for each object to form a plurality of global tracks;

providing a display of a 3D model and a map display overlaid with at least a portion of the plurality of global tracks, the 3D model being of a site overlaid with a plurality of video streams; and

providing a second display, proximate to the first display, of a list of recently fused tracks.