Situation awareness system

• US 6,396,535 B1
• Filed: 02/16/1999
• Issued: 05/28/2002
• Est. Priority Date: 02/16/1999
• Status: Expired due to Fees

• Current Assignee: Mitsubishi Electric Research Laboratories, Inc. (Mitsubishi Electric Corporation)
• Original Assignee: Mitsubishi Electric Research Laboratories, Inc. (Mitsubishi Electric Corporation)
• Inventors: Waters, Richard C.
• Primary Examiner(s): RAO, ANAND SHASHIKANT
• Application Number:
  US09/250,956
• Time in Patent Office: 1,197 Days
• Field of Search: 348/143-149,160,169,170,150-159
• US Class Current: 348/159
• CPC Class Codes: G01S 11/12 using electromagnetic waves other than radio waves
• CPC Class Codes: G06T 7/269 using gradient-based methods
• CPC Class Codes: G06T 7/97 Determining parameters from multiple pictures depth or shape recovery from multiple images G06T7/55; stereo camera calibration G06T7/85
• CPC Class Codes: G06V 20/54 of traffic, e.g. cars on the road, trains or boats
• CPC Class Codes: G06V 20/58 Recognition of moving objects or obstacles, e.g. vehicles or pedestrians; Recognition of traffic objects, e.g. traffic signs, traffic lights or roads

Abstract

A situation awareness system includes a plurality of cameras. Each camera acquires a sequence of images of a particular part of an area of interest. There is overlap between the parts so that the system can obtain depth information about objects in the area of interest. An analyzer identifies moving objects in the areas of interest, attributes of the moving objects, and events related to the moving objects. A display device displays the attributed objects and events as annotated graphic elements and alerts.

Claims

1. A real-time situation awareness system, comprising:

a plurality of cameras acquiring a plurality of video streams of overlapping parts of an area of interest;

analyzer means for reducing the plurality of video streams to attributed data objects and events using template matching;

means for discarding the plurality of video streams after reducing the plurality of video streams to attributed data objects and events; and

synthesizer means for rendering the attributed data objects and events as annotated graphic elements and alerts representative of the area of interest on an output device.

2. The system of claim 1 further comprising:

means for temporally analyzing an optical flow in successive images of a single video stream to generate motion fields;

means for spatially registering adjacent images of multiple video stream to obtain depth images; and

means for segmenting the motion fields and depth images to generate partially attributed data objects.

3. The system of claim 2 further comprising:

means for analyzing the partially attributed data objects using an application specific database to generate fully attributed data objects and events.

4. The system of claim 3 further comprising:

sensors providing the analyzing step with secondary data and signals.

5. The system of claim 1 wherein the synthesizer means produces a display having a point of view substantially orthogonal to the point of view of the cameras.

6. The system of claim 1 wherein the area of interest is a panoramic scene.

7. The system of claim 1 wherein annotations for the graphic elements include directions of movement.

8. The system of claim 5 wherein user control signals determine an orientation of the point of view of the display.

9. A method for generating a real-time situation awareness display, comprising the steps of:

acquiring a plurality of video streams of overlapping parts of an area of interest;

reducing the plurality of video streams to attributed data objects and events using template matching;

discarding the plurality of video streams after reducing the plurality of video streams to attributed data objects and events; and

rendering the attributed data objects and events as annotated graphic elements and alerts representative of the area of interest on an output device.

10. The method of claim 9 further comprising:

temporally analyzing an optical flow in successive images of a single video stream to generate motion fields;

spatially registering adjacent images of multiple video stream to obtain depth images; and

segmenting the motion fields and depth images to generate partially attributed data objects.

11. The method of claim 10 further comprising:

analyzing the partially attributed data objects using an application specific database to generate fully attributed data objects and event.