Multi-Sensor Surveillance System with a Common Operating Picture

• US 20120304085A1
• Filed: 05/23/2011
• Published: 11/29/2012
• Est. Priority Date: 05/23/2011
• Status: Active Grant

• Current Assignee: The Boeing Co.
• Original Assignee: The Boeing Co.
• Inventors: Owechko, Yuri, Nijamkin, Alejandro, Kim, Kyungnam, Flores, Arturo, Korchev, Dmitriy V.
• Granted Patent: US 9,746,988 B2
• Time in Patent Office: Days
• US Class Current: 715/763
• CPC Class Codes: G06F 3/04815 Interaction with a metaphor-based environment or interaction object displayed as three-dimensional, e.g. changing the user viewpoint with respect to the environment or object
• CPC Class Codes: G06T 2207/30196 Human being; Person
• CPC Class Codes: G06T 2207/30232 Surveillance
• CPC Class Codes: G08B 13/19641 Multiple cameras having overlapping views on a single scene
• CPC Class Codes: G08B 13/19671 Addition of non-video data, i.e. metadata, to video stream
• CPC Class Codes: G08B 13/19682 Graphic User Interface [GUI] presenting system data to the user, e.g. information on a screen helping a user interacting with an alarm system
• CPC Class Codes: H04N 7/181 for receiving images from a plurality of remote sources
• CPC Class Codes: H04N 7/185 from a mobile camera, e.g. for remote control

Abstract

A method and apparatus for processing video data streams for an area. Objects are identified in the area from images in the video data streams. The video data streams are generated by cameras. First locations are identified for the objects using the images. The first locations are defined using a coordinate system for the images. Graphical representations are formed for the objects using the images. The graphical representations are displayed for the objects in second locations in a model of the area on a display system with respect to features in the area that are represented in the model. The second locations are defined using a geographic coordinate system for the model. A first location in the first locations for an object in the objects corresponds to a second location in the second locations for a corresponding graphical representation in the graphical representations.

Claims

1. A method for processing video data streams for an area, the method comprising:

identifying a number of objects in the area from images in the video data streams for the area, wherein the video data streams are generated by cameras;

identifying a first number of locations for the number of objects using the images in the video data streams, wherein the first number of locations is defined using a coordinate system for the images;

forming a number of graphical representations for the number of objects using the images in the video data streams generated by the cameras; and

displaying the number of graphical representations for the number of objects in a second number of locations in a model of the area on a display system with respect to a number of features in the area that is represented in the model, wherein the second number of locations is defined using a geographic coordinate system for the model and a first location in the first number of locations for an object in the number of objects corresponds to a second location in the second number of locations for a corresponding graphical representation in the number of graphical representations.

2. The method of claim 1 further comprising:

identifying the second number of locations in the model of the area that correspond to the first number of locations for the number of objects.

3. The method of claim 1 further comprising:

identifying a number of orientations for the number of objects using the images in the video data streams; and

wherein the step of displaying the number of graphical representations for the number of objects in the second number of locations in the model of the area on the display system with respect to the number of features in the area that is represented in the model comprises;

displaying the number of graphical representations for the number of objects in the second number of locations in the model of the area on the display system with the number of orientations with respect to the number of features in the area that is represented in the model.

4. The method of claim 3, wherein the step of displaying the number of graphical representations for the number of objects in the second number of locations in the model of the area on the display system with respect to the number of features in the area that is represented in the model further comprises:

displaying information about the number of objects in association with the number of graphical representations for the number of objects.

5. The method of claim 1, wherein the first number of locations is identified from a first portion of the images in the video data streams generated at a first time and further comprising:

updating the first number of locations for the number of objects using a second portion of the images in the video data streams that are generated at a second time that is later than the first time to form an updated first number of locations;

identifying an updated second number of locations in the model of the area using the updated first number of locations; and

displaying the number of graphical representations for the number of objects in the updated second number of locations in the model of the area on the display system with respect to the number of features in the area that is represented in the model, wherein the number of graphical representations for the number of objects move in the model in a manner that represents movement of the number of objects in the area.

6. The method of claim 1, wherein the model is selected from at least one of a two-dimensional model of the area, a three-dimensional model of the area, a map, and a satellite image.

7. The method of claim 1 further comprising:

receiving the video data streams for the area from a number of platforms, wherein a platform in the number of platforms is selected from one of a mobile platform, a stationary platform, a land-based structure, an aquatic-based structure, a space-based structure, a vehicle, an aircraft, an unmanned aerial vehicle, and an unmanned ground vehicle.

8. The method of claim 1 further comprising:

displaying a number of videos containing the object in the number of objects in response to a user input selecting a graphical representation in the number of graphical representations for the object in the model.

9. The method of claim 1, wherein the model is a common operating picture for the area.

10. The method of claim 1, wherein the object in the number of objects is selected from one of a vehicle, a person, a building, an aircraft, an automobile, a truck, a tank, a train, and a ship.

11. An apparatus comprising:

a display system; and

a computer system configured to identify a number of objects in an area from images in video data streams for the area in which the video data streams are generated by cameras;

identify a first number of locations for the number of objects, wherein the first number of locations is defined using a coordinate system for the images;

form a number of graphical representations for the number of objects using the images in the video data streams generated by the cameras; and

display the number of graphical representations for the number of objects in a second number of locations in a model of the area on a display system with respect to a number of features in the area that is represented in the model, wherein the second number of locations is defined using a geographic coordinate system for the model and a first location in the first number of locations for an object in the number of objects that corresponds to a second location in the second number of locations for a corresponding graphical representation in the number of graphical representations.

12. The apparatus of claim 11, wherein the computer system is further configured to identify the second number of locations in the model of the area that corresponds to the first number of locations for the number of objects.

13. The apparatus of claim 11, wherein the computer system is further configured to identify a number of orientations for the number of objects using the images in the video data streams;

and wherein in being configured to display the number of graphical representations for the number of objects in the second number of locations in the model of the area on the display system with respect to the number of features in the area that is represented in the model, the computer system is configured to display the number of graphical representations for the number of objects in the second number of locations in the model of the area on the display system with the number of orientations with respect to the number of features in the area that is represented in the model.

14. The apparatus of claim 13, wherein in being configured to display the number of graphical representations for the number of objects in the second number of locations in the model of the area on the display system with respect to the number of features in the area that is represented in the model, the computer system is configured to display information about the number of objects in association with the number of graphical representations for the number of objects.

15. The apparatus of claim 11, wherein the first number of locations is identified from a first portion of the images in the video data streams generated at a first time, and wherein the computer system is further configured to update the first number of locations for the number of objects using a second portion of the images in the video data streams that is generated at a second time that is later than the first time to form an updated first number of locations;

identify an updated second number of locations in the model of the area using the updated first number of locations; and

display the number of graphical representations for the number of objects in the updated second number of locations in the model of the area on the display system with respect to the number of features in the area that is represented in the model, wherein the number of graphical representations for the number of objects move in the model in a manner that represents movement of the number of objects in the area.

16. The apparatus of claim 11, wherein the model is selected from at least one of a two-dimensional model of the area, a three-dimensional model of the area, a map, and a satellite image.

17. The apparatus of claim 11, wherein the computer system is further configured to receive the video data streams for the area from a number of platforms, wherein a platform in the number of platforms is selected from one of a mobile platform, a stationary platform, a land-based structure, an aquatic-based structure, a space-based structure, a vehicle, an aircraft, an unmanned aerial vehicle, and an unmanned ground vehicle.

18. The apparatus of claim 11, wherein the computer system is further configured to display a number of videos containing the object in the number of objects in response to a user input selecting a graphical representation in the number of graphical representations for the object in the model.

19. A computer program product comprising:

a computer readable storage medium;

first program code for identifying a number of objects in an area from images in video data streams for the area, wherein the video data streams are generated by cameras;

second program code for identifying a first number of locations for the number of objects using the images in the video data streams, wherein the first number of locations is defined using a coordinate system for the images;

third program code for forming a number of graphical representations for the number of objects using the images in the video data streams generated by the cameras;

fourth program code for displaying the number of graphical representations for the number of objects in a second number of locations in a model of the area on a display system with respect to a number of features in the area that is represented in the model, wherein the second number of locations is defined using a geographic coordinate system for the model and a first location in the first number of locations for an object in the number of objects that corresponds to a second location in the second number of locations for a corresponding graphical representation in the number of graphical representations; and

wherein the first program code, the second program code, the third program code, and the fourth program code are stored on the computer readable storage medium.

20. The computer program product of claim 19 further comprising:

fifth program code for identifying a number of orientations for the number of objects using the images in the video data streams, wherein the fifth program code is stored on the computer readable storage medium; and

wherein the fourth program code for displaying the number of graphical representations for the number of objects in the second number of locations in the model of the area on the display system with respect to the number of features in the area that is represented in the model comprises;

program code for displaying the number of graphical representations for the number of objects in the second number of locations in the model of the area on the display system with the number of orientations with respect to the number of features in the area that is represented in the model.