Multi-sensor surveillance system with a common operating picture

• US 9,746,988 B2
• Filed: 05/23/2011
• Issued: 08/29/2017
• Est. Priority Date: 05/23/2011
• Status: Active Grant

• Current Assignee: The Boeing Co.
• Original Assignee: The Boeing Co.
• Inventors: Kim, Kyungnam, Owechko, Yuri, Flores, Arturo, Nijamkin, Alejandro, Korchev, Dmitriy V.
• Primary Examiner(s): WIENER, ERIC A
• Application Number:
  US13/113,295
• Publication Number:
  US 20120304085A1
• Time in Patent Office: 2,290 Days
• CPC Class Codes: G06F 3/04815 Interaction with a metaphor-based environment or interaction object displayed as three-dimensional, e.g. changing the user viewpoint with respect to the environment or object
• CPC Class Codes: G06T 2207/30196 Human being; Person
• CPC Class Codes: G06T 2207/30232 Surveillance
• CPC Class Codes: G08B 13/19641 Multiple cameras having overlapping views on a single scene
• CPC Class Codes: G08B 13/19671 Addition of non-video data, i.e. metadata, to video stream
• CPC Class Codes: G08B 13/19682 Graphic User Interface [GUI] presenting system data to the user, e.g. information on a screen helping a user interacting with an alarm system
• CPC Class Codes: H04N 7/181 for receiving images from a plurality of remote sources
• CPC Class Codes: H04N 7/185 from a mobile camera, e.g. for remote control

Abstract

A method and apparatus for processing video data streams for an area. Objects are identified in the area from images in the video data streams. The video data streams are generated by cameras. First locations are identified for the objects using the images. The first locations are defined using a coordinate system for the images. Graphical representations are formed for the objects using the images. The graphical representations are displayed for the objects in second locations in a model of the area on a display system with respect to features in the area that are represented in the model. The second locations are defined using a geographic coordinate system for the model. A first location in the first locations for an object in the objects corresponds to a second location in the second locations for a corresponding graphical representation in the graphical representations.

Claims

1. A method for processing video data streams for an area, the method comprising:

identifying a number of objects in the area from images in the video data streams for the area, wherein the number of objects includes an unmanned aerial vehicle and wherein the video data streams are generated by cameras;

identifying a first number of locations for the number of objects using the images in the video data streams, wherein the first number of locations is defined using a coordinate system for the images;

forming a number of graphical representations for the number of objects using the images in the video data streams generated by the cameras, wherein the number of graphical representations includes an aircraft icon corresponding to the unmanned aerial vehicle;

displaying the number of graphical representations for the number of objects in a second number of locations in a model of the area on a display system with respect to a number of features in the area that is represented in the model, wherein the model comprises a satellite image of the area, the aircraft icon is displayed having an orientation in the model that corresponds to an orientation of the unmanned aerial vehicle over the area, the second number of locations is defined using a geographic coordinate system for the model, and a first location in the first number of locations for an object in the number of objects corresponds to a second location in the second number of locations for a corresponding graphical representation in the number of graphical representations;

displaying a second area in the model, the second area representing a field of view for the unmanned aerial vehicle, wherein a selection of a graphical representation of the graphical representations causes an image to be displayed in association with the graphical representation, the image is a portion of an image received in a video data stream of the video data streams in which an object corresponding to the graphical representation was identified, the number of graphical representations further includes a person icon present in the second area, and the person icon represents a person identified from the video data stream received from the unmanned aerial vehicle; and

displaying a path in the model in association with the person icon, wherein the path is one of the number of graphical representations and is an expected path for the person icon, the path has a direction that represents a direction of movement for the person represented by the person icon, the selection of the graphical representation of the person icon causes a window to be displayed in association with the person icon, the image is displayed in the window and comprises a video that is a presentation of the video data stream received from the unmanned aerial vehicle in which the person was identified, and a graphical indicator is positioned over the video in the window to provide a location in the video for the person corresponding to person icon.

2. The method of claim 1 further comprising:

identifying the second number of locations in the model of the area that correspond to the first number of locations for the number of objects.

3. The method of claim 1 further comprising:

identifying a number of orientations for the number of objects using the images in the video data streams; and

wherein the step of displaying the number of graphical representations for the number of objects in the second number of locations in the model of the area on the display system with respect to the number of features in the area that is represented in the model comprises;

displaying the number of graphical representations for the number of objects in the second number of locations in the model of the area on the display system with the number of orientations with respect to the number of features in the area that is represented in the model.

4. The method of claim 3, wherein the step of displaying the number of graphical representations for the number of objects in the second number of locations in the model of the area on the display system with respect to the number of features in the area that is represented in the model further comprises:

displaying information about the number of objects in association with the number of graphical representations for the number of objects.

5. The method of claim 1, wherein the first number of locations is identified from a first portion of the images in the video data streams generated at a first time and further comprising:

updating the first number of locations for the number of objects using a second portion of the images in the video data streams that are generated at a second time that is later than the first time to form an updated first number of locations;

identifying an updated second number of locations in the model of the area using the updated first number of locations; and

displaying the number of graphical representations for the number of objects in the updated second number of locations in the model of the area on the display system with respect to the number of features in the area that is represented in the model, wherein the number of graphical representations for the number of objects move in the model in a manner that represents movement of the number of objects in the area.

6. The method of claim 1, wherein the model is selected from at least one of a two-dimensional model of the area, a three-dimensional model of the area, a map, and a satellite image.

7. The method of claim 1 further comprising:

receiving the video data streams for the area from a number of platforms, wherein a platform in the number of platforms is selected from one of a mobile platform, a stationary platform, a land-based structure, an aquatic-based structure, a space-based structure, a vehicle, an aircraft, an unmanned aerial vehicle, and an unmanned ground vehicle.

8. The method of claim 1 further comprising:

displaying a number of videos containing the object in the number of objects in response to a user input selecting a graphical representation in the number of graphical representations for the object in the model.

9. The method of claim 1, wherein the model is a common operating picture for the area.

10. The method of claim 1, wherein the object in the number of objects is selected from one of a vehicle, a person, a building, an aircraft, an automobile, a truck, a tank, a train, and a ship.

11. An apparatus comprising:

a display system; and

a computer system configured to;

identify a number of objects in an area from images in video data streams for the area in which the video data streams are generated by cameras, wherein the number of objects includes an unmanned aerial vehicle;

identify a first number of locations for the number of objects, wherein the first number of locations is defined using a coordinate system for the images;

form a number of graphical representations for the number of objects using the images in the video data streams generated by the cameras, wherein the number of graphical representations includes an aircraft icon corresponding to the unmanned aerial vehicle;

display the number of graphical representations for the number of objects in a second number of locations in a model of the area on a display system with respect to a number of features in the area that is represented in the model, wherein the model comprises a satellite image of the area and wherein the aircraft icon is displayed having an orientation in the model that corresponds to an orientation of the unmanned aerial vehicle over the area, wherein the second number of locations is defined using a geographic coordinate system for the model and a first location in the first number of locations for an object in the number of objects that corresponds to a second location in the second number of locations for a corresponding graphical representation in the number of graphical representations;

display a second area in the model, the second area representing a field of view for the unmanned aerial vehicle, wherein a selection of a graphical representation of the graphical representations causes an image to be displayed in association with the graphical representation, the image is a portion of an image received in a video data stream of the video data streams in which an object corresponding to the graphical representation was identified, the number of graphical representations further includes a person icon present in the second area, and the person icon represents a person identified from the video data stream received from the unmanned aerial vehicle; and

display a path in the model in association with the person icon, wherein the path is one of the number of graphical representations and is an expected path for the person icon, the path has a direction that represents a direction of movement for the person represented by the person icon, the selection of the graphical representation of the person icon causes a window to be displayed in association with the person icon, the image is displayed in the window and comprises a video that is a presentation of the video data stream received from the unmanned aerial vehicle in which the person was identified, and a graphical indicator is positioned over the video in the window to provide a location in the video for the person corresponding to person icon.

12. The apparatus of claim 11, wherein the computer system is further configured to identify the second number of locations in the model of the area that corresponds to the first number of locations for the number of objects.

13. The apparatus of claim 11, wherein the computer system is further configured to identify a number of orientations for the number of objects using the images in the video data streams;

and wherein in being configured to display the number of graphical representations for the number of objects in the second number of locations in the model of the area on the display system with respect to the number of features in the area that is represented in the model, the computer system is configured to display the number of graphical representations for the number of objects in the second number of locations in the model of the area on the display system with the number of orientations with respect to the number of features in the area that is represented in the model.

14. The apparatus of claim 13, wherein in being configured to display the number of graphical representations for the number of objects in the second number of locations in the model of the area on the display system with respect to the number of features in the area that is represented in the model, the computer system is configured to display information about the number of objects in association with the number of graphical representations for the number of objects.

15. The apparatus of claim 11, wherein the first number of locations is identified from a first portion of the images in the video data streams generated at a first time, and wherein the computer system is further configured to update the first number of locations for the number of objects using a second portion of the images in the video data streams that is generated at a second time that is later than the first time to form an updated first number of locations;

identify an updated second number of locations in the model of the area using the updated first number of locations; and

display the number of graphical representations for the number of objects in the updated second number of locations in the model of the area on the display system with respect to the number of features in the area that is represented in the model, wherein the number of graphical representations for the number of objects move in the model in a manner that represents movement of the number of objects in the area.

16. The apparatus of claim 11, wherein the model is selected from at least one of a two-dimensional model of the area, a three-dimensional model of the area, a map, and a satellite image.

17. The apparatus of claim 11, wherein the computer system is further configured to receive the video data streams for the area from a number of platforms, wherein a platform in the number of platforms is selected from one of a mobile platform, a stationary platform, a land-based structure, an aquatic-based structure, a space-based structure, a vehicle, an aircraft, an unmanned aerial vehicle, and an unmanned ground vehicle.

18. The apparatus of claim 11, wherein the computer system is further configured to display a number of videos containing the object in the number of objects in response to a user input selecting a graphical representation in the number of graphical representations for the object in the model.

19. A computer program product comprising:

a non-transitory computer readable storage medium;

first program code for identifying a number of objects in an area from images in video data streams for the area, wherein the number of objects includes an unmanned aerial vehicle, wherein the video data streams are generated by cameras;

second program code for identifying a first number of locations for the number of objects using the images in the video data streams, wherein the first number of locations is defined using a coordinate system for the images;

third program code for forming a number of graphical representations for the number of objects using the images in the video data streams generated by the cameras, wherein the number of graphical representations includes an aircraft icon corresponding to the unmanned aerial vehicle;

fourth program code for displaying the number of graphical representations for the number of objects in a second number of locations in a model of the area on a display system with respect to a number of features in the area that is represented in the model, wherein the model comprises a satellite image of the area and wherein the aircraft icon is displayed having an orientation in the model that corresponds to an orientation of the unmanned aerial vehicle over the area, wherein the second number of locations is defined using a geographic coordinate system for the model and a first location in the first number of locations for an object in the number of objects that corresponds to a second location in the second number of locations for a corresponding graphical representation in the number of graphical representations;

fifth program code for displaying a second area in the model, the second area representing a field of view for the unmanned aerial vehicle, wherein a selection of a graphical representation of the graphical representations causes an image to be displayed in association with the graphical representation, the image is a portion of an image received in a video data stream of the video data streams in which an object corresponding to the graphical representation was identified, the number of graphical representations further includes a person icon present in the second area, and the person icon represents a person identified from the video data stream received from the unmanned aerial vehicle; and

sixth program code for displaying a path in the model in association with the person icon, wherein the path is one of the number of graphical representations and is an expected path for the person icon, the path has a direction that represents a direction of movement for the person represented by the person icon, the selection of the graphical representation of the person icon causes a window to be displayed in association with the person icon, the image is displayed in the window and comprises a video that is a presentation of the video data stream received from the unmanned aerial vehicle in which the person was identified, and a graphical indicator is positioned over the video in the window to provide a location in the video for the person corresponding to person icon;

wherein the first program code, the second program code, the third program code, the fourth program code, the fifth program code, and the sixth program code are stored on the non-transitory computer readable storage medium.

20. The computer program product of claim 19 further comprising:

seventh program code for identifying a number of orientations for the number of objects using the images in the video data streams, and

eighth program code for displaying the number of graphical representations for the number of objects in the second number of locations in the model of the area on the display system with the number of orientations with respect to the number of features in the area that is represented in the model;

wherein the seventh program code and the eighth program code are stored on the non-transitory computer readable storage medium.