DYNAMICALLY ADJUSTABLE SITUATIONAL AWARENESS INTERFACE FOR CONTROL OF UNMANNED VEHICLES

• US 20170041587A1
• Filed: 04/29/2015
• Published: 02/09/2017
• Est. Priority Date: 04/29/2015
• Status: Active Grant

• Current Assignee: Northrop Grumman Systems Corporation (Northrop Grumman Corporation)
• Original Assignee: Northrop Grumman Systems Corporation (Northrop Grumman Corporation)
• Inventors: WEI, JEROME H.
• Granted Patent: US 10,142,609 B2
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: B64C 39/024 of the remote controlled vehicle type, i.e. RPV
• CPC Class Codes: B64U 2101/30 for imaging, photography or videography
• CPC Class Codes: B64U 2201/00 UAVs characterised by their flight controls
• CPC Class Codes: B64U 2201/20 Remote controls
• CPC Class Codes: G05D 1/0011 associated with a remote control arrangement
• CPC Class Codes: H04N 13/139 Format conversion, e.g. of frame-rate or size
• CPC Class Codes: H04N 13/161 Encoding, multiplexing or demultiplexing different image signal components for multi-view video sequence encoding H04N19/597
• CPC Class Codes: H04N 13/167 Synchronising or controlling image signals
• CPC Class Codes: H04N 13/25 using two or more image sensors with different characteristics other than in their location or field of view, e.g. having different resolutions or colour pickup characteristics; using image signals from one sensor to control the characteristics of another sensor
• CPC Class Codes: H04N 13/254 in combination with electromagnetic radiation sources for illuminating objects
• CPC Class Codes: H04N 7/181 for receiving images from a plurality of remote sources
• CPC Class Codes: H04N 7/185 from a mobile camera, e.g. for remote control

Abstract

An apparatus includes an image collection module that monitors at least one parameter to dynamically regulate an amount of data and resolution to be allocated to at least one object in a scene collected from an image data set. A situational awareness interface (SAI) renders a 3-D video of the scene to an operator based on the amount of data and resolution allocated from the image data set by the image collection module and receives operator commands for an unmanned vehicle (UV) that interacts with the scene.

Claims

1. An apparatus, comprising:

an image collection module that monitors at least one parameter to dynamically regulate an amount of data and resolution to be allocated to an area of a scene collected from an image data set; and

a situational awareness interface (SAI) to render a 3-D video of the scene to an operator based on the amount of data and resolution allocated from the image data set by the image collection module and to receive operator commands for an unmanned vehicle (UV) that interacts with the scene.

2. The apparatus of claim 1, wherein the at least one parameter indicates an amount of available bandwidth parameter in which to render the image data set, a quality of service parameter from a network service provider, a data per second parameter indicating current network performance, or a resolution parameter to indicate an amount of detail to be rendered for the 3-D video scene.

3. The apparatus of claim 2, wherein the SAI provides feedback from the operator to the image collection module to allocate resolution bandwidth to a particular object within the area of the scene, wherein the feedback includes a voice command, a gaze tracking device input, or a cross hair adjustment via a joystick input.

4. The apparatus of claim 1, wherein image collection module processes data from at least two data sets that are generated from at least two sensors that includes an electro-optical (EO) sensor data set, a Laser Illuminated Detection and Ranging (LIDAR) image sensor data set, an acoustic sensor, an infrared sensor, an ultraviolet sensor, and a visible light sensor.

5. The apparatus of claim 4, wherein the image collection module includes an object coordinate mapper to map situational data received from the sensors to video coordinates of the 3-D video scene.

6. The apparatus of claim 5, wherein the image collection module includes an object identifier to determine object types detected in the area of the scene.

7. The apparatus of claim 6, wherein the object identifier includes a classifier to determine the object types based on probabilities associated with a shape or frequency band emitted from the object.

8. The apparatus of claim 1, wherein the SAI includes a virtual reality headset or multiple monitors to render the 3-D scene to the operator.

9. The apparatus of claim 8, further comprising a controller to receive operator commands from the SAI and to send control commands to the UV based on the operator commands.

10. The apparatus of claim 8, wherein object classifications, procedural data, or operational data is overlaid on to the 3-D scene to facilitate situational awareness of the operator.

11. A system, comprising:

a first sensor configured to generate an electro-optical (EO) image data set characterizing a scene;

a second sensor configured to generate a Laser Illuminated Detection and Ranging (LIDAR) image data set characterizing the scene;

an image collection module configured to dynamically regulate an amount of data and resolution to be allocated to at least one object within an area of a scene from the EO image data set and the LIDAR image data set based on at least one parameter to generate a fused image data set to provide a 3-D video of the scene; and

a situational awareness interface to render the 3-D video of the scene from the fused image data set to an operator and to receive operator commands for an unmanned vehicle (UV) that interacts with the scene.

12. The system of claim 11, wherein the at least one parameter indicates an amount of available bandwidth parameter in which to render the image data set, a quality of service parameter from a network service provider, a data per second parameter indicating current network performance, or a resolution parameter to indicate an amount of detail to be rendered for the 3-D video scene.

13. The system of claim 11, wherein the SAI provides feedback from the operator to the image collection module to allocate resolution bandwidth to a particular object within the area of the scene, wherein the feedback includes a voice command, a gaze tracking device input, or a cross hair adjustment via a joystick input.

14. The system of claim 11, wherein image collection module fuses data from other data sets that includes an acoustic sensor, an infrared sensor, an ultraviolet sensor, and a visible light sensor.

15. The system of claim 14, wherein the image collection module includes an object coordinate mapper to map situational data received from the sensors to video coordinates of the 3-D video scene.

16. The system of claim 15, wherein the image collections module includes an object identifier to determine object types detected in the area of the scene.

17. The system of claim 16, wherein the object identifier includes a classifier to determine the object types based on probabilities associated with a shape or frequency band emitted from the object.

18. The system of claim 11, further comprising a controller to receive operator commands from the SAI and to send control commands to the UV based on the operator commands.

19. A method, comprising:

receiving image data sets, via a controller, from at least two sensors, fusing the image data sets, via the controller, to generate a 3-D scene for an operator of an unmanned vehicle (UV) based on the image data sets;

determining, via the controller, an available bandwidth to render the scene at an interface for the operator; and

adjusting, via the controller, the resolution of an area in the scene based on the available bandwidth.

20. The method of claim 19, further comprising classifying objects in the scene to determine object types based on probabilities associated with a shape or frequency band emitted from the object.