Dynamically adjustable situational awareness interface for control of unmanned vehicles

• US 10,142,609 B2
• Filed: 04/29/2015
• Issued: 11/27/2018
• Est. Priority Date: 04/29/2015
• Status: Active Grant

• Current Assignee: Northrop Grumman Systems Corporation (Northrop Grumman Corporation)
• Original Assignee: Northrop Grumman Systems Corporation (Northrop Grumman Corporation)
• Inventors: Wei, Jerome H.
• Primary Examiner(s): Jiang, Zaihan
• Application Number:
  US14/699,733
• Publication Number:
  US 20170041587A1
• Time in Patent Office: 1,308 Days
• Field of Search: 348 43
• CPC Class Codes: B64C 39/024 of the remote controlled vehicle type, i.e. RPV
• CPC Class Codes: B64U 2101/30 for imaging, photography or videography
• CPC Class Codes: B64U 2201/00 UAVs characterised by their flight controls
• CPC Class Codes: B64U 2201/20 Remote controls
• CPC Class Codes: G05D 1/0011 associated with a remote control arrangement
• CPC Class Codes: H04N 13/139 Format conversion, e.g. of frame-rate or size
• CPC Class Codes: H04N 13/161 Encoding, multiplexing or demultiplexing different image signal components for multi-view video sequence encoding H04N19/597
• CPC Class Codes: H04N 13/167 Synchronising or controlling image signals
• CPC Class Codes: H04N 13/25 using two or more image sensors with different characteristics other than in their location or field of view, e.g. having different resolutions or colour pickup characteristics; using image signals from one sensor to control the characteristics of another sensor
• CPC Class Codes: H04N 13/254 in combination with electromagnetic radiation sources for illuminating objects
• CPC Class Codes: H04N 7/181 for receiving images from a plurality of remote sources
• CPC Class Codes: H04N 7/185 from a mobile camera, e.g. for remote control

Abstract

An apparatus includes an image collection module that monitors at least one parameter to dynamically regulate an amount of data and resolution to be allocated to at least one object in a scene collected from an image data set. A situational awareness interface (SAI) renders a 3-D video of the scene to an operator based on the amount of data and resolution allocated from the image data set by the image collection module and receives operator commands for an unmanned vehicle (UV) that interacts with the scene.

Claims

1. An apparatus, comprising:

an image collection module that monitors at least one parameter to dynamically regulate an amount of data and resolution to be allocated to an area of a scene collected from an image data set, the image collection module including an object identifier having a classifier to determine object types detected in the area of the scene based on probabilities associated with a frequency band emitted from the object;

a situational awareness interface (SAI) to render a 3-D video of the scene to an operator based on the amount of data and resolution allocated from the image data set by the image collection module and to receive operator commands for an unmanned vehicle (UV) that interacts with the scene, the SAI receiving feedback from the operator to allocate resolution bandwidth to an object within the area of the scene; and

a bandwidth detector that, based on the feedback from the operator, renders the object in the scene at a first resolution and other objects in the scene at a second resolution where the first resolution is higher than the second resolution.

2. The apparatus of claim 1, wherein the at least one parameter indicates an amount of available bandwidth parameter in which to render the image data set, a quality of service parameter from a network service provider, a data per second parameter indicating current network performance, or a resolution parameter to indicate an amount of detail to be rendered for the 3-D video scene.

3. The apparatus of claim 2, wherein the SAI provides the feedback from the operator to the image collection module to allocate resolution bandwidth to the object within the area of the scene, and wherein the feedback includes a voice command, a gaze tracking device input, or a cross hair adjustment via a joystick input.

4. The apparatus of claim 1, wherein image collection module processes data from at least two data sets that are generated from at least two sensors that includes an electro-optical (EO) sensor data set, a Laser Illuminated Detection and Ranging (LIDAR) image sensor data set, an acoustic sensor, an infrared sensor, an ultraviolet sensor, and a visible light sensor.

5. The apparatus of claim 4, wherein the image collection module includes an object coordinate mapper to map situational data received from the sensors to video coordinates of the 3-D video scene.

6. The apparatus of claim 1, wherein the classifier determines the object types based on probabilities associated with a shape of the object.

7. The apparatus of claim 1, wherein the SAI includes a virtual reality headset or multiple monitors to render the 3-D scene to the operator.

8. The apparatus of claim 7, further comprising a controller to receive operator commands from the SAI and to send control commands to the UV based on the operator commands.

9. The apparatus of claim 7, wherein object classifications, procedural data, or operational data is overlaid on to the 3-D scene to facilitate situational awareness of the operator.

10. A system, comprising:

a first sensor that generates an electro-optical (EO) image data set characterizing a scene;

a second sensor that generates a Laser Illuminated Detection and Ranging (LIDAR) image data set characterizing the scene;

an image collection module that dynamically regulates an amount of data and resolution to be allocated to at least one object within an area of a scene from the EO image data set and the LIDAR image data set based on at least one parameter to generate a fused image data set to provide a 3-D video of the scene, the image collection module including an object identifier having a classifier to determine object types detected in the area of the scene based on probabilities associated with a frequency band emitted from the object;

a situational awareness interface (SAI) to render the 3-D video of the scene from the fused image data set to an operator and to receive operator commands for an unmanned vehicle (UV) that interacts with the scene, the SAI receiving feedback from the operator to allocate resolution bandwidth to an object within the area of the scene; and

a bandwidth detector that, based on the feedback from the operator, renders the object in the scene at a first resolution and other objects in the scene at a second resolution where the first resolution is higher than the second resolution.

11. The system of claim 10, wherein the at least one parameter indicates an amount of available bandwidth parameter in which to render the image data set, a quality of service parameter from a network service provider, a data per second parameter indicating current network performance, or a resolution parameter to indicate an amount of detail to be rendered for the 3-D video scene.

12. The system of claim 10, wherein the SAI provides the feedback from the operator to the image collection module to allocate resolution bandwidth to the object within the area of the scene, and wherein the feedback includes a voice command, a gaze tracking device input, or a cross hair adjustment via a joystick input.

13. The system of claim 10, wherein image collection module fuses data from other data sets that includes an acoustic sensor, an infrared sensor, an ultraviolet sensor, and a visible light sensor.

14. The system of claim 13, wherein the image collection module includes an object coordinate mapper to map situational data received from the sensors to video coordinates of the 3-D video scene.

16. The system of claim 10, further comprising a controller to receive operator commands from the SAI and to send control commands to the UV based on the operator commands.

15. The system of 10, wherein the classifier determines the object types based on probabilities associated with a shape of the object.

17. A method, comprising:

receiving image data sets, via a controller, from at least two sensors,

fusing the image data sets, via the controller, to generate a 3-D scene for an operator of an unmanned vehicle (UV) based on the image data sets;

determining, via the controller, an available bandwidth to render the scene at an interface for the operator;

adjusting, via the controller, the resolution of an area in the scene based on the available bandwidth;

receiving feedback from the operator to allocate resolution bandwidth to an object within the area of the scene;

rendering, based on the feedback from the operator, the object in the scene at a resolution higher than a resolution of other objects in the scene; and

classifying objects in the scene to determine object types based on probabilities associated with a frequency band emitted from the object.

18. The method of claim 17, further comprising classifying objects in the scene to determine the object types based on probabilities associated with a shape of the object.