Remote information collection, situational awareness, and adaptive response system for improving advance threat awareness and hazardous risk avoidance

• US 10,474,144 B2
• Filed: 03/10/2017
• Issued: 11/12/2019
• Est. Priority Date: 08/01/2016
• Status: Active Grant

• Current Assignee: the united states of america as represented by the secretary of the navy
• Original Assignee: the united states of america as represented by the secretary of the navy
• Inventors: Wieneke, Garry
• Primary Examiner(s): Kong, Sze-Hon
• Application Number:
  US15/455,802
• Publication Number:
  US 20180032071A1
• Time in Patent Office: 977 Days
• CPC Class Codes: G05D 1/0027 involving a plurality of vehicles, e.g. fleet or convoy travelling fleet control of land vehicles from a control room G05D1/0297; traffic control systems for road vehicles G08G1/00; for marine craft G08G3/00; for aircraft G08G5/00
• CPC Class Codes: G05D 1/0038 by providing the operator with simple or augmented images from one or more cameras located onboard the vehicle, e.g. tele-operation images analyzed by a computer and used for automatic navigation G05D1/0246
• CPC Class Codes: G05D 1/0044 by providing the operator with a computer generated representation of the environment of the vehicle, e.g. virtual reality, maps maps used for automatic navigation G05D1/0274; flight directors G01C23/005
• CPC Class Codes: G05D 1/0214 in accordance with safety or protection criteria, e.g. avoiding hazardous areas monitoring the location of vehicles within a certain area, e.g. forbidden or allowed areas, in traffic control systems for road vehicles G08G1/13

Abstract

An adaptive situational awareness, decision support, and automated response system operable to receive sensor data or situational awareness data for a region of interest along or in advance of a path of travel of a vehicle obtained from at least a mobile reconnaissance sensor platform and select/execute one or more response programming or plans to operate one or more vehicle equipment items, e.g., a remote interaction, control or jamming system, from a stored database, manual inputs, or a distributive network based on matching of plans/programs with at least one of the sensor outputs or situational awareness data is provided. Apparatus and methods are also provided that enable an exemplary system to obtain data associated with an area surrounding the vehicle and provide this data to other system, subsystems, and personnel.

Claims

1. A Remote Information Collection, Situational Awareness, and Adaptive Response System (RICSAARS) comprising:

a first mobile platform comprising;

a first communication system,

a first position determination and navigation system (PDNS) configured to determine the first mobile platform'"'"'s position and output position data, and

a reconfigurable control and situational awareness system (RCSAS) configured to generate RCSAS data comprising a first and second RCSAS data in response to an operator'"'"'s input, wherein the first RCSAS data comprises movement control instructions for a second mobile platform and the second RCSAS data comprises sensor control instructions for the second mobile platform; and

the second mobile platform comprising;

a second communication system,

a movement control system configured to maneuver the second mobile platform within a first predetermined distance from the first mobile platform in proximity of a first designated route of travel,

a second PDNS configured to determine the second mobile platform'"'"'s position and output position data, and

a sensor system configured to detect at least one of one or more detectable emissions, at least one optical pattern, or characteristics of the environment surrounding the second mobile platform and further configured to identify a first plurality of conditions of interest in an area within a second predetermined distance and proximity to the first designated route of travel to generate sensor output data;

wherein the first and second communication systems are configured to transfer the sensor output data from the sensor system to the RCSAS, selectively transfer the first plurality of RCSAS data from the RCSAS to the movement system, and selectively transfer the second plurality of RCSAS data from the RCSAS to the sensor system;

wherein the movement control system is further configured to read the first plurality of RCSAS data to reconfigure the movement control system to maneuver the second mobile platform within a third predetermined distance from the first mobile platform in proximity of a second designated route of travel;

wherein the sensor system is further configured to read the second plurality of RCSAS data to reconfigure the sensor system to identify a second plurality of conditions of interest within a fourth predetermined distance from the first mobile platform in proximity of the second designated route of travel;

wherein the first communication system is electronically coupled to the RCSAS and the first PDNS, wherein the second communication system is electronically coupled to the sensor system, the movement control system, and the second PDNS.

2. The Remote Information Collection, Situational Awareness, and Adaptive Response System (RICSAARS) of claim 1, wherein the first mobile platform further comprises:

a reconfigurable control system comprising a transmitter configured to interact with, override or jam, or interfere with a first and second plurality of external receivers and control systems (ERCS) within a predetermined radius of the first mobile platform, wherein each of the first and second plurality of ERCS is associated with one or more detectable emissions, at least one optical pattern, or characteristics including conditions of terrain or area surrounding at least one of said first or second plurality of ERCS, wherein the first plurality of ERCS also have one or more of a first plurality of predetermined or known operating configurations, wherein the reconfigurable control system further comprises a plurality of loadset instructions comprising a first plurality of loadset instructions operable to operate the reconfigurable control system to interact with, control, or interfere with said first plurality of ERCS;

wherein the reconfigurable control and situational awareness system (RCSAS) is further configured to generate RCSAS data comprising a third RCSAS data comprising a second plurality of loadset instructions, wherein said RCSAS selectively replaces or alters said first plurality of loadset instructions to create said second plurality of loadset instructions configured to alter operation of the reconfigurable control system to enable a first plurality of responses or a second plurality of responses respectively to the first and second plurality of ERCS, wherein said first plurality of responses respectively comprises interaction with, jamming, or interference with ones of said second plurality of ERCS having one or more of a second plurality of operating configurations that operate respective second plurality of ERCS differently than said first plurality of ERCS;

wherein the RCSAS is electronically coupled to the reconfigurable control system.

3. The Remote Information Collection, Situational Awareness, and Adaptive Response System (RICSAARS) of claim 1, wherein the sensor system comprises:

an electronic magnetic spectrum scanning device,

an optical video camera, and

a thermal sensing device.

4. The Remote Information Collection, Situational Awareness, and Adaptive Response System (RICSAARS) of claim 1, wherein the transmission of data between the first communication system and the second communication system uses a radio frequency (RF) system or a laser communication data transfer system that permits line of sight (LOS) interfaces.

5. The Remote Information Collection, Situational Awareness, and Adaptive Response System (RICSAARS) of claim 1, wherein the RCSAS comprises a graphical user interface (GUI) configured to display key information related to the first plurality of conditions of interest, wherein the operator uses the GUI to generate RCSAS data in the RCSAS.

6. The Remote Information Collection, Situational Awareness, and Adaptive Response System (RICSAARS) of claim 5, wherein the GUI is further configured to display an interactive geospatial map, wherein the interactive geospatial map comprises a pre-made map of an area within a fifth predetermined distance of the first designated route of travel, wherein the interactive geospatial map plots condition of interest indicators onto the map at locations representing where the first and second pluralities of conditions of interest were detected by the sensor system, wherein the GUI is further configured to allow the operator to manually add or remove condition of interest indicators to the interactive geospatial map.

7. The Remote Information Collection, Situational Awareness, and Adaptive Response System (RICSAARS) of claim 6, wherein the first mobile platform further comprises a virtual reality difference identification (VRDI) system comprising a virtual reality headset and a movement tracking system, wherein the virtual reality headset is configured to display a virtual field of view comprising a plurality of images and the condition of interest indicators, wherein the condition of interest indicators are superimposed over the plurality of images, wherein the movement tracking system is configured to detect when the operator moves his field of view and adjust the virtual field of view by a predetermined amount corresponding to the operator'"'"'s movement.

8. The Remote Information Collection, Situational Awareness, and Adaptive Response System (RICSAARS) of claim 6, wherein the GUI displays the first and second routes of travel.

9. A Remote Information Collection, Situational Awareness, and Adaptive Response System (RICSAARS) comprising:

a mobile platform comprising;

a reconfigurable control and situational awareness system (RCSAS) configured to receive a plurality of inputs from a plurality of sources, process said plurality of inputs, generate a plurality of outputs, and transmit said plurality of outputs to a plurality of systems or subsystems;

a lifting, propulsion, and directional control mechanism,

a sensor system is mounted to an unmanned aerial vehicle and comprising;

an electronic magnetic spectrum scanning device,

an optical video camera, and

a thermal sensing device;

a distribution network configured to transmit a plurality of sensor data and a plurality of images collected by said sensor system to said RCSAS,

a graphic user interface configured to receive said plurality of outputs from said RCSAS and display a plurality of visual images and a visual representation of said plurality of sensor data to provide an operator with information associated with an environment of a path of intended travel,

a plurality of systems or subsystems configured to receive said plurality of outputs from said RCSAS, wherein said plurality of systems or subsystems are configured to automatically modify or adjust said plurality of systems or subsystems in response to a conditions of interest, or require user inputs to modify or adjust said plurality of systems or subsystems in response to the conditions of interest, and

a reconfigurable control system comprising a transmitter configured to interact with, override or jam, or interfere with a first and second plurality of external receivers and control systems (ERCS) within a predetermined radius of the first mobile platform, wherein each of the first and second plurality of ERCS is associated with one or more detectable emissions, at least one optical pattern, or characteristics including conditions of terrain or area surrounding at least one of said first or second plurality of ERCS, wherein the first plurality of ERCS also have one or more of a first plurality of predetermined or known operating configurations, wherein the reconfigurable control system further comprises a plurality of loadset instructions comprising a first plurality of loadset instructions operable to operate the reconfigurable control system to interact with, control, or interfere with said first plurality of ERCS;

wherein the reconfigurable control and situational awareness system (RCSAS) is further configured to generate a second plurality of loadset instructions, wherein said RCSAS selectively replaces or alters said first plurality of loadset instructions to create said second plurality of loadset instructions configured to alter operation of the reconfigurable control system to enable a first plurality of responses or a second plurality of responses respectively to the first and second plurality of ERCS, wherein said first plurality of responses respectively comprises interaction with, jamming, or interference with ones of said second plurality of ERCS having one or more of a second plurality of operating configurations that operate respective second plurality of ERCS differently than said first plurality of ERCS.

10. A Remote Information Collection, Situational Awareness, and Adaptive Response System (RICSAARS) comprising:

a first and second mobile vehicle, wherein the first mobile vehicle comprises;

a propulsion system and directional control system configured to navigate and maneuver the first mobile vehicle along a route to be traveled by the second mobile vehicle and maintain a first predetermined distance ahead of the second mobile vehicle, wherein movement of the first mobile vehicle is controlled autonomously by a first processor or remotely by an operator in the second mobile vehicle,

a first position determination and navigation system (PDNS) configured to determine the first mobile vehicle position and output position data,

a first processor and a first storage medium configured to store and execute a plurality of machine readable instructions, wherein the first processor is further configured to analyze and distribute a plurality of sensor data,

a first communication system configured to transmit a first plurality of electronic signals to a second communication system and receive a second plurality of electronic signals from a second communication system, wherein the first plurality of electronic signals comprises the plurality of sensor data, and

a plurality of sensors configured to detect a plurality of conditions of interest along the route to be traveled by the second mobile vehicle and generate the plurality of sensor data, wherein the plurality of sensors are electronically coupled with the first processor, wherein the plurality of sensors comprises of an electronic magnetic spectrum scanning device configured to detect a plurality of electromagnetic frequencies, an optical video camera configured to capture a plurality of images, and a thermal sensing device configured to capture a plurality of thermal images;

wherein the second mobile vehicle comprises;

a second propulsion system and directional control system configured to enable movement of the second mobile vehicle along a route of travel,

a second PDNS configured to determine the first mobile vehicle position and output position data,

a second processor and a second storage medium configured to store and execute a plurality of machine readable instruction, wherein the second processor is coupled with the second communication system and further configured to analyze the plurality of sensor data to detect the plurality of conditions of interest,

a second communication system configured to transmit a second plurality of electronic signals to the first mobile vehicle, and receive the first plurality of electronic signals from the first mobile vehicle,

a third communication system configured to transmit and receive a third plurality of electronic signals with a cloud network, wherein the cloud network is configured to further process and analyze the plurality of sensor data,

a graphical user interface (GUI) is configured to display the plurality of images, the plurality of thermal images, the plurality of sensor data, and the plurality of conditions of interest with corresponding condition of interest indicators, wherein the graphical user interface is further configured to enable the operator to manually alter or modify an operation or function of a plurality of reconfigurable subsystems, the first mobile vehicle, or the plurality of sensors, wherein the GUI is further configured to display an interactive geospatial map,

a reconfigurable system coupled with the second processor and configured to permit the processor to autonomously execute a plurality of machine readable instructions to alter or modify an operation or function of the reconfigurable system when the plurality of condition(s) of interest are detected within the plurality of sensor data,

a virtual reality difference identification (VRDI) system comprising a virtual reality headset and a movement tracking system, wherein the virtual reality headset is configured to display a virtual field of view comprising the plurality of images, the plurality of thermal images, the plurality of sensor data, the plurality of conditions of interest such that condition of interest indicators are superimposed over the plurality of images and plurality of thermal images, wherein the movement tracking system is configured to detect when the operator moves his field of view and adjust the virtual field of view by a predetermined amount corresponding to the operator'"'"'s movement, and

a reconfigurable control system comprising a transmitter configured to interact with, override or jam, or interfere with a first and second plurality of external receivers and control systems (ERCS) within a predetermined radius of the first mobile vehicle, wherein each of the first and second plurality of ERCS is associated with one or more detectable emissions, at least one optical pattern, or characteristics including conditions of terrain or area surrounding at least one of said first or second plurality of ERCS, wherein the first plurality of ERCS also have one or more of a first plurality of predetermined or known operating configurations, wherein the reconfigurable control system further comprises a plurality of loadset instructions comprising a first plurality of loadset instructions operable to operate the reconfigurable control system to interact with, control, or interfere with said first plurality of ERCS;

wherein the reconfigurable control and situational awareness system (RCSAS) is further configured to generate a second plurality of loadset instructions, wherein said RCSAS selectively replaces or alters said first plurality of loadset instructions to create said second plurality of loadset instructions configured to alter operation of the reconfigurable control system to enable a first plurality of responses or a second plurality of responses respectively to the first and second plurality of ERCS, wherein said first plurality of responses respectively comprises interaction with, jamming, or interference with ones of said second plurality of ERCS having one or more of a second plurality of operating configurations that operate respective second plurality of ERCS differently than said first plurality of ERCS;

wherein the first and second plurality of signals are transmitted between first and second communication systems with a plurality of RF signals and optical communication lasers;

wherein the first processor is further configured to generate a plurality of avoidance machine readable instructions comprising instructions for maneuvering to avoid lights and sounds consistent with known hazards recorded in the first storage medium and cloud network, obstacles, terrain, and fire, wherein the first processor is further configured to execute the avoidance machine readable instructions through the propulsion system and directional control system;

wherein the interactive geospatial map comprises a pre-made map of an area within a second predetermined distance of the route of travel, wherein the interactive geospatial map plots the condition of interest indicators onto the map at locations representing where the conditions of interest were detected by the plurality of sensors, wherein the operator can manually add or remove condition of interest indicators to the interactive geospatial map;

wherein the GUI displays the first and second routes of travel.

11. A method of operating a Remote Information Collection, Situational Awareness, and Adaptive Response System (RICSAARS) comprising:

maneuvering and moving an unmanned mobile scout vehicle along a predetermined route of travel, and maintaining a predetermined distance ahead of a mobile control vehicle, wherein the predetermined distance includes a minimum distance required for the mobile control vehicle to defeat or avoid a sensor detectable pattern or a condition of interest detected by a plurality of sensors affixed to the unmanned mobile scout vehicle prior to the mobile control vehicle entering a predetermined effective range of the detected condition of interest;

operating the plurality of sensors coupled with a first control system onboard the unmanned mobile scout vehicle to collect a plurality of sensor output on or near the predetermined route of travel to be analyzed by the first control system;

analyzing and matching the plurality of sensor output with one or more stored condition of interest data profiles on a first storage medium coupled with a first processor;

transmitting a warning signal using a transmitter on the unmanned mobile scout vehicle to a receiver on the mobile control vehicle based upon matching of at least one of the plurality of sensor output with one or more the stored condition of interest profiles, wherein the warning signal includes a plurality first elements;

selectively, transmitting said warning signal or the plurality of sensor output using said transmitter on said unmanned mobile scout vehicle or a transmitter on said mobile control vehicle to a cloud network for further analysis by a remote source;

selecting a matching protective response action or activity from a library of a plurality of protection response actions or activities stored on a second storage medium mounted on said mobile control vehicle coupled with a second processor based upon matching one or more of said plurality of first elements with one or more of a plurality second elements corresponding to said library of protection response actions or activities;

transmitting said plurality of sensor output, said warning signal, and said matching protective response action or activity to a display showing a graphical user interface, wherein said graphical user interface displays a visual representation of said plurality or sensor output and matching protective response action or activity to an operator onboard said mobile control vehicle, and permit said operator to input a plurality of control actions;

transmitting said matching protective response action or activity data to a reconfigurable response system affixed to said mobile control vehicle, wherein said matching protective response action or activity data includes a plurality of machine readable reconfiguring instructions or signals configured to program and/or operate said reconfigurable response system to defeat or avoid a detected condition of interest; and

altering or modifying an operation or function of said reconfigurable response system to respond based on a plurality of machine readable reconfiguring instructions or signals that defeat or destroy said detected condition of interest to be encountered by said mobile control vehicle based upon said protective response action or activity.

12. A method of claim 11, wherein said sensor detectable pattern or condition of interest includes a road side bomb with a triggering mechanism, or a destructive traffic/road/electromagnetic environment autopilot disruptive conditions that could interfere with autopilot functions.

13. A method of claim 11, wherein said predetermined effective range includes a blast radius of an explosive device or an area where electromagnetic interference would occur.

14. A method of claim 11, wherein said plurality of sensor output includes an electromagnetic spectrum pattern, an image pattern, or a thermal pattern.

15. A method of claim 11, wherein a protective response action or activity includes an electromagnetic jammer profile suitable to jam a signal in a zone of interest, alter an autopilot function, disengage a vehicle'"'"'s autopilot system, or produce a hazard warning indication to alert the operator of a condition of interest.

16. A method of claim 11, wherein said first and second elements associated with said warning signal and said protective response action or activity, respectively, comprises a warning identifier code, electromagnetic patterns, optical image patterns, and thermal image patterns.

17. A method of claim 11, further comprising:

determining to avoid or continue on said predetermined route of travel based upon a match or no match of said plurality of first and second elements, wherein an altered route of travel will entirely avoid said predetermined effective range of said condition of interest when no matching protective response action or activity is selected.