4D interactive mission analytics for visualization of unmanned vehicle performance indicators

• US 10,261,652 B1
• Filed: 07/07/2016
• Issued: 04/16/2019
• Est. Priority Date: 07/07/2016
• Status: Active Grant

• Current Assignee: Northrop Grumman Systems Corporation (Northrop Grumman Corporation)
• Original Assignee: Northrop Grumman Systems Corporation (Northrop Grumman Corporation)
• Inventors: Patel, Miteshkumar K., Oddo, Louis, Parsons, William, Cao, Feng, Fung, Henry H., Parekh, Devang R.
• Primary Examiner(s): Nguyen, Tan Q
• Application Number:
  US15/204,690
• Time in Patent Office: 1,013 Days
• Field of Search: 701 291
• CPC Class Codes: G05D 1/0044 by providing the operator with a computer generated representation of the environment of the vehicle, e.g. virtual reality, maps maps used for automatic navigation G05D1/0274; flight directors G01C23/005
• CPC Class Codes: G06F 3/013 Eye tracking input arrangements G06F3/015 takes precedence
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0481 based on specific properties of the displayed interaction object or a metaphor-based environment, e.g. interaction with desktop elements like windows or icons, or assisted by a cursor's changing behaviour or appearance
• CPC Class Codes: G06F 3/04847 Interaction techniques to control parameter settings, e.g. interaction with sliders or dials
• CPC Class Codes: G06F 3/167 Audio in a user interface, e.g. using voice commands for navigating, audio feedback
• CPC Class Codes: G06T 11/206 Drawing of charts or graphs
• CPC Class Codes: G06T 2200/24 involving graphical user interfaces [GUIs]

Abstract

A system includes an analytics collector that receives world state data to provide status relating to a plurality of mission analytics of an unmanned vehicle or an unmanned vehicle mission planner. An asset filter filters the status from the analytics collector with respect to mission analytics of a subset of selected assets. An analytic aggregator collects the filtered status from the asset filter and generates a visual analytics file based on one or more selected analytics for the subset of selected assets. A rendering pipeline processes the visual analytics file from the analytic aggregator and generates a formatted output file describing a visualization of the plurality of mission analytics from the visual analytics file.

Claims

1. A system, comprising:

a memory to store computer-executable components; and

a processor to execute the computer-executable components from the memory, the computer-executable components comprising;

an analytics collector that receives world state data to provide status relating to a plurality of mission analytics of an unmanned vehicle or an unmanned vehicle mission planner, the mission analytics relate to the ability of the unmanned vehicle and its associated payload resources to execute a given mission plan;

an asset filter to filter the status from the analytics collector with respect to mission analytics of a subset of selected assets;

an asset selector to specify the subset of selected assets to the asset filter;

an analytic aggregator to collect the filtered status from the asset filter and to generate a visual analytics file based on one or more selected analytics for the subset of selected assets;

an analytic selector to specify the one or more selected analytics to the analytic aggregator; and

a rendering pipeline to process the visual analytics file from the analytic aggregator and to generate a formatted output file describing a visualization of the plurality of mission analytics from the visual analytics file, the formatted output file generated as a collection of display objects, wherein each of the display objects represent at least one mission analytic from the one or more selected analytics of the subset of selected assets and each of the specified display objects represent a current or future status of the mission analytics over a specified time window.

2. The system of claim 1, further comprising a graphical user interface (GUI) to display the display objects from the formatted output file, the GUI providing a visualization of the display objects that represent a current or future status of the selected mission analytics of for the subset of selected assets over a specified time window.

3. The system of claim 2, wherein the display objects represent at least one of a fuel status for an unmanned vehicle, a payload collection opportunity for an unmanned vehicle, or and a payload collection status for an unmanned vehicle in a present or future time window.

4. The system of claim 2, wherein the display objects represent at least one of time of arrival of next payload collections for an unmanned vehicle, a communications availability for the unmanned vehicle, and a weighted route score for an unmanned vehicle relating to alternative routes for an unmanned vehicle.

5. The system of claim 4, wherein the weighted route score is stored in a data structure comprising:

a route field to describe a loaded route for the unmanned vehicle;

a behavior field to describe three-dimensional spatial capability of the unmanned vehicle and its resource capability along the loaded route that includes at least one of a latitude, a longitude, and an altitude parameter for the loaded route of the unmanned vehicle; and

a when field that describes a starting time or a duration for the loaded route of the unmanned vehicle and a starting time or a duration for deployment of the resources that drive mission plan inputs for the unmanned vehicle mission planner.

6. The system of claim 5, further comprising a time specifier to specify time projections of the loaded route, specify viewing of the mission analytics over time, and specify deployment of resources over time, the time specifier comprises a time slider graphical input to specify a current time or the future time for the visualization output on the GUI, a voice input to specify the current time or the future time for the visualization output on the GUI, a hand or figure gesture to specify the current time or the future time for the visualization output on the GUI, or a retinal scan input to specify the current time or the future time for the visualization output on the GUI.

7. The system of claim 5, further comprising a state aggregator to process input data feeds to update the behavior field of the data structure in real time, the data feeds include at least one of a tasking feed, a mission plan feed, a weather feed, a threat feed, a sensor feed, a tracks feed, and a communications feed.

8. The system of claim 7, further comprising a prediction engine to project the data feeds from the state aggregator into a future visualization of the loaded routed, a future visualization of the resources that drive the mission plan inputs, and a future visualization of at least one alternative route for the unmanned vehicle.

9. The system of claim 4, wherein mission analytics for the alternative routes are projected over time.

10. The system of claim 2, wherein the display objects represent a status from at least one of a weather prediction model for an unmanned vehicle, an unmanned vehicle tasking model, an unmanned vehicle threat detection model, and an unmanned vehicle tracking model.

11. The system of claim 1, wherein the display objects are specified relative to a selected unmanned vehicle or specified relative to a plurality of unmanned vehicles in a swarm.

12. A non-transitory computer readable medium having computer executable instructions stored thereon, the computer executable instructions to:

provide status relating to a plurality of mission analytics of a plurality of unmanned vehicles and a plurality of unmanned vehicle mission planners, the mission analytics relate to the ability of the plurality of unmanned vehicles and their respective payload resources to execute a respective mission plan or respective alternative mission plan generated by the plurality of unmanned vehicle mission planners;

filter the status with respect to mission analytics of a subset of selected assets;

specify the subset of selected assets;

collect the filtered status from the asset filter and generate a visual analytics file based on one or more selected analytics for the subset of selected assets;

specify the one or more selected analytics; and

process the visual analytics file and to generate a formatted output file describing a visualization of the plurality of mission analytics from the visual analytics file, the formatted output file generated as a collection of display objects, wherein each of the display objects represent at least one mission analytic from the one or more selected analytics of the subset of selected assets and each of the specified display objects represent a current or future status of the mission analytics over a specified time window.

13. The computer readable medium of claim 12, further comprising instructions to display the display objects from the visual analytics file, the instructions provide a visualization of the display objects that represent a current or future status of the mission analytics over a specified time window.

14. The computer readable medium of claim 13, wherein the display objects represent at least one of a fuel status for an unmanned vehicle, a payload collection opportunity for an unmanned vehicle, or and a payload collection status for an unmanned vehicle in a present or future time window.

15. The computer readable medium of claim 13, wherein the display objects represent at least one of time of arrival of next payload collections for an unmanned vehicle, a communications availability for the unmanned vehicle, and a weighted route score for an unmanned vehicle relating to alternative routes for an unmanned vehicle.

16. The computer readable medium of claim 15, wherein the weighted route score is stored in a data structure comprising:

a route field to describe a loaded route for the unmanned vehicle;

a behavior field to describe three-dimensional spatial capability of the unmanned vehicle and its resource capability along the loaded route that includes at least one of a latitude, a longitude, and an altitude parameter for the loaded route of the unmanned vehicle; and

a when field that describes a starting time or a duration for the loaded route of the unmanned vehicle and a starting time or a duration for deployment of the resources that drive mission plan inputs for the unmanned vehicle mission planner.

17. The computer readable medium of claim 15, wherein mission analytics for the alternative routes are projected over time.

18. The computer readable medium of claim 13, wherein the display objects represent a status from at least one of a weather prediction model for an unmanned vehicle, an unmanned vehicle tasking model, an unmanned vehicle threat detection model, and an unmanned vehicle tracking model.

19. A computer implemented method, comprising:

providing status relating to a plurality of mission analytics of an unmanned vehicle or an unmanned vehicle mission planner, the mission analytics relate to the ability of the unmanned vehicle and its associated payload resources to execute a given mission plan;

generating an analytics file from the status describing the plurality of mission analytics;

generating a formatted output file from the analytics file describing a visualization of the plurality of mission analytics from the analytics file, the formatted output file generated as a collection of display objects, each of the display objects represent at least one mission analytic from the plurality of mission analytics;

specifying a subset of the display objects to be visualized in the formatted output file; and

generating a mission analytics visualization output from the subset of display objects specified via the formatted output file, each of the specified display objects represent a current or future status of the mission analytics over a specified time window.

20. The method of claim 19, projecting mission analytics for alternative routes for an unmanned vehicle or an unmanned vehicle swarm over the specified time window.