System and method for operator control of heterogeneous unmanned system teams

• US 10,168,674 B1
• Filed: 04/22/2014
• Issued: 01/01/2019
• Est. Priority Date: 04/22/2013
• Status: Active Grant

• Current Assignee: National Technology & Engineering Solutions Of Sandia LLC (Honeywell International Inc.)
• Original Assignee: National Technology & Engineering Solutions Of Sandia LLC (Honeywell International Inc.)
• Inventors: Buerger, Stephen, Love, Joshua Alan, Neely, Jason C., Amai, Wendy A., Little, Charles Q.
• Primary Examiner(s): Bendidi, Rachid
• Application Number:
  US14/258,986
• Time in Patent Office: 1,715 Days
• Field of Search: 700245-264, 901 47
• CPC Class Codes: G05B 15/02 electric
• CPC Class Codes: G05D 1/0088 characterized by the autonomous decision making process, e.g. artificial intelligence, predefined behaviours using knowledge based models G06N5/00
• CPC Class Codes: G06Q 10/00 Administration; Management
• CPC Class Codes: Y10S 901/47 Optical

Abstract

A control architecture and associated control methods are under development to allow a single user to control a team of multiple heterogeneous UMS as they conduct multi-faceted (i.e. multi-objective) missions in real time is disclosed. The control architecture is hierarchical, modular and layered and enables operator interaction at each layer, ensuring the human operator is in close control of the unmanned team at all times. The architecture and key data structures are introduced. Two approaches to distributed collaborative control of heterogeneous unmanned systems are described, including an extension of homogeneous swarm control and a novel application of distributed model predictive control. Initial results are presented, demonstrating heterogeneous UMS teams conducting collaborative missions.

Claims

1. A method, comprising:

assigning an objective to one or more unmanned vehicles by a user;

estimating a cost to perform the objective by one or more agents;

determining control instructions to be executed by one or more unmanned vehicles that minimizes a total cost of the objective;

transmitting the control instructions to be executed by the one or more unmanned vehicles to the one or more unmanned vehicles;

using a processor which functions as a high level optimizer to continue to estimate the cost and modify the assignment of the objective to minimize the cost during the performance of the objective;

executing the objective while minimizing the cost of the objective; and

transmitting sensor data from at least one of the one or more unmanned vehicles to the user;

wherein the cost is tunable during performing the objective by the user selecting between different cost metrics.

2. The method of claim 1, wherein executing the objective comprises executing individual or collaborative behavior of the one or more unmanned vehicles.

3. The method of claim 1, wherein assigning the objective comprises an objective location to be reached by at least one of the one or more unmanned vehicles.

4. The method of claim 1, wherein the assigning the objective comprises instructions by a user for at least one of the one or more unmanned vehicles to perform tasks at an objective location.

5. The method of claim 4, wherein the tasks are selected from a group comprising gathering sensor data, manipulating or grasping objects, switching to lower power states to conserve energy, positioning communications antennae to act as a relay, and delivering payloads.

6. The method of claim 1, wherein assigning the objective is executed by the processor which functions as the high level optimizer.

7. The method of claim 6, wherein the processor which functions as the high level optimizer is at least partially located at a user'"'"'s location.

8. The method of claim 6, wherein the processor which functions as the high level optimizer is at least partially located at a location of at least one of the one or more unmanned vehicles.

9. The method of claim 1, wherein determining control instructions is executed by a processor which functions as a mid-level controller.

10. The method of claim 9, wherein the processor which functions as the mid-level controller is at least partially located at a user'"'"'s location.

11. The method of claim 9, wherein the processor which functions as the mid-level controller is at least partially located at a location of at least one of the one or more unmanned vehicles.

12. The method of claim 1, wherein estimating the cost to perform the objective is executed by a processor which functions as a mid-level controller.

13. The method of claim 1, wherein the objective comprises movement commands selected from a group comprising waypoints, spatial curves to move along, spatial areas through which to move, and spatial volumes through which to move.

14. The method of claim 1, wherein the objective comprises multi-agent behaviors.

15. The method of claim 1, wherein at least one of the one or more unmanned vehicles is selected from a group consisting of unmanned ground vehicles, unmanned aerial vehicles, unmanned water vehicles and unmanned underwater vehicles.

16. The method of claim 1, wherein the one or more unmanned vehicles comprises two or more unmanned vehicles.

17. The method of claim 16, wherein determining control instructions to be executed by the two or more unmanned vehicles comprises determining which unmanned vehicle can perform the objective at a minimum cost.

18. The method of claim 1, further comprising:

modifying the objective by the user.

19. A non-transitory machine-readable medium in communication with at least one processor, the non-transitory machine-readable medium storing a system architecture having instructions, which, when executed by the at least one processor of a machine, causes one or more unmanned vehicles to perform one or more operations selected from a group consisting of moving to a location, delivering a package, and obtaining sensor data;

wherein the system architecture comprises at least one processor which functions as a high level optimizer that receives an objective from a user and at least one processor which functions as a mid-level behavior controller that receives an assignment list from the at least one processor which functions as the high level optimizer and provides an objective cost list to the at least one processor which functions as the high level optimizer thereby optimizing the objective performed by the one or more unmanned vehicles; and

wherein the at least one processor which functions as the high level optimizer continually updates the objective cost while the objective is being performed and updates the objective of one or more unmanned vehicles; and

wherein the objective cost is tunable during performing the objective by the user selecting between different cost metrics.

20. The non-transitory machine-readable medium of claim 19, wherein the at least one processor which functions as the high level optimizer is centralized at a user interface.

21. The non-transitory machine-readable medium of claim 19, wherein the at least one processor which functions as the high level optimizer is distributed with the one or more unmanned vehicles.

22. The non-transitory machine-readable medium of claim 19, wherein the at least one processor which functions as the mid-level behavior controller is distributed to the one or more unmanned vehicles.

23. The non-transitory machine-readable medium of claim 19, wherein the system architecture further comprising a low-level control architecture distributed to the one or more unmanned vehicles.

24. The non-transitory machine-readable medium of claim 19, wherein upon completion of the operation, the one or more unmanned vehicles performing the one or more operations reports results of the one or more operations to the user.

25. A system, comprising:

a user interface;

at least one processor of a machine;

at least one processor which functions as a high level optimizer module configured to receive an objective from the user interface;

at least one processor which functions as a mid-level behavior controller configured to determine a cost associated with the one or more unmanned vehicles performing the objective; and

one or more agents configured to receive the objective from the at least one processor which functions as the mid-level behavior controller;

wherein the system continuously updates the cost and modifies the objective of one or more unmanned vehicles; and

wherein the cost is tunable during performing the objective by the user selecting between different cost metrics.

26. The system of claim 25, wherein at least one unmanned vehicle of the one or more unmanned vehicles has a sensor package.

27. The system of claim 26, wherein the sensor package is selected from a group consisting of visual and/or infrared (IR) cameras, chemical and/or biological detectors, radiation detectors, three dimensional imaging systems, seismic sensors, acoustic sensors, radars, range finders, temperature or pressure sensors.

28. The system of claim 25, wherein at least one of the one or more unmanned vehicles is mobile.

29. The system of claim 28, wherein the at least one mobile unmanned vehicle is selected from a group consisting of unmanned ground vehicles, unmanned aerial vehicles, unmanned water vehicles and unmanned underwater vehicles.

30. The system of claim 25, wherein at least one of the one or more unmanned vehicles is configured to gather sensor data, manipulate or grasp objects, switch to lower power states to conserve energy, position communications antennae to act as a relay, and delivering payloads.

31. The system of claim 25, wherein the at least one processor which functions as the high-level optimizer module is at least partially co-located with the user interface.

32. The system of claim 25, wherein the at least one processor which functions as the high-level optimizer module is at least partially co-located with the one or more unmanned vehicles.

33. The system of claim 25, wherein the at least one processor which functions as the mid-level behavior controller module is at least partially co-located with the user interface.

34. The system of claim 25, wherein the at least one processor which functions as the mid-level behavior controller module is at least partially co-located with the one or more unmanned vehicles.