Artificial intelligence augmented reality command, control and communications system

• US 9,983,581 B1
• Filed: 09/14/2017
• Issued: 05/29/2018
• Est. Priority Date: 09/14/2017
• Status: Active Grant

• Current Assignee: SparkCognition Inc., The Boeing Co.
• Original Assignee: SparkCognition Inc.
• Inventors: Husain, Syed Mohammad Amir, Allen, John Rutherford
• Primary Examiner(s): Figueroa, Jaime
• Application Number:
  US15/705,027
• Time in Patent Office: 257 Days
• Field of Search: 701 1, 701 27
• CPC Class Codes: B64C 39/024 of the remote controlled vehicle type, i.e. RPV
• CPC Class Codes: B64U 2101/15 for conventional or electronic warfare
• CPC Class Codes: B64U 2101/19 for use as targets or decoys
• CPC Class Codes: B64U 2201/10 autonomous, i.e. by navigating independently from ground or air stations, e.g. by using inertial navigation systems [INS]
• CPC Class Codes: B64U 2201/20 Remote controls
• CPC Class Codes: B64U 50/19 using electrically powered motors
• CPC Class Codes: B64U 50/30 Supply or distribution of electrical power
• CPC Class Codes: G05D 1/0027 involving a plurality of vehicles, e.g. fleet or convoy travelling fleet control of land vehicles from a control room G05D1/0297; traffic control systems for road vehicles G08G1/00; for marine craft G08G3/00; for aircraft G08G5/00
• CPC Class Codes: G05D 1/0038 by providing the operator with simple or augmented images from one or more cameras located onboard the vehicle, e.g. tele-operation images analyzed by a computer and used for automatic navigation G05D1/0246
• CPC Class Codes: G05D 1/0088 characterized by the autonomous decision making process, e.g. artificial intelligence, predefined behaviours using knowledge based models G06N5/00
• CPC Class Codes: H04N 7/18 Closed-circuit television [CCTV] systems, i.e. systems in which the video signal is not broadcast
• CPC Class Codes: H04N 7/181 for receiving images from a plurality of remote sources

Abstract

A method and system comprises a plurality of electronically controlled distributed devices and a supervisory node. The supervisory node comprises a communications interface, a processor, and a display. The supervisory node is configured to communicate with the plurality of electronically controlled distributed devices via the communications interface. The supervisory node is adapted to receive sensor information, to receive functionality information and device status information, to determine useful life prognostics from the functionality information, to obtain human defined policy and strategy directives, to assess the useful life prognostics and device status information based on the human defined policy and strategy directives to provide device assessments, to construct device commands for the plurality of electronically controlled distributed devices based on the device assessments using the processor, and to communicate the device commands to the plurality of electronically controlled distributed via the communications interface.

Claims

1. A system comprising:

a plurality of electronically controlled distributed devices; and

a supervisory node comprising a communications interface, a processor coupled to the communications interface, and a display coupled to the processor, the supervisory node configured to communicate with the plurality of electronically controlled distributed devices via the communications interface, the supervisory node adapted to receive sensor information from the plurality of electronically controlled distributed devices, to receive functionality information and device status information from the plurality of electronically controlled distributed devices, to determine useful life prognostics from the functionality information using the processor, to obtain human defined policy and strategy directives, to assess the useful life prognostics and device status information based on the human defined policy and strategy directives using the processor to provide device assessments, to construct device commands for the plurality of electronically controlled distributed devices based on the device assessments using the processor, and to communicate the device commands to the plurality of electronically controlled distributed devices via the communications interface.

2. The system of claim 1, wherein the supervisory node is further configured to receive from a first electronically controlled distributed device of the electronically controlled distributed devices a streaming camera feed and to display the streaming camera feed for viewing.

3. The system of claim 2, wherein the supervisory node is further configured to perform machine vision analysis of the streaming camera feed and to construct the device commands based on the streaming camera feed and the device assessments.

4. The system of claim 2, wherein the supervisory node is further configured to compare the streaming camera feed to stored camera images and to provide a battle damage assessment based on the comparison.

5. The system of claim 2, wherein supervisory node is further configured to display the streaming camera feed via a synthetic display including a virtual reality display, an augmented reality display, or both.

6. The system of claim 1, wherein the supervisory node is further configured to autonomously construct and communicate a first device command of the device commands and to receive explicit authorization prior to issuance of a second device command of the device commands.

7. The system of claim 1, wherein the supervisory node is further configured to perform outcome simulations based on historical data obtained from a data repository and to construct device commands for the plurality of electronically controlled distributed devices based on the device assessments and the outcome simulations.

8. A method comprising:

communicating, by a supervisory node, with a plurality of electronically controlled distributed devices;

receiving, at the supervisory node, sensor information from the plurality of electronically controlled distributed devices;

receiving, at the supervisory node, functionality information and device status information from the plurality of electronically controlled distributed devices;

determining, at the supervisory node, useful life prognostics from the functionality information;

obtaining, at the supervisory node, human defined policy and strategy directives;

assessing, at the supervisory node, the useful life prognostics and device status information based on the human defined policy and strategy directives to provide device assessments;

constructing, at the supervisory node, device commands for the plurality of electronically controlled distributed devices based on the device assessments; and

communicating, from the supervisory node, the device commands to the plurality of electronically controlled distributed devices.

9. The method of claim 8, further comprising:

receiving, at the supervisory node from a first electronically controlled distributed device of the electronically controlled distributed devices, a streaming camera feed and displaying the streaming camera feed for viewing.

10. The method of claim 9, further comprising:

performing, at the supervisory node, machine vision analysis of the streaming camera feed, wherein the device commands are constructed based on the streaming camera feed and the device assessments.

11. The method of claim 9, further comprising:

comparing, at the supervisory node, the streaming camera feed to stored camera images and providing a battle damage assessment based on the comparison.

12. The method of claim 9, further comprising:

displaying, at the supervisory node, the streaming camera feed via a synthetic display including a virtual reality display, an augmented reality display, or both.

13. The method of claim 8, further comprising:

autonomously constructing at and communicating from the supervisory node a first device command of the device commands; and

receiving, at the supervisory node, explicit authorization prior to issuance of a second device command of the device commands.

14. The method of claim 8, further comprising:

performing, at the supervisory node, outcome simulations based on historical data obtained from a data repository and constructing the device commands for the plurality of electronically controlled distributed devices based on the device assessments and the outcome simulations.

15. A system comprising:

a plurality of electronically controlled distributed devices; and

a supervisory node comprising a communications interface, a human command input interface, a processor coupled to the communications interface and to the human command input interface, and a display coupled to the processor, the supervisory node configured to communicate with the plurality of electronically controlled distributed devices via the communications interface, the supervisory node adapted to receive sensor information from the plurality of electronically controlled distributed devices, to receive functionality information and device status information from the plurality of electronically controlled distributed devices, to determine useful life prognostics from the functionality information using the processor, to obtain human defined policy and strategy directives via the human command input interface, to assess the useful life prognostics and device status information based on the human defined policy and strategy directives using the processor to provide device assessments, to construct device commands for the plurality of electronically controlled distributed devices based on the device assessments using the processor, and to communicate the device commands to the plurality of electronically controlled distributed devices via the communications interface, the processor configured to arbitrate between automated control instructions generated by the processor and human-generated control instructions received via the human command input interface and to construct the device commands based on the device assessments and arbitrated control instructions obtained by arbitrating.

16. The system of claim 15, wherein the supervisory node is further configured to receive from a first electronically controlled distributed device of the electronically controlled distributed devices a streaming camera feed and to display the streaming camera feed for viewing.

17. The system of claim 16, wherein the supervisory node is further configured to perform machine vision analysis of the streaming camera feed and to construct the device commands based on the streaming camera feed and the device assessments.

18. The system of claim 16, wherein supervisory node is further configured to display the streaming camera feed via a synthetic display including a virtual reality display, an augmented reality display, or both.

19. The system of claim 15, wherein the supervisory node is further configured to autonomously construct and communicate a first device command of the device commands and to receive explicit authorization prior to issuance of a second device command of the device commands.

20. The system of claim 15, wherein the supervisory node is further configured to perform outcome simulations based on historical data obtained from a data repository and to construct device commands for the plurality of electronically controlled distributed devices based on the device assessments and the outcome simulations.