Virtual and augmented reality cockpit and operational control systems

• US 9,335,764 B2
• Filed: 05/26/2015
• Issued: 05/10/2016
• Est. Priority Date: 05/27/2014
• Status: Active Grant

• Current Assignee: Recreational Drone Event Systems, LLC
• Original Assignee: Recreational Drone Event Systems, LLC
• Inventors: Herz, Steven D., Kovalik, Alfred N.
• Primary Examiner(s): Shafi, Muhammad
• Application Number:
  US14/721,268
• Publication Number:
  US 20150346722A1
• Time in Patent Office: 350 Days
• Field of Search: 701/2
• US Class Current: 1/1
• CPC Class Codes: G01S 19/13 Receivers
• CPC Class Codes: G05D 1/0016 characterised by the operator's input device input arrangements for computing systems in general G06F3/00
• CPC Class Codes: G05D 1/0027 involving a plurality of vehicles, e.g. fleet or convoy travelling fleet control of land vehicles from a control room G05D1/0297; traffic control systems for road vehicles G08G1/00; for marine craft G08G3/00; for aircraft G08G5/00
• CPC Class Codes: G05D 1/0038 by providing the operator with simple or augmented images from one or more cameras located onboard the vehicle, e.g. tele-operation images analyzed by a computer and used for automatic navigation G05D1/0246

Abstract

Architecture for a multimodal, multiplatform switching, unmanned vehicle (UV) swarm system which can execute missions in diverse environments. The architecture includes onboard and ground processors to handle and integrate multiple sensor inputs generating a unique UV pilot experience for a remote drone pilot (RDP) via a virtual augmented reality cockpit (VARC). The RDP is monitored by an operational control system and an experienced control pilot. A ground processor handles real-time localization, forwarding of commands, generation and delivery of augmented content to users, along with safety features and overrides. The UVs onboard processors and autopilot execute the commands and provide a redundant source of safety features and override in the case of loss of signal. The UVs perform customizable missions, with adjustable rules for differing skill levels. RDPs experience real-time virtual piloting of the UV with augmented interactive and actionable visual and audio content delivered to them via VARC systems.

Claims

1. A system to control a plurality of unmanned vehicles, comprising:

an operational control system;

at least one virtual augmented reality cockpit in two way communication with said control system; and

a plurality of unmanned vehicles in two way communication with said control system;

wherein said operational control system sets command parameters for each one of said virtual augmented reality cockpits and said control system sets safety limits for each one of said unmanned vehicles; and

each one of said virtual augmented reality cockpits has control over one of said unmanned vehicles except that said operational control system over-rides said virtual augmented reality cockpit if either said command parameters or said safety limits are out of preset ranges.

2. The system of claim 1 further including a universal geo-fence that defines a permissible travel three dimensional space for said plurality of unmanned vehicles.

3. The system of claim 2 wherein said operational control system is configured to over-ride said virtual augmented reality cockpit if one of said plurality of unmanned vehicles crosses said universal geo-fence.

4. The system of claim 2 wherein said command parameters are independently set for each one of said virtual augmented reality cockpits.

5. The system of claim 4 wherein said command parameters are based on a pilot profile setting based on pilot experience and are selected from the group consisting of full control, partial control and view-only.

6. The system of claim 5 wherein said partial control is configured to enable said virtual augmented reality cockpit to have maneuverability within said universal geo-fence subject to throttle/acceleration and velocity restrictions.

7. The system of claim 4 wherein said safety limits are selected from the group consisting of weather, battery charge, communication integrity, obstacle avoidance, speed and location.

8. The system of claim 7 wherein said operational control system is configured to over-ride said virtual augmented reality cockpit if said location deviates from said permissible three dimensional defined by said universal geo-fence.

9. The system of claim 8 wherein location is determined by one or more of a global positioning system (GPS), a distributed image network, field located visual quick response (QR) codes and benchmarks.

10. The system of claim 7 wherein said operational control system defines a three dimensional personal geo-fence around each one of said plurality of unmanned vehicles.

11. The system of claim 10 wherein said operational control system is configured to over-ride said virtual augmented reality cockpit if one of said unmanned vehicles encroaches on the personal geo-fence of another of said plurality of unmanned vehicles.

12. The system of claim 11 wherein said operational control system configures said plurality of unmanned vehicles to exhibit swarm behavior.

13. The system of claim 12 wherein said operational control system is configured to toggle between swarm behavior and independent travel.

14. The system of claim 2 wherein said plurality of unmanned vehicles are selected from the group consisting of aerial vehicles, ground vehicles, surface water vehicles, underwater vehicles and combinations thereof.

15. The system of claim 14 wherein at least one of said plurality of unmanned vehicles includes a camera effective to capture a view from said unmanned vehicle and said virtual augmented reality cockpit includes a system effective to display said view.

16. The system of claim 15 wherein said virtual augmented reality cockpit includes one or more of physical, voice and neural control systems responsive to said view.

17. The system of claim 14 wherein said virtual augmented reality cockpit has a range of motion configured to simulate motion of at least one of said plurality of unmanned vehicles.

18. The system of claim 14 wherein said operational control system adds augmented content configured to correspond to said view.

19. The system of claim 18 wherein augmented content is selected from the group consisting of a stored digital file including audio content, interactive real-time up-dates, three dimensional virtual reality and combinations thereof.

20. The system of claim 19 wherein said operational control system is configured to select a desired language for said audio content.

21. The system of claim 19 wherein said virtual augmented reality cockpit has selective control over said augmented content.