AUGMENTED REALITY TO DISPLAY FLIGHT DATA AND LOCATE AND CONTROL AN AERIAL VEHICLE IN REAL TIME

• US 20170140657A1
• Filed: 11/07/2016
• Published: 05/18/2017
• Est. Priority Date: 11/09/2015
• Status: Abandoned Application

• Current Assignee: Black Swift Technologies LLC
• Original Assignee: Black Swift Technologies LLC
• Inventors: Elston, Jack S., Stachura, Maciej, Dixon, Cory
• Application Number:
  US15/345,473
• Publication Number:
  US 20170140657A1
• Time in Patent Office: Days
• CPC Class Codes: G06T 11/00 2D [Two Dimensional] image generation
• CPC Class Codes: G06V 20/10 Terrestrial scenes scenes under surveillance with static cameras G06V20/52; scenes perceived from the exterior of a vehicle G06V20/56; scenes perceived from the interior of a vehicle G06V20/59
• CPC Class Codes: G08G 5/0013 with a ground station
• CPC Class Codes: G08G 5/0034 Assembly of a flight plan
• CPC Class Codes: G08G 5/0039 Modification of a flight plan
• CPC Class Codes: G08G 5/0069 specially adapted for an unmanned aircraft
• CPC Class Codes: G08G 5/0082 for monitoring traffic from a ground station radar or analogous systems specially adapted for traffic control G01S13/91
• CPC Class Codes: G08G 5/045 Navigation or guidance aids, e.g. determination of anti-collision manoeuvers radar or analogous systems specially adapted for anti-collision between aircraft G01S13/933

Abstract

A system for displaying information related to the flight of an aerial vehicle that is comprised of a display, a camera that captures real-time video input from its surrounding environment, and a computing device that is coupled to, and communicates with the camera, a display, and one or more aerial vehicles. The computing device maps the current location and orientation of the camera to a display coordinate system. The flight data of the aerial vehicle is also mapped on the same display coordinate system. The computing device displays, on the display, the real-time video output that is comprised of the visual integration of the flight data and the real-time video input in the display coordinate system in relation to the present location and orientation of the camera.

Claims

1. A system for displaying information related to flight of an aerial vehicle comprising:

a display;

a camera that operates to capture real-time video input from a surrounding environment; and

a computing device, communicatively coupled with the display, the camera, and an aerial vehicle, wherein the computing device operates to;

map a present location and an orientation of the camera to a display coordinate system;

map flight data for the aerial vehicle to the display coordinate system; and

display, on the display, a real-time video output comprising a visual integration of the flight data with the real-time video input in the display coordinate system according to a present location and orientation of the camera.

2. The system of claim 1, wherein the flight data for the aerial vehicle comprises a flight plan that indicates a planned future position of the aerial vehicle.

3. The system of claim 1, wherein the camera communicates the present location and orientation to the computing device.

4. The system of claim 1, wherein the flight data indicates a current position of the aerial vehicle.

5. The system of claim 1, wherein the flight data indicates an estimated future position of the aerial vehicle.

6. The system of claim 1, wherein the computing device is cooperative with the aerial vehicle.

7. The system of claim 1, wherein the computing device further operates to:

receive a real-time updates of the flight data; and

update, in real-time on the display, the real-time video output in accordance with the real-time updates.

8. The system of claim 1, wherein the computing device further operates to:

receive a flight path command via a user interface; and

alter the flight plan in real time in response to the flight path command.

9. The system of claim 1, wherein:

the aerial vehicle is one of a plurality of aerial vehicles;

the computing device is communicatively coupled with the plurality of aerial vehicles; and

the visual integration further comprises visual integration of flight data for the plurality of aerial vehicles with the real-time video input in the display coordinate system image according to the present location and orientation of the camera.

10. The system of claim 1, wherein the computing device further operates to display the real-time video output by displaying an arrow indicating a direction of a present location of the aerial vehicle with respect to the real-time video input when the aerial vehicle is not visible within the real-time video input.

11. The system of claim 1, wherein the aerial vehicle is in wireless communication with a ground station and the ground station is in wireless communication with the computing device.

12. The system of claim 1, wherein the aerial vehicle is in communication directly with the computing device.

13. The system of claim 1, wherein the aerial vehicle is in communication with the computing device via a network.

14. The system of claim 1, wherein the computing device, the display, and the camera are integrated into a single device.

15. A process for displaying flight data of at least one aerial vehicle from the ground, the process comprising:

receiving with a computing device signals relayed from a location device, the location device coupled with a camera, the signals specifying a spatial orientation of the camera;

receiving with the computing device signals relayed from at least one aerial vehicle, the signals indicating a location of the at least one aerial vehicle;

computing the spatial orientation of the camera in relation to the location of the at least one aerial vehicle;

transmitting to a display the spatial orientation of the camera in relation to the location of the at least one aerial vehicle;

rendering, on the display, a real time image relayed from the camera visually integrated with a visual indication of the spatial orientation of the camera in relation to the location of the at least one aerial vehicle;

receiving with the computing device updated signals from the location device and the at least one aerial vehicle, the updated signals specifying an updated orientation of the camera in relation to the at least one aerial vehicle; and

rendering, on the display, a real time image relayed from the camera visually integrated with a visual indication of the updated orientation of the camera in relation to the location of the at least one aerial vehicle.

16. The process of claim 15 further comprising:

displaying a predetermined flight plan representing a future position of the at least one aerial vehicle.

17. The process of claim 15 further comprising:

displaying, on the display, one or more obstacles that intersect a future flight path of the at least one aerial vehicle.

18. The process of claim 15 further comprising:

adjusting a future flight path of a cooperative aerial vehicle in real-time to avoid any obstacles displayed on the display.

19. A system for facilitating avoidance of obstacles by a cooperative aerial vehicle comprising:

a display;

a camera that operates to capture a real-time video input from a surrounding environment, the camera coupled to an inertial measurement unit (IMU); and

a computing device communicatively coupled with the display, the camera, and a cooperative aerial vehicle, wherein the computing device operates to;

map a present location and orientation of the camera, the present location and orientation relayed to the computing device;

map flight data for the cooperative aerial vehicle to a display coordinate system;

display, on the display, real-time video output comprising visual integration of the flight data with the real-time video input in the display coordinate system according to the present location and orientation of the camera; and

revise on the real-time video output, in real time, a position of a visual indication of a future location of the aerial vehicle.

20. The system of claim 19, wherein the computing device further operates to detect a user interaction that changes the position of the visual indication and in response to the detecting, updates a flight plan in the flight data for the cooperative aerial vehicle.