Augmented reality cross-cueing systems and methods

• US 10,353,389 B2
• Filed: 11/16/2016
• Issued: 07/16/2019
• Est. Priority Date: 11/16/2016
• Status: Active Grant

• Current Assignee: Rapid Imaging Technologies, LLC
• Original Assignee: Rapid Imaging Technologies, LLC
• Inventors: Abernathy, Michael Franklin
• Primary Examiner(s): Casillas, Roland J
• Application Number:
  US15/352,959
• Publication Number:
  US 20180136813A1
• Time in Patent Office: 972 Days
• CPC Class Codes: G05D 1/0038 by providing the operator with simple or augmented images from one or more cameras located onboard the vehicle, e.g. tele-operation images analyzed by a computer and used for automatic navigation G05D1/0246
• CPC Class Codes: G05D 1/0044 by providing the operator with a computer generated representation of the environment of the vehicle, e.g. virtual reality, maps maps used for automatic navigation G05D1/0274; flight directors G01C23/005
• CPC Class Codes: G06F 3/0481 based on specific properties of the displayed interaction object or a metaphor-based environment, e.g. interaction with desktop elements like windows or icons, or assisted by a cursor's changing behaviour or appearance

Abstract

Cross-cueing systems allow users to instantly understand the geographic location of objects in the video scene, as well as instantly understand the location within the scene of objects on a map. This provides an unprecedented level of situation awareness for unmanned vehicle operators and video surveillance system operators.

Claims

1. A system comprising:

a ground station comprising a graphical user interface (GUI) and a cursor control device, wherein the GUI comprises a map view, a camera view, a map cursor, and a camera cursor, wherein the ground station is configured to communicate with an unmanned vehicle (UV) comprising a geographic position sensor and a camera, wherein the camera produces a live video stream that is transmitted to the ground station, and wherein the live video stream is displayed in the camera view;

an electronic map comprising map imagery wherein at least a portion of the map imagery is displayed in the map view;

wherein the map cursor is displayed within the map view, wherein the camera cursor is displayed within the camera view;

wherein a user operates the cursor control device to move the camera cursor to indicate a place, and wherein the map cursor is automatically positioned to also indicate the place.

2. The system of claim 1 wherein operating the cursor control device to move the map cursor to indicate a second place causes the camera cursor to be automatically positioned to also indicate the second place.

3. The system of claim 2 wherein the electronic map further comprises height data of geographic features, wherein line of sight data is calculated at least in part on UV location data, wherein the line of sight data indicates that the second place is at an occluded map location, wherein the second place is labeled in the map view indicating the second place is at an occluded map location, and wherein the second place is not labeled in the camera view because the occluded map location is not visible to the UV.

4. The system of claim 2 wherein the electronic map further comprises height data of geographic features, wherein line of sight data is calculated at least in part on UV location data, wherein the line of sight data indicates that the second place is at an occluded map location, and wherein a label in the camera view indicates a location of the second place and indicates that the location is not visible to the UV.

5. The system of claim 2 wherein the electronic map further comprises height data of geographic features, wherein line of sight data is calculated at least in part on UV location data, wherein the line of sight data indicates that the second place is at an occluded map location, and wherein the camera cursor is changed in appearance to indicate that the second place is not visible to the UV.

6. The system of claim 1 wherein the electronic map further comprises height data of geographic features, wherein line of sight data is calculated at least in part on UV location data, wherein the map view indicates an occluded area that is occluded from view of the camera.

7. The system of claim 1 wherein the electronic map further comprises height data of geographic features, wherein line of sight data is calculated at least in part on UV location data, wherein the line of sight data indicates that the place is not visible from the UV, and wherein at least one of the map cursor and the camera cursor indicates that the place is not visible from the UV.

8. The system of claim 1 wherein the electronic map further comprises height data of geographic features, wherein line of sight data is calculated at least in part on UV location data, wherein the line of sight data indicates that the place is a map feature that occludes an area behind the map feature, and wherein the map cursor is positioned to indicate the map feature.

9. The system of claim 1 wherein the UV comprises a distance measuring device, wherein aiming the distance measuring device does not also aim the camera, wherein the place is visible in the camera view, wherein the distance measuring device measures a distance to the place, and wherein the map cursor is positioned to indicate the place.

10. The system of claim 1 wherein the place is visible in the live video stream wherein the live video stream is processed to calculate a distance to the place, and wherein the map cursor is positioned to indicate the place.

11. The system of claim 1 wherein the map view comprises a UV icon indicating the position and heading of the UV, wherein the map view comprises a graphic indicating the camera view, and wherein the camera is pointed in a direction other than the heading.

12. The system of claim 1 wherein the map view comprises a graphic showing map imagery that is within the camera view.

13. The system of claim 1 further comprising satellite imagery wherein at least a portion of the satellite imagery is displayed in the map view.

14. A method comprising:

displaying a map view and a camera view wherein a graphical user interface (GUI) comprises the camera view and the map view, wherein a ground station comprises the GUI and a cursor control device;

receiving a live video stream from an unmanned vehicle (UV) comprising geographic position sensor and a camera producing the live video stream;

displaying the live video stream and a camera cursor in the camera view;

displaying map imagery and a map cursor in the map view;

accepting a cursor control input wherein a user operates the cursor control device to move the camera cursor to indicate a place; and

automatically positioning the map cursor to also indicate the place.

15. The method of claim 14 further comprising:

accepting location data from the UV wherein the location data comprises a geographic position of the UV and a vector, and wherein the vector indicates a pointing direction of the camera wherein the pointing direction of the camera is not the heading.

16. The method of claim 14 further comprising:

accepting another cursor control input wherein a user operates the cursor control device to move the map cursor to indicate a second place; and

automatically positioning the camera cursor to also indicate the second place.

17. The method of claim 14 further comprising:

indicating on the map view a position and a heading of the UV; and

indicating a camera view area on the map view by drawing a visible boundary on the map view.

18. The method of claim 14 further comprising:

calculating line of sight data based on UV location data and an electronic map; and

modifying the camera cursor and the map cursor to indicate that the place is not visible from the UV.

19. The method of claim 14 further comprising:

calculating line of sight data based on UV location data and an electronic map; and

positioning the map cursor over a map feature wherein the line of sight data indicates that the map feature occludes the place.

20. A system comprising:

a ground station comprising a graphical user interface (GUI) and a cursor control device, wherein the GUI comprises a map view, a camera view, a map cursor, and a camera cursor;

an unmanned vehicle (UV) wherein the UV comprises laser ranging device, a geographic position sensor, a heading sensor, and a camera producing a live video stream, a camera turret that aims the camera without changing the UV heading, and a turret that that aims the laser ranging device without aiming the camera, wherein the live video stream is transmitted to the ground station and displayed in the camera view;

an electronic map comprising map imagery wherein at least a portion of the map imagery is displayed in the map view, wherein the map cursor is displayed within the map view, wherein the camera cursor is displayed within the camera view, wherein operating the cursor control device to move the map cursor to indicate a second place causes the camera cursor to be automatically positioned to also indicate the second place; and

wherein operating the cursor control device to move the camera cursor to indicate a second place causes the map cursor to be automatically positioned to also indicate the second place;

a plurality of annotated map locations, wherein at least one of the annotated map locations is labeled in the map view and in the camera view;

height data of geographic features wherein the electronic map further comprises the height data of geographic features;

line of sight data calculated at least in part on UV location data, wherein the line of sight data indicates that one of the annotated map locations is occluded from view by the UV, wherein the one of the annotated map locations is not labeled in the camera view, wherein the line of sight data indicates that another one of the annotated map locations is occluded from view by the UV, wherein the another one of annotated map locations is labeled in the camera view indicating that the another one of annotated map locations is not visible to the UV, wherein the line of sight data indicates that yet another one of the annotated map locations is occluded from view by the UV, wherein the yet another one of the annotated map locations is labeled in the map view indicating that yet another one of the annotated map locations is not visible to the UV, wherein the map view indicates an area that is not visible to the UV, wherein the line of sight data indicates when the place is not visible from the UV, wherein at least one of the map cursor and the camera cursor indicates when the place is not visible from the UV, and wherein the map cursor is positioned to indicate the map feature when the place is not visible from the UV;

wherein the UV measures a distance to the place when the place is visible to the UV, wherein the image data is processed to calculate a confirming distance to the place;

wherein at least a portion of the map view displays satellite imagery;

wherein the map view comprises a UV icon indicating the position and heading of the UV; and

wherein the map view comprises a graphic indicating the camera view area by drawing a visible boundary on the map view.