AUGMENTED VIDEO SYSTEM PROVIDING ENHANCED SITUATIONAL AWARENESS

• US 20140267723A1
• Filed: 01/30/2013
• Published: 09/18/2014
• Est. Priority Date: 01/30/2013
• Status: Active Grant

• Current Assignee: Insitu Incorporated (The Boeing Co.)
• Original Assignee: Insitu Incorporated (The Boeing Co.)
• Inventors: Davidson, Darcy Jr., Trowbridge, Theodore T.
• Granted Patent: US 9,380,275 B2
• Time in Patent Office: Days
• US Class Current: 348/147
• CPC Class Codes: G01C 11/00 Photogrammetry or videogrammetry, e.g. stereogrammetry; Photographic surveying
• CPC Class Codes: G01C 11/02 Picture taking arrangements specially adapted for photogrammetry or photographic surveying, e.g. controlling overlapping of pictures
• CPC Class Codes: G01C 11/36 Videogrammetry, i.e. electronic processing of video signals from a single source or from different sources to give parallax or range information
• CPC Class Codes: G05D 1/0038 by providing the operator with simple or augmented images from one or more cameras located onboard the vehicle, e.g. tele-operation images analyzed by a computer and used for automatic navigation G05D1/0246
• CPC Class Codes: G06T 11/00 2D [Two Dimensional] image generation
• CPC Class Codes: G06T 11/60 Editing figures and text; Combining figures or text
• CPC Class Codes: G06T 17/05 Geographic models
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: H04N 7/183 for receiving images from a single remote source

Abstract

A facility, comprising systems and methods, for providing enhanced situational awareness to captured image data is disclosed. The disclosed techniques are used in conjunction with image data, such as a real-time or near real-time image stream captured by a camera attached to an unmanned system, previously captured image data, rendered image data, etc. The facility enhances situational awareness by projecting overlays onto captured video data or “wrapping” captured image data with previously-captured and/or “synthetic world” information, such as satellite images, computer-generated images, wire models, textured surfaces, and so on. The facility also provides enhanced zoom techniques that allow a user to quickly zoom in on an object or area of interest using a combined optical and digital zoom technique. Additionally, the facility provides a digital lead indicator designed to reduce operator-induced oscillations in commanding an image capturing device.

Claims

1. A method for providing enhanced situational awareness to image data captured by an image capturing device carried by an unmanned aircraft vehicle system, the method comprising:

storing, for at least one of a plurality of objects of interest, position data for the at least one object of interest;

storing, for at least one of a plurality of geographic areas, topographical data for the at least one geographic area;

receiving image data captured by the image capturing device carried by the unmanned aircraft vehicle, wherein at least one of a plurality of frames of the received image data has associated metadata comprising a timestamp, an orientation of the unmanned aircraft vehicle, an orientation of the image capturing device relative to the unmanned aircraft vehicle, configuration information for the image capturing device, a position for the unmanned aircraft vehicle, and a velocity for the unmanned aircraft vehicle; and

for at least one of a plurality of frames of the received image data,

determining a transformation matrix based on metadata associated with the frame, and

for each of at least one overlay,

applying the determined transformation matrix data to the overlay to transform the overlay, and

providing the transformed overlay for display in association with a display of the frame.

4. The method of claim 1, further comprising:

in response to receiving a request to change the orientation of the image capturing device,

determining a horizontal field of view for the image capturing device,

determining a vertical field of view for the image capturing device,

determining a horizontal slew rate for changing the orientation of the image capturing device,

determining a vertical slew rate for changing the orientation of the image capturing device,

determining a width of a displayed image,

determining a height of a displayed image,

determining an advance period,

calculating a coordinate based on the determined horizontal field of view, the determined vertical field of view, the determined horizontal slew rate, the determined vertical slew rate, the determined width, the determined height, and the determined advance period, and

providing for display an indication of a digital lead indicator at the calculated coordinate.

5. The method of claim 1, further comprising:

in response to receiving a request to perform a zoom operation, digitally zooming a video frame captured by the image capturing device,

providing for display at least a portion of the digitally-zoomed video frame in place of a displayed video frame,

determining whether the image capturing device has completed the requested zoom operation, and

in response to determining that the image capturing device has completed the requested zoom operation, replacing a digitally-zoomed video frame with a video frame captured by the image capturing device.

6. The method of claim 1 wherein the at least one overlay includes an object of interest overlay, a map annotation overlay, a heading and groundspeed overlay, a flight path overlay, and a waypoint and route overlay.

7. The method of claim 1 wherein the at least one overlay includes an engine out boundaries overlay, a terrain contour overlay, a map overlay, a building overlay, and a city block overlay.

8. The method of claim 1 wherein the at least one overlay includes a 3D airspace overlay, a detectability overlay, an automatic identification system (AIS) overlay, and an approach information overlay.

2. The method of example 1 wherein the topographical data includes at least one of wire frame data, map data, or terrain contour data.

3. The method of example 1, further comprising:

storing, for at least one of a plurality of geographic areas, bathymetric data for the at least one geographic area.

9. A computer-readable storage medium storing instructions that, when executed by a computing system having a processor, cause the computing system to perform operations comprising:

storing terrain information for a geographic region;

identifying a first image stream captured by a first image capturing device and including images of a first area within the geographic region;

identifying a second image stream other than the first image stream captured by a second image capturing device other than the first image capturing device and including images of a second area within the geographic region other than the first area within the geographic region;

rendering a hybrid environment comprising at least a portion of the stored terrain information for the geographic region, at least one image from the first image stream, and at least one image from the second image stream, wherein the rendering comprises;

determining a position of a virtual camera relative to the hybrid environment,

determining a position of the first image capturing device relative to the geographic region,

generating a first transformation matrix based on the determined position of the virtual camera relative to the hybrid environment and the determined position of the first image capturing device relative to the geographic region,

applying the generated first transformation matrix to images of the first stream to generate first transformed frames,

determining a position of the second image capturing device relative to the geographic region,

generating a second transformation matrix based on the determined position of the virtual camera relative to the hybrid environment and the determined position of the second image capturing device relative to the geographic region,

applying the generated second transformation matrix to images of the second image stream to generate second transformed frames,

adding at least one of the generated first transformed frames and at least one of the generated second transformed frames to a rendering of the stored terrain information for the geographic region.

10. The computer-readable storage medium of claim 9 wherein the first image capturing device is carried by an aircraft vehicle system and wherein the rendered hybrid environment includes a graphical representation of the aircraft vehicle system at a location that approximates the actual location of the aircraft vehicle system relative to the geographic region.

11. The computer-readable storage medium of claim 9, the operations further comprising:

for each of a plurality of overlays,

for each of a plurality of objects of the overlay,

transforming a position of the object into screen coordinates, and

providing an indication of the object for display at the screen coordinates.

12. The computer-readable medium of claim 11, the operations further comprising:

for at least one overlay,

for at least one object of the overlay,

receiving, from a user, a label for the object,

storing the received label, and

providing for display an indication of the received label.

13. A computing system comprising a processor, the computing system comprising:

a component configured to display at least a portion of a video frame captured by an image capturing device;

a component configured to receive a request to perform an optical zoom operation of the image capturing device;

a component configured to initiate the requested optical zoom operation of the image capturing device;

a component configured to, while the image capturing device is performing the zoom operation,

digitally zoom a video frame captured by the image capturing device,

provide for display at least a portion of the digitally-zoomed video frame in place of the displayed video frame, and

determine whether the image capturing device has completed the zoom operation; and

a component configured to replace a digitally-zoomed video frame with a video frame captured by the image capturing device after the image capturing device has completed the zoom operation.

14. The computing system of claim 13, further comprising:

means for augmenting image data captured by the image capturing device with overlay information; and

means for synchronizing image data captured by the image capturing device with associated metadata.

15. The computing system of claim 14, wherein the means for augmenting comprises means for converting a position of at least one overlay object into screen coordinates.

16. The computing system of claim 13, further comprising:

means for rendering a digital lead indicator.

17. The computing system of claim 13, further comprising:

means for rendering a dynamic reticle.

18. A computer-readable storage medium storing instructions that, if executed by a computing system having a processor, cause the computing system to perform operations comprising:

providing images captured by an image capturing device for display; and

in response to receiving a request to change the orientation of the image capturing device,

determining a horizontal field of view for the image capturing device,

determining a vertical field of view for the image capturing device,

determining a horizontal slew rate for changing the orientation of the image capturing device,

determining a vertical slew rate for changing the orientation of the image capturing device,

determining a width of the displayed images,

determining a height of the displayed images,

determining an advance period,

calculating a coordinate based on the determined horizontal field of view, the determined vertical field of view, the determined horizontal slew rate, the determined vertical slew rate, the determined width, the determined height, and the determined advance period, and

displaying an indication of a digital lead indicator at the calculated coordinate.

19. The computer-readable storage medium of claim 18 wherein the horizontal and vertical slew rates are determined based on a magnitude of user input.

20. The computer-readable storage medium of claim 18 wherein the coordinate (X, Y) is calculated according to the following equations: