SYSTEM AND METHOD FOR REAL-TIME OVERLAY OF MAP FEATURES ONTO A VIDEO FEED

• US 20170039765A1
• Filed: 05/04/2015
• Published: 02/09/2017
• Est. Priority Date: 05/05/2014
• Status: Active Grant

• Current Assignee: Motorola Solutions, Inc.
• Original Assignee: Avigilon Fortress Corporation (Motorola Solutions, Inc.)
• Inventors: Zhou, Shirley, Scanlon, Andrew W., Madden, Don, Choe, Tae Eun
• Granted Patent: US 9,996,976 B2
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: G06T 17/05 Geographic models
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/10028 Range image; Depth image; 3D point clouds
• CPC Class Codes: G06T 2207/10032 Satellite or aerial image; Remote sensing
• CPC Class Codes: G06T 2207/30212 Military
• CPC Class Codes: G06T 2207/30241 Trajectory
• CPC Class Codes: G06T 7/344 involving models
• CPC Class Codes: G06T 7/521 from laser ranging, e.g. using interferometry; from the projection of structured light
• CPC Class Codes: G06T 7/55 from multiple images

Abstract

A method is provided for augmenting video feed obtained by a camera of a aerial vehicle to a user interface. The method can include obtaining a sequence of video images with or without corresponding sensor metadata from the aerial vehicle; obtaining supplemental data based on the sequence of video images and the sensor metadata; correcting an error in the sensor metadata using a reconstruction error minimization technique; creating a geographically-referenced scene model based on a virtual sensor coordinate system that is registered to the sequence of video images; overlaying the supplemental information onto the geographically-referenced scene model by rendering geo-registered data from a 3D perspective that matches a corrected camera model; creating a video stream of a virtual representation from the scene from the perspective of the camera based on the overlaying; and providing the video stream to a UI to be render onto a display.

Claims

1. A method for providing an augmented video feed obtained by a camera of a manned or unmanned aerial vehicle (“

UAV”

) to a user interface (“

UI”

), the method comprising;

obtaining a sequence of video images with or without corresponding sensor metadata from the aerial vehicle;

obtaining supplemental data based on the sequence of video images and the sensor metadata;

correcting, by a processor, an error in the sensor metadata using a reconstruction error minimization technique;

creating, by a processor, a geographically-referenced scene model based on a virtual sensor coordinate system that is registered to the sequence of video images;

overlaying the supplemental information onto the geographically-referenced scene model by rendering geo-registered data from a 3D perspective that matches a corrected camera model;

creating a video stream of a virtual representation from the scene from the perspective of the camera based on the overlaying; and

providing the video stream to a UI to be render onto a display.

2. The method of claim 1, wherein the supplemental data comprises one or more of:

static geo-referenced datasets, dynamical geo-referenced datasets, traffic conditions, elevation data, terrain data, social media information, waypoint data, light detection and ranging (“

LIDAR”

) data, airspace symbology data, 3D model data, and road maps.

3. The method of claim 1, wherein the error in the sensor metadata comprises one or more of:

missing data, temporal drift, and spatial drift.

4. The method of claim 1, wherein the correcting the error in the sensor metadata is performed for each frame in the sequence of video images, wherein a correction of one video image is used to refine the correction for a subsequent video image.

5. The method of claim 1, wherein the correcting the error in the sensor metadata for one image frame is based on another image frame or map data corresponding to a scene that is represented in the one image.

6. The method of claim 1, further comprising:

constructing a depth map using light detection and ranging (“

LIDAR”

) or digital elevation maps (“

DEM”

);

determining that one or more pixels representing overlay objects has low or no visibility for a video frame based the depth map; and

rendering the one or more pixels in a manner to represent that the one or more pixels are occluded.

7. The method of claim 1, wherein the correcting the error in the sensor metadata further comprises:

performing a first registration between a video image in the sequence of video images and a corresponding map data;

determining an anchor frame from the sequence of video images; and

performing a second registration between the video image and the corresponding map data using the anchor frame by minimizing a reprojection error.

8. The method of claim 7, wherein the minimizing the reprojection error is performed using the reconstruction error minimization technique according to:

min(H)R_g(v,m,H)=min(H)Σ

_j=1ⁿ(|p_j^v−

Hp_j^m|+|p_j^m−

H<sup>−

1</sup>p_j^v|)/n, where

subject to H to be close to rigid body geometry, and v is a frame of input image, m is cropped map imagery, R_g is reconstruction error of feature points, j where j=1, . . . , n, where n is a number of corresponding points.

9. The method of claim 1, wherein the reconstruction error is minimized using a Lagrange optimization technique to obtain a final homography between map data and the video images.

10. The method of claim 1, wherein the supplemental data comprises information beyond a field-of-view of the camera of the aerial vehicle.

11. The method of claim 1, further comprising:

obtaining a series of waypoint data, wherein the series comprises a beginning waypoint, one or more intermediate waypoints, and a destination waypoint;

generating a flight path based on the series of waypoint data; and

outputting the flight path to the UI.

12. The method of claim 1, further comprising generating the UI that shows the supplemental data overlaid over the video stream and the sensor metadata.

13. A device for providing an augmented video feed obtained by a camera of a manned or unmanned aerial vehicle (“

UAV”

) to a user interface (“

UI”

), the device comprising;

a memory containing instructions; and

at least one processor, operably connected to the memory, the executes the instructions to perform a method for providing an augmented video feed obtained by a camera of a manned or unmanned aerial vehicle (“

UAV”

) to a user interface (“

UI”

), comprising;

obtaining a sequence of video images with or without corresponding sensor metadata from the aerial vehicle;

obtaining supplemental data based on the sequence of video images and the sensor metadata;

correcting, by a processor, an error in the sensor metadata using a reconstruction error minimization technique;

creating, by a processor, a geographically-referenced scene model based on a virtual sensor coordinate system that is registered to the sequence of video images;

overlaying the supplemental information onto the geographically-referenced scene model by rendering geo-registered data from a 3D perspective that matches a corrected camera model;

creating a video stream of a virtual representation from the scene from the perspective of the camera based on the overlaying; and

providing the video stream to a UI to be render onto a display.

14. The device of claim 13, wherein the at least one processor is further operable to perform the method comprising:

constructing a depth map using light detection and ranging (“

LIDAR”

) or digital elevation maps (“

DEM”

);

determining that one or more pixels representing overlay objects has low or no visibility for a video frame based the depth map; and

rendering the one or more pixels in a manner to represent that the one or more pixels are occluded.

15. The device of claim 14, wherein the correcting the error in the sensor metadata further comprises:

performing a first registration between a video image in the sequence of video images and a corresponding map data;

determining an anchor frame from the sequence of video images; and

performing a second registration between the video image and the corresponding map data using the anchor frame by minimizing a reprojection error.

16. The device of claim 15, wherein the minimizing the reprojection error is performed using the reconstruction error minimization technique according to:

min(H)R_g(v,m,H)=min(H)Σ

_j=1ⁿ(|p_j^v−

Hp_j^m|+|p_j^m−

H<sup>−

1</sup>p_j^v|)/n, where

subject to H to be close to rigid body geometry, and v is a frame of input image, m is cropped map imagery, R_g is reconstruction error of feature points, j where j=1, . . . , n, where n is a number of corresponding points.

17. The device of claim 16, wherein the reconstruction error is minimized using a Lagrange optimization technique to obtain a final homography between map data and the video images.

18. A computer readable storage medium comprising instructions for causing one or more processors to perform a method, the method for discerning a vehicle at an access control point, the device comprising:

obtaining a sequence of video images with or without corresponding sensor metadata from the aerial vehicle;

obtaining supplemental data based on the sequence of video images and the sensor metadata;

correcting, by a processor, an error in the sensor metadata using a reconstruction error minimization technique;

creating, by a processor, a geographically-referenced scene model based on a virtual sensor coordinate system that is registered to the sequence of video images;

overlaying the supplemental information onto the geographically-referenced scene model by rendering geo-registered data from a 3D perspective that matches a corrected camera model;

creating a video stream of a virtual representation from the scene from the perspective of the camera based on the overlaying; and

providing the video stream to a UI to be render onto a display.

19. The computer readable storage medium of claim 18, wherein the correcting the error in the sensor metadata further comprises:

performing a first registration between a video image in the sequence of video images and a corresponding map data;

determining an anchor frame from the sequence of video images; and

performing a second registration between the video image and the corresponding map data using the anchor frame by minimizing a reprojection error.

20. The computer readable storage medium of claim 19, wherein the minimizing the reprojection error is performed using the reconstruction error minimization technique according to:

min(H)R_g(v,m,H)=min(H)Σ

_j=1ⁿ(|p_j^v−

Hp_j^m|+|p_j^m−

H<sup>−

1</sup>p_j^v|)/n, where

subject to H to be close to rigid body geometry, and v is a frame of input image, m is cropped map imagery, R_g is reconstruction error of feature points, j where j=1, . . . , n, where n is a number of corresponding points.