Method and apparatus for digital video latency reduction by real-time warping

• US 8,917,322 B2
• Filed: 04/01/2011
• Issued: 12/23/2014
• Est. Priority Date: 04/01/2011
• Status: Active Grant

• Current Assignee: Lockheed Martin Corporation (Martin Marietta Corporation)
• Original Assignee: Lockheed Martin Corporation (Martin Marietta Corporation)
• Inventors: Olson, Andrew C., Brinck, Brian E., Konyn, John N.
• Primary Examiner(s): Perungavoor, Sath V
• Assistant Examiner(s): Le, Peter D
• Application Number:
  US13/078,832
• Publication Number:
  US 20120249784A1
• Time in Patent Office: 1,362 Days
• Field of Search: 348/143
• US Class Current: 348/143
• CPC Class Codes: G06T 3/18 Image warping, e.g. rearranging pixels individually
• CPC Class Codes: H04N 7/183 for receiving images from a single remote source

Abstract

In one aspect, video latency reduction by real-time warping is described. In one aspect, an original geometric image model of a digital video frame is adjusted according to a video frame latency, to form an adjusted geometric image model. A geometric image model may represent a field of view from a remote camera used to capture the digital video frame. The adjusted geometric image model may be overlaid onto the original geometric image model to capture a warped image. In one aspect the warped image is re-projected according to the adjusted geometric image model to form a re-projected image. The re-projected image may then be displayed to approximate a real-time field of view from a camera used to capture the digital video frame. In one aspect, an attitude and runway alignment of an unmanned aerial vehicle may be controlled using a displayed, re-projected image. Other aspects are described and claimed.

Claims

1. A method for digital video latency reduction of a received digital video frame captured by a remote camera attached to a vehicle, the method comprising:

receiving the digital video frame captured by the remote camera, wherein the received digital video frame is associated with an original image model representing a field of view from the remote camera at a time the digital video frame is captured;

detecting a change in the vehicle position and attitude;

adjusting the original image model of the received digital video frame based on the detected change in the vehicle position and attitude and a time difference between a time the digital video frame is received and the time the digital video frame is captured, to form an adjusted image model representing an approximation of a future field of view of what could be seen from the remote camera at the time the digital video frame is received, wherein the time the digital video frame is received is later in time than the time the digital video frame is captured due to video latency;

overlaying the adjusted image model onto the original image model of the received digital video frame to capture a warped image; and

re-projecting the warped image according to the adjusted image model to form a re-projected image that approximates a view from the remote camera used to capture the received digital video frame at the time the digital video frame is received, wherein the re-projected image accounts for a digital video latency between the time the digital video frame is captured and the time the digital video frame is received.

2. The method of claim 1, wherein the vehicle is an unmanned aerial vehicle, the method further comprising:

displaying a plurality of re-projected images based on a plurality of received digital video frames captured by the remote camera on a display at a remote location; and

controlling an attitude and runway alignment of the unmanned aerial vehicle in response to user commands from viewing the plurality of displayed, re-projected images.

3. The method of claim 1, further comprising:

storing a camera attitude for each recorded digital video frame at a time that each digital video frame is recorded according to received navigation data.

4. The method of claim 3, wherein the camera attitude for each recorded frame is received over a wireless link.

5. The method of claim 1, wherein overlaying further comprises:

mapping the received digital video frame onto the adjusted image model to form the warped video frame; and

reducing a field of view of the warped video frame to eliminate one or more edges of the warped video frame to form the re-projected image.

6. The method of claim 1, wherein adjusting further comprises:

re-mapping the received digital video frame to determine the original image model corresponding to a field of view of the remote camera at a digital video frame capture time; and

modifying the original image model of the received digital video frame to approximate the real-time field of view from the remote camera at the time the digital video frame is received to form the adjusted image model.

7. The method of claim 1, further comprising:

rendering, using a graphics processing unit, a composite two-dimensional display of the digital video frame by applying the video frame to a three-dimensional geometry corresponding to a field of view of the remote camera at the time the video frame is captured to approximate a real-time view from a vehicle including the camera.

8. A non-transitory computer readable medium having processor-executable software instruction to perform a method digital video latency reduction of a received digital video frame captured by a remote camera attached to a vehicle, comprising:

re-mapping the digital video frame to determine an original geometric image model corresponding to a field of view from the remote camera used to capture the digital video frame at a digital video frame capture time;

detecting a change in the vehicle position and attitude;

modifying the original geometric image model according to an approximate field of view from the remote camera at a digital video frame receive time based on the detected change in the vehicle position and attitude and a time difference between the digital video frame receive time and digital video frame capture time, to form an adjusted geometric image model, wherein the digital video frame receive time is later than the digital video frame capture time due to video latency;

overlaying the adjusted geometric image model onto the original geometric image model to capture a warped image; and

reducing a field of view of the warped video frame image to eliminate one or more edges of the warped video frame image to form a re-projected image that approximates a view from the remote camera at the digital video frame receive time; and

displaying the re-projected image to approximate a real-time field of view from the camera at the video frame receive time, wherein the re-projected image accounts for a digital video latency between the digital video frame capture time and digital video frame receive time.

9. The non-transitory computer readable medium of claim 8, further comprising:

maintaining an attitude and runway alignment of an unmanned aerial vehicle using a displayed, re-projected image.

10. The non-transitory computer readable medium of claim 9, wherein re-mapping further comprises:

adjusting the original geometric image model of a digital video frame according to a delay between a video frame receive time and a video frame capture time to form the adjusted geometric image model.

11. The non-transitory computer readable medium of claim 8, wherein displaying further comprises:

rendering, using a graphics processing unit, a composite two-dimensional display of the video frame by applying the video frame to a three-dimensional geometry corresponding to a field of view of the camera at the video frame capture time.

12. The non-transitory computer readable medium of claim 11, further comprising:

storing a camera attitude for each recorded frame, at a time that each frame is recorded, according to received navigation data, wherein the camera attitude for each recorded frame is received over a wireless link.

13. The non-transitory computer readable medium of claim 8, wherein overlaying further comprises:

mapping the video frame onto the adjusted geometric image model to form the warped video frame image.

14. The non-transitory computer readable medium of claim 8, wherein overlaying further comprises:

warping each frame of digital video to approximate a geometry of a future video frame in real-time.

15. A system for digital video latency reduction of a received digital video frame captured by a remote camera attached to a vehicle, comprising:

a transceiver configured to receive the digital video frame and a camera geometry at a digital video frame capture time for the digital video frame;

a memory operable to store the received digital video frame and the camera geometry;

a graphics processing unit operable to;

determine an original geometric image model corresponding to a field of view from the remote camera used to capture the digital video frame, at the digital video frame capture time,

detect a change in the vehicle position and attitude;

overlay an adjusted geometric image model onto the original geometric image model to capture a warped image, the adjusted geometric image model being based on the detected change in the vehicle position and attitude and a time difference between a digital video frame receive time and the digital video frame capture time, and

corresponding to an approximate field of view from the remote camera at the digital video frame receive time, wherein the time the digital video frame was received is later in time than the time the digital video frame was captured due to video latency, and

reduce a field of view of the warped video frame to eliminate one or more edges of the warped image to form a re-projected image;

a display operable to display the re-projected image to approximate a field of view from the camera used to capture the video frame according to a location of the camera at the video frame receive time, wherein the re-projected image accounts for a digital video latency between the digital video frame capture time and the digital video frame receive time; and

a controller operable to control the vehicle, including the remote camera, according to a re-projected image.

16. The system of claim 15, wherein the graphics processing unit is further operable to warp each frame of digital video to approximate a geometry of a future video frame in real-time.

17. The system of claim 15, wherein the graphics processing unit is further operable to re-map the digital video frame to determine the original geometric image model corresponding to a field of view of the camera at the video frame capture time and to modify the original geometric image to approximate the real-time field of view of the camera at the video frame receive time to form the adjusted geometric image.

18. The system of claim 15, further comprising an application programming interface to control an attitude and runway alignment of an unmanned aerial vehicle using a displayed, re-projected image.

19. The system of claim 15, wherein the vehicle comprises an unmanned aerial vehicle.

20. The system of claim 19, wherein the vehicle comprises an inertial navigation system operable to provide a camera geometry for each recorded digital video frame at a time the frame is captured.