Cooperative nesting of mechanical and electronic stabilization for an airborne camera system
First Claim
1. A method for stabilizing an image of an object being taken from a video camera, the video camera being moved by a transport mechanism and being controlled by a line-of-sight controller, the line-of-sight controller having an orientation and an angular velocity, the method comprising:
- receiving a plurality of images of the object; and
for each of the plurality of received images,receiving a distance from the video camera to the object,determining a difference between the location of the object within the image and the location of the object within a previously captured image,calculating an inter-frame stabilization adjustment based on the distance to the object and the difference between the location of the object within the image and the location of the object within a previously captured image, the inter-frame stabilization adjustment for adjusting the position of a display area of the received images,adjusting the position of the display area of the received images based on the inter-frame stabilization adjustment, wherein the received images are larger than the display area and the adjusting moves the display area relative to the received images, andcontrolling the line-of-sight controller at least in part by,calculating a line-of-sight adjustment for the line-of-sight controller based on the inter-frame stabilization adjustment,adjusting the orientation of the line-of-sight controller based on the calculated line-of-sight adjustment,calculating an angular velocity for the line-of-sight controller based on the inter-frame stabilization adjustment, andsetting the angular velocity of the line-of-sight controller to the calculated angular velocity.
2 Assignments
0 Petitions
Accused Products
Abstract
A method and system for stabilizing images being taken by a video camera using electromechanical stabilization. The stabilization system performs inter-frame stabilization based on the velocity of a vehicle on which the video camera is mounted and the pan rate of a line-of-sight controller of the video camera. The inter-frame stabilization is performed by a software component by moving a display area (or viewport) within a larger image area. The stabilization system converts an inter-frame stabilization adjustment into a pan rate adjustment so that the line-of-sight controller will keep the desired object within the image area of the camera.
-
Citations
26 Claims
-
1. A method for stabilizing an image of an object being taken from a video camera, the video camera being moved by a transport mechanism and being controlled by a line-of-sight controller, the line-of-sight controller having an orientation and an angular velocity, the method comprising:
-
receiving a plurality of images of the object; and for each of the plurality of received images, receiving a distance from the video camera to the object, determining a difference between the location of the object within the image and the location of the object within a previously captured image, calculating an inter-frame stabilization adjustment based on the distance to the object and the difference between the location of the object within the image and the location of the object within a previously captured image, the inter-frame stabilization adjustment for adjusting the position of a display area of the received images, adjusting the position of the display area of the received images based on the inter-frame stabilization adjustment, wherein the received images are larger than the display area and the adjusting moves the display area relative to the received images, and controlling the line-of-sight controller at least in part by, calculating a line-of-sight adjustment for the line-of-sight controller based on the inter-frame stabilization adjustment, adjusting the orientation of the line-of-sight controller based on the calculated line-of-sight adjustment, calculating an angular velocity for the line-of-sight controller based on the inter-frame stabilization adjustment, and setting the angular velocity of the line-of-sight controller to the calculated angular velocity. - View Dependent Claims (2, 3, 4, 5, 6, 7, 8, 9, 10, 11)
-
-
12. A method for stabilizing an image of an object being taken from a video camera, the video camera being moved by a transport mechanism and being controlled by a line-of-sight controller, the line-of-sight controller having an orientation and a pan rate, the image being displayed on a display device, the method comprising:
-
determining a difference in the location of the object within the image from one frame to the next frame; adjusting the display of the image based on the determined difference to remove small-amplitude jitter; and controlling the line-of-sight controller by, calculating a line-of-sight adjustment for the line-of-sight controller based at least in part on the determined difference, adjusting the orientation of the line-of-sight controller based on the calculated line-of-sight adjustment to account for large-amplitude jitter, calculating a pan rate for the line-of-sight controller based at least in part on the determined difference, and setting the pan rate of the line-of-sight controller to the calculated pan rate. - View Dependent Claims (13, 14, 15, 16, 17, 18)
-
-
19. A method for stabilizing images being taken from a video camera mounted on a moving vehicle, the video camera having a line of sight being controlled by a line-of-sight controller, the line-of-sight controller having an orientation and a rate of rotation, the method comprising:
-
calculating initial coordinates for a viewport, the viewport corresponding to a portion of an image that is to be displayed; calculating inter-frame stabilization adjustments based on the change in location of an object in a succession of image frames to account for a velocity of the vehicle, the inter-frame stabilization adjustments used to electronically move the viewport from one frame to the next frame; moving the viewport in accordance with the calculated inter-frame stabilization adjustments so that the viewport does not remain centered relative to the images taken from the video camera; displaying a portion of an image corresponding to the moved viewport; and controlling the line-of-sight controller at least in part by, calculating a line-of-sight adjustment for the line-of-sight controller based on the inter-frame stabilization adjustments, adjusting the orientation of the line-of-sight controller in accordance with the calculated line-of-sight adjustment, calculating a rate of rotation for the line-of-sight controller based on the inter-frame stabilization adjustments, and setting the rate of rotation of the line-of-sight controller to the calculated rate of rotation. - View Dependent Claims (20, 21, 22, 23, 24)
-
-
25. A method in a camera stabilization system for stabilizing the display of images received from a video camera attached to an aircraft and controlled by a gimbal-based line-of-sight controller, the method comprising:
-
receiving a first image from the video camera; receiving a second image from the video camera; determining the position of an object in the first image; determining the position of the object in the second image; determining an image pixel offset in the scan direction, IPO(S), based on the difference in the position of the object in the first and second images; determining an image pixel offset in the tilt direction, IPO(T), based on the difference in the position of the object in the first and second images; determining a pixel offset in the scan direction, PO(S), based on IPO(S); determining a pixel offset in the tilt direction, PO(T), based on IPO(T); adjusting the display of an image on a display device of the camera stabilization system based on PO(S) and PO(T); converting PO(S) to a corresponding scan angle based on the field of view of the video camera; converting PO(T) to a corresponding tilt angle based on the field of view of the video camera; adjusting a scan rate of the line-of-sight controller based on the scan angle; adjusting a tilt rate of the line-of-sight controller based on the tilt angle; and determining aircraft pixel offsets caused by the movement of the aircraft by, receiving an indication of the velocity of the aircraft in the earth reference frame, VaircraftE, receiving a matrix, CBE, corresponding to the orientation of the aircraft in the earth reference frame, receiving a matrix, CCB, corresponding to the orientation of the video camera, calculating a transformation matrix, CCE, for transforming from the earth reference frame to the camera reference frame, wherein CCE=CCBCBE, calculating a line of sight, LE, of the video camera in the earth reference frame, wherein LE=CCET(1,0,0)T, determining the distance, K, to an object at the center of the image, determining the velocity of the aircraft in the camera reference frame, VaircraftC, wherein VaircraftC=CCE*VaircraftE, calculating a normalized velocity of the aircraft VaircraftC=VaircraftC/K, calculating a first difference in scan units Δ
S1C, wherein Δ
S1C=VaircraftC(S)*Δ
T, wherein VaircraftC(S) corresponds to the normalized velocity of the aircraft in the scan direction, and wherein Δ
T corresponds to a frame refresh period,calculating a first difference in tilt units Δ
T1C, wherein Δ
T1C=VaircraftC(T)*Δ
T, wherein VaircraftC(T) corresponds to the normalized velocity of the aircraft in the tilt direction,calculating an aircraft pixel offset in the scan direction APO(S), wherein APO(S)=Δ
S1C*P/Z, wherein P corresponds to a pixel density associated with the video camera, and wherein Z corresponds to a zoom factor associated with the video camera,calculating an aircraft pixel offset in the tilt direction APO(T), wherein APO(T)=Δ
T1C*P/Zwherein PO(S) is determined based on IPO(S) and APO(S), and wherein PO(T) is determined based on IPO(T) and APO(T) so that both the display of the image and the line-of-sight controller are adjusted based on IPO(S) and IPO(T). - View Dependent Claims (26)
-
Specification