Cooperative nesting of mechanical and electronic stabilization for an airborne camera system

• US 7,876,359 B2
• Filed: 12/01/2003
• Issued: 01/25/2011
• Est. Priority Date: 01/17/2003
• Status: Active Grant

• Current Assignee: Insitu Incorporated (The Boeing Co.)
• Original Assignee: Insitu Incorporated (The Boeing Co.)
• Inventors: von Flotow, Andreas H., Mercadal, Mathieu, McGeer, Brian T.
• Primary Examiner(s): Tran; Thai
• Assistant Examiner(s): JONES, HEATHER RAE
• Application Number:
  US10/726,334
• Publication Number:
  US 20040183917A1
• Time in Patent Office: 2,612 Days
• Field of Search: None
• US Class Current: 348/208.14
• CPC Class Codes: G01C 11/025 by scanning the object
• CPC Class Codes: G02B 27/646 compensating for small deviations, e.g. due to vibration or shake movement of one or more optical elements for control of motion blur in cameras, projectors or printers G03B2205/0007; image stabilisation in cameras peculiar to the presence or use of an electronic image sensor H04N23/68
• CPC Class Codes: G03B 15/006 Apparatus mounted on flying objects
• CPC Class Codes: H04N 23/68 for stable pick-up of the scene, e.g. compensating for camera body vibrations
• CPC Class Codes: H04N 23/681 Motion detection
• CPC Class Codes: H04N 23/682 Vibration or motion blur correction

Abstract

A method and system for stabilizing images being taken by a video camera using electromechanical stabilization. The stabilization system performs inter-frame stabilization based on the velocity of a vehicle on which the video camera is mounted and the pan rate of a line-of-sight controller of the video camera. The inter-frame stabilization is performed by a software component by moving a display area (or viewport) within a larger image area. The stabilization system converts an inter-frame stabilization adjustment into a pan rate adjustment so that the line-of-sight controller will keep the desired object within the image area of the camera.

Claims

1. A method for stabilizing an image of an object being taken from a video camera, the video camera being moved by a transport mechanism and being controlled by a line-of-sight controller, the line-of-sight controller having an orientation and an angular velocity, the method comprising:

receiving a plurality of images of the object; and

for each of the plurality of received images,

receiving a distance from the video camera to the object,

determining a difference between the location of the object within the image and the location of the object within a previously captured image,

calculating an inter-frame stabilization adjustment based on the distance to the object and the difference between the location of the object within the image and the location of the object within a previously captured image, the inter-frame stabilization adjustment for adjusting the position of a display area of the received images,

adjusting the position of the display area of the received images based on the inter-frame stabilization adjustment, wherein the received images are larger than the display area and the adjusting moves the display area relative to the received images, and

controlling the line-of-sight controller at least in part by,

calculating a line-of-sight adjustment for the line-of-sight controller based on the inter-frame stabilization adjustment,

adjusting the orientation of the line-of-sight controller based on the calculated line-of-sight adjustment,

calculating an angular velocity for the line-of-sight controller based on the inter-frame stabilization adjustment, and

setting the angular velocity of the line-of-sight controller to the calculated angular velocity.

2. The method of claim 1 wherein the transport mechanism is an airborne vehicle.

3. The method of claim 1 wherein the line of sight of the video camera is derived from the line-of-sight controller.

4. The method of claim 1 wherein the inter-frame stabilization adjustment specifies the number of pixels in scan and tilt directions.

5. The method of claim 1 wherein the controlling of the line-of-sight controller specifies rate of scan and tilt movement.

6. The method of claim 1 wherein the distance to the object is provided by a range finder.

7. The method of claim 1 wherein the distance to the object is calculated based on the line of sight of the video camera and the difference in altitude of the object and the video camera.

8. The method of claim 1 wherein the velocity of the transport mechanism is relative to the object.

9. The method of claim 1 wherein the velocity of the transport mechanism is relative to an earth frame of reference.

10. The method of claim 1 wherein the calculated inter-frame stabilization adjustment factors in field of view of the display.

11. The method of claim 1, further comprising:

receiving an indication of a user-specified image flow; and

controlling the line-of-sight controller in accordance with the user-specified image flow so that the object moves relative to the display area.

12. A method for stabilizing an image of an object being taken from a video camera, the video camera being moved by a transport mechanism and being controlled by a line-of-sight controller, the line-of-sight controller having an orientation and a pan rate, the image being displayed on a display device, the method comprising:

determining a difference in the location of the object within the image from one frame to the next frame;

adjusting the display of the image based on the determined difference to remove small-amplitude jitter; and

controlling the line-of-sight controller by,

calculating a line-of-sight adjustment for the line-of-sight controller based at least in part on the determined difference,

adjusting the orientation of the line-of-sight controller based on the calculated line-of-sight adjustment to account for large-amplitude jitter,

calculating a pan rate for the line-of-sight controller based at least in part on the determined difference, and

setting the pan rate of the line-of-sight controller to the calculated pan rate.

13. The method of claim 12 wherein the determining of the difference includes analyzing scan and tilt rate of the line-of-sight controller.

14. The method of claim 12 wherein the determining of the difference includes analyzing velocity of the transport mechanism.

15. The method of claim 12 wherein the determining of the difference includes analyzing line of sight of the video camera.

16. The method of claim 12 wherein the determining of the difference includes analyzing orientation of the video camera and the transport mechanism.

17. The method of claim 12 wherein the determining of the difference includes recognizing the object within the images.

18. The method of claim 12 wherein controlling the line-of-sight controller includes compensating for a user-specified image flow.

19. A method for stabilizing images being taken from a video camera mounted on a moving vehicle, the video camera having a line of sight being controlled by a line-of-sight controller, the line-of-sight controller having an orientation and a rate of rotation, the method comprising:

calculating initial coordinates for a viewport, the viewport corresponding to a portion of an image that is to be displayed;

calculating inter-frame stabilization adjustments based on the change in location of an object in a succession of image frames to account for a velocity of the vehicle, the inter-frame stabilization adjustments used to electronically move the viewport from one frame to the next frame;

moving the viewport in accordance with the calculated inter-frame stabilization adjustments so that the viewport does not remain centered relative to the images taken from the video camera;

displaying a portion of an image corresponding to the moved viewport; and

controlling the line-of-sight controller at least in part by,

calculating a line-of-sight adjustment for the line-of-sight controller based on the inter-frame stabilization adjustments,

adjusting the orientation of the line-of-sight controller in accordance with the calculated line-of-sight adjustment,

calculating a rate of rotation for the line-of-sight controller based on the inter-frame stabilization adjustments, and

setting the rate of rotation of the line-of-sight controller to the calculated rate of rotation.

20. The method of claim 19 wherein the calculating of the inter-frame stabilization adjustments factors in scan and tilt rate of the line-of-sight controller.

21. The method of claim 19 wherein the calculating of the inter-frame stabilization adjustments factors in line of sight of the video camera.

22. The method of claim 19 wherein the calculating of the inter-frame stabilization adjustments factors in orientation of the video camera and the vehicle.

23. The method of claim 19 wherein the calculating of the inter-frame stabilization adjustments includes recognizing an object within the images.

24. The method of claim 19 wherein the calculated line-of-sight adjustment specifies a scan and tilt rate for the line-of-sight controller.

25. A method in a camera stabilization system for stabilizing the display of images received from a video camera attached to an aircraft and controlled by a gimbal-based line-of-sight controller, the method comprising:

receiving a first image from the video camera;

receiving a second image from the video camera;

determining the position of an object in the first image;

determining the position of the object in the second image;

determining an image pixel offset in the scan direction, IPO(S), based on the difference in the position of the object in the first and second images;

determining an image pixel offset in the tilt direction, IPO(T), based on the difference in the position of the object in the first and second images;

determining a pixel offset in the scan direction, PO(S), based on IPO(S);

determining a pixel offset in the tilt direction, PO(T), based on IPO(T);

adjusting the display of an image on a display device of the camera stabilization system based on PO(S) and PO(T);

converting PO(S) to a corresponding scan angle based on the field of view of the video camera;

converting PO(T) to a corresponding tilt angle based on the field of view of the video camera;

adjusting a scan rate of the line-of-sight controller based on the scan angle;

adjusting a tilt rate of the line-of-sight controller based on the tilt angle; and

determining aircraft pixel offsets caused by the movement of the aircraft by,

receiving an indication of the velocity of the aircraft in the earth reference frame, V_aircraft^E,

receiving a matrix, C_BE, corresponding to the orientation of the aircraft in the earth reference frame,

receiving a matrix, C_CB, corresponding to the orientation of the video camera,

calculating a transformation matrix, C_CE, for transforming from the earth reference frame to the camera reference frame, wherein C_CE=C_CBC_BE,

calculating a line of sight, L^E, of the video camera in the earth reference frame, wherein L^E=C_CE^T(1,0,0)^T,

determining the distance, K, to an object at the center of the image,

determining the velocity of the aircraft in the camera reference frame, V_aircraft^C, wherein V_aircraft^C=C_CE*V_aircraft^E,

calculating a normalized velocity of the aircraft V_aircraft^C=V_aircraft^C/K,

calculating a first difference in scan units Δ

S₁^C, wherein Δ

S₁^C=V_aircraft^C(S)*Δ

T, wherein V_aircraft^C(S) corresponds to the normalized velocity of the aircraft in the scan direction, and wherein Δ

T corresponds to a frame refresh period,

calculating a first difference in tilt units Δ

T₁^C, wherein Δ

T₁^C=V_aircraft^C(T)*Δ

T, wherein V_aircraft^C(T) corresponds to the normalized velocity of the aircraft in the tilt direction,

calculating an aircraft pixel offset in the scan direction APO(S), wherein APO(S)=Δ

S₁^C*P/Z, wherein P corresponds to a pixel density associated with the video camera, and wherein Z corresponds to a zoom factor associated with the video camera,

calculating an aircraft pixel offset in the tilt direction APO(T), wherein APO(T)=Δ

T₁^C*P/Z

wherein PO(S) is determined based on IPO(S) and APO(S), and

wherein PO(T) is determined based on IPO(T) and APO(T)

so that both the display of the image and the line-of-sight controller are adjusted based on IPO(S) and IPO(T).

26. The method of claim 25, further comprising:

determining camera pixel offsets caused by the rotation of the video camera by,

receiving the instantaneous camera scan rate IS,

receiving the instantaneous camera tilt rate IT,

calculating a second difference in scan units, Δ

S₂^C, wherein Δ

S₂^C=IS*Δ

T,

calculating a second difference in tilt units, Δ

T₂^C, wherein Δ

T₂^C=IT*Δ

T,

calculating a camera pixel offset in the scan direction, CPO(S), based on Δ

S₂^C,

calculating a camera pixel offset in the tilt direction, CPO(T), based on Δ

T₂^C

wherein PO(S) is determined based on IPO(S), APO(S), and CPO(S), and

wherein PO(T) is determined based on IPO(T), APO(T), and CPO(T).