Cooperative nesting of mechanical and electronic stabilization for an airborne camera system

• US 20040183917A1
• Filed: 12/01/2003
• Published: 09/23/2004
• Est. Priority Date: 01/17/2003
• Status: Active Grant

• Current Assignee: Insitu Incorporated (The Boeing Co.)
• Original Assignee: Insitu Incorporated (The Boeing Co.)
• Inventors: McGeer, Brian T., von Flotow, Andreas H., Mercadal, Mathieu
• Granted Patent: US 7,876,359 B2
• Time in Patent Office: Days
• US Class Current: 348/208.99
• CPC Class Codes: G01C 11/025 by scanning the object
• CPC Class Codes: G02B 27/646 compensating for small deviations, e.g. due to vibration or shake movement of one or more optical elements for control of motion blur in cameras, projectors or printers G03B2205/0007; image stabilisation in cameras peculiar to the presence or use of an electronic image sensor H04N23/68
• CPC Class Codes: G03B 15/006 Apparatus mounted on flying objects
• CPC Class Codes: H04N 23/68 for stable pick-up of the scene, e.g. compensating for camera body vibrations
• CPC Class Codes: H04N 23/681 Motion detection
• CPC Class Codes: H04N 23/682 Vibration or motion blur correction

Abstract

A method and system for stabilizing images being taken by a video camera using electromechanical stabilization. The stabilization system performs inter-frame stabilization based on the velocity of a vehicle on which the video camera is mounted and the pan rate of a line-of-sight controller of the video camera. The inter-frame stabilization is performed by a software component by moving a display area (or viewport) within a larger image area. The stabilization system converts an inter-frame stabilization adjustment into a pan rate adjustment so that the line-of-sight controller will keep the desired object within the image area of the camera.

Claims

1. A method for stabilizing an image of an object being taken from a video camera, the video camera being moved by a transport mechanism and being controlled by a line-of-sight controller, the method comprising:

receiving a plurality of images of the object; and

for each of the plurality of received images,

receiving a velocity and orientation of the transport mechanism;

receiving an orientation of the camera relative to the transport mechanism;

receiving a scan and tilt rate of the camera;

receiving a distance from the camera to the object;

calculating an inter-frame stabilization adjustment based on the velocity and orientation of the transport mechanism, the orientation of the camera, the scan and tilt rate of the camera, and distance to the object;

calculating a line-of-sight adjustment for the line-of-sight controller based on the inter-frame stabilization; and

controlling the line-of-sight controller in accordance with the calculated line-of-sight adjustment.

2. The method of claim 1 wherein the transport mechanism is an airborne vehicle.

3. The method of claim 1 wherein the line of sight of the camera is derived from the line-of-sight controller.

4. The method of claim 1 including adjusting the display of the images based on the inter-frame stabilization adjustment.

5. The method of claim 4 wherein the received images are larger than the displayed images and the adjusting of the display of the images moves an area of a displayed image within the received image.

6. The method of claim 4 wherein the inter-frame stabilization adjustment specifies the number of pixels in scan and tilt directions.

7. The method of claim 1 wherein the controlling of the line-of-sight controller specifies rate of scan and tilt movement.

8. The method of claim 1 wherein the distance to the object is provided by a range finder.

9. The method of claim 1 wherein the distance to the object is calculated based on the line of sight of the camera and the difference in altitude of the object and the camera.

10. The method of claim 1 wherein the velocity of the transport mechanism is relative to the object.

11. The method of claim 1 wherein the velocity of the transport mechanism is relative to an earth frame of reference.

12. The method of claim 1 wherein the calculated inter-frame stabilization adjustment factors in field of view of the display.

13. A method for stabilizing an image of an object being taken from a video camera, the video camera being moved by a transport mechanism and being controlled by a line-of-sight controller, the method comprising:

determining a difference in the location of the object within the image from one frame to the next frame;

calculating an inter-frame stabilization adjustment based on the determined difference;

calculating a line-of-sight adjustment for the line-of-sight controller based on the inter-frame stabilization; and

controlling the line-of-sight controller in accordance with the calculated line-of-sight adjustment.

14. The method of claim 13 wherein the determining of the difference includes analyzing scan and tilt rate of the line-of-sight controller.

15. The method of claim 13 wherein the determining of the difference includes analyzing velocity of the transport mechanism.

16. The method of claim 13 wherein the determining of the difference includes analyzing line of sight of the camera.

17. The method of claim 13 wherein the determining of the difference includes analyzing orientation of the camera and the transport mechanism.

18. The method of claim 13 wherein the determining of the difference includes recognizing the object within the images.

19. The method of claim 13 wherein the calculated line-of-sight adjustment specifies a scan and tilt rate for the line-of-sight controller.

20. An apparatus for stabilizing imagery from an airborne video camera, comprising:

a mechanical line-of-sight controller for controlling line of sight of the video camera at a specified line-of-sight adjustment rate; and

an electronic stabilization component that provides frame-to-frame image stabilization based on the specified line-of-sight adjustment rate and that provides to the mechanical line-of-sight controller a new line-of-sight adjustment rate derived from an amount of frame-to-frame image stabilization.

21. The apparatus of claim 20 wherein the amount of frame-to-frame image stabilization is additionally based on velocity and orientation of an airborne transport vehicle, orientation of the camera relative to the airborne transport vehicle, and distance from the camera to an object within the image.

22. The apparatus of claim 20 wherein the line-of-sight adjustment rate includes a scan rate and a tilt rate.

23. The apparatus of claim 20 wherein an image received from the video camera is larger than a displayed image and the electronic stabilization component provides frame-to-frame image stabilization by adjusting the location of the displayed image within a received image.

24. The apparatus of claim 20 wherein the specified line-of-sight adjustment rate includes a user-specified image flow.

25. The apparatus of claim 20 wherein the mechanical line-of-sight controller is a motorized gimbal system.

26. The apparatus of claim 20 wherein the frame-to-frame adjustment keeps an object of the images at the same location when displayed.

27. An apparatus for stabilizing imagery from an airborne video camera, comprising:

a mechanical line-of-sight controller for controlling line of sight of the video camera at a specified line-of-sight adjustment rate; and

an electronic stabilization component that provides frame-to-frame image stabilization based on location of an object within the images and that provides to the mechanical line-of-sight controller a new line-of-sight adjustment rate derived from an amount of frame-to-frame image stabilization.

28. The apparatus of claim 27 wherein an amount of frame-to-frame image stabilization is additionally based on velocity and orientation of an airborne transport vehicle, orientation of the camera relative to the airborne transport vehicle, and distance from the camera to the object within the image.

29. The apparatus of claim 27 wherein an amount of frame-to-frame image stabilization is additional based on the specified line-of-sight adjustment rate.

30. The apparatus of claim 27 wherein the line-of-sight adjustment rate includes a scan rate and a tilt rate.

31. The apparatus of claim 27 wherein an image received from the video camera is larger than a displayed image and the electronic stabilization component provides frame-to-frame image stabilization by adjusting the location of the displayed image within a received image.

32. The apparatus of claim 27 wherein the specified line-of-sight adjustment rate includes a user-specified image flow.

33. The apparatus of claim 27 wherein the mechanical line-of-sight controller is a motorized gimbal system.

34. The apparatus of claim 27 wherein the frame-to-frame adjustment keeps the object of the images at the same location when displayed.

35. A method for stabilizing images being taken from a video camera mounted on a moving vehicle, the camera having a line of sight being controlled by a line-of-sight controller, the method comprising:

calculating inter-frame stabilization adjustments to account for velocity of the vehicle;

displaying the images in accordance with the calculated inter-frame stabilization adjustments;

calculating line-of-sight adjustments for the line-of-sight controller based on the inter-frame stabilization adjustments; and

controlling the line-of-sight controller in accordance with the calculated line-of-sight adjustments.

36. The method of claim 35 wherein the calculating of the inter-frame stabilization adjustments factors in scan and tilt rate of the line-of-sight controller.

37. The method of claim 35 wherein the calculating of the inter-frame stabilization adjustments factors in line of sight of the camera.

38. The method of claim 35 wherein the calculating of the inter-frame stabilization adjustments factors in orientation of the camera and the vehicle.

39. The method of claim 35 wherein the calculating of the inter-frame stabilization adjustments includes recognizing an object within the images.

40. The method of claim 35 wherein the calculated line-of-sight adjustment specifies a scan and tilt rate for the line-of-sight controller.