Method and apparatus for fixating a camera on a target point using image alignment

• US 6,396,961 B1
• Filed: 08/31/1998
• Issued: 05/28/2002
• Est. Priority Date: 11/12/1997
• Status: Expired due to Term

• Current Assignee: SRI International, Inc.
• Original Assignee: Sarnoff Corporation (SRI International, Inc.)
• Inventors: Mandelbaum, Robert, Eledath, Jayakrishnan, Hansen, Michael Wade, Mishra, Deepam, Wixson, Lambert Ernest
• Primary Examiner(s): Boudreau, Leo
• Assistant Examiner(s): PATEL, KANJIBHAI B
• Application Number:
  US09/143,910
• Time in Patent Office: 1,366 Days
• Field of Search: 382/293,276,294,107,106,154,103,255,295,296,282,284 348/174,135,137,580,584,598 345/419,425 701/25
• US Class Current: 382/294
• CPC Class Codes: G06T 7/70 Determining position or orientation of objects or cameras camera calibration G06T7/80
• CPC Class Codes: G06T 7/80 Analysis of captured images to determine intrinsic or extrinsic camera parameters, i.e. camera calibration
• CPC Class Codes: H04N 23/68 for stable pick-up of the scene, e.g. compensating for camera body vibrations
• CPC Class Codes: H04N 23/6811 based on the image signal
• CPC Class Codes: H04N 23/682 Vibration or motion blur correction
• CPC Class Codes: H04N 23/687 by shifting the lens or sensor position

Abstract

A method for fixating a camera, mounted on a motorized mount, on a target point. The method includes receiving a plurality of images representative of a scene; selecting, within the plurality of images, a first display reference image containing the target point at a first coordinate location; estimating, in a current image within the plurality of images, a transformation that maps the current image to the first display reference image; estimating a second coordinate location of the target point in the current image using the transformation; computing an image slip between the target point in the current image and the target point in the first display reference image; converting the image slip into an angular correction for fixating the camera; modifying the angular correction by using closed-loop control with the motorized mount; and warping the current image using the transformation to align the current image with the first display reference image. This warping step provides a stable view of the scene even in the presence of camera movements that cannot be eliminated using mechanical stabilization, and in the presence of motor control lags and oscillations due to measurement and actuator latencies.

Claims

1. A method for fixating a camera, mounted on a motorized mount, on a target point comprising the steps of:

a) receiving a plurality of images representative of a scene;

b) selecting, within the plurality of images, a first display reference image containing the target point at a first coordinate location;

c) estimating, in a current image within the plurality of images, a transformation that maps the current image to the first display reference image;

d) estimating a second coordinate location of the target point in the current image using the transformation;

e) computing an image slip between the target point in the current image and the target point in the first display reference image;

f) converting the image slip into an angular correction for fixating the camera; and

g) implementing the angular correction by using closed-loop control with the motorized mount.

2. The method of claim 1, wherein step (e) includes computing the image slip by subtracting the second coordinate location from the location of a point in the current image.

3. The method of claim 2, wherein step (g) includes closed-loop control using one of a proportional plus integral plus derivative (PID) control system and a Kalman filtering based control system.

4. The method of claim 1 wherein step (c) includes estimating the transformation using one of an affine model algorithm and a projective model algorithm.

5. The method of claim 1 further comprising the step of:

(i) determining a magnitude of selected coefficients of the transformation, and replacing the first display reference image with a second display reference image when the determined magnitude exceeds a threshold.

6. The method of claim 1 wherein a targeting reference image is selected having known target point coordinates, and the method further includes the steps of:

(i) estimating a further transformation that maps the targeting reference image to the display reference image; and

(j) adjusting the target point of the display reference image to the target point of the targeting reference image using the further transformation.

7. The method of claim 6 wherein the targeting reference image is obtained from a database, and the database includes targeting reference images having been synthesized from one of a previous image, a known image mosaic, and a known geometric model of a scene.

8. The method of claim 6 wherein the targeting reference image is obtained from a database, and the database includes targeting reference images having been synthesized from a previous image and a known geometric model of a scene.

9. The method of claim 6 wherein the targeting reference image is obtained from a database, and the database includes targeting reference images having been synthesized from a known image mosaic and a known geometric model of a scene.

10. The method of claim 1 wherein step (f) includes

estimating a relative orientation between the camera and the target point in a coordinate system;

and

providing the angular correction in the same coordinate system.

11. The method of claim 10 wherein the coordinate system is a world coordinate system and the angular correction uses Horn'"'"'s quaternion method.

12. The method of claim 1 further including the step of:

h) warping the current image using the transformation to align the current image with the first display reference image.

13. The method of claim 12, wherein step (h) includes sending the warped image to one of a display for viewing and an image processor for processing.

14. In fixating a camera on a target point, a method for maintaining a database of targeting reference images, comprising the steps of:

a) receiving a plurality of images representative of a scene;

b) selecting a targeting reference image from the database;

c) estimating, in a current image within the plurality of images, a first transformation that maps the current image to the targeting reference image;

d) comparing the first transformation to other transformations between the current image and the other targeting reference images in the database; and

e) adding the current image to the database, if the first transformation is sufficiently different from the other transformations between the current image and the other targeting reference images in the database.

15. The method of claim 14 further including the following steps:

f) estimating a coordinate location of the target point in the current image using the transformation;

g) computing an image slip between the target point in the current image and the target point in the targeting reference image;

h) converting the image slip into an angular correction for fixating the camera; and

i) implementing the angular correction by using closed-loop control with a motorized mount receiving the camera.

16. The method of claim 15 further including the step of:

j) warping the current image using the first transformation to align the current image with the targeting reference image.

17. The method of claim 14 wherein step (e) includes the step of adding the current image to the database if a magnitude of selected coefficients of the first transformation is greater by a threshold value than any other magnitude of selected coefficients of transformations between the current image and each of the other targeting reference images.

18. Apparatus for fixating a camera on a target point comprising:

a motorized mount for fixating the camera;

a computer processor for processing a plurality of images representative of a scene including;

means for selecting within the plurality of images a first display reference image containing the target point at a first coordinate location;

means for deriving, from a current image within the plurality of images, a transformation that maps the current image to the first display reference image;

means for estimating a second coordinate location of the target point in the current image using the transformation;

means for computing an image slip between a point in the current image and a point in the first display reference image;

means for converting the image slip into an angular correction for fixating the camera; and

a closed loop control system which implements the angular correction by controlling the motorized mount.

19. The apparatus of claim 18 wherein means for modifying the angular correction includes a closed-loop control system between the computer processor and the motorized mount, the closed-loop control system using one of a proportional plus integral plus derivative (PID) control algorithm and a Kalman filtering based control algorithm.

20. The apparatus of claim 18 wherein the transformation is one of an affine model algorithm and a projective model algorithm.

21. The apparatus of claim 18 including means for replacing the first display reference image with a second display reference image when a magnitude of selected coefficients of the transformation exceeds a predetermined magnitude.

22. The apparatus of claim 18 including:

means for warping the current image using the transformation to align the current image with the first display reference image.

23. The apparatus of claim 22 including:

an output device for presenting the warped current image as the output image of the camera.

24. A computer-readable medium having stored thereon a plurality of instructions, the plurality of instructions including instructions which, when executed by a processor cause the processor to perform the steps of:

a) receiving a plurality of images representative of a scene;

b) selecting, from within the plurality of images, a first display reference image containing a target point at a first coordinate location;

c) estimating in a current image, within the plurality of images, a transformation that maps the current image to the first display reference image;

d) estimating a second coordinate location of the target point in the current image using the transformation;

e) computing an image slip between a point in the current image and a point in the first display reference image;

f) converting the image slip into an angular correction for fixating a camera mounted on motorized mount; and

g) implementing the angular correction by using closed-loop control with the motorized mount.

25. The computer readable medium of claim 24, wherein the current image is presented to a display for viewing.

26. The computer readable medium of claim 24, wherein the current image is presented to an image processor for further processing.

27. The method of claim 24 further including the step of:

h) warping the current image using the transformation for alignment with the display reference image.