Apparatus and method for self-calibrating visual time-to-contact sensor

• US 5,559,695 A
• Filed: 12/27/1994
• Issued: 09/24/1996
• Est. Priority Date: 12/27/1994
• Status: Expired due to Term

• Current Assignee: Hughes Electronics Corporation (Rtx Corporation)
• Original Assignee: Hughes Aircraft Company (Rtx Corporation)
• Inventors: Daily, Michael J.
• Primary Examiner(s): Teska, Kevin J.
• Assistant Examiner(s): LOUIS JACQUES, JACQUES H
• Application Number:
  US08/364,158
• Time in Patent Office: 637 Days
• Field of Search: 364/424.01, 364/424.02, 364/514, 364/516, 364/517, 364/468, 348/219, 348/116, 348/355, 348/349, 348/79, 348/347, 348/31, 348/229, 348/241, 348/220, 348/321, 348/14, 348/20, 348/15, 348/674, 395/162, 395/81-84, 395/94, 395/904-905, 226/15-20, 358/3, 358/442, 358/406, 358/455, 352/140, 354/402, 354/81, 354/400, 354/140
• US Class Current: 701/1
• CPC Class Codes: G01S 11/12 using electromagnetic waves other than radio waves
• CPC Class Codes: G05D 1/0253 extracting relative motion information from a plurality of images taken successively, e.g. visual odometry, optical flow determining position or orientation from images G06T7/70
• CPC Class Codes: G05D 1/0276 using signals provided by a source external to the vehicle involving a plurality of vehicles G05D1/0287; automatically controlling vehicle speed responsive to externally generated signals B60K31/0058

Abstract

The invention relates to a method and a self-calibrating visual time-to-contact (TTC) sensor for a mobile manned or autonomous unmanned vehicle which operates in a high speed manner and which permits the continuous, adaptive motion of the vehicle through the vehicle'"'"'s environment. More specifically, the invention relates to an apparatus and method for novel use of active sensor control for aligning a camera, mounted on the vehicle, to track the direction of motion of the vehicle in order to successfully navigate in complex environments and to simplify difficult processing steps previously attempted through algorithmic means. Rather than attempting to explicitly find a focus of expansion (FOE) from image data, the invention continuously calibrates the sensor to point in the direction of the FOE, using weighted global average of the horizontal and vertical component of the optical flow. The pan and tilt angles of the visual TTC sensor are iteratively changed by the scanning mechanism in the opposite direction by small increments, biased by the global magnitude of the optical flow. In this way, the visual TTC sensor is not constrained to point in any direction at the start, and after several iterations it will point near the FOE and continue to do so as the vehicle moves. By actively centering the FOE in the image sequence, better accuracy is possible. By avoiding the focus of expansion calculation, the method becomes self-calibrating and it is faster and more robust to vibrations and sensor misalignments occurring in changing and complex indoor or outdoor environments.

Claims

1. A self-calibrating visual sensor system, mounted on a mobile vehicle, comprising:

a visual time-to-contact (TTC) sensor that determines a time before said sensor will contact an object in its path, said TTC sensor generating a plurality of image frames descriptive of a travel area in the direction of movement of the mobile vehicle;

an image sensor processor, coupled to said visual TTC sensor, comprising an image memory for storing at least two image frames outputted by the visual TTC sensor;

an image processing unit comprising an image grabber, coupled to the image memory and operable for reading the stored image frames from the image memory, and a visual control means for continuous calibration of the visual TTC sensor to point in the direction of focus of expansion (FOE) by adjusting pan and tilt angles of the visual TTC sensor; and

a video bus between the image processing unit and the image sensor processor for transfer of image frames.

2. The self-calibrating visual sensor system of claim 1, wherein said visual TTC sensor comprises at least one electronic camera, with image digitization capability, generating a plurality of pixels stored in the image memory which have a value indicative of the intensity of brightness in the field of view of said electronic camera.

3. The electronic camera of claim 2, comprising a noise reduction filter to smooth high-frequency noise by generating a weighted sum of the adjacent pixel values.

4. The self-calibrating visual sensor system of claim 2, wherein each said electronic camera is used without pre-calibration of the area of movement and the field of view of the camera.

5. The self-calibrating visual sensor system of claim 2, wherein each said electronic camera is used without predetermining the position of the camera with respect to the mobile vehicle or the obstacles located in the mobile vehicle'"'"'s environment.

6. The self-calibrating visual sensor system of claim 1, wherein said visual TTC sensor further comprises a scanning mechanism, a pan and tilt interface and a vision interface.

7. The self-calibrating visual sensor system of claim 1, wherein said visual control means for continuous calibration of the visual TTC sensor iteratively computes an average optical flow by calculating a weighted global average of the horizontal and vertical components of optical flow for successive image frames, the pan and tilt angles to point the visual TTC sensor closer to the focus of expansion (FOE), and TTC value, for adjusting the course of movement of the mobile vehicle until the desired position is achieved.

8. The self-calibrating visual sensor system of claim 7, wherein said visual control means for continuous calibration of the visual TTC sensor computes the pan and tilt angles by computing optical flow bias and combining it with the average optical flow.

9. The self-calibrating visual sensor system of claim 7, wherein said computation of the average optical flow within said visual control means further comprises smoothing the optical flow with a Gaussian function at each pixel, at high values of the determinant, by replacing values with the weighted average of the neighborhood data.

10. The self-calibrating visual sensor system of claim 1, wherein said calibration is accomplished without explicit calculation of the FOE.

11. A method of self-calibrating a visual time-to-contact (TTC) sensor that determines a time before said sensor will contact an object in its path, said TTC sensor mounted on a mobile vehicle, for obstacle detection or avoidance, comprising the following steps:

(a) active centering of the visual TTC sensor at the focus of expansion FOE using optical flow by continuously calibrating the pan and tilt of the sensor to point in the direction of the FOE;

(b) computing a coarse estimate of time-to-contact (TTC) value along the direction of the mobile vehicle'"'"'s motion, in order to accurately predict the time-to-contact with stationary and moving obstacles in the mobile vehicle'"'"'s immediate path, and enable warning or evasive action; and

(c) continuously repeating the steps (a) to (b) for adjusting the course of movement of the mobile vehicle until the desired position is achieved.

12. A method of self-calibrating a visual time-to-contact (TTC) sensor that determines a time before said sensor will contact an object in its path, said TTC sensor mounted on a mobile vehicle, for continuous calibration of the visual TTC sensor to point in the direction of focus of expansion (FOE), without explicit calculation of the FOE, comprising the following steps:

(a) grabbing an image at time t1;

(b) moving the mobile vehicle forward with approximately linear motion;

(c) grabbing second image at time t2;

(d) computing an average optical flow by calculating a weighted global average of the horizontal and vertical components of the optical flow between time t1 and t2;

(e) computing pan and tilt angles to point the visual TTC sensor closer to the focus of expansion (FOE);

(f) moving the visual TTC sensor with the calculated pan and tilt angles toward the FOE;

(g) computing the TTC value for the image pair for time t1 and t2 and calculating the average of the most recent TTC values, if the FOE is centered in the image; and

(i) iteratively repeating the steps (a) to (g) for adjusting the course of movement of the mobile vehicle until the desired position is achieved.

13. A method of self-calibrating a visual time-to-contact (TTC) sensor, from claim 12, wherein the step (d) for computation of the average optical flow is accomplished by computing second partial spatial and temporal derivatives of the image brightness intensity.

14. A method of self-calibrating a visual time-to-contact (TTC) sensor, from claim 12, wherein the step (g) for determination that the FOE is centered in the image is made when the weighted global average of the horizontal and vertical components of the optical flow are below a threshold.

15. A method of self-calibrating a visual time-to-contact (TTC) sensor, from claim 12, wherein the step (e) for computation of the pan and tilt angles is accomplished by computing optical flow bias and combining it with the average optical flow.

16. A method of self-calibrating a visual time-to-contact (TTC) sensor, from claim 12, wherein the TTC value is computed in the step (g) by calculating distance from the FOE to the image point in the image plane, divided by the magnitude of the optical flow at that distance.

17. A method of self-calibrating a visual time-to-contact (TTC) sensor, from claim 12, wherein the step (a) further comprises smoothing of the input images with a Gaussian smoothing function.

18. A method of self-calibrating a visual time-to-contact (TTC) sensor, from claim 12, wherein the step (d) for computation of the average optical flow further comprises smoothing the optical flow with a Gaussian function at each pixel, at high values of the determinant, by replacing values with the weighted average of the neighborhood data.

19. A method of self-calibrating a visual time-to-contact (TTC) sensor, from claim 12, wherein no steps of pre-calibration are necessary.

20. A method of self-calibrating a visual time-to-contact (TTC) sensor, from claim 12, wherein the continuous calibration is possible regardless of which direction the mobile vehicle is pointing to.

21. A method of self-calibrating a visual time-to-contact (TTC) sensor, from claim 12, wherein the continuous calibration is possible even when the FOE of an image sequence is outside the field of view of the visual TTC sensor, or if the FOE is located within the field of view of the visual TTC sensor but not centered in the image plane.