Estimating distance to an object using a sequence of images recorded by a monocular camera

• US 8,164,628 B2
• Filed: 10/30/2006
• Issued: 04/24/2012
• Est. Priority Date: 01/04/2006
• Status: Active Grant

• Current Assignee: MobilEye Vision Technologies Ltd. (Intel Corporation)
• Original Assignee: Mobileye Technologies Limited (Intel Corporation)
• Inventors: Stein, Gideon P., Ferencz, Andras D., Avni, Ofer
• Primary Examiner(s): Chang, Jungwon
• Application Number:
  US11/554,048
• Publication Number:
  US 20070154068A1
• Time in Patent Office: 2,003 Days
• Field of Search: 340/425.5, 340/435, 382/104, 382/103, 348/148, 348/135, 701/301
• US Class Current: 348/148
• CPC Class Codes: B60R 11/04 Mounting of cameras operative during drive; Arrangement of controls thereof relative to the vehicle
• CPC Class Codes: B60R 2001/1253 with cameras, video cameras or video screens
• CPC Class Codes: B60R 2300/105 using multiple cameras
• CPC Class Codes: B60R 2300/802 for monitoring and displaying vehicle exterior blind spot views
• CPC Class Codes: B60R 2300/8093 for obstacle warning
• CPC Class Codes: G01B 11/026 by measuring distance between sensor and object G01B11/0608 takes precedence
• CPC Class Codes: G01B 11/14 for measuring distance or clearance between spaced objects or spaced apertures G01B11/26 takes precedence; rangefinders G01C3/00
• CPC Class Codes: G01C 22/00 Measuring distance traversed on the ground by vehicles, persons, animals or other moving solid bodies, e.g. using odometers, using pedometers
• CPC Class Codes: G01C 3/00 Measuring distances in line of sight; Optical rangefinders tapes, chains or wheels for measuring length G01B3/00; active triangulation systems, i.e. using the transmission and reflection of electromagnetic waves other than radio waves, G01S17/48
• CPC Class Codes: G01C 3/22 using a parallactic triangle with variable angles and a base of fixed length at, near, or formed by the object
• CPC Class Codes: G01S 11/12 using electromagnetic waves other than radio waves
• CPC Class Codes: G01S 17/93 for anti-collision purposes
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/20024 Filtering details
• CPC Class Codes: G06T 2207/30196 Human being; Person
• CPC Class Codes: G06T 2207/30252 Vehicle exterior; Vicinity of vehicle
• CPC Class Codes: G06T 2207/30256 Lane; Road marking
• CPC Class Codes: G06T 2207/30261 Obstacle
• CPC Class Codes: G06T 7/55 from multiple images
• CPC Class Codes: G06V 20/58 Recognition of moving objects or obstacles, e.g. vehicles or pedestrians; Recognition of traffic objects, e.g. traffic signs, traffic lights or roads

Abstract

In a computerized system including a camera mounted in a moving vehicle. The camera acquires consecutively in real time image frames including images of an object within the field of view of the camera. Range to the object from the moving vehicle is determined in real time. A dimension, e.g. a width, is measured in the respective images of two or more image frames, thereby producing measurements of the dimension. The measurements are processed to produce a smoothed measurement of the dimension. The dimension is measured subsequently in one or more subsequent frames. The range from the vehicle to the object is calculated in real time based on the smoothed measurement and the subsequent measurements. The processing preferably includes calculating recursively the smoothed dimension using a Kalman filter.

Claims

1. In a computerized system including a camera mounted in a moving vehicle, wherein the camera acquires consecutively in real time a plurality of image frames including respectively a plurality of images of an object within a field of view of the camera, a method for determining a range of the object from the moving vehicle, the method comprising the steps of:

(a) measuring in at least two of said image frames a dimension in the respective images of the object thereby producing at least two measurements of said dimension;

(b) processing said at least two measurements thereby producing a smoothed measurement of said dimension;

(c) measuring said dimension in at least one of said image frames subsequent to said at least two image frames, thereby producing at least one subsequent measurement of said dimension; and

(d) calculating the range in real time based on said smoothed measurement and said at least one subsequent measurement.

2. The method, according to claim 1, wherein range is determined solely from information derived from the images.

3. The method, according to claim 1, wherein said dimension is a real dimension of said object.

4. The method, according to claim 1, wherein said processing includes calculating recursively said smoothed dimension using a Kalman filter.

5. The method, according to claim 1, wherein the object is a second vehicle and wherein said dimension is a width as measured in the images of said second vehicle.

6. The method, according to claim 1, wherein the object is a pedestrian and wherein said dimension is a height of said pedestrian.

7. The method, according to claim 1, wherein said smoothed measurement is a smoothed width W_v, wherein said at least one subsequent measurement is a width w_i, wherein f is a focal length of the camera, wherein the range is Z calculated by:

8. The method, according to claim 1, wherein said (a) measuring and said (c) measuring of said dimension are performed by:

(i) detecting at least one horizontal edge in each of said images;

(ii) detecting two vertical edges based on end points of said at least one horizontal edge; and

(iii) detecting a lower edge, by detecting respective lower ends of said two vertical edges, wherein said dimension is a width measured between said lower ends.

9. The method, according to claim 8, wherein said (i) detecting at least one horizontal edge is performed by mapping grayscale levels of a plurality of pixels in at least one of said image frames, thereby classifying each of said picture elements as imaging a portion of a road surface or not imaging a portion of a road surface.

10. The method, according to claim 8, wherein said (i) detecting at least one horizontal edge is performed at sub-pixel accuracy by processing over said at least two image frames.

11. The method, according to claim 8, wherein said (ii) detecting two vertical edges is performed at sub-pixel accuracy by processing over said at least two image frames.

12. The method, according to claim 8, wherein said (iii) detecting a lower edge, is performed at sub-pixel accuracy by processing over said at least two image frames.

13. The method, according to claim 8, wherein said lower edge coincides with a portion of an image of a road surface, whereby said lower edge is a bottom edge.

14. The method, according to claim 8, wherein a height of said lower edge is determined based on at least one imaged feature selected from the group consisting of:

an image of a shadow on a road surface and an image of self-illumination of the object on a road surface.

15. The method, according to claim 1, whereby said calculating the range is dependent on an imaged height of the horizon, further comprising the step of:

(e) locating said imaged height of the horizon.

16. The method, according to claim 15, wherein said locating is refined by at least one technique selected from the group consisting of:

(i) measuring the shape of the imaged road,

(ii) detecting the vanishing point from at least one road marking selected from the group consisting of;

lane structure, lane marking, and horizontal line marking;

(iii) detecting relative motion of imaged points and velocity of the moving vehicle,

(iv) compensating for pitch angle variations of the camera, and

(v) detecting ego motion of the camera.

17. A computerized system including a camera mounted in a moving vehicle, wherein the camera acquires consecutively in real time a plurality of image frames including respectively a plurality of images of an object within a field of view of the camera, wherein the computerized system performs the method steps of claim 1.

18. A computerized system including a camera mounted in a moving vehicle, wherein the camera acquires consecutively in real time a plurality of image frames including respectively a plurality of images of an object within a field of view of the camera, wherein the system determines in real time a range from the moving vehicle to the object, the system comprising:

(a) a measurement mechanism adapted to measure in at least two of said image frames a dimension in the respective images of the object thereby producing a series of at least two measurements of said dimension; and

(b) a processor which processes said at least two measurements thereby producing a smoothed measurement of said dimension;

wherein said measurement mechanism is adapted to measure said dimension in at least one of said image frames subsequent to said at least two image frames, thereby producing at least one subsequent measurement of said dimension; and

wherein said processor is adapted to calculate the range in real time based on said smoothed measurement and said at least one subsequent measurement.

19. The computerized system, according to claim 18, wherein said object is selected from the group consisting of:

pedestrian, motorcycle, automotive vehicle, animal and bicycle.

20. The computerized system, according to claim 18, wherein said measurement mechanism is adapted to perform sub-pixel measurements on said image frames.