Apparatus and method for automatically detecting objects

• US 7,747,039 B2
• Filed: 11/29/2005
• Issued: 06/29/2010
• Est. Priority Date: 11/30/2004
• Status: Active Grant

• Current Assignee: Nissan Motor Co., Ltd.
• Original Assignee: Nissan Motor Co., Ltd.
• Inventors: Fujimoto, Kazumi
• Primary Examiner(s): Tucker; Wesley
• Assistant Examiner(s): Bitar; Nancy
• Application Number:
  US10/575,283
• Publication Number:
  US 20080273750A1
• Time in Patent Office: 1,673 Days
• Field of Search: 382/128, 382/103, 382/104, 382/153, 382/107, 348/119, 348/118, 348/148, 701/301, 701/300, 340/436, 340/435, 340/901, 340/904, 340/937
• US Class Current: 382/103
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/30256 Lane; Road marking
• CPC Class Codes: G06T 7/12 Edge-based segmentation
• CPC Class Codes: G06T 7/215 Motion-based segmentation
• CPC Class Codes: G06V 20/58 Recognition of moving objects or obstacles, e.g. vehicles or pedestrians; Recognition of traffic objects, e.g. traffic signs, traffic lights or roads
• CPC Class Codes: G06V 20/588 Recognition of the road, e.g. of lane markings; Recognition of the vehicle driving pattern in relation to the road
• CPC Class Codes: G06V 40/10 Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands

Abstract

A device automatically detects boundary lines on the road from an image captured by a camera mounted on the vehicle. The device includes a controller that performs image processing on the image to compute the velocity information for each pixel in the image, and, on the basis of the computed velocity information for each pixel in the image, extracts the pixels that contain velocity information, detects the oblique lines formed by the extracted pixels, and detects the boundary lines on the road on the basis of the detected oblique lines.

Claims

1. An apparatus for detecting objects in one or more images captured by an image pickup device mounted on a vehicle, comprising:

(a) memory on which is stored pixels of at least one image captured by the image pickup device; and

(b) a controller operatively coupled to the memory and adapted to compute velocity information for each pixel in an image using a sequential series of images;

extract from the image extracted pixels having a velocity component based on the velocity information, wherein the velocity component comprises a movement direction and a movement velocity in a lateral direction;

define regions for detecting a road boundary;

detect oblique lines made of pixels having a velocity component based on grouping those extracted pixels having a velocity component in the regions; and

generate a signal indicative of a road boundary in the image based on the oblique lines.

2. The apparatus of claim 1, wherein the controller is further adapted to judge that oblique lines in the image are road boundaries when the vehicle is traveling and the oblique lines are positioned on the image with bilateral symmetry and respectively comprise pixels having velocity components with different movement directions.

3. The apparatus of claim 1, wherein the controller is further adapted to judge that oblique lines in the image are road boundaries when respective slopes of the oblique lines decrease from a center of the image toward an outside of the image.

4. The apparatus of claim 1, wherein the controller is further adapted to detect a change point where a velocity direction of a detected oblique line changes from one image to a subsequent image, and to judge that the change point is a pitching balance point where a line of sight orientation of the image pickup device is horizontal with respect to a road surface.

5. The apparatus of claim 1, wherein the controller is further adapted to identify a moving object that is approaching a predicted path of the vehicle by grouping those extracted pixels having a same velocity component with the movement direction being from a side toward the predicted path of the vehicle;

and to generate a collision danger signal indicative of a risk of collision between the vehicle and the moving object.

6. The apparatus of claim 5, wherein the controller is further adapted to transform into a real space road model the oblique lines judged as the road boundary and the moving object and to determine the risk of collision between the vehicle and the moving object based on a relative positional relationship between the road boundary and the moving object as established in the real space road model.

7. The apparatus of claim 5, wherein the controller is further adapted to generate the collision danger signal at one of a plurality of values corresponding to collision risk levels.

8. The apparatus of claim 5, further comprising an audio alert operatively coupled to the controller and activated by the collision danger signal.

9. The apparatus of claim 5 further comprising an automatic braking device operatively coupled to the controller and activated by the collision danger signal.

10. A vehicle, comprising:

(a) a camera mounted including an image-pickup element generating as output a plurality of images;

(b) memory on which is stored the plurality of images generated by the image pickup element; and

(c) a controller operatively coupled to the memory and adapted to compute velocity information for each pixel in an image using a sequential series of images;

extract those pixels having a velocity component based on the velocity information, wherein the velocity component comprises a movement direction and a movement velocity in a lateral direction;

define regions for detecting a road boundary;

detect oblique lines based on grouping those extracted pixels having a velocity component in the regions; and

generate a signal indicative of a road boundary in the image based on the oblique lines.

11. The vehicle of claim 10, wherein the controller is further adapted to judge that oblique lines in the image are road boundaries when the vehicle is traveling and the oblique lines are positioned on the image with bilateral symmetry and respectively comprise pixels having velocity components with different movement directions.

12. The vehicle of claim 10, wherein the controller is further adapted to judge that oblique lines in the image are road boundaries when respective slopes of the oblique lines decrease from a center of the image toward an outside of the image.

13. The vehicle of claim 10, wherein the controller is further adapted to detect a change point where a velocity direction of a detected oblique line changes from one image to a subsequent image, and to judge that the change point is a pitching balance point where a line of sight orientation of the camera is horizontal with respect to a road surface.

14. The vehicle of claim 10, wherein the controller is further adapted to identify a moving object that is approaching a predicted path of the vehicle by grouping those extracted pixels having a same velocity component with the movement direction being from a side toward the predicted path of the vehicle;

and to generate a collision danger signal indicative of a risk of collision between the vehicle and the moving object.

15. The vehicle of claim 14, wherein the controller is further adapted to transform into a real space road model the oblique lines judged as the road boundary and the moving object, and to determine the risk of collision between the vehicle and the moving object based on a relative positional relationship between the road boundary and the moving object as established in the real space road model.

16. The vehicle of claim 14, wherein the controller is further adapted to generate the collision danger signal at one of a plurality of values corresponding to collision risk levels.

17. The vehicle of claim 14, further comprising an audio alert operatively coupled to the controller and activated by the collision danger signal.

18. The vehicle of claim 14, further comprising an automatic braking device operatively coupled to the controller and activated by the collision danger signal.

19. An apparatus for detecting objects in one or more images captured by an image pickup device mounted on a vehicle and traveling on a road, comprising:

(a) velocity information computing means for processing a sequential series of images to compute velocity information for each pixel in an image;

(b) pixel extracting means for extracting pixels having a velocity component based on the velocity information computed by the velocity information computing means, wherein the velocity component comprises a movement direction and a movement velocity in a lateral direction;

(c) defining means for defining regions for detecting a road boundary;

(d) oblique line detecting means for detecting oblique lines made of pixels having a velocity component and extracted by the pixel extracting means by grouping those extracted pixels having the velocity component in the regions; and

(e) boundary line detecting means for detecting at least one boundary line on the road present in the image on the basis of the oblique line detected by the oblique line detecting means.

20. The apparatus of claim 19, wherein the boundary line detecting means judges that the oblique lines are road boundaries when the vehicle is traveling and the oblique lines detected by the oblique line detecting means are positioned with bilateral symmetry on the images and respectively comprise pixels having velocity components with different movement directions.

21. The apparatus of claim 19, wherein when multiple oblique lines are detected by the oblique line detecting means, the boundary line detecting means detects the oblique lines as boundary lines on the road when respective slopes of the oblique lines decrease from a center of the image in front of the vehicle toward an outside of the image.

22. The apparatus of claim 19, further comprising balance point judgment means for detecting a change point where a velocity direction of a detected oblique line detected by the oblique line detecting means changes from one image to a subsequent image, and judging that the change point is a pitching balance point where a line of sight orientation of the image pickup device is horizontal with respect to a road surface.

23. The apparatus of claim 22, further comprising:

object detecting means for grouping those pixels extracted by the pixel extracting means having a same velocity component with the movement direction being from a side toward a predicted movement path of the vehicle, and detecting moving objects that approach the predicted movement path of the vehicle;

transformation means for transforming the oblique lines judged as the boundary lines on the road by the boundary line detecting means and the moving object detected by the object detecting means into a real space road model; and

degree of collision danger judgment means for accessing a degree of collision danger between the vehicle and the moving body on the basis of a relative positional relationship between the boundary lines on the road together with the moving object established in the real space road model by the transformation means.

24. The apparatus of claim 23, wherein:

the degree of collision danger judgment means classifies the degree of collision danger between the vehicle and the moving object into multiple risk levels and makes a judgment according to a degree of risk based on the multiple risk levels; and

further comprising risk avoidance means for controlling the vehicle to avoid a collision between the vehicle and the moving object according to the degree of risk assessed by the degree of collision danger judgment means.

25. A method for detecting objects in an image captured of the pathway of a vehicle, comprising:

(a) computing velocity information for each pixel in the image using a sequential series of images;

(b) extracting from the image extracted pixels having a velocity component based on the velocity information, wherein the velocity component comprises a movement direction and a movement velocity in a lateral direction;

(c) defining regions for detecting a road boundary;

(d) detecting oblique lines made of pixels having a velocity component by grouping the extracted pixels having a velocity component in the regions; and

(e) detecting one or more boundary lines on the vehicle pathway in the image from the oblique lines.

26. The method of claim 25, wherein detecting one or more boundary lines comprises judging that the oblique lines are road boundaries when the vehicle is traveling and the oblique lines are positioned with bilateral symmetry on the images and each oblique line comprises pixels having velocity components with a movement direction different from a movement direction of pixels of the other oblique line.

27. The method of claim 25, wherein multiple oblique lines are detected as the boundary lines on the vehicle pathway when respective slopes of the oblique lines decrease from a center of the image in front of the vehicle toward an outside of the image.

28. The method of claim 25, further comprising:

detecting a change point where a velocity direction of an oblique line detected by the oblique line detecting means changes from one image to a subsequent image, and judging that the change point is a pitching balance point where a line of sight orientation of an image pickup device is horizontal with respect to a road surface.

29. The method of claim 25, further comprising:

grouping the pixels having movement directions from a side toward a predicted movement path of the vehicle from among the extracted pixels, and detecting a moving object that approaches the predicted movement path ahead of the vehicle where a group of pixels has a same velocity component;

transforming into a real space road model the oblique lines judged as the boundary lines on the road and the moving object; and

accessing a degree of collision danger between the vehicle and the moving object on the basis of a relative positional relationship between the boundary lines on the road and the moving object established in the real space road model.

30. The method of claim 29, further comprising:

classifying the degree of collision danger between the vehicle and the moving object into one of a plurality of risk levels and making a collision danger judgment according to the classified degree; and

generating a signal to control the vehicle to avoid a collision between the vehicle and the moving object according to the collision danger judgment.