Vehicle environment monitoring device

• US 7,672,510 B2
• Filed: 08/27/2004
• Issued: 03/02/2010
• Est. Priority Date: 09/01/2003
• Status: Active Grant

• Current Assignee: Arriver Software AB (Qualcomm, Inc.)
• Original Assignee: Honda Motor Co., Ltd. (Honda Motor Company)
• Inventors: Saka, Masakazu, Tsuji, Takayuki, Nagaoka, Nobuharu
• Primary Examiner(s): Bali; Vikkram
• Assistant Examiner(s): ENTEZARI, MICHELLE M
• Application Number:
  US10/928,768
• Publication Number:
  US 20050063565A1
• Time in Patent Office: 2,013 Days
• Field of Search: None
• US Class Current: 382/170
• CPC Class Codes: B60R 21/013 including means for detecting collisions, impending collisions or roll-over
• CPC Class Codes: B60R 21/0134 responsive to imminent contact with an obstacle , e.g. using radar systems
• CPC Class Codes: B60R 21/34 Protecting non-occupants of a vehicle, e.g. pedestrians B60R19/02 takes precedence
• CPC Class Codes: G06T 2207/10048 Infrared image
• CPC Class Codes: G06T 2207/30261 Obstacle
• CPC Class Codes: G06T 7/66 of image moments or centre of gravity
• CPC Class Codes: G06V 40/10 Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands

Abstract

A vehicle environment monitoring device identifies objects from an image taken by an infrared camera. Object extraction from images is performed in accordance with the state of the environment as determined by measurements extracted from the images as follows: N1 binarized objects are extracted from a single frame. Height of a grayscale objects corresponding to one of the binarized objects are calculated. If a ratio of the number of binarized objects C, where the absolute value of the height difference is less than the predetermined value ΔH, to the total number N1 of binarized objects, is greater than a predetermined value X1, the image frame is determined to be rainfall-affected. If the ratio is less than the predetermined value X1, the image frame is determined to be a normal frame. A warning is provided if it is determined that the object is a pedestrian and if a collision is likely.

Claims

1. A vehicle environment monitoring device, which detects bodies present in the environment of a vehicle from infrared images taken by infrared cameras, which comprises:

a binarized object extraction device which by binarizing grayscale images of the infrared images, extracts the bodies as binarized objects;

a binarized object feature quantity calculation device which calculates a feature quantity of the binarized objects;

an object image extraction device which extracts an image of grayscale objects from the grayscale image, the grayscale objects corresponding to the binarized objects;

an object image feature quantity calculation device which calculates a feature quantity of the grayscale objects;

a feature quantity comparison device which compares to each other the feature quantity extracted by the binarized object feature quantity calculation device and the feature quantity extracted by the object image feature quantity calculation device; and

a body state determination device which determines the state of the bodies as affected by a surrounding environment thereof based on the results of the comparison performed in the feature quantity comparison device;

wherein said feature quantity comparison device compares for a plurality of bodies the feature quantity extracted by said binarized object feature quantity calculation device and the feature quantity extracted by said object image feature quantity calculation device, and

said body state determination device determines the state of said bodies according to a proportion of bodies specified by the results of the comparison of said feature quantity comparison device.

5. A vehicle environment monitoring device according to claim 1, wherein said feature quantity comparison device compares the height of the binarized object extracted by said binarized object feature quantity calculation device and the height of the grayscale object extracted by said object image feature quantity calculation device.

6. A vehicle environment monitoring device according to claim 1, wherein said feature quantity comparison device compares the width of the binarized object extracted by said binarized object feature quantity calculation device and the width of the grayscale object extracted by said object image feature quantity calculation device.

11. A vehicle environment monitoring device according to claim 1, wherein said feature quantity comparison device compares for each body of a plurality of bodies the feature quantity extracted by said binarized object feature quantity calculation device and the feature quantity extracted by said object image feature quantity calculation device, and said body state determination device determines the state of said bodies as affected by the surrounding environment thereof by calculation of a proportion comprising the number of bodies of said plurality of bodies wherein the results of the comparison of said feature quantity comparison device indicate a specified body state, relative to the total number of bodies of said plurality of bodies, and comparison of the proportion to a threshold value.

12. A vehicle environment monitoring device according to claim 5 wherein the grayscale object is extracted by said object image feature quantity calculation device by applying a plurality of mask regions to the object in the grayscale image, and wherein the region extracted as the grayscale object includes mask regions which meet each of the following criteria:

i. the amount of luminance variation within the mask region is high;

ii. there is a high degree of correlation in the mask region between the image of each respective camera; and

iii. the calculated distance to the grayscale object is the same as the calculated distance to the corresponding binarized object.

13. A vehicle environment monitoring device according to claim 1, wherein said feature quantity comparison device compares at least one of a height and a width of the binarized object extracted by said binarized object feature quantity calculation device to at least one of the height and the width of the grayscale object extracted by said object image feature quantity calculation device.

18. A vehicle environment monitoring device according to claim 1, wherein the body state determination device determines the state of the bodies as affected by a surrounding environment thereof, including rainfall.

2. A vehicle environment monitoring device, which detects bodies present in the environment of a vehicle from infrared images taken by infrared cameras, which comprises:

a binarized object extraction device which by binarizing grayscale images of the infrared images, extracts the bodies as binarized objects;

a binarized object feature quantity calculation device which calculates a feature quantity of the binarized objects;

an object image extraction device which extracts an image of grayscale objects from the grayscale image, the grayscale objects corresponding to the binarized objects;

an object image feature quantity calculation device which calculates a feature quantity of the grayscale objects;

a feature quantity comparison device which compares to each other the feature quantity extracted by the binarized object feature quantity calculation device and the feature quantity extracted by the object image feature quantity calculation device; and

a body state determination device which determines the state of the bodies as affected by a surrounding environment thereof based on the results of the comparison performed in the feature quantity comparison device;

wherein said feature quantity comparison device compares the feature quantity extracted by said binarized object feature quantity calculation device and the feature quantity extracted by said object image feature quantity calculation device across a plurality of image frames, and

said body state determination device determines the state of said bodies according to a proportion of image frames where the proportion of bodies specified based on the comparison results of said feature quantity comparison device is at or above a predetermined level.

3. A vehicle environment monitoring device which detects bodies present in the environment of a vehicle from infrared images taken by infrared cameras, comprising:

a binarized object extraction device which by binarizing grayscale images of said infrared images, extracts said bodies as binarized objects;

a binarized object feature quantity calculation device which calculates a feature quantity of said binarized objects;

an object image extraction device which extracts an image of a grayscale object from said grayscale image, the grayscale object corresponding to a binarized object;

an object image feature quantity calculation device which calculates a feature quantity of said grayscale image;

a pedestrian recognition device which determines which objects are pedestrians using the feature quantity extracted by said binarized object feature quantity calculation device and the feature quantity extracted by said object image feature quantity calculation device; and

a feature quantity comparison device which compares to each other the feature quantity extracted by said binarized object feature quantity calculation device and the feature quantity extracted by said object image feature quantity calculation device to each other,

wherein said pedestrian recognition device changes a pedestrian recognition processing method thereof based on the comparison results of said feature quantity comparison device,

said feature quantity comparison device compares for a plurality of bodies the feature quantity extracted by said binarized object feature quantity calculation device and the feature quantity extracted by said object image feature quantity calculation device, and

said pedestrian recognition device changes said pedestrian recognition processing method according to a proportion of bodies specified based on the results of the comparison of said feature quantity comparison device.

9. A vehicle environment monitoring device according to claim 3 wherein said feature quantity comparison device compares the height of the binarized object extracted by said binarized object feature quantity calculation device and the height of the grayscale object extracted by said object image feature quantity calculation device.

10. A vehicle environment monitoring device according to claim 3, wherein said feature quantity comparison device compares the width of the binarized object extracted by said binarized object feature quantity calculation device and the width of the grayscale object extracted by said object image feature quantity calculation device.

16. A vehicle environment monitoring device according to claim 3, wherein the pedestrian recognition device changes a pedestrian recognition processing method thereof based on a state of the bodies as affected by a surrounding environment thereof which is determined from the comparison performed in the feature quality comparison device.

4. A vehicle environment monitoring device which detects bodies present in the environment of a vehicle from infrared images taken by infrared cameras, comprising:

a binarized object extraction device which by binarizing grayscale images of said infrared images, extracts said bodies as binarized objects;

a binarized object feature quantity calculation device which calculates a feature quantity of said binarized objects;

an object image extraction device which extracts an image of a grayscale object from said grayscale image, the grayscale object corresponding to a binarized object;

an object image feature quantity calculation device which calculates a feature quantity of said grayscale image;

a pedestrian recognition device which determines which objects are pedestrians using the feature quantity extracted by said binarized object feature quantity calculation device and the feature quantity extracted by said object image feature quantity calculation device; and

a feature quantity comparison device which compares to each other the feature quantity extracted by said binarized object feature quantity calculation device and the feature quantity extracted by said object image feature quantity calculation device to each other,

wherein said pedestrian recognition device changes a pedestrian recognition processing method thereof based on the comparison results of said feature quantity comparison device,

said feature quantity comparison device compares the feature quantity extracted by said binarized object feature quantity calculation device and the feature quantity extracted by said object image feature quantity calculation device across a plurality of image frames, and

said pedestrian recognition device changes said pedestrian recognition processing method according to a proportion of image frames where the proportion of bodies specified based on the comparison results of said feature quantity comparison device is at or above a predetermined level.

7. A vehicle environment monitoring device which detects bodies present in the environment of a vehicle from infrared images taken by infrared cameras, comprising:

a binarized object extraction device which extracts said bodies as a plurality of binarized objects, by plural binarizations of a grayscale image from said infrared cameras according to a plurality of luminance threshold values;

a binarized object feature quantity calculation device which respectively calculates a feature quantity of each of said plurality of binarized objects with different luminance threshold values;

a feature quantity comparison device which compares to each other the feature quantities of said plurality of binarized objects with different luminance threshold values extracted by said binarized object feature quantity calculation device; and

a body state determination device which determines the state of said bodies as affected by a surrounding environment thereof based on the comparison results of said feature quantity comparison device;

wherein said feature quantity comparison device compares to each other the feature quantities of said plurality of binarized objects with different luminance threshold values extracted by said binarized object feature quantity calculation device, for a plurality of bodies, and

said body state determination device determines the state of said bodies according to a proportion of bodies specified based on the results of the comparison of said feature quantity comparison device.

14. A vehicle environment monitoring device according to claim 7, wherein said feature quantity comparison device compares to each other the area of the plurality of binarized objects with the different luminance threshold values extracted by said binarized object feature quantity calculation device.

8. A vehicle environment monitoring device which detects bodies present in the environment of a vehicle from infrared images taken by infrared cameras, comprising:

a binarized object extraction device which plurally extracts each of said bodies to form a plurality of binarized objects, by binarizing grayscale images from said infrared cameras according to a plurality of luminance threshold values, each of the plurality of binarized objects comprising a unique luminance threshold value;

a binarized object feature quantity calculation device which respectively calculates a feature quantity of said plurality of binarized objects with different luminance threshold values;

an object image extraction device which extracts images of grayscale objects corresponding to said binarized objects from said grayscale images;

an object image feature quantity calculation device which calculates a feature quantity of said grayscale objects;

a pedestrian recognition device which recognizes pedestrians using the feature quantity extracted by said binarized object feature quantity calculation device and the feature quantity extracted by said object image feature quantity calculation device; and

a feature quantity comparison device which compares to each other the feature quantity of said plurality of binarized objects with different luminance threshold values extracted by said binarized object feature quantity calculation device, and

said pedestrian recognition device changes a pedestrian recognition processing method thereof based on the comparison results of said feature quantity comparison device,

wherein said feature quantity comparison device compares to each other the feature quantity of said plurality of binarized objects with different luminance threshold values extracted by said binarized object feature quantity calculation device, for a plurality of bodies, and

said pedestrian recognition device changes said pedestrian recognition processing method according to a proportion of bodies specified based on the results of the comparison of said feature quantity comparison device.

15. A vehicle environment monitoring device according to claim 8, wherein said feature quantity comparison device compares to each other the area of the plurality of binarized objects with the different luminance threshold values extracted by said binarized object feature quantity calculation device.

17. A vehicle environment monitoring device according to claim 8, wherein the pedestrian recognition device changes a pedestrian recognition processing method thereof based on a state of the bodies as affected by a surrounding environment thereof which is determined from the comparison performed in the feature quality comparison device.