Environment recognition device

• US 7,804,980 B2
• Filed: 08/10/2006
• Issued: 09/28/2010
• Est. Priority Date: 08/24/2005
• Status: Active Grant

• Current Assignee: DENSO Corporation
• Original Assignee: DENSO Corporation
• Inventors: Sasaki, Mikio
• Primary Examiner(s): Bella; Matthew C
• Assistant Examiner(s): Newman; Michael A
• Application Number:
  US11/502,485
• Publication Number:
  US 20070047809A1
• Time in Patent Office: 1,510 Days
• Field of Search: 382/103, 382/104, 382/106, 382/107, 382/168, 382/170, 382/209, 382/217, 382/220, 382/228, 340/905, 340/435, 340/903, 701/117
• US Class Current: 382/103
• CPC Class Codes: G06V 10/422 for representing the structure of the pattern or shape of an object therefor
• CPC Class Codes: G06V 10/56 relating to colour
• CPC Class Codes: G06V 20/56 exterior to a vehicle by using sensors mounted on the vehicle
• CPC Class Codes: G06V 20/584 of vehicle lights or traffic lights

Abstract

A color information creating part calculates the average color of captured image data in units of macro blocks, and determines to which color index the average color of each macro block corresponds. Color arrangement information formed by associating color information with the macro blocks is stored in a color index memory. A histogram creating part creates color histograms representative of the distribution of macro blocks added with color information corresponding to aimed color indexes. Based on the color arrangement information and the color histograms, a color application execution part recognizes objects, surroundings, or the situation of a driver'"'"'s own vehicle without performing area splitting (shape acquisition) of an image.

Claims

1. An environment recognition device comprising:

an on-board camera mounted on a vehicle;

an object specifying unit that specifies a target object to be recognized;

an image data acquiring unit that acquires image data based on a color image in front of the vehicle imaged by the on-board camera;

an average color calculating unit that calculates, based on the image data acquired by the image data acquiring unit, an average color of each of pixel blocks produced by splitting the color image by a preset size;

a color information adding unit that determines to which of color indexes provided in advance the average color calculated in the average color calculating unit belongs, and stores a determination result in association with the pixel block as color information;

a color histogram creating unit that accumulates along one direction on the color image the number of the pixel blocks that aimed color indexes match the color information associated by the color information adding unit, and creates a color histogram formed by one-dimensionally arranging accumulated values, for each of the color indexes and in time series in units of frames of the image data; and

an object information adding unit that creates information about a target object specified by the object specifying unit on the basis of the color histograms created by the color histogram creating unit, and stores the created information as object information in association with the image data acquired by the image data acquiring unit.

2. The environment recognition device according to claim 1,

wherein the direction in which the color histogram creating unit accumulates the number of the pixel blocks is a vertical direction in the color image, and individual elements of the color histogram are associated with positions of a horizontal direction orthogonal to the vertical direction.

3. The environment recognition device according to claim 1,

wherein the color histogram creating unit sets, according to the target object, a range of pixel blocks to be accumulated.

4. The environment recognition device according to claim 1, further comprising:

a color-specific evaluation value calculating unit that, for each of the color histograms created in the color histogram creating unit, uses weight coefficients set to be larger for positions in which the target object exists more probably, of the positions of a horizontal direction to which individual elements of the color histograms correspond, performs weighting additions for elements of the color histograms, and calculates results of accumulating operation results of the weighting additions for a monitoring period set according to the target object as color-specific evaluation values,

wherein the object information adding unit uses the sum of the color-specific evaluation values calculated in the color-specific evaluation value calculating unit as a total evaluation value, and when the total evaluation values is larger than a preset evaluation threshold, determines that the target object exists in the color image, and outputs the determination result as the object information.

5. The environment recognition device according to claim 4,

wherein the color-specific evaluation value calculating unit sets a monitoring period to a value inversely proportional to a vehicle speed of the vehicle when the vehicle speed is equal to or greater than a preset lower limit, and sets the monitoring period to a preset fixed value when the vehicle speed is smaller than the preset lower limit.

6. The environment recognition device according to claim 4,

wherein the color-specific evaluation value calculating unit comprises:

a variable element creating unit that defines an arrangement formed by respectively associating standard distances according to objects with the color indexes as a distance vector, defines an arrangement formed by respectively associating load coefficient vectors formed by associating the load coefficients with individual elements of the color histograms with the color indexes as a load coefficient matrix, and creates the distance vector and the load coefficient matrix that are suitable to the target object specified in the object specifying unit;

a period vector creating unit that defines an arrangement formed by respectively associating the monitoring periods with the color indexes as a period vector, and creates the period vector on the basis of the distance vector created in the variable element creating unit, a frame rate of the color image, and a vehicle speed of the vehicle mounting the device;

a matrix calculation unit that defines an arrangement formed by respectively associating color histograms created in the color histogram creating unit with the color indexes as a color histogram matrix, and calculates an evaluation unit vector that performs a matrix calculation between the histogram matrix and the load coefficient matrix created in the variable element creating unit to calculate an evaluation unit vector formed by arranging the weighting addition values respectively corresponding to the color indexes; and

an accumulation operation unit that creates a color-specific evaluation vector formed by arranging the color-specific evaluation values respectively associated with the color indexes by accumulating individual elements of the evaluation unit vector calculated in the matrix calculation unit according to the contents of the period vector created in the period vector creating unit,

wherein the object information adding unit calculates a total evaluation value of the target object by adding individual elements of the color-specific evaluation vector created in the accumulation operation unit.

7. The environment recognition device according to claim 3, further comprising:

a scene classification dictionary storage unit that stores a scene classification dictionary storing plural scenes representative of a typical view of the color image, the plural scenes being respectively associated with objects likely to exist in the scenes,

wherein the object specifying unit successively reads an object associated with a scene specified from the outside from the scene classification dictionary storage unit, and specifies it as the target object.

8. An environment recognition device comprising:

an on-board camera mounted on a vehicle;

an object specifying unit that specifies a target object to be recognized;

an image data acquiring unit that acquires image data based on a color image in front of the vehicle imaged by the on-board camera;

an average color calculating unit that calculates the average color of each of pixel blocks produced by splitting the color image by a preset size, based on the image data acquired by the image data acquiring unit;

a color information adding unit that determines to which of color indexes provided in advance the average color calculated in the average color calculating unit belongs, and stores the determination result in association with the pixel block as color information;

an object information adding unit that creates, based on the color information respectively associated with the pixel blocks by the color information adding unit, information on the target object specified in the object specifying unit, and stores the created information as object information in association with the image data acquired by the image data acquiring unit;

a template storage unit that stores plural templates representative of a position of a wiper in the color image;

a binarizing unit that creates a binary image added with different values for pixel blocks added with color information corresponding to a black color index by the color information adding unit, and pixel blocks added with other color information;

a wiper detecting unit that detects a wiper in a position other than a stop position when not used, by performing pattern matching between the templates stored in the template storage unit and the binary image created by the binarizing unit; and

a frequency calculating unit that calculates the number of times a wiper was detected in the wiper detecting unit for a preset observation period,

wherein the object information adding unit outputs information indicating that the wiper is operating, when the frequency calculated in the frequency calculating unit exceeds a preset operation determination threshold, as the object information.

9. The environment recognition device according to claim 8, further comprising:

a cumulative addition unit that, based on the image data captured from the image acquiring unit when a wiper of the vehicle mounting the device is operating, classifies the binary image created in the binarizing unit into groups by macro blocks added with color information corresponding to a black color index, the macro blocks being identified by the addresses of macro blocks on an edge of the binary image, and creates a cumulatively added image formed by cumulatively adding the binary image belonging to same groups for each of the groups in units of the pixel blocks; and

a template creating unit that, for the cumulatively added image created in the cumulative addition unit, creates a template formed by adding different values for pixel blocks with a cumulatively added value equal to or greater than a preset existence determination threshold and pixel blocks with a cumulatively added value smaller than the existence determination threshold, as the template, and stores the template in the template storing unit.

10. The environment recognition device according to claim 8, further comprising:

a candidate extracting unit that defines a pixel block added with color information corresponding to a red color index by the color information adding unit as a red pixel block, and extracts the two red pixel blocks placed in a horizontal direction as tail light candidates; and

an inter-vehicle distance estimating unit that estimates an inter-vehicle distance by determining which of vehicle width conditions a positional relationship of the tail light candidates extracted in the candidate extracting unit satisfies, the vehicle width conditions being respectively associated with plural stages of inter-vehicle distances,

wherein the object information adding unit outputs a vehicle position represented by the positions of the tail light candidates and the inter-vehicle distance estimated in the inter-vehicle distance estimating unit as the object information.

11. The environment recognition device according to claim 10, further comprising:

a lighting determination unit that determines whether the tail light candidates extracted in the candidate extracting unit are in a turned-on condition, based on the luminance of the tail light candidate,

wherein the object information adding unit outputs a determination result in the lighting determination unit as the object information.

13. The environment recognition device according to claim 8, further comprising:

an abnormal condition determining unit that determines an occurrence of an abnormal condition when the object information stored in the object information adding unit and information about the target object supplied from outside of the device do not match.

12. An environment recognition device comprising:

an on-board camera mounted on a vehicle;

an object specifying unit that specifies a target object to be recognized;

an image data acquiring unit that acquires image data based on a color image in front of the vehicle imaged by the on-board camera;

an average color calculating unit that calculates the average color of each of pixel blocks produced by splitting the color image by a preset size, based on the image data acquired by the image data acquiring unit;

a color information adding unit that determines to which of color indexes provided in advance the average color calculated in the average color calculating unit belongs, and stores the determination result in association with the pixel block as color information;

an object information adding unit that creates, based on the color information respectively associated with the pixel blocks by the color information adding unit, information on the target object specified in the object specifying unit, and stores the created information as object information in association with the image data acquired by the image data acquiring unit;

a candidate extracting unit that defines a pixel block added with color information corresponding to a red color index by the color information adding unit as a red pixel block, and extracts the two red pixel blocks placed in a horizontal direction as tail light candidates;

an inter-vehicle distance estimating unit that estimates an inter-vehicle distance by determining which of vehicle width conditions a positional relationship of the tail light candidates extracted in the candidate extracting unit satisfies, the vehicle width conditions being respectively associated with plural stages of inter-vehicle distances;

wherein the object information adding unit outputs a vehicle position represented by the positions of the tail light candidates and the inter-vehicle distance estimated in the inter-vehicle distance estimating unit as the object information;

vehicle color extracting unit that extracts the color of macro blocks that exist in a section between the tail light candidates extracted in the candidate extracting unit or vertically to the section, as the color of a preceding vehicle,

wherein the object information adding unit outputs an extraction result in the vehicle color extracting unit as the object information.

14. An environment recognition device, comprising:

an on-board camera mounted on a vehicle;

an image data acquiring unit that acquires image data based on a color image in front of the vehicle imaged by the on-board camera;

an average color calculating unit that calculates, based on the image data acquired by the image data acquiring unit, an average color of each of pixel blocks produced by splitting the color image by a preset size;

a color information adding unit that determines to which of color indexes provided in advance the average color calculated in the average color calculating unit belongs, and stores the determination result in association with the pixel block as color information;

a color histogram creating unit that accumulates along one direction on the color image the number of the pixel blocks that aimed color indexes match the color information associated by the color information adding unit, and creates a color histogram formed by one-dimensionally arranging the accumulated values, for each of the color indexes and in time series in units of frames of the image data;

a difference calculating unit that calculates an inter-frame difference of color histograms created in the color histogram creating unit, for each of the color indexes; and

a caution calling situation detecting unit that detects, based on a calculation result in the difference calculating unit, a caution calling situation appearing on a color screen.

15. An environment recognition device, comprising:

an on-board camera mounted on a vehicle;

an image data acquiring unit that acquires image data based on a color image in front of a vehicle imaged by an on-board camera;

an average color calculating unit that calculates, based on the image data acquired by the image data acquiring unit, an average color of each of pixel blocks produced by splitting the color image by a preset size;

a color information adding unit that determines to which of color indexes provided in advance the average color calculated in the average color calculating unit belongs, and stores the determination result in association with the pixel block as color information;

a full screen accumulated value creating unit that creates a full screen accumulated value produced by accumulating over the whole of the color image the number of the pixel blocks that aimed color indexes match the color information associated by the color information adding unit, in time series in units of frames of the image data;

a difference calculating unit that calculates an inter-frame difference of the full screen accumulated value created in the full screen accumulated value creating unit; and

a caution calling situation detecting unit that detects, based on a calculation result in the difference calculating unit, a caution calling situation appearing on a color screen.

16. The environment recognition device according to claim 15,

wherein the full screen accumulated value creating unit creates the full screen accumulated value on a red color index, and the caution calling situation detecting unit detects as the caution calling situation a case where weather data supplied from an outside indicates that it is raining, and the inter-frame difference of the full screen accumulated value calculated in the difference calculating unit is equal to or greater than a preset approach threshold.

17. An environment recognition device comprising:

an on-board camera mounted on a mobile unit;

an image data acquiring unit that acquires image data based on a color image imaged by the camera;

an average color calculating unit that calculates, based on the image data acquired by the image data acquiring unit, an average color of a pixel block for each of pixel blocks produced by splitting the color image by a preset size;

a color information adding unit that determines to which of color indexes provided in advance the average color calculated in the average color calculating unit belongs, and stores a determination result in association with the pixel block as color information;

a certainty degree calculating unit that calculates, with matters and attributes to be extracted from the color image as objects, based on the color information stored in the color information adding unit, a certainty degree that an aimed pixel block constitutes an aimed object, for each of the pixel blocks and for each of the objects; and

an object determining unit that determines an object to be associated with the pixel block, based on a calculation result in the certainty degree calculating unit;

wherein

the certainty degree calculating unit comprises;

plural partial certainty degree calculating units that obtain the certainty degree by using mutually different methods;

a certainty degree integration unit that obtains a certainty degree by integrating calculation results in the plural partial certainty degree calculating units for each of the objects; and

wherein

at least one of the plural partial certainty degree calculating units obtains the certainty degree by statistical estimation based on statistical analysis;

in the statistical estimation, a measurement matrix is formed by arranging the certainty degrees and factor information that influence the calculation of the certainty degrees in association with each other, and

the certainty degree is calculated based on regression coefficients obtained by subjecting a measurement matrix to regression analysis, and extracted information which is an actual measurement value of the factor information extracted from the image data acquired by the image data acquiring unit.

18. The environment recognition device according to claim 17,

wherein the certainty degree calculating unit includes a target object selecting unit that selects, based on the color information, the objects subject to the calculation of the certainty degree.

19. The environment recognition device according to claim 17,

wherein at least one of the plural partial certainty degree calculating units obtains the certainty degree by rule base estimation based on preset restriction rules.

20. The environment recognition device according to claim 17, further comprising:

a measurement matrix creating unit that creates the measurement matrix based on information obtained from the image data acquired by the image data acquiring unit, and instruction data inputted from an outside; and

a regression coefficient calculating unit that determines a regression coefficient from the measurement matrix created by the measurement matrix creating unit.

21. The environment recognition device according to claim 20, further comprising:

a lacking information estimating unit that, when information is lacking in the factor information constituting the measurement matrix created by the measurement matrix creating unit, estimates lacking information by regression analysis.

22. The environment recognition device according to claim 17,

wherein the factor information includes information representative of physical characteristics of pixel blocks constituting the image.

23. The environment recognition device according to claim 17,

wherein the factor information includes information representative of a situation during image photographing.

24. The environment recognition device according to claim 17,

wherein the certainty degree calculating unit includes a certainty degree correcting unit that corrects a certainty degree obtained in either of the partial certainty degree calculating unit and the certainty degree integrating unit, using rule base estimation based on preset restriction rules.

25. The environment recognition device according to claim 24,

wherein the certainty degree correcting unit at least uses the preset restriction rules for estimating whether an object shown in the pixel block is a moving matter or a standing matter, based on a motion vector obtained from the image data.

26. The environment recognition device according to claim 24,

wherein the certainty degree correcting unit determines a past pixel block corresponding to a present pixel block, based on the motion vector obtained from the image data, and at least uses the restriction rules about information on the determined past pixel block.

27. The environment recognition device according to claim 24,

wherein the certainty degree correcting unit at least uses the restriction rules about characteristics of a same color area formed by pixel blocks that have same color information and are adjacent to each other.

28. The environment recognition device according to claim 17, further comprising:

a view pattern storage unit that stores view patterns representative of typical, static environments, by splitting a screen of the color image by partial areas having same characteristics; and

a view pattern selecting unit that selects a view pattern matching the image data acquired by the image data acquiring unit,

wherein the certainty degree calculating unit uses characteristics of the partial region to which the pixel block belongs, for the calculation of the certainty degree.

31. The environment recognition device according to claim 24,

wherein the object determining unit determines an object having a largest certainty degree calculated in the certainty degree calculating unit as an object to be associated with the pixel block.

32. The environment recognition device according to claim 31,

wherein the object determining unit excludes objects which obviously cannot exist in the color image from determination candidates, based on information about objects obtained from the outside.

29. An environment recognition device comprising:

an on-board camera mounted on a mobile unit;

an image data acquiring unit that acquires image data based on a color image imaged by the camera;

an average color calculating unit that calculates, based on the image data acquired by the image data acquiring unit, an average color of a pixel block for each of pixel blocks produced by splitting the color image by a preset size;

a color information adding unit that determines to which of color indexes provided in advance the average color calculated in the average color calculating unit belongs, and stores a determination result in association with the pixel block as color information;

a certainty degree calculating unit that calculates, with matters and attributes to be extracted from the color image as objects, based on the color information stored in the color information adding unit, a certainty degree that an aimed pixel block constitutes an aimed object, for each of the pixel blocks and for each of the objects;

an object determining unit that determines an object to be associated with the pixel block, based on a calculation result in the certainty degree calculating unit;

a view pattern storage unit that stores view patterns representative of typical, static environments, by splitting a screen of the color image by partial areas having same characteristics; and

a view pattern selecting unit that selects a view pattern matching the image data acquired by the image data acquiring unit,

wherein the certainty degree calculating unit uses characteristics of the partial region to which the pixel block belongs, for the calculation of the certainty degree;

a model storage unit that stores road structure models representative of typical patterns of a three-dimensional structure in a vicinity of roads shown in the color image in association with the view patterns; and

a distance map creating unit that creates a distance map showing three-dimensional positions of individual pixel blocks by projecting a road structure model corresponding to a view pattern selected in the view pattern selecting unit into the screen comprised of the pixel blocks,

wherein the certainty degree calculating unit uses information of the three-dimensional positions of the pixel blocks determined from the distance map, for the calculation of the certainty degree.

30. The environment recognition device according to claim 29, further comprising:

a distance map modifying unit that, for each of partial areas constituting the view pattern selected in the view pattern selecting unit, defines a range of color information belonging to the partial area, uses the definition as a determination condition, and when color information of the pixel block does not satisfy the determination condition set for a partial area in which the pixel block is located, determines the pixel block as being different from a background area and modifies the distance map.